Carrick joined up with the squad earlier this week with a knock he suffered during United's 3-1 UEFA Champions League final defeat against Barcelona last weekend and has been unable to train.
Now, following examination by England's medical staff, it has been decided to release Carrick back to Old Trafford, leaving coach Fabio Capello with a 24-man squad.
It was far from certain Carrick would have started anyway, with Scott Parker, Frank Lampard and Jack Wilshere tipped to form a midfield trio against the Swiss at Wembley on Saturday.
Carrick is the third player to withdraw from the squad. Fulham goalkeeper David Stockdale has been replaced by West Ham United's Robert Green while Jermain Defoe dropped out with a wrist injury.