The visitors dominated the majority of the game and duly took the lead after 20 minutes.
Teenage midfielder Xhaka collected a pass from Eren Derdiyok before finishing smartly from the edge of the penalty area.
Slovenia came close to an equaliser eight minutes later but Swiss goalkeeper Diego Benaglio tipped Tim Matavz's header over the crossbar.
Derdiyok came close to scoring a second goal for the visitors but his 30th minute effort hit the post before the assistant referee flagged him offside.
Napoli midfielder Inler doubled Switzerland's lead six minutes into the second-half.
The 28-year-old fired in a long-range effort that Slovenia goalkeeper Jasmin Handanovic could not keep out.
The result was tainted for Switzerland when skipper Barnetta was sent off for a second bookable offence.
The Schalke winger had already been booked for a foul on Miso Brecko during the first-half and was shown a second yellow card for bringing down the same player in the 75th minute.
Slovenia enjoyed much more of the ball as they looked to make their numerical advantage count.
Substitute Zlatan Ljubijankic spurned the best chance for the hosts with 10 minutes remaining as he missed the target from close range.
The Omiya Ardija forward also headed over from 12 yards as Switzerland continued to defend their clean sheet.