The hosts largely dominated the match, with Japan goalkeeper Eiji Kawashima required to produce a number of saves, but they were unable to find the net and the visitors punished them in the 88th minute when Shinji Kagawa claimed the winner.
Kagawa's goal, following a speedy counter-attack, gave Japan their first win over Les Bleus in six matches, and means they can approach Tuesday's friendly against Brazil in Poland with renewed optimism.
"It's a very, very good result that will give the players confidence, but our objective remains the 2014 World Cup," said Zaccheroni, who led Japan to victory in last year's AFC Asian Cup in Qatar.
"We played a good second half. We managed to play in the spaces between their lines. The goal came after a great counter-attack, but in terms of chances, France were superior.
"We defended very well tonight but the players were a bit timid in the first half. At the end of the match, France put everything into attack and found themselves unbalanced."
Japan are currently six points clear of second-placed Jordan in Asian qualifying Group B for the 2014 FIFA World Cup Brazil™.
The Blue Samurai's next competitive match is a trip to Oman on 14 November.