Klinsmann, who took over on 29 July, is still seeking his first victory with the Americans, after drawing 1-1 last month with Mexico in his debut match.
"For sure you're not happy with the result, but I am happy with the performance," Klinsmann said. "The players did what I wanted to do. We had our chances in the first half. Overall a very good performance. Unfortunately there wasn't a goal."
Wallace, a 23-year-old midfielder for the Portland Timbers of Major League Soccer, scored less than five minutes after coming on as a substitute for his international debut.
Real Salt Lake forward Alvaro Saborio took the ball deep down the right side for Costa Rica, then crossed for Michael Barrantes, whose left-foot shot deflected off the left arm of USA goalkeeper Tim Howard. The rebound came to Wallace, who opened his international account with a diving header.
The Americans pressed in vain for the equaliser, with play becoming more and more physical, forward Brek Shea and substitute Sacha Kljestan being issued yellow cards in the 78th and 79th minutes, respectively.
"We would like to develop a style of play that is more attacking, more aggressive," Klinsmann said. The 47-year-old wanted to see how well such talents as Jozy Altidore and Landon Donovan combined and pressed home his desire for an aggressive style.
"The things we saw were what we wanted to see," Klinsmann said. "We wanted to see how they combined. They were patient, moved the ball around, played the whole 90 minutes through. They kept the pace up and the rhythm high. What we needed was a goal."
No luck for USA
Donovan nearly delivered it in the seventh minute, his shot going just wide of the near post after a charge to the right from inside the penalty area, the misfire leaving the Los Angeles Galaxy forward with his head in his hands.
Los Ticos only managed their first shots in first-half stoppage time, but forced two saves from Howard before the end of a scoreless first half.
Altidore was offside just inside the area in the 55th minute to thwart a USA attack, while Costa Rica's Randall Brenes was denied by Howard from inside the area in the 59th minute, moments before he was withdrawn for eventual hero Wallace.