Goals by Martin Jakubko and Kornel Salata gave Slovakia a 2-0 lead after just 16 minutes.
Valerijs Sabala pulled one back shortly after half-time but it appeared that the visitors would claim all three points - only for Renars Rode to head home at the end.
Jan Durica had the first effort of the game when his header went wide in the sixth minute before Slovakia opened the scoring three minutes later.
Tomas Kona's corner from the right found Jakubko in the six-yard box and he headed the ball low into the bottom right-hand corner.
Slovakia then doubled their lead in the 16th minute and again it was from a Kona corner, his delivery headed home into the bottom right of the goal by Salata.
Latvia hauled themselves back into the game in the 47th minute when Sabala drove home from 20 yards.
Eduards Visnakovs headed over before his shot was blocked by Slovakia goalkeeper Matus Kozacik.
Slovakia appeared on the verge of holding on for victory, only for Rode to head home after Kaspars Gorkss had headed back Arturs Zjuzins' corner.