Victory not enough for Vienna
Karl Daxbacher's side had to win and hope AZ Alkmaar lost at home to Metalist Kharkiv, who had already been confirmed as group winners, but a 1-1 draw in the Netherlands meant Vienna's interest in the competition was ended despite second-half goals from Michael Liendl and Nacer Barazite.
The hosts created the first chance in the fourth minute when Florian Klein crossed and Roland Linz directed his overhead kick over the bar.
Vienna goalkeeper Heinz Lindner gathered an eighth-minute strike from Jiloan Hamad while at the other end Liendl's free-kick was charged down by the Malmo wall.
Linz threatened again after 23 minutes with a 25-yard drive which Malmo goalkeeper Johan Dahlin did well to tip around the post, before Zlatko Junuzovic blasted a chance over the top three minutes later.
The home side continued to dominate and Pontus Jansson did well to block a Liendl shot after Malmo had failed to adequately clear Alexander Gorgon's cross.
Liendl claims first
Lindner fielded an early second-half effort from Daniel Larsson and Gorgon fired over the top ten minutes into the second period after Junuzovic's centre teed him up.
Dahlin did well to keep out a close-range header from Liendl just before the hour mark, but the Vienna midfielder was on target in the 62nd minute.
Florian Mader's cross was punched away by Dahlin but only as far as Liendl, who fired the ball back beyond the goalkeeper from 20 yards.
Math Ranegie headed a chance wide for the visitors in the 67th minute and Larsson volleyed as Malmo threatened a recovery, but the game was effectively over with ten minutes to go when Junuzovic teed up Barazite from the left channel, and he made no mistake from close range.
A flurry of substitutions disrupted the flow of the game, with Gorgon heading another chance wide for Vienna in the 90th minute and substitutes Tomas Jun and Marko Stankovic combining in injury time, with the latter drilling the former's cross over from six yards.