Norwich City new boy Nathan Redmond gave the Canaries their first English Premier League victory of the season with a 1-0 victory over Southampton at Carrow Road.
Redmond, a £3.2m capture from Birmingham City this summer, produced a superb cut-back from the left wing before firing a low shot into the bottom corner past Artur Boruc in the 68th minute. The visitors had gone close in the first half when Adam Lallana hit a post, the midfielder also being denied a penalty when Bradley Johnson appeared to handle his shot.
At Upton Park, Jermaine Pennant scored his first goal for 63 matches to give Stoke City a 1-0 triumph at West Ham United.
After Hammers goalkeeper Jussi Jaaskelainen had made a trio of fine saves in the first half, he could do nothing about Pennant's superb 25-yard free-kick in the 82nd minute, just five minutes after coming on as a substitute.
There was also a late winner for Newcastle United as they edged past Fulham 1-0 at St James' Park, with Hatem Ben Arfa thumping home an 86th-minute left-footed strike for United's first goal of the season.
One of the main stories from the game was the appearance of Yohan Cabaye in the 65th minute. The France midfielder was included in the 18 for the first time this season despite his future remaining uncertain, with Newcastle having turned down a £10m offer from Arsenal in the opening week of the season.
After a stunning 3-2 win over Manchester City last weekend, Cardiff City managed to hold visitors Everton to a goalless draw on home soil. Under new manager Roberto Martinez, the Toffees have started the league season with three successive draws.
In the early kick-off, Manchester City bounced back from that shock defeat at Cardiff with a routine 2-0 victory over Hull City at the Etihad Stadium.
On the back of a clean sheet from Joe Hart, who had been heavily criticised in the wake of his poor performance against Cardiff, second-half goals from Alvaro Negredo and Yaya Toure were enough to see off Hull.
For Negredo, a £20m summer buy from Seville, it was his second goal in two matches after his late strike in south Wales, and again after coming off the bench. The Spaniard opened the scoring in the 65th minute with a close-range header, but it was not until the 90th that City finally made the game safe via a stunning free-kick from Toure.
Promoted Crystal Palace recorded their first win of the season after late goals from Dwight Gayle and Stuart O'Keefe secured a 3-1 victory at home to Sunderland in the day's evening game.
Danny Gabbidon's fortuitous early opener was cancelled out by a 64th-minute header from fit-again Sunderland striker Steven Fletcher, but Gayle restored the hosts' lead with a penalty 11 minutes from time after visiting captain John O'Shea had been sent off for bringing him down in the box.
Substitute O'Keefe added a fine third in injury time, curling a superb strike into the top-left corner from the edge of the area.