It was a sweet moment for Kenny Dalglish's side after their win over Manchester City in the League Cup semi-finals in midweek and against a background of tensions between the two bitter rivals.
Daniel Agger opened the scoring with a 21st-minute header for the hosts, but Park Ji-Sung equalised six minutes before the break and a replay looked on the cards until Kuyt netted the winner two minutes from time.
The match between Queens Park Rangers and Chelsea provided little in the way of action throughout the first hour, but the Blues claimed a 1-0 victory courtesy of a 62nd-minute penalty from Juan Mata after Clint Hill had fouled Daniel Sturridge.
In the day's later kick-offs, once again it was Crawley who provided the shock of the day, the League Two side defeating Championship Hull 1-0 thanks to Matt Tubbs' 57th-minute goal to reach the fifth round for the second successive year.
League One side Sheffield looked to have followed up their fourth-round win over West Ham by knocking out another Championship side as Clinton Morrison lobbed them ahead at Blackpool, but Kevin Phillips equalised in injury time from the spot.
Last year's runners-up Stoke City made a great start, taking the lead in the fifth minute through Cameron Jerome, and Robert Huth added a second nine minutes from time for a 2-0 win at local rivals Derby.
Bolton scrape by Swansea
Bolton Wanderers, the team Stoke beat 5-0 in the semi-finals in 2011, are also through thanks to Chris Eagles' 56th-minute winner against Swansea City with visiting goalkeeper Gerhard Tremmel culpable. Luke Moore had put the Swans ahead, but it was quickly cancelled out just before half-time by Darren Pratley, scoring for the first time since a goal in last season's play-off semi-final.
A goal from Simeon Jackson five minutes from time earned Norwich City a 2-1 win at West Bromwich Albion in another all Premier League clash after Marc Antoine Fortune had cancelled out Grant Holt's first-half opener.
League Two Swindon were the heroes of the third round after knocking out top-flight Wigan Athletic, but they went down 2-0 to Leicester today as Jermaine Beckford made it five goals in the competition this season.
Birmingham were comfortable 4-0 winners over Sheffield United at Bramall Lane thanks to goals from 17-year-old Nathan Redmond, Wade Elliott and a brace from Adam Rooney. Meanwhile, Rickie Lambert's 31st-minute goal looked like it would be enough for Championship high-fliers Southampton to beat Millwall, but Darius Henderson struck four minutes from time to secure a 1-1 draw.
In the all League One clash, Damion Stewart's own goal on 12 minutes put Stevenage into the fifth round for the first time in their history as they beat Notts County 1-0.
The biggest shock of the day came in the final match as Brighton dumped out Premier League side Newcastle United with a 1-0 victory at the Amex Stadium.
It was a deja vu moment for Magpies defender Mike Williamson, who deflected in Will Buckley's close-range effort for the only goal 14 minutes from time. Williamson also scored an own goal last season when Newcastle lost to then League Two side Stevenage in the third round of the competition.