In Tuesday's other third-round replays, League Two Bury secured a 2-1 win over Championship Norwich and will play Southampton at St Mary's while Millwall face a trip to Coventry after a 2-1 win over Walsall.
On Wednesday, League Two Hereford beat League One Tranmere and will travel to Cardiff for their fourth round tie.
And Manchester City saw off Premier League rivals West Ham to secure a fourth round clash away to Sheffield United.
Giant-killers Oldham and Huddersfield have drawn each other, while League Two's Mansfield host Middlesbrough.
The possibility of Liverpool hosting a team five leagues below them has caught the imagination.
"It's a fantastic draw - it couldn't have gone much better," Havant player-manager Charlie Oatway told BBC Radio 5 Live before his side's win over Swansea.
"I think we'll have the old tin hats on if we get that far. But it's great to see that light at the end of the tunnel in the shape of a big club like Liverpool."
Meanwhile, Mansfield manager Billy Dearden, whose side are second from bottom in League Two, admitted getting past Middlesbrough would be a tall order.
"They are a top-class team, we'll have to be at our best to get any kind of result," said the Stags boss, whose side beat Brighton on Saturday.
"But it's a tremendous draw. We've got a Premier League club at home and we're looking forward to it."
The draw - conducted by former FA Cup winners Alan Cork and John Aldridge - ensured that at least seven non-Premier League teams will make it through to the last 16.
The ties will all take place over the weekend of 26-27 January.
Full draw for the fourth round of the FA Cup
Arsenal
v
Newcastle
Coventry
v
Millwall
Oldham
v
Huddersfield Town
Barnet
v
Bristol Rovers
Wigan
v
Chelsea
Liverpool
v
Havant & Waterlooville
Southend
v
Barnsley
Southampton
v
Bury
Man Utd
v
Tottenham
Portsmouth
v
Plymouth
Derby
v
Preston
Watford
v
Wolves
Peterborough
v
West Brom
Sheff Utd
v
Man City
Mansfield
v
Middlesbrough
Hereford
v
Cardiff
Ties to be played on 26 & 27 January
