Defending champion Stuart Bingham also suffered an unexpected loss as he went down 5-4 to fellow Englishman Matthew Selt only 12 months after he usurped Mark Williams to lift his first ranking event as a professional in Bendigo.
Selt's reward from winning a turgid match in which both men spurned copious amount of scoring chances is a meeting against Ryan Day in the last 16.
A moody Carter - who lost to Ronnie O'Sullivan in the World Championship final two months ago - could not find any fluency as world number 84 Cao moved four frames clear boosted by a 99 in the opening frame and a 47 in the fourth.
The 21-year-old appeared to be hesitant when play resumed after the mid-session interval with Carter snatching a scrappy fifth frame before runs of 69 and 61 saw the Essex player close to 4-3 behind.
Carter had chances to forced the deciding frame, but missed a cut on black using the rest as Cao - who ousted Mark Allen in the opening round of the World Championship - finished off matters to seal a dust-up with Martin Gould or Ken Doherty in the last 16.
2005 world champion Shaun Murphy narrowly avoided following Carter out of the tournament as he led Marcus Campbell 3-0 and 4-1 only to scramble over the line with a 5-4 win.
Murphy appeared likely to complete a comfortable day's work as he ran in 65, 55 and 78 in moving 3-0 ahead only to allow Campbell to grid his way back.
Campbell from Dumbarton in Scotland plodded along as runs of 51 and 57 helped him restore parity at 4-4, but Murphy came to his senses in dominating the final frame to progress amid scenes of some relief.
2002 world champion Peter Ebdon edged past Michael Holt 5-4. He will face Ding Junhui or Alan McManus in the last 16 while there was a 5-1 win for Marco Fu over Joe Perry.
Perry rolled in a 128 in claiming his solitary frame with Fu contributing 108. Fu will face Mark Selby in the last 16 if he can account for Jamie Burnett on Wednesday.
Mark Davis - fresh from winning the Six-Red World Championship in Bangkok - proved too strong for Jack Lisowski in a 5-2 success, but is likely to encounter serious heat against home favourite Neil Robertson in the second round.