The win makes Robson the first British woman in the last 16 of a Grand Slam since Sam Smith reached the fourth round at Wimbledon in 1998.
The former French Open champion was outplayed by the 18-year-old, and said after the match: "I was feeling that I still had a chance but I was making a lot of mistakes the whole match.
"That gave her a lot of confidence and gave her free points, made her feel like she could win."
Former Wimbledon junior champion Robson had sent three-times champion Clijsters into retirement in round two and followed up with another nerveless display full of strong serving and huge hitting, especially on her forehand.
Li lifted her game to snatch the tie-break 7-5 and force a decider but Robson continued to attack at every opportunity.
At 2-2 in the third set, Robson thought she had broken serve but umpire Louise Engzell ruled that the point should be replayed, even though Li had missed with her reply.
But Robson kept her head, breaking twice to lead 5-2 and then serving out the match to set up a meeting with defending champion Samantha Stosur, who beat American Varvara Lepchenko 7-6 6-2.
"I knew if I kept sticking with her and playing my shots it might pay off and it did," Robson said.
"I had a few break points early on and I gave them up with average returning, so when I got a chance I knew I had to go for it.
"She's a great hitter so knew I had to go for it. I'm going to go back to the gym and try to recover for the next match."
Meanwhile, fellow Brit Jonathan Marray was sent packing from the men's doubles after he and his partner Frederik Nielsen of Denmark were beaten 6-1 6-7(6) 6-4 by Jesse Levine of the US and Marinko Matosevic of Australia.
Better news came in the first qualifying round for the junior boy's tournament, where fourth seeded Brit Evan Hoyt downed Naoki Nakagawa of Japan 7-6(4) 4-6 7-5 .