Third-seeded Swiss Wawrinka, who followed up his title in Melbourne with victory at last month's Monte Carlo Masters, started strongly against the 70th-ranked Austrian, who at 20 years and eight months is the youngest player in the top 100.
However, he lost his way in the second set and was unable to recover in the third as Thiem pulled off his first ever win over a top 10 player to secure passage to the third round and a meeting with 15th seed Mikhail Youzhny of Russia or unseeded Spaniard Feliciano Lopez.
Wawrinka said he had found trouble adjusting to the conditions in the Spanish capital, where the thinner air at altitude helps the balls fly faster than a clay court at sea level.
"It's never easy to (adjust), especially here, it's flying a little bit," added the 29-year-old, who lost to Nadal in last year's Madrid final.
"He was finding his game and I was hesitating with my game and wasn't playing my best game.
"He's a good player. We all know that. We all see that, especially since the beginning of the year.
"I'm not surprised because I know him. He was going for it and he deserved to win."
Thiem added: "I cannot really believe it.
"Of course I was a little bit nervous, but it wasn't the reason for the first set, his pace was just too high for me.
"I played a good game at the beginning of the second set to break him and then I started to get used to more and more his pace and angles and his game.
"This year I played very good so far. Played also very consistent. I think that's it's more important, to play almost every day against these top guys to increase your own level."
Thiem was joined in the last 16 by fifth seed David Ferrer, who had to dig deep to see off Spanish compatriot Albert Ramos 7-6(6) 5-7 6-3.
Top seed and defending champion Rafa Nadal, the world number one, begins his title defence with a second-round match against unseeded Argentine Juan Monaco on Wednesday before seventh seed Andy Murray of Britain takes on Nadal's Spanish compatriot Nicolas Almagro.
Nadal's chances of success have been boosted by the withdrawal of both second-ranked Serb Novak Djokovic, who has an arm injury, and Roger Federer.
The Swiss world number four opted to miss the tournament so he could be present at the birth of his twin sons Leo and Lenny earlier on Tuesday.
Earlier, big serving Marin Cilic breezed past Joao Sousa, conceding just two games to secure a 6-1 6-1 victory in only 54 minutes.
Cilic, who has two ATP World Tour titles to his name, hit seven aces and lost just eight points on serve and will now play qualifier Paul-Henri Mathieu in the third round.
"The crucial part was in the first set," said Cilic. "I think I played well and I converted both of the break points I had. That made the difference and it became a little bit easier. After I broke him for 2-1 in the second set, the match went quickly."
- Sports & Recreation