The 13-times European Cup winners went behind in the 57th minute on Tuesday night when Shakthar substitute Dentinho capitalised on a lapse in concentration from defender Raphael Varane to score.
Israeli forward Manor Solomon then wrapped up the home side's shock win over the Spanish champions with eight minutes remaining, with a solo run on the counter and finish into the bottom corner.
Dentinho came on in the first half to replace striker Junior Moraes, who was injured after being hauled down by Varane in a bid to prevent the forward racing towards goal unchallenged.
Shakhtar, who also beat Madrid 3-2 away in October, are second in Group B on seven points, level with Zinedine Zidane's Real side but ahead due to their superior head-to-head record.
The result leaves Real third in the group, and likely needing to better Shakhtar’s result in the final round of matches in order to qualify.
Real will face leaders Borussia Monchengladbach - who could qualify if they beat Inter Milan tonight - while Shakhtar will take on Antonio Conte’s Inter.
Additional reporting by Reuters.