The Light Blues had never overcome a 2-0 defeat away from home in the first leg of a European tie but after huffing and puffing in the first half they went ahead with a James Tavernier penalty just before the break.
Striker Antonio Colak, who scored twice for Malmo at Ibrox a year ago to knock Rangers out of the Champions League qualifiers, headed in a second in the 58th minute to level the tie.
More drama followed in the 78th minute when Malik Tillman, the 20-year-old attacker on loan from Bayern Munich, scored his first Gers goal with a header to turn the tie around and take the Govan side into the play-offs, with USG's Lazare Amani sent off in injury time.
Some of last season's games at Ibrox when Rangers reached the Europa League final were epic and this one reached similar heights. There was excitement before the game around the return of Gers winger Ryan Kent from an ankle knock, with left-back Borna Barisic and midfielder Scott Arfield also back while Ridvan Yilmaz, Scott Wright and Steven Davis dropped to the bench.
Alfredo Morelos, who came off the bench on Saturday to score the clincher in a 2-0 win against Kilmarnock after a five-month absence due to a thigh injury, started again among the substitutes but there was a feeling the Colombia striker would be required at some point.
Willed on by their enthusiastic but occasionally anxious fans, Rangers hastily tried to unsettle USG, who were unchanged from the first leg.
Tillman knocked a Tom Lawrence corner past the near post before Colak headed a free-kick from Tavernier over the bar. The visitors, meanwhile, barely left their own half but with their first corner in the 19th minute, defender Siebe Van Der Heyden glanced Simon Adingra's delivery just past the far post and Ibrox breathed a sigh of relief.
In the 28th minute Union Saint-Gilloise keeper Anthony Moris reacted quickly to tip a header from Colak over the bar, from a terrific Lawrence free-kick, and the visitors held out again.
It looked like they would see out the first half but they were undone in the 44th minute when referee Anastasios Sidiropoulos judged that Barisic's deep cross into the box had struck the arm of Van Der Heyden and Tavernier duly sent Moris the wrong way with his spot-kick and the Champions League dream was on.
Rangers pressed again from the start of the second half and again USG filed back into defence looking to spring on the counter.
The second goal came when Lawrence's raking pass from the left found Tavernier inside the box and he cut the ball back to Arfield. The midfielder's drive from 16 yards was parried by Moris and Colak was first to the loose ball to head in from a yard out as Ibrox exploded in noise and colour again.
USG were rocking and moments later Moris had to dive to his right to push away a decent drive from Gers midfielder John Lundstram. Rabbi Matondo replaced Arfield in the 64th minute and the game began to stretch with spaces opening up.
USG tried to emerge from their shell and Light Blues keeper Jon McLaughlin raced from his goal to thwart Adingra, who was put through by Loic Lapoussin.
Then came Rangers' third as Barisic's looping cross into the box saw Tillman rise early and above Moris to head into the empty net.
Ben Davies and Morelos immediately replaced James Sands and Colak and Moris rushed from his goal to deny Morelos 35 yards out after the striker had been sent clear by Kent, whose cross from the left minutes later just evaded Morelos.
Amani, booked in the 75th minute, was sent off after picking up a second yellow to complete an extraordinary night in Govan.
Additional reporting by PA.