It was not just that Manchester City had led by two goals with 90 minutes on the clock, a place in the Champions League final against Liverpool basically theirs – although that was plainly the greatest, deepest agony.

It was not that this semi-final should have long since been over. After the first leg, which City had dominated. Or before Real Madrid’s stoppage-time sorcery, in which the substitute, Rodrygo, cast the spells, scoring two scarcely believable goals to force extra-time.

Related: Magic beats control once again in Real Madrid’s enchanted canyon | Barney Ronay

The City substitute, Jack Grealish, had seen a shot miraculously hacked off the line by Ferland Mendy in the 87th minute and he then watched Thibaut Courtois stick out a toe to divert a shot from him just past the far post.

It was the way that the footballing gods, with whom Real appear to have a deal with options, tormented them. Rodrygo had almost completed a stoppage-time hat-trick at the end of normal time, stealing in to extend Ederson, when Phil Foden received a quick free-kick and saw glory beckon. His shot flew high.

On into extra time and there was a measure of inevitability about how Real got the next goal, not to mention the identity of the scorer. It was a tired challenge from Rúben Dias on Karim Benzema, who had nipped in front of the City defender and on to a low Rodrygo cross – an obvious penalty. And everybody knew what was coming next.

Benzema had kept Real in the tie at the end of the first-leg with an ice-in-the-veins panenka. This time, he slotted low inside Ederson’s left-hand post. It was his 15th goal of the Champions League campaign and, incredibly, his 10th in the knockout rounds. Overall for Real this season, Benzema has 43 goals in 43 appearances.

The tie had been spiky from start to finish and how the Bernabéu crowd howled when Pep Guardiola, their bête noire from his Barcelona days, remonstrated with the officials as the City dream headed towards dust.

Did City have anything left? Foden drew a diving save out of Courtois with a header at the end of the first period of extra time but the broad answer was no. Their pain knew no bounds. Kevin De Bruyne had said that winning a first Champions League would change the external narrative around the club. This only reinforced it.

City have fallen short previously under Guardiola – in the competition that they crave more than any other – not least in last season’s final against Chelsea. But never like this. They will attempt to bounce back by retaining their Premier League title but the wounds from this will remain open for a long time. Could they impact on the league campaign?

Related: Real Madrid 3-1 Manchester City: Champions League semi-final ratings

And so Real advance, breathlessly, into a repeat of the 2018 final – in which they beat Liverpool – their sights set on a fifth triumph in this competition in eight years. They have defied logic in the past but surely never as much as they have done this season.

Real were largely outplayed by Paris Saint-Germain and Chelsea in the previous rounds only to squeeze through and here, it was no exaggeration to say that they did not create a clear-cut chance until Rodrygo turned home Benzema’s cross to cancel out Riyad Mahrez’s rocket on the night. Rodrygo’s second – the dagger to City hearts – was a header from a Dani Carvajal cross that deflected in front of him at the very last. His reaction was excellent.

City had defended resiliently, limiting Real to scraps, and they had come to feel that one opening, one moment, could give them the comfort they wanted. With 73 minutes gone, they got it. It was Bernardo Silva who played the final pass, teasing it into the path of the onrushing Mahrez, who had been peripheral up until that point.

No longer. City’s leading scorer exploded the sweetest left-footed drive high inside Courtois’ near post and that appeared to be that. It did not even seem to matter when Grealish could not take his chances. How wrong that feeling would be.

It had been possible to forget that City did actually win the first leg, what with the angst over the missed chances – at least half a dozen big ones; the feeling that the single-goal cushion represented a booby prize. Or a trap.

Guardiola’s team would create the better openings during the first half of normal time here, with De Bruyne laying on the clearest one for Silva, Courtois making a smart reaction save. De Bruyne also teed up Gabriel Jesus, who swiped narrowly wide, while there was a sweet hit on the volley by Foden from 25 yards that worked Courtois.

The pre-match confidence in Madrid had been ludicrously high. Ditto the sense of theatre. “Another magical night from the kings of Europe,” trumpeted the tifo above an image of Benzema, who was mocked up as an emperor. It is all a part of the show, all a part of what City had to tame.

It was edgy, the passions simmering throughout and first boiling over when Casemiro took down De Bruyne with a scissors-style tackle. It led to a confrontation between Luka Modric and Aymeric Laporte, after which both were booked. Casemiro surprisingly was not and he got away with another one in the 33rd minute when he dragged down Foden.

A key clash was the one between Vinícius Júnior and the fit again City right-back, Kyle Walker, who barely conceded an inch. Walker was the symbol of his team’s defiance until he was forced off moments before Mahrez’s goal. At that point, it was impossible to foresee the conclusion. Real would find a way.