So near and yet so far. South Africa are into the Rugby World Cup final but this was the night when English rugby finally roared again. They were denied at the very last by a nerveless long-range penalty from Handré Pollard which took the Springboks through to a final against New Zealand next Saturday but in every other category this was one of the great English efforts.

Because these white tornadoes were utterly unrecognisable from the England who stuttered into this tournament. They flew into everything, harried ceaselessly and played the tricky wet conditions splendidly. At times South Africa seemed to be out of answers as their opponents, spearheaded by Owen Farrell, Courtney Lawes, the youthful George Martin and Ben Earl, dragged them to the edge of reason.

Related: ‘Players should be incredibly proud’: Borthwick praises England after defeat

Even a few days ago it felt barely conceivable this English squad could come so achingly close. Given the cricket World Cup result involving these two nations in India, the pre-game omens had not been especially good either. Little did we know. Until the savage late twist this was right up there with the sensational semi-final win over the All Blacks in Yokohama four years ago.

England were equally as physical this time against a Bok side still apparently sapped by their remarkable win over France the previous weekend. Tactically South Africa were also unusually off key, hauling off their starting fly-half Manie Libbok after barely half an hour. Until the very last it seemed as though England, 12-6 up at half-time thanks to four Farrell penalties, would finally gain revenge for their 2019 final defeat to these same opponents.

They reckoned without the massive bearded lock RG Snyman, who muscled over for the only try of the game in the 69th minute, just when England seemed to be driving up the home straight. Farrell, in the kind of contest he relishes most, had also kicked a long 53rd-minute drop goal to give his side a 15-6 lead, only for Snyman and Pollard to drag their team back from the brink and set up an all-southern hemisphere final against the All Blacks. From England’s perspective it was so cruel. At least they banished some of their demons from 2019. The scrum was a proper contest, the breakdown a constant battle. The previous evening’s semi-final between Argentina and the All Blacks had been horribly one-sided, suggesting a significant gap between the world’s top four nations and everyone else. In this specific type of close-quarter slugfest, though, England are a different proposition.

The weather also played its part. Initially it had been a foul evening: wild, windy and damp. By kick-off the gusts had reduced in intensity but showers were still peppering the stadium. Which suited England fine. Anything to level the playing field slightly and make South Africa think twice. The collective anticipation was equally impossible to ignore. Say what you like about big-time rugby but the sense of theatre is hard to beat.

And England, fully aware of the need to make an early statement, could not have made a more purposeful start. They won the first ruck penalty, giving Farrell his first three points, nicked a lineout and held firm in the early scrums. When Bongi Mbonambi was then pinged for a crooked throw, it added to the sense of South African nervousness.

Related: Dominant England played the game of their lives. And South Africa still won | Jonathan Liew

Farrell kicked a second penalty, this time for Siya Kolisi contesting a ruck that the referee Ben O’Keeffe felt had already been won. If there was little danger of England putting much width on the ball there was nothing wrong with their kicking game and Freddie Steward was impressively solid beneath the high ball.

With tempers fraying – a furious Farrell was marched back 10 metres at one stage – the battle was now raging in earnest. A third Farrell penalty gave England more encouragement, with Mbonambi’s lineout problems continuing. South Africa did create one promising passage, with Kurt-Lee Arendse bursting around the front of a lineout, but were denied by a vital turnover from the omnipresent Lawes.

Not long after Libbok received the shepherd’s crook as the Springboks attempted to revert to the tried and tested. In many ways it was also recognition of the obvious. This was never going to be a game of artistic subtlety and Libbok, good player that he is, was less well suited to its requirements than Pollard.

History will record the switch paid off handsomely, another little win for the Springbok management. Pollard, so influential in 2019, was not in the original Bok squad because of injury but more than justified his call-up, calming his side down and keeping his nerve from the tee in the 78th minute. England, in addition to a crucial scrum penalty and a misjudged lineout at crucial moments, never really looked like scoring a try and, ultimately, it came back to bite them.

Still, it has been one hell of a ride. In August it seemed possible that England might not make it out of their pool. To fall just two points short of a World Cup final, in that context, shows serious character. Up to a dozen of this squad may not make the next World Cup and for a few this will be the end of the Test road. If so, all concerned can go out with their heads held high.