Marcus Rashford gave the visitors the lead against the run of play at the Emirates but Arsenal captain Martin Odegaard equalised seconds later.
The home side had a second-half penalty award overturned following a VAR check and United were denied what could have been a late winner when substitute Alejandro Garnacho was ruled offside by VAR.
Last year's Premier League runners-up kept up the pressure in eight minutes of added time.
There was still time for another goal as substitute Jesus burst forward, beating Onana to leave United shellshocked.
"We work and do everything we do every day for moments like this," Arsenal manager Mikel Arteta told Sky, praising his side for their never-say-die attitude.
"This is sport at the top level, it is about the small margins. It could have gone either way.
"I am really happy, obviously. They are a great team, really well coached and they make life difficult for you. When we gave the goal away that we did you are always up against it."
The result leaves Arsenal fifth in the table after their third win in four games, two points behind leaders Manchester City, while United are 11th.
Arsenal shot out of the blocks, pinning United back in the warm September sunshine, and should have taken the lead in the 13th minute but new signing Kai Havertz miskicked horrendously with the goal at his mercy.
- Quickfire goals -
The game exploded into life shortly before the half-hour mark when United took the lead against after a speedy counter-attack.
Christian Eriksen picked up a loose ball in midfield and burst forward, producing a perfectly weighted pass for Rashford, running down the left-hand channel.
The England forward, largely anonymous in the early stages, cut inside and past two defenders before unleashing a shot past the despairing dive of Aaron Ramsdale in the 27th minute.
But the Gunners were level just a minute later when Odegaard, unmarked in the area, blasted home from just inside the box after being found by the impressive Gabriel Martinelli.
Arsenal were quicker to settle after the break and were awarded a penalty just before the hour mark when Havertz tumbled in the box but referee Anthony Taylor overturned his decision following a VAR check.
Arteta made three changes, including bringing on Jesus for the lively Eddie Nketiah.
Garnacho appeared to have won the game for United in the 88th minute following a swift break but VAR ruled he was marginally off-side to the delight of the Arsenal fans.
That announcement sent the decibel-level soaring but it rose higher as the home supporters celebrated their two dramatic late goals with wild abandon.
United lost all six matches away to other members of the top seven last season, conceding 3.66 goals per away match on average.
They went down 2-0 at Tottenham last month and Sunday's result spells more away-day misery for the misfiring Reds.
"We deserved to win this game but you don't always get what you deserve," Ten Hag told the BBC.
"I look more at how we step up and if you see from the Wolves game to this game there is big progression. Tonight was a team. I am really happy with this performance and the progress of our performance."