Two goals from Merouane Zemmama and one each for Andy Halliday and Ishmael Miller saw off the spirited Ryman League side but it was Goldberg's 25-yard curling shot into the top corner that brought all four sides of the stadium to their feet.
It was a richly deserved highlight moment for Goldberg - son of former Crystal Palace owner Mark who lost his fortune in football - and his team-mates, who had given Tony Mowbray's men plenty to think about.
Goalkeeper Liam O'Brien - brought in from Barnet on Friday after an injury to regular number one Matt Armstrong-Ford - guessed right to save a 17th-minute penalty from Miller.
Goldberg had the ball in the net towards the end of the first half before being narrowly flagged for offside, while grooming product salesman Lee Carey provided a hairy moment for Boro when he forced a superb save out of Jason Steele on the hour.
If the ingredients for an upset were everywhere, they took some time to arrive as the visitors, ranked 131 places below Boro in the league structure, barely managed to touch the ball outside their own half in the first half-hour.
Miller and Halliday both missed early chances and the former spurned the chance to put his side in front from the spot, trickling a feeble penalty which O'Brien saved low to his right after the striker had been pushed by Tom Vickers.
It was recalled midfielder Zemmama who finally grabbed the lead for Boro after 22 minutes when he curled the ball into the top corner from 25 yards.
But far from folding, Hastings managed to finish the first period with a flourish, Jack Dixon almost fashioning a shooting chance from a 34th-minute corner before Vickers denied Miller with a superb interception at the other end.
Hastings' Jamie Crellin saw a goal-bound effort blocked out for a corner by Kevin Thomson in the 38th minute, then Goldberg hared through, rounded Steele and put the ball in the net before being pulled back for a marginal offside.
And on the stroke of half-time Hastings had their best chance yet when Bailo Camara sent in a dangerous cross from the right and Halliday almost deflected the ball past his own goalkeeper Steele, who parried it out for a corner.
But if the visitors went in at the break still with hope, it looked to have been shattered just two minutes after the interval when Halliday prodded in Boro's second from a Luke Williams corner.
After Carey forced Steele into a superb one-handed save, Zemmama made it 3-0 to Boro in the 68th minute with a heavily deflected shot from the edge of the box.
But almost immediately after the restart, Hastings pushed forward through Goldberg, who bore down on goal before looking up and letting fly with a long-range effort which beat Steele and nestled in the top corner of the net.
It was clear by their celebrations just how much the goal meant to Goldberg and the 1,068-strong army of travelling fans - just two off a remarkable 1,066 coincidence - who had provided most of the atmosphere in contrast to the sparsely populated home sections.
Miller, who had frustrated the home fans all afternoon, finally got on the scoresheet with five minutes to go when he drove the ball home low past O'Brien, but it was Hastings who rightly took the plaudits on the final whistle.
- Sports & Recreation
- Merouane Zemmama
- Ishmael Miller
- Andy Halliday
- Jason Steele