The Skrill Premier hosts, looking to make the fourth round for the first time since 1998, took the lead when Ross Hannah tapped home from six yards after 25 minutes.
Oliver Norwood equalised for the Terriers, though, with a stunning free-kick six minutes after the break.
But after 62 minutes Grimsby captain Craig Disley headed home from 10 yards to restore their lead.
However, Martin Paterson, who spent three months on loan at Blundell Park over seven years ago, side-footed home and Aswad Thomas diverted Huddersfield substitute Duane Holmes' left-wing cross into his own net in the final minute.
"We started off well and I thought we controlled the game for long spells in the first half," said Robins.
"But we knew they were going to put us under pressure - it's the FA Cup, and that was a real FA Cup tie and like a throwback to when I was playing.
"The pitch was poor and they were tough conditions - it was all the things that you want as a neutral in an FA Cup tie.
"But as the manager of an away team coming to a Conference side, it's not what you want at all.
"We knew it was going to be tough, they put us under a lot of pressure but had we scored early on with one of the numerous chances we had we might have had a lot easier afternoon.
"There were lots of cameras around looking for an upset, and we disappointed them, which is great."
- Sports & Recreation
- Craig Disley
- Ross Hannah
- Martin Paterson
- Aswad Thomas