The colt's owner said the horse had a swollen left front tendon after a morning workout and while it was not considered life-threatening, J. Paul Reddam was not prepared to risk any further damage and will retire I'll Have Another to stud.
"He's not lame, he could have run," Reddam said. "But if he can't compete at the top level, he's done enough. History is going to have to wait for another day."
I'll Have Another's withdrawal is a devastating blow for America's struggling race industry, which has been waiting 34 years for another Triple Crown winner.
More than 100,000 people were expected to flock to Belmont Park to watch the race live while millions more were expected to watch on television.
I'll Have Another's trainer Doug O'Neill said the three-year-old looked fine after a final workout early Friday but his leg started to swell after he cooled down and scans revealed the early onset of tendonitis.
"This is the beginning of a tendon. Could he run and compete? Yes. But would it be in his best interests? No," O'Neill said.
"This is extremely tough for all of us. Though it's far from tragic, no one died or anything like that, it's extremely disappointing and I feel so sorry for the whole team."
I'll Have Another, a flashy chestnut colt with a powerful finishing burst, won the Kentucky Derby and Preakness Stakes in his last two starts and was attempting to become just the 12th horse, and first since Affirmed in 1978, to complete the treble.
Eleven horses have won the first two legs since Affirmed only to fail at the last hurdle.
I'll Have Another was the odds-on favourite to win the gruelling 1-1/2 mile (2,414 metre) Belmont Stakes and end the 34-year drought.
Instead, he became the first horse since Bold Venture in 1936 to miss the Belmont after winning the Kentucky Derby and the Preakness Stakes.
"It's just a freakish thing," O'Neill said. "When you have a human or an equine athlete, when you give 110 percent every time you step on the court or the track, you're suspect to injury."
What makes the Triple Crown so difficult to achieve is the makeup and timing of the races, each held in different states, over different distances and during a span of five weeks. The winner also invariably faces different opposition each time.
The final race is a lung-bursting test of speed and stamina and the pressure on connections to try and complete the Triple Crown has ended in disaster before.
In 1969, Majestic Prince was withdrawn from the Belmont after suffering a tendon injury but his connections changed their mind and entered the horse even though he was not fit.
He finished a distant second to Arts And Letters and never raced again.
A decade later, Spectacular Bid stepped on a safety pin that became embedded in his hoof and caused an infection that almost killed him. He also raced and was beaten.
In 1999, Charismatic broke two bones in his foreleg while leading in the final stages of the race. Veterinarians saved his life but he never raced again either.
"I really thought he was going to run off tomorrow and really show something," said Reddam. "We were all a bit shocked, but we have to do what's best for the horse."