O'Grady, who retired this week, told a newspaper on Wednesday that he had used the banned blood-booster erythropoietin (EPO) before the notorious 1998 Tour.
The admission came after a French Senate inquiry named him among riders with "suspicious" test results in a damning report into the 1998 race.
A successful Olympic track cyclist, O'Grady won a madison gold at the 2004 Athens Games, following a team pursuit silver at the 1992 Barcelona Games and a pair of bronzes at Atlanta in 1996.
"It's sad," Australian Olympic Committee spokesman Mike Tancred said.
"He won't be remembered as a fantastic competitor that we all thought he was. Instead he'll be remembered as an athlete who succumbed to the temptation of drugs in sport just to get an edge on his fellow riders.
"In regard to his medals, it's a matter for the international federation, so the UCI will consider the medals and they will then make some recommendation to the IOC."
The AOC had already called on O'Grady to step down from its Athletes' Commission, a 10-member advisory body comprised of respected athletes.
"Members of our London Olympic team ... are entitled to be angry knowing they had supported an athlete who had cheated," AOC president John Coates said.
O'Grady, one of Australia's most celebrated cyclists, could also stand to lose his three national citations, which include an Order of Australia Medal awarded in 2005.
The French Senate inquiry found the top two finishers at the 1998 tour - Italian Marco Pantani and Germany's Jan Ullrich - were among 18 riders who had tested positive for EPO.
O'Grady was among 12 riders whose tests were said to be "suspicious" and the 39-year-old did not waste time confirming he had used EPO to the Adelaide Advertiser, insisting he had acted alone in sourcing it.
He announced his retirement on Tuesday after helping his GreenEdge team to a time trial victory in this year's Tour, his 17th appearance tying George Hincapie's record.
He had been expected to race on in 2014, however, and push for a record 18th Tour.
GreenEdge was rocked by doping revelations last year involving sports director and former cyclist Matt White, who was implicated in the United States Anti-Doping Agency's dossier on disgraced drug cheat Lance Armstrong.
The Australian team sacked White after he admitted to doping with Armstrong's US Postal team, but reinstated him earlier this year after he completed a six-month ban.
GreenEdge said it supported O'Grady "as a person and an advocate for a clean sport".
"Like the majority of the riders in his generation, he was also exposed to the issues and wrongdoings of the sport and made some wrong choices in that environment," GreenEdge general manager Shayne Bannan said.
Cycling Australia also declined to condemn O'Grady, blaming the era and the European "environment".
"The late 1990's was clearly a dark period in cycling's international history," the governing body said in a statement.
Coates, though, said the "everybody else was doing it" line was no defence for cheating.
"This was a shameful period for the sport of cycling which has been well documented, that is no excuse for the decision taken by Stuart O'Grady," he added.
- Sports & Recreation