Luger Nodar Kumaritashvili died after crashing into a steel pillar in his last training run hours before the Games opening ceremony on February 12.
Rogge said that the tragedy would cast a lasting cloud over the entire Games.
An initial report from the federation responsible for the sport pointed the finger at the athlete, saying it was human error that caused the accident.
"If you look at it legally, the IOC is not legally bound to the track," Rogge said. "It is the international federation but we are morally responsible."
Athletes and some officials had said before and after the accident that the Whistler sliding track was too fast.
Rogge said he was awaiting a coroner's and a police report and had also demanded the luge and bobsleigh federations review the infrastructure regulations and the qualification process for athletes.
He also said the IOC had written to the 2014 Games organisers in Sochi, Russia, to make sure that safety came first in the sliding track construction.
Following the crash, race officials erected a wall at the place of the incident to cover the pillars and moved the starting spot for the competitions to slow down the athletes who at times reached speeds of over 150kph.
"The design of the track is the responsibility of the international federation, building the track is the responsibility of VANOC (Games organisers), the running of the competition is in the hands of the international federation and the IOC must make sure that we have good Games," he said.
"It is not a responsibility in juridical (legal) terms, it is a moral responsibility (we have)."
When asked specifically who was responsible for the athlete's death, Rogge responded: "Everyone. The death of Kumaritashvili will cast a shadow over the Games. It will be part of the Games. Just like what happened in Munich is part of the Games. What happened in Atlanta is part of the Games."
Palestinians militants attacked and took hostage several Israeli team members at the 1972 Munich Olympics, later killing 11 of them. One German policeman and five of the eight attackers also died.
A bomb exploded at Atlanta's Olympic park during the 1996 Games, killing one person and injuring about 100.