The German complained of a lack of boost pressure during his pre-race installation lap and was noticeably lacking in power when taking off from 12th place on the grid.
Vettel told his team over the radio before the race started that "the engine is not running smoothly" and after a few laps around Albert Park circuit headed back to the pits.
Vettel's retirement snapped a winning streak that started at the Belgian Grand Prix in August.
Mercedes' Lewis Hamilton, the last driver to beat Vettel when he won the Hungarian Grand Prix in July, also retired on Sunday with an engine problem.
"On the second formation lap already we lost power," Vettel told Britain's Sky television.
"We didn't understand why and then basically I had no power from the start. The cars kept passing me. I was down. We tried to recover as much as we could.
"At some stage I thought there was a slight improvement but obviously we realised there was a bigger problem with the engine for some reason. I think it looked like we lost a couple of cylinders," explained the German.
Sunday's race was the first for Formula One's new V6 turbo engine with its complicated energy recovery systems and Red Bull had arrived in Melbourne with a big question mark over their Renault power unit after problems in testing.
"When you don't have the power from the engine, you can't make the whole system function properly and then you lose even more power and so we had to stop." said Vettel.
"No doubt we'll fix these issues. The question is how soon," he added.
"We are working hard...obviously we started a bit on the back foot but we've learned this weekend an awful lot. We also learned the car is quick so if the package comes together I think we are very competitive."
- Motor Racing