The American, who won the gruelling event at the last two Olympics, won his morning heat but only snuck into the finals in eighth place after a desperate lunge on his final stroke.
"I didn't expect those guys to go that fast in the heats," said Phelps. "I think the only thing that matters is getting a spot. You can't get the gold medal from the morning."
Phelps stopped the clock at four minutes and 13.33 seconds, just 0.07 ahead of Hungary's Laszlo Czech, who won the silver in Beijing four years ago, but missed the final after qualifying ninth.
Ryan Lochte, the reigning world champion and favourite to win gold, was third fastest overall in 4:12.35.
"It didn't feel so good, but that was my first race, and my first race is always the worst one," Lochte said. "It's hard, it's a tough field. But he's (Phelps) in."
Japan's Kosuke Hagino (4:10.01) qualified fastest ahead of South Africa's Chad le Clos (4:12.24).
Olympic men's 400m freestyle champion Park Tae-Hwan of South Korea was disqualified in the first big shock of the swimming competition, but was later reinstated.
Park, defending the Olympic title he won in Beijing four years ago, was initially disqualified after officials ruled that he false started in his morning heat but won an appeal to overturn the decision.
China's Sun Yang is Park's main rival to win the gold in the final later on Saturday.
He set the fastest overall time of three minutes 45.07 seconds, just ahead of American Peter Vanderkaay (3:45.80).
Germany's Paul Biedermann, the world record holder for the event, missed out on the final after finishing 12th overall in 3:48.50.
Japan's Kosuke Kitajima safely navigated the first stage of his bid to become possibly the first male swimmer to win the same event at three Olympics when he made it through the men's 100m breaststroke heats.
Kitajima finished second fastest overall, in a time of 59.62 seconds. He was just 0.01, the closest margin to a deadheat in swimming, behind Australia's Christian Sprenger, who was in the same heat.
The event, however, was tinged with sadness because of the death of world champion Alexander Dale Oen in April.
Norway's Dale Oen, who won the world title in Shanghai last year three days after a bombing and shooting attack in Oslo that killed 77 people, had been at a training camp in Arizona when he collapsed and died.
A medical examiner ruled the 26-year-old had died of severe coronary artery disease and his compatriot Sara Nordenstam said his absence was felt in their team and at the pool.
"I know a lot of his competitors will think of him. Cameron (van der Burgh) from South Africa has dedicated his swim to Alex," Nordenstam said before the 100 breaststroke heats.
"Kitajima says he wishes he could compete with Alex. Something is missing when he is not here."
Van der Burgh qualified sixth fastest in 59.79 with the first 12 swimmers all finishing in under a minute.
Kitajima won the 100-200m breaststroke double at each of the past two Olympics and could become the first man to win the same event three times.
The only other swimmer who could beat him for the feat is Michael Phelps, who is competing in four different events that he won at the past two Olympics.
"I don't feel pressure, I just try to enjoy it," Kitajima said. "I can swim at a very high level, so I'll try to improve in the semi-final."
Dana Vollmer of the United States set the fastest qualifying time in the heats of the women's 100m butterfly.
The reigning world champion broke Inge de Bruijn's 12-year-old Olympic record of 56.61 when she sailed through her heat in 56.25 seconds, almost a second ahead of her nearest rival, China's Lu Ying (57.17).
Australia's Alicia Coutts was third overall while Swedish world record holder Sarah Sjostrom qualified fourth fastest in 57.45.
"I am really happy with how fast it was, and I feel like it is only going to get faster," said Vollmer.
"I am going to keep working on what I have been working on, having a tighter rhythm, hitting my walls... Just making sure I do all those extra things."
The top 16 qualified for Saturday night's semi-finals, with the fastest eight from the semis advancing to Sunday's final.
Australia's Jessicah Schipper, a bronze medallist in the event in Beijing four years ago, missed out after finishing 24th overall, while Poland's Otylia Jedrzejczak, the 2004 Olympic champion over 200m, was 25th.
Elizabeth Beisel sailed into the final of the women's 400m individual medley with the fastest qualifying time.
The American, who is looking to add the Olympic gold to the world title she won in Shanghai last year, was just 0.05 ahead of China's Ye Shiwen, setting the stage for an epic final that could rival the men's event.
Australia's Stephanie Rice, the defending Olympic champion and world record holder, got into the final in seventh place after saving her energy for the final.
"There are some really hot times," said Rice. "These are races where you have to be quick to make the final because everyone wants to be in there, but at the same time you have to try and conserve something because it is such a taxing race."
Australia stormed into the final of the women's 4x100 metres freestyle final when they clocked the fastest time in the morning heats.
The quartet of Emily Seebohm, Brittany Elmslie, Yolane Kukla and Libby Trickett won their heat in a combined time of three minutes, 36.34 seconds.
The United States were second, just 0.19 behind, with the defending Olympic champions, the Netherlands third, and China fourth.
The eight countries that qualified for the final were separated by less than two seconds but should all go much faster in the evening after resting their best swimmers for the heats.
- Kosuke Kitajima