As Williams took a shock front-row sweep with Massa and Valtteri Bottas, Formula 1's dominant force Mercedes had to settle for third with championship leader Rosberg, while both Hamilton's Q3 runs went awry.
The Briton's first attempt was wasted when he made a mistake at Turn 8 and ran off the track, leading to what was already a slow time being disallowed for breaching track limits rules.
Hamilton then spun off under braking for Turn 2 on his final pole shot.
Asked if he could explain the incident, Hamilton replied: "I can't really. I just spun. I hit the brakes and locked the rears. I'm not really sure [why] - I have to look at the data."
He remains optimistic that he can recover in Sunday's race at the Red Bull Ring.
"I'm OK, it's a long race tomorrow," said Hamilton. "There aren't that many corners here but there are long straights and we have the best engine, so hopefully I'll be able to work my way up."
Although Hamilton did not set a time in Q3, he moved up from 10th to ninth as Nico Hulkenberg's lap was disallowed for a track limits infringement.
Fernando Alonso was just over half-a-second off the pace in fourth place.
Daniel Ricciardo was fifth fastest ahead of McLaren's Kevin Magnussen, with Daniil Kvyat a superb seventh.
Kimi Raikkonen was the slowest of those actually to register a lap time in Q2, ending up eighth.
Sergio Perez was 11th after being bumped out of a Q3 position late on, missing out by just under a tenth of a second.
But he will start 16th thanks to the five-place grid penalty he received for causing a collision with Massa in the Canadian GP two weeks ago.
The Mexican's former team-mate Jenson Button, whose morning practice session was ruined by a rear brake problem, and Sebastian Vettel, whose lack of pace in the final sector cost him, were just behind Perez.
Pastor Maldonado outqualified Lotus team-mate Romain Grosjean for the first time this year, the pair sandwiching 15th-placed Jean-Eric Vergne's Toro Rosso.
Adrian Sutil was the fastest of those to be knocked out in Q1, ending up just four-thousandths slower than Maldonado.
He was comfortably ahead of Sauber team-mate Esteban Gutierrez, with Jules Bianchi 19th fastest.
Kamui Kobayashi did a good job to split the Marussias, a tenth faster than Max Chilton, with Marcus Ericsson unable to improve on his second run and ending up last.
- Sports & Recreation
- Motor Racing
- Valtteri Bottas