Hirscher, part of the Austrian men's team that left Vancouver empty handed, had been tipped as a clear medal contender in both races he is entering, the giant slalom and slalom.
"What good is fourth place to me? I might as well go home again," he told reporters after the second leg. "I didn't need to come just because it's nice here. It would have been better to quit the race than come fourth.
"I've got an Olympic jinx."
In addition to coming fourth in the giant slalom in Vancouver, the 24-year old claimed fifth place in the slalom four years ago. Asked if he would not draw motivation for Saturday's slalom, he shook his head and kept repeating "no".
"You can write up your front page title 'Rockstar Hirscher fourth again. Yippie.'"
Hirscher has been on the podium in 12 races this season. He has already won the overall World Cup - considered the ultimate skiing title - in 2012 and 2013, a feat not achieved since his compatriot Stephan Eberharter won it in 2002 and 2003.
But Hirscher said he had not felt bad throughout the race, just made "a few little mistakes" and he could not do more.
He could only look on as U.S. American Ted Ligety completed his second leg in the 14th fastest time, running down his clear lead from the first leg.
At that point, Hirscher was in third position and watched Ligety almost have to climb uphill at one stage to reach a gate. "I thought that could have been it. It's incredible.
"It's the worst race not to get onto the podium but someone has to be first loser."
By contrast, team mate Matthias Mayer, who won gold in the downhill race early on in the Sochi Games, helping the Austrian men's team banish the memory of medal-less Vancouver, felt happier about his sixth place.
"For me it was pretty good. My best result in GS was 16th place last year at Beaver Creek. This year it was a 22 in St Moritz and now I'm sixth here. So I can be happy," Mayer said.
"Of course it's bad for Austria. A fourth position for Marcel. You don't want to be fourth at Olympic Games."
There had been some controversy about Mayer starting in Wednesday's race as he had only started in three World Cup GS races ahead of the Olympics - although five are usually required - prompting French and U.S. officials to complain.
"I heard about it after the first leg that some guys had some problems that I started but last week I got my green light from the FIS to start here, so I see no problem."
- Sports & Recreation
- giant slalom