Still with a lusty rendition of La Marseillaise ringing in their ears, Fabien Galthié's players trailed to a Mark Telea try within two minutes of the kick-off.
But Thomas Ramos converted a penalty to bring them straight back and he added another one to send France 6-5 ahead after 20 minutes.
He swapped penalties with New Zealand's Richie Mo'unga and there was a slender 9-8 lead for the hosts at half-time.
That advantage disappeared soon after the restart though as Rieko Ioane set up Telea for his second try of the encounter.
But at 13-9, that was as good as it got for New Zealand.
Fly-half Matthieu Jalibert sent Damien Penaud clear to restore France's ascendancy and Ramos added the two points for the conversion.
Ramos notched up two more penalties - one while New Zealand were down to 14 men - to make it 22-13.
In the closing stages and Melvyn Jaminet added another try to gloss up the scoreline.
“It was a really, really tough game," player-of-the match Grégory Alldritt told the British broadcaster ITV.
“I said before, the one who was going to win was the team that played for 80 minutes. It’s a great night for us.
“It was really, really hot," the 26-year-old added. "There was fantastic support in the stadium. We are looking forward to the next game. We are not champions now so we have to keep working and go step by step.”
"We have nothing to lose, for us it's just about being proud to be against them, to challenge them."