New Zealand had not lost a Commonwealth match since the sport made its debut in 1998 and they looked to be heading for a fifth consecutive gold when Sherwin Stowers crossed in the first half.
South Africa, who beat Samoa 35-7 to reach the final, came storming back and Seabelo Senatla grabbed two quick tries either side of halftime before Cecil Afrika scored with three minutes remaining.
Joe Webber pulled a late try for New Zealand, setting up a nervous few minutes for Neil Powell's side but the Africans held on for victory.
"It could have gone either way," New Zealand captain DJ Forbes told reporters. "They put pressure on us but we had a chance to win the game.
"But it didn't go to plan and that's rugby.
"We're happy with second best, so to speak. We are part of the legacy that has gone before us - we'd never been beaten.
"As a country we have a lot of chances to win medals but as a New Zealand rugby player it is disappointing to come second."
Australia, who lost 19-7 to New Zealand in the semi-finals, beat Samoa 24-0 to win bronze.
Rugby will make its return to the Olympics in its abbreviated format in Rio in 2016.
- Sports & Recreation
- New Zealand
- Sherwin Stowers
- Cecil Afrika