Togo were the targets of an attack during the 2010 African Nations Cup in Angola when separatist gunmen ambushed their bus in the Cabinda province, killing three members of the squad and forcing them to withdraw from the tournament.
FIFA have moved Friday's Group I qualifier to Tripoli after clashes between protesters and a militia killed at least 31 people on Saturday in the eastern Libyan city of Benghazi, where the game was initially scheduled to take place.
However, Romao said the move was not enough to convince him to play.
"I'm about to take off from Lome to come back to France," Romao, who plays for Olympique Marseille, said on Twitter.
"After having been through a traumatising experience in 2010 at the African Nations Cup in Angola, I don't want my family to live with that stress again," added the 29-year-old, who has 50 international caps.
"Tripoli or Benghazi, what's the difference? I will change my mind only if FIFA's officials who have made the decision come with us."
Libya, whose former ruler Muammar Gaddafi was toppled during the 2011 civil war, drew at home with Democratic Republic of Congo in their first home game in more than two years on Friday, played under high security in Tripoli.
They top the four-team group with six points from four games, with Togo bottom on four.
- Unrest, Conflicts & War
- Sports & Recreation
- African Nations Cup