The Scot, who said he felt "physically good" after beating young American Frances Tiafoe in the first round in New York in his first competitive match in nine months, backed that up with a courageous 6-3 3-6 7-5 win over the fifth-seeded German, a semi-finalist at the Australian Open earlier this year.
A single break of serve lifted Murray to the first set before his opponent converted one of his four break points in the second to level the match.
But the 33-year-old Murray, who gained a wild card into this tournament, regrouped and broke for a 3-1 lead in the decider and then moved 4-1 in front, only for Zverev to reel off four straight games to go 5-4 ahead and give himself the chance to serve for the match.
However, Murray broke again to level before holding serve, leaving Zverev serving to stay in the match.
He was unable to do so as Murray converted his first match point to seal victory in two hours and 31 minutes.
It was a hugely encouraging display from the former world number one, who has admitted has had to lower his expectations since battling injury problems, with the US Open starting on August 31.
"I don't think either of us played particularly well when we were close to the finishing line, but (that was) maybe to be expected since we haven't played any matches for such a long period," Murray said.
"He didn't serve so well at the end, which obviously helped me but I fought hard and it is a good effort to win that after not playing for a while."
Murray will take on Canadian Milos Raonic, who knocked out Britain's Dan Evans, in the last 16.
Murray leads Raonic 9-3 in career meetings, including winning their past eight encounters, one of them the 2016 Wimbledon final.
"I have played him a lot. Huge serve," Murray said.
"I saw a bit of his match today against Dan Evans. He is serving well, playing aggressive, which is when he plays his best."
"We have played each other in some big matches. We played at the US Open, Cincinnati, Wimbledon final, the Australian Open. He has had his injury struggles as well, so it is nice to see him back."
Earlier in the day, Johanna Konta eased through her opening round match, sealing a 6-2 6-0 win over Kirsten Flipkens.
It was her first win since the resumption of the WTA Tour.