The Italians, who have only won one domestic game in two seasons, had led by 10 points just after half-time.
Ulster clawed their way back to lead by four going into the closing minutes but had to survive a yellow card to prop James French and Zebre pressure in their 22 before a knock on from the Italians secured the win for Dan McFarland’s side.
Michael Lowry got one of Ulster’s six tries and he was relieved to leave Parma with maximum points – but admits there is a lot for Ulster to work on.
“Delighted with the win and five points but it was far from perfect,” said Lowry. “We definitely have a lot to work on defensively and accuracy wise, but I felt we attacked pretty well at times.
“Hopefully, we are starting to see our attacking shape coming through and it is real exciting, it was a nice dry day and we want to have a craic and enjoy it.
“We definitely enjoyed it and, fair play to Zebre, they were always going to be up for it, they were really physical and gave us a really good challenge up front.
"A massive part of the game is set-piece and I thought Zebre took it to us there, which was quite frustrating and we’ll have to be a lot better there against the Bulls.”
Lowry was playing in the unusual position of wing to accommodate Will Addison’s return from injury.
“I am always up for a new challenge and as long as I get on the pitch that’s the main thing for me and I don’t really mind where I play,” he said. “I am up for learning all different positions on the field and it just helps your overall game.
“I just want to play rugby and if that is on the wing it’s on the wing or if it is at full back it is full back.”
Ulster face the Bulls on Sunday evening as they unveil the new artificial pitch at Kingspan Stadium.
Zebre captain Giovanni Licata scored the first of the game’s 12 tries after four minutes with out half Geronimo Prisciantelli converting.
Ulster out half Jake Flannery crossed for a try converted by Nathan Doak before Licata got his second touchdown which Prisciantelli converted.
Lowry went over on 16 minutes for a converted try.
Zebre regained the lead with full back Lorenzo Pani converted try six minutes later.
Jacob Stockdale replied for Ulster with Doak converting.
Ulster lock Kieran Treadwell was sin-binned for illegally stopping a maul and Zebre profited with Scott Georgey getting the home side’s bonus point try to give the Italians a 26-21 half-time lead.
Winger Simone Gesi extended Zebre’s advantage with a try a minute after the restart.
Two tries for captain Tom Stewart either side of a Stockdale touchdown with Doak converting two of them gave Ulster a nine-point lead.
Gesi’s second try on 70 minutes set up a nervy finish which was compounded by French’s late red card but, despite a man advantage, Zebre couldn’t breach the Ulster line for a winning score.