Palace twice came from behind, with Eberechi Eze’s penalty quickly cancelling out Vitalii Mykolenko’s early opener and Odsonne Edouard capitalising on a howler from James Tarkowski to level up after Abdoulaye Doucoure’s goal.
But they had no answer a third time after Gueye kept his cool to seal victory and propel his side to consecutive away wins.
It all came after an electric start. Palace failed to clear their lines as Mykolenko’s shot was blocked, and as the ball broke wide on the right the Ukrainian found space inside the box and climbed highest to nod Jack Harrison’s cross past Sam Johnstone after just 55 seconds.
Palace fans may have been stunned but their team quickly hit back.
Within three minutes they were level, and it was Eze, back in the side after a hamstring injury, who danced into Everton’s box and drew a foul from Jarrad Branthwaite.
VAR checked and saw no reason to overturn referee Sam Barrott’s penalty award, leaving the Eze the task of calmly rolling the ball past Jordan Pickford.
Selhurst Park howled for a second spot-kick when Eze again went down under apparent pressure inside the box. This time the referee deemed the forward had dived, and rather than a penalty, a yellow card was Eze’s reward.
Edouard forced Pickford into a save low to his right in added time at the end of the half in what was a rare instance of attacking threat from the home team.
Everton had won three of their previous six in the league, and after the frustration of failing to hold on to their early lead they began the second half in similarly urgent fashion.
A corner from the visitors’ right was cleared only to the edge of the box, where Amadou Onana scooped the ball square to Mykolenko. For the second time in the game he was given too much time to line up an effort on goal, and as his volley cannoned back off a post there was Doucoure unmarked to tap home.
The advantage looked fragile. On the hour mark, Mykolenko and Gueye almost produced a comical own-goal, getting in one another’s way as Jeffrey Schlupp’s cross dropped into the box and they required Pickford’s fingertips to keep them from bundling the ball over their own goal line.
Edouard was growing as a threat and with 25 minutes to go he handed Jefferson Lerma a golden chance to level, coming inside from the left of the box and cutting the ball back, only for Lerma, free on the edge of the box, to fire wide.
Michael Oliseh came off the bench for his first appearance of the season and drew a roar of anticipation from around Selhurst Park with a shot from 20 yards that deflected narrowly over.
Palace were by now dominant, and their second equaliser came courtesy of a defensive calamity. A high, headed ball into the box looked an easy mop-up job for Tarkowski, but rather than nod it clear he left the ball for his goalkeeper, and in stole Edouard to tap home.
Still Palace could not hold on to their point, and Everton roared back at them once more with four minutes to go, this time decisively.
Doucoure received the ball in midfield and looked up to see Gueye racing through the centre. Doucoure’s pass was weighted expertly and Gueye needed barely to break stride as he evaded Tyrick Mitchell’s lunging challenge and guided it beyond Johnstone.