The former world no.1 missed two match points against a player ranked 12, eventually slipping to a 7-6 (4) 2-6 7-6 (2) defeat.
Murray was furious with Fognini, who is no stranger to overstepping the mark, when he shouted as the Scot was about to put away a volley in the 11th game of the deciding set.
Murray confronted his opponent at the change of ends, telling the Italian to "shut up" as umpire Fergus Murphy tried to dissuade him from taking it any further.
Having failed to serve out the match at 5-4, Murray had another chance at 6-5 but Fognini broke back once more and then dominated the tie-break, clinching victory after three hours and nine minutes.
The handshake between the pair was brief, with Murray continuing to press his point to Murphy before leaving the court.
Murray has had many tough tussles with Fognini over the years, winning four and losing three of their previous seven matches.
The Italian landed the first blow with a break of the Murray serve to lead 2-1 only for Murray to reel off three games in a row, securing the second break with a backhand winner flashed brilliantly cross-court.
However, Fognini produced some high-quality points of his own to get back on serve immediately and, after both men had survived pressure on their own serves, it was the Italian who edged the tie-break, a double fault proving costly for Murray.
The Scot responded very well, breaking Fognini for 3-2 in the second set and then making it five games in a row after his opponent had a mental lapse.
He was back fully focused at the start of the decider, though, and it was nip and tuck all the way through to the dramatic final stages.
Cameron Norrie was also beaten in the second round, losing 6-3 6-1 to US Open finalist Daniil Medvedev.
Meanwhile, Murray will be welcomed back to the Australian Open in January a year after the tournament bade him an emotional farewell.
Tournament director Craig Tiley confirmed the former world number one, who can enter events using a protected ranking of two, will be in Melbourne for the first grand slam of 2020.
"For sure Andy will be here," Tiley told Australian radio station Triple M on Tuesday. "I was on the phone to his agent this morning. He is going to be in Australia early.
"He is ready to return. Remember he said goodbye a year ago. It's great in that period he had surgery and has rebounded really well."
Reporting by Press Association.