The 55-year-old had been in charge at the Memorial Ground for 11 months and won just 12 times.
An official statement said the final decision came after the defeat at Bootham Crescent, a match which saw Rovers concede four first-half goals to keep them in the npower League Two relegation zone, just two points clear of rock-bottom Barnet.
Chairman Nick Higgs told Rovers' official website: "We had hoped that the appointment of Mark McGhee would give the club some stability and long-term progress, but unfortunately that has not been the case.
"We have decided that we can no longer continue with the current situation and had to make the difficult decision to part company with Mark. We decided we had to act now before it was too late."
McGhee, who replaced Paul Buckle as manager in January, had seen his side win just one of their last 10 games.
- Sports & Recreation
- Bootham Crescent