A second-half goal from Harry Kane and a brace from Lucas Moura piled the pressure on Jose Mourinho, who faced chants of "you're getting sacked in the morning" from the Spurs fans.
There was a mixed reception on the final whistle from the Old Trafford crowd as United followed up defeat at Brighton with another loss, prompting questions over the future of the former Chelsea manager and Neville admitted he has a huge job on his hands if he is to try and reverse the club's fortunes.
"The manager is always the person where the buck stops," he said on Sky's Monday Night Football.
READ MORE: Gossip: Man Utd star Pogba 'makes Barcelona happy', Henry 'to reject Bordeaux'
READ MORE: Jose Mourinho rant raises more questions than answers
READ MORE: Mourinho’s press conference transcript - Every word spoken
"Jose Mourinho has been the most successful manager along with Pep Guardiola in the last 20 years in football and he is going through a difficult moment now over the last couple of seasons with Manchester United.
"This is now the biggest task he has had in his life and the biggest challenge he has had in football.
"He has got to come out of it because I have to say I look at the squad and the team and I'm not sure what the best XI is.
"He is going to have to back certain players and just stick with it because at the moment they are changing every game.
"The buck stops with the manager, it always does and it has to. At the moment there doesn't look to be a balanced outfit there at all."
Former England goalkeeper Peter Shilton was also critical of the United defence, which Mourinho had been keen to strengthen on the summer transfer window.
He said on Twitter: "Big defeat for United tonight any good side has to keep clean sheets that is the base of a great team! Not good enough for Mourinho or United tonight !! Clean sheets or sack!!"
After Mourinho stopped to applaud the home fans on his way to the tunnel at full time, former Celtic striker Chris Sutton added on BBC Radio 5 Live: "Maybe that stance by Jose Mourinho at the end was trying to show unity, but who is he kidding? Has Mourinho just walked into this job? No.
"He has had two years to build this team and he has to take responsibility. His managerial record is fantastic, but if United want to accept mediocrity, then that is what they have. Man City and Liverpool are in a different league."