Eyebrows were raised on Monday when Sherwood was handed an 18-month contract as Spurs' new head coach, and it was clear from the Boxing Day display against Albion that he still has much work to do.
Tottenham held onto the lead for less than two minutes and they were regularly exposed at the back by some fluid counter-attacking from the visitors, who arrived at White Hart Lane having not won in their previous seven games.
Emmanuel Adebayor - whose error led to the Olsson goal - had an off day up front and had it not been for some good goalkeeping from Hugo Lloris, Tottenham could have suffered another damaging defeat.
A smattering of boos rang out at the final whistle to sum up how the home supporters viewed the performance.
"The fans will be disappointed, obviously. I understand that," Sherwood said.
"We are at home and, with respect, against West Brom you should be winning.
"It's a game we should have won. We should have played better and we are capable of playing better.
"I thought we could be a bit more adventurous around the penalty box and show a little more guile."
Sherwood was appointed caretaker following the dismissal of Andre Villas-Boas and he delivered an encouraging win over Southampton last weekend.
The former Spurs midfielder laid out his unconditional commitment to attacking football in that match and he repeated that stance at White Hart Lane on Boxing Day.
Sherwood omitted a holding midfielder from the line-up, and Tottenham were exposed as a result, but he insists the team will become a more cohesive unit once they have spent more time working together at the end of the hectic festive schedule.
Still, with his only previous managerial experience coming at youth level, he knows his appointment is a punt on Levy's part.
"It's a gamble because I've never done it before," the 44-year-old said.
"But there are a lot of good managers who had never done it before and become a success.
"I was never a professional footballer until I stepped on to the field."
The fact that he only has a contract until the end of next season has prompted suggestions that Sherwood is keeping the seat warm for someone with more experience.
"I regard (that) as an insult, but it is only an insult if I go and win all the games," Sherwood said.
"What can we do? If we win all the games and someone else comes in and keeps it and brings someone else in after I've won the league then I'm going to Real Madrid," he added with a smile.
Sherwood says the players have responded well to his appointment, and bemoaned the fact that he had almost a full team of unfit players unavailable.
On paper this was a perfect first league match for the newly-appointed Tottenham boss. West Brom sacked their manager Steve Clarke two weeks ago after they started tumbling down the table.
Caretaker manager Keith Downing has steadied the ship somewhat, and had it not been for a couple of good saves from Lloris, the former Wolves midfielder could have recorded his first win.
Former Real Betis manager Pepe Mel is understood to be the front runner to succeed Clarke, and Downing - who does not want to be considered - wants the vacancy to be filled soon.
"I hope (an appointment) is close for the players' sake," said Downing, who will stay in London with his players ahead of Saturday's game against West Ham.
"I know there's a lot of speculation but I have really tried to concentrate (on the game).
"As far as I know (I will be in charge for Saturday). I'm down (in London). We have planned for the next two or three days. We are ready to go and if anybody steps in then I will gladly support whoever comes in.
"I spoke to (director of football) Richard Garlick but the interview process is ongoing."
- Sports & Recreation
- Jonas Olsson