Villa at least ended a run of four straight defeats with a 1-1 draw at home to Swansea, but there were more boos from the Villa Park faithful as the visitors dominated.
Of the seven teams below Villa in the Premier League table, five have already parted company with a manager this season.
He said: "I've got a good relationship with Randy, but that doesn't mean I'm immune to it. I always have that fear. I always had that fear as a footballer and I always do when I'm the manager. I think you've got to have that.
"I don't think it will get any tougher than what happened last year and he was great with me then. I spoke to him last night and he's been good now as well."
In the same week last season Villa lost all three games, conceding 15 goals and scoring none, and when Crystal Palace's Dwight Gayle netted the winner in injury time on Thursday it appeared festive cheer would again be entirely absent from Villa Park.
Things began well on Saturday with Gabriel Agbonlahor netting after only seven minutes - the first time Villa have scored in the first half of any home league game this season.
Agbonlahor rushed to Lambert and the squad celebrated exuberantly in a huddle in front of the Villa bench.
"I didn't know that was going to happen," said the boss. "They're a group that are really close together, which you need."
But Swansea quickly took control of the game and there was no doubt they deserved their equaliser, which was headed in by Roland Lamah in the 37th minute.
The visitors dominated the second half completely, but could not turn possession into chances, with Brad Guzan having only one real save to make from Jonathan De Guzman's free-kick.
Lambert cited the absence of injured trio Christian Benteke, Ron Vlaar and Jores Okore, saying: "It's a game that you'd love to win, but a point stops that sequence of losing, which was vital for us.
"We're without big players, and it's a lot for us to carry. We're a young squad and when you lose people like Benteke, Vlaar and Okore, it's a big chunk of our spine.
"We (his management team) have only been here 17 months and I knew the first few years was going to be really tough, but we're halfway there when you look at the table. It's a big point.
"Swansea are a good side, they pass the ball really well, but Guzan hasn't really had many saves.
"Some games you play really well and end up losing, some games you have to fight tooth and nail to get something, and we certainly did that."
- Sports & Recreation
- Randy Lerner