Allardyce felt mistakes at the back by his men in the second half of the Barclays Premier League contest - which was 1-1 at half-time after Peter Odemwingie had cancelled out Andy Carroll's early opener - certainly cost them dear as the Potters wrapped up victory.
Marko Arnautovic scored after squeezing through a crowd of West Ham players and Odemwingie then finished off a counter-attack move by lashing the ball in.
But the Hammers boss also pointed the finger at Pawson, who did not give a penalty for what Allardyce viewed as a "blatant" handball by Stoke defender Marc Muniesa in the home box shortly after Arnautovic had made it 2-1.
"I think in the cold light of day, we have to blame ourselves," Allardyce said.
"But there is a big part to blame the referee for as well.
"There is absolutely no doubt about that in my mind - I have just reviewed the situation on the laptop and watched a blatant handball that stops Andy Carroll chesting the ball down and maybe having a chance to shoot and score a goal.
"It is arm to ball, not ball to arm, and the referee is in a fantastic position to see it.
"So I am totally and utterly baffled by that decision.
"But the other thing is that it is really our fault for the poor defending we showed in the second half - on two occasions, where we just had to defend correctly, and when we were in the ascendancy, particularly with the first goal (Arnautovic's).
"We were in control, and it was looking like if anyone was going to score at that stage, it would be us.
"But then we fall asleep, and I just don't know where our defending went then.
"So those two bits - one from the referee and one from our defence - is why we are not here with some points on the board."
- Sports & Recreation
- Marko Arnautovic
- Peter Odemwingie
- Andy Carroll