Tom Naylor's first-half header was enough to beat the League One leaders, who are unbeaten in 26 games in the league, in the first round of the famous competition.
The Blues will host another League One club in Leyton Orient in the second round.
Webb said: “It is really special. It almost feels like a third or fourth rounder. We have had a lot of dark years, but hopefully this year is the time we can all celebrate. But there are many games to come and lots of hurdles to overcome.
“We needed them (Portsmouth) to be a little bit off their game and we had to be on ours. I think both things married up today.
"I am biased but I think the best team won today. I thought we deserved the win. I am very proud.
"I think the buzz word from today was passion and I think the fans saw that today. The boys have pushed themselves to the limit.
"I think Portsmouth will go up. Their manager is a class act, the way he spoke to us before and after the game.”
Despite the joy of the win, the camp are keen not to get carried away with a top-of-the-table clash at home to Barnet next up.
"It is hard to play it down because we are ever so excited at what has happened, but believe me all focus now turns to Barnet. That is a fact. That is a bigger game than today.
"The gaffer keeps everyone grounded – he has won promotions before and he wants another – there is a long way to go yet. It is important that we don’t let the cup become a distraction.
"The celebrations were really calm. We can have our celebrations if we get promoted. I think we would have bigger celebrations if we were to beat Barnet next week.”