An Andy Barcham 20-yard rocket after 90 seconds saw Pompey take the lead at Fratton Park in their first match since sacking Guy Whittingham.
Veteran striker David Connolly missed a golden chance to double the lead in the 58th minute but he dragged his finish wide and they would pay for their missed chances.
The Shrimpers looked to be dead and buried in the 65th minute when substitute Cauley Woodrow saw red for elbowing Wallace after only being on the pitch for four minutes.
But Atkinson started the comeback in the 75th minute when he slotted under Trevor Carson.
Corr turned the game on its head and ensured Pompey left the field to a chorus of boos five minutes later, heading home a cross from captain John White.
- Sports & Recreation
- David Connolly