An own goal from Latics striker Matt Smith sent Forest in ahead at the break but the tie turned on its head to leave Alex McLeish still searching for his first win as Forest manager.
Reds defender Danny Collins was sent off after an hour at the City Ground as Oldham struck three times in the second half, twice from Robbie Simpson, to give Paul Dickov's outfit a deserved victory after a turbulent week that began with his backroom staff being placed on gardening leave.
Jose Baxter's goal gave the visitors the breathing space they required before Billy Sharp came off the bench to net a consolation deep into stoppage time.
Forest included Jonathan Greening on the subs' bench following his loan spell with Barnsley, but striker Matt Derbyshire was ineligible having played for Latics in the earlier rounds of the competition and Marcus Tudgay sidelined with a broken collarbone.
Oldham were without suspended midfielder James Wesolowski, and winger Cristian Montano also missed out with a torn thigh muscle, although Simpson returned from a three-match ban.
Forest were the first to show when an early effort from Andy Reid went wide of the target, and it was from the Irish midfielder's 13th minute cross that Smith put through his own goal.
Two long-range efforts from Radoslaw Majewski fizzed wide of Oldham goalkeeper Dean Bouzanis' post as the hosts went for a second.
And it required a top class save from Bouzanis to prevent Dexter Blackstock from doubling Forest's advantage.
Perhaps crucially, Reid sustained an injury and was replaced by Lewis McGugan eight minutes from the break.
Within a minute of entering the action, McGugan had a chance of increasing Forest's tally after Cliff Byrne's foul on Simon Cox gave the hosts a free-kick 35 yards out.
He struck the ball well enough but his effort struck the defensive wall and the Latics cleared the danger.
Blackstock got on the end of a Chris Cohen corner five minutes from the break, but Simpson cleared the danger.
Baxter fashioned two opportunities for Oldham at the start of the second half but missed the target on both occasions.
It was nevertheless a spirited start to the second period for Dickov's side who were encouraged to push back their Championship hosts.
And when the lively Simpson equalised from 25 yards after 54 minutes, it was no more than they deserved.
Simpson then added his second four minutes later when he headed home a Dean Furman cross.
All seemed lost for Forest when Collins received his marching orders for a professional foul on Smith and Baxter added insult to injury by rifling home the resulting free-kick - Oldham's third goal in seven minutes.
Sharp's fine left-footed volley in added-time proved to be was too little, too late for McLeish's men.
- Sports & Recreation
- Robbie Simpson