Forest defender Danny Collins was sent off after an hour at the City Ground as Oldham struck three times in the second half, twice from Robbie Simpson, to give Paul Dickov's outfit a deserved victory.
Forest included Jonathan Greening on the subs' bench following his loan spell with Barnsley, but striker Matt Derbyshire was ineligible and Marcus Tudgay sidelined with a broken collarbone.
Oldham were without banned James Wesolowski, and winger Cristian Montano also missed out with a torn thigh muscle, although Simpson returned from suspension to make his mark.
Forest were the first to show when an early effort from Andy Reid went wide of the target, but after 13 minutes the Forest star crossed from the right and Oldham striker Matt Smith put through his own goal from just six yards.
Forest had begun the third-round tie brightly and the early goal had their opponents rattled.
Two long-range efforts from Radoslaw Majewski fizzed wide of Oldham goalkeeper Dean Bouzanis' post as the hosts went for a second.
And on the half hour it was only a top-drawer save from Bouzanis that kept the scoreline down, the Latics goalkeeper denying Dexter Blackstock whose effort from 12 yards looked destined for the net.
Reid had taken a knock in the first half and eight minutes from the break he was replaced by Lewis McGugan.
And within a minute of entering the action, the midfielder had a chance of increasing Forest's tally after Cliff Byrne's foul on Simon Cox gave the hosts a free-kick 35 yards out. McGugan struck the ball well enough but his effort struck the defensive wall and the Latics cleared the danger.
Blackstock got on the end of a Chris Cohen corner five minutes from the break, but Simpson cleared the danger.
Oldham's Jose Baxter fashioned two opportunities at the start of the second half but missed the target on both occasions.
It was nevertheless a spirited start to the second period for Dickov's side who were encouraged to push back their Championship hosts.
And when the lively Simpson equalised from 25 yards after 54 minutes, it was no more than they deserved.
Simpson then added his second four minutes later when he headed home a Dean Furman cross, and it was game over when Danny Collins received his marching orders for a professional foul on Matt Smith and Baxter rifled home the resulting free-kick from 20 yards for Oldham's third goal in seven minutes.
Billy Sharp's added-time effort for Forest was too little, too late.
- Sports & Recreation
- Robbie Simpson
- Danny Collins
- Matt Derbyshire