Rovers striker Ryan Lowe took his tally for the season to 12 goals in all competitions as she stooped to head home in the 32nd minute before converting from the penalty spot after 56 minutes.
U's defender Brian Wilson was at fault for Lowe's penalty, bringing down Jean-Louis Akpa Akpro, before former Crystal Palace striker Clinton Morrison netted an injury-time consolation for the visitors.
The defeat for Colchester was in stark contrast to their 3-1 win over MK Dons in midweek and Dunne did not hold back in telling his players what he thought.
"I'll accept players missing chances when they get into positions because generally you will score," said Dunne.
"But I won't accept what I think was crucial and that was the fact that we switched off twice from set pieces and that's what has lost the game - it's unacceptable.
"People had their jobs to do and they didn't do them and we can't do that. That's why we will be forever fighting and fighting.
"The concentration levels we showed against MK Dons were incredible but we need that to happen every week. We need that consistency and that's what I demand.
"The two goals they scored were unacceptable. We practised for it and it's an individual switch off for the first goal and they switched off from a short corner for their second.
"They're two switch-off moments that have cost us the match because other than that, I thought we played well."
- Sports & Recreation
- MK Dons