The 26-year-old has been at Fratton Park since mid-February and has made eight appearances for the struggling Sky Bet League Two side.
But with the Bees suffering a heavy 3-0 defeat at Rotherham in midweek, boss Mark Warburton wants Diagouraga back in his own ranks.
Warburton told his club's official website: "The idea of sending Toumani out on loan was for him to get some valuable game time, but we now need him back."
- Sports & Recreation
- Mark Warburton
- Fratton Park