The 32-year-old said he heard the chants throughout the match at Cagliari's Stadio Sant'Elia and asked referee Daniele Minelli to stop the game.
Instead, the former Ghana midfielder was shown a yellow card in the 90th minute and then angrily left the pitch, leaving Pescara, who lost 1-0, to play out stoppage time with 10 men.
La Gazzetta dello Sport quoted the former Inter and AC Milan player telling Premium Sport: "You saw what happened. The fans were chanting at me during the first half.
"There was a little kid doing it with his parents standing nearby. So I went over to him and told him not to do it. I gave him my shirt, to teach him that you're not supposed to do things like that. I needed to set an example so he grows up to be nice.
"Then in the second half it happened in that stand again and so I spoke to the referee. And then he really p***ed me off. He told me that I'm not allowed to speak to the fans. I asked him, 'But didn't you hear?'
Read the full article on eurosport.co.uk: Pescara midfielder Sulley Muntari booked after reporting racial abuse to referee