The brothers, Manchester United defender Rio and then-QPR player Anton, were among several prominent black players who opted not to wear shirts issued by campaigning group Kick It Out before a round of Premier League fixtures last October.
Carlisle wrote in his new autobiography 'You Don't Know Me, But... A Footballer's Life', that he was "bitterly disappointed" with the boycott and - exempting Roberts - labelled players who took a stance against the T-shirt campaign without explaining their reasons "s***houses".
In extracts published in the Daily Mail, he added: "Kick It Out is one of few bodies doing something positive about the issue, but there are some who evidently feel it is not doing enough.
"Rio and Anton Ferdinand were among a group of high-profile abstainers along with Joleon Lescott, who has not worn one since Newcastle's Turkish midfielder Emre was cleared of racially abusing his Everton team-mate Joseph Yobo, and Jason Roberts.
"At least Jason was vocal about his reasons. I don't mind people disagreeing but at least say why."
Reading striker Roberts joined the boycott in protest at what he saw as lenient sanctions handed to Liverpool's Luis Suarez and in particular Chelsea captain John Terry, the latter for racially abusing Anton Ferdinand.
He told the BBC at the time: "I'm totally committed to kicking racism out of football but when there's a movement I feel represents the issue in the way that speaks for me and my colleagues, then I will happily support it.
"I think people feel let down by what used to be called 'Let's Kick Racism Out of Football'. People don't feel like they have been strong enough. Unless they're independent, unless they don't have to explain their actions to anyone then they won't be held accountable."
Carlisle continued: "Jason apart, the Premier League 'big' players happily throw in a grenade but do nothing when it explodes, nothing other than appear to hide behind their advisers.
"That's because they are s***houses. They have made their 'statement' for all to see but how many are prepared to stand in front of the cameras and be counted?
"The impression conveyed is that I have failed (as PFA chairman) to get people to unite and support the cause. Not only that, I feel under attack from a group of players questioning my position and representation of them on the issue. This has really hurt me."
- Sports & Recreation
- Joleon Lescott
- Jason Roberts
- Premier League
- Anton Ferdinand