The England and Wales Cricket board has criticised comments from the former Yorkshire chairman Colin Graves in which he referred to incidents of racism at the club as “banter”.

The club has admitted four charges issued by the ECB in relation to their handling of allegations of racism and bullying made by their former player Azeem Rafiq and faces a sanctions hearing on 27 June.

Related: Mitchell Starc: ‘The easy money is in franchise cricket, it’s the fast track to notoriety’

One of the charges the club admitted is a failure to address a systemic use of racist and or discriminatory language over a prolonged period. In an interview, however, Graves said no racism allegations were raised to him when he was chairman between 2012 and 2015.

The ECB said it was disappointed by the remarks, adding: “We must never again hear the accounts like Azeem Rafiq’s, where racist slurs are used as part of normalised language.

“These events, along with many issues experienced by Azeem and others during their time at Yorkshire, have been upheld more than once, including during proceedings overseen by the Cricket Discipline Committee (CDC).

“We vehemently disagree that this is ‘just banter’ and believe any debate in regard should stop immediately. Racism isn’t banter.”

On Sky Sports News on Monday, Graves said: “I think there have been odd occasions where words have been said that people may regret afterwards.

“I don’t think it was done on a racist, savage basis. I think there was a lot of – I know people don’t like the word banter – but I think there could have been a lot of banter in there about it, and I know people don’t like that.

“But when you play cricket and you’re part of cricket teams, and you’re in cricket dressing rooms, that’s what happened in the past. But the world has changed, society has changed, it’s not acceptable. I understand that, I accept it, full stop.”

On the allegations of racism, Graves, who withdrew his application to return as Yorkshire chairman last week, said: “When I was chairman of the club, when some of these allegations were made, I can tell you now, nothing was brought up within the club.”

Six former Yorkshire players were sanctioned last month over the use of racist and or discriminatory language. A seventh – the former England captain Michael Vaughan – was cleared of using such language.