Taylor rejected an offer to keep the reins of the test team after his removal as limited over captain, a move local media linked to his soured relations with head coach Mike Hesson.
The disgruntled batsman has also opted out of New Zealand's tour of South Africa under McCullum, who is now in charge of the team in all three formats.
McCullum said he had no role in captaincy decision.
"That's probably the thing that has come out this week which has annoyed me the most," McCullum was quoted as saying by Fairfax NZ News on Saturday.
"It cuts right to the bone, that people would question your integrity and your character like that," the 31-year-old stumper-batsman told reporters in Lincoln.
"To be totally honest, it's highly insulting for people to cast aspersions when they certainly don't know any of the circumstances.
"I had absolutely no involvement in that recommendation whatsoever. I was asked if I would lead the one-day and Twenty20 team two days ago, and yesterday I was asked if I'd lead the test team after Ross turned it down.
"That's about as much involvement as I've had in the entire situation."
McCullum said Taylor still had a big role to play with New Zealand.
"Ross is an integral member of this team, he's vitally important for us increasing our performances and he's also a very popular member of the team as well, so there are a lot of guys who will be feeling for him at the moment.
"We'll welcome him back when he's good and ready." (Reporting by Amlan Chakraborty in New Delhi; editing by Peter Rutherford)
- Sports & Recreation