"The thing I find hurtful about it is people are questioning my integrity and what I say... that this is something that I would be embarrassed about, or I would hide," he said in an interview with ABC TV.
"I don't want to offend anyone, whether they're friends that are gay or whatever else, by getting angry about it, frustrated about it.
"The only part of it I find frustrating is that people think I'm lying."
The five-times Olympic champion said speculation that he was gay had perhaps arisen because he did not fit into the stereotypical idea of how a successful Australian athlete should behave.
"I behave differently, I guess, I like different things," he added.
"I'm a nerd, I'm just someone who just happened to be good at a sport as well... I like beautiful things in the world, I like the aesthetics of those things."
Thorpe is promoting his autobiography, "This is Me", in which he reveals that he has also struggled with severe depression and had on occasions considered suicide and turned to heavy consumption of red wine to manage the illness.
"I was probably in my mid-teens when I realised something wasn't quite right," Thorpe, who won his first world title as a 14 year old, said.
"And at the time, I didn't know that it was depression and it wasn't until a little bit later that I actually went to seek some help.
"I didn't actually have the words to put around what I was feeling, and I think a lot of young people struggle with that."
Despite failing to qualify for a spot on the Australia team for the London Olympics after his return from retirement, Thorpe said he was "in a good place" as he continued training in an attempt to compete at the 2013 world championships.
"I'm happy with what I'm doing. I'm loving swimming, I'm enjoying my life, things are good," he said.
- Sports & Recreation
- Mental Health