The Swagger Jagger singer appeared on the ITV reality singing contest in 2010 aged 16, mentored by judge Cheryl Cole, and became the target of an online hate campaign before finishing in fourth place.
Lloyd told the Guardian newspaper: “What baffles me is that they created a character but then, after the show, expected me to be a fully fledged, respected artist. I had to do damage control. And still, at 26 years old, I have to do damage control. Years later, I do have some anger towards people for doing that to me.
“I definitely don’t say hello and shake hands with certain people.”
She also revealed she refuses to perform her Number 1 single Swagger Jagger, the first song she released after The X Factor, as it reminds her of the “angry, afraid” person she was then.
Lloyd was targeted by the tabloid press for her Romani Gypsy heritage and dubbed “the Primark Cheryl Cole” on social media.
Lloyd said that she – along with other contestants – was taken to talk to two mental health professionals about how she was coping with the public scrutiny.
But she believes “reality TV baggage” almost destroyed her career.
Last year the government launched an inquiry into the treatment of reality TV contestants following the death of a guest on The Jeremy Kyle Show and Lloyd said she hopes things have improved since.
She said: “That awareness will continue, hopefully – where people think: ‘D’you know what, I don’t want to have any involvement in this process of destroying another person.’”
Lloyd is now mother to 20-month-old daughter Delilah-Rae with her husband of six years and is set to release her third album later this year.