Towers had been due to fight Browne in final eliminator for the Commonwealth title in Melbourne on Thursday but his unsavoury past has come back to haunt him.
Towers, who has a 14-0 record, served six and a half years of a 13 year jail sentence for kidnapping back in 1997 and that has prevented the 6ft 8in Sheffield heavyweight from obtaining a visa to face Browne (16-0).
Hatton Promotions head of Boxing Richard Poxon stated: "We believe that all steps had been taken for Richard to take part in the fight.
"However, our sole focus must now be on securing a worthy opponent for Lucas to face on the 25th".
Towers, who used to go by the name Hayles, was involved in a gruesome crime back in 1997 in which a man was kidnapped, held at gunpoint and then tortured with a steam iron, a hammer and stun gun for three days while a gang sought a ransom of £150,000.
Towers played no part in the man's torture but was involved in the kidnapping which led to him being jailed.
On his release from prison he returned to the gym he used to work-out in and took up boxing under the tutelage of legendary trainer Brendan Ingle, who has worked with the likes of Herol Graham and Naseem Hamed in the past.
He could only fight amateur contests while still on probation but has now forged a successful, unbeaten professional career. He has been tipped by his promoter Ricky Hatton to win British and Commonwealth titles.
He has also turned his life around and often gives talks to children about the dangers of falling in with the wrong crowd.
Towers told the Independent last year: "The turning point for me was when my mum came on a visit and told me my little brother was struggling and that he needed me out.
"He had nobody - there was never an influential figure there as a dad. I just felt so helpless and I thought to myself, 'What am I doing, what am I doing?'
"It was killing me. My mindset changed. I told myself. 'It's got to start here.' It did. I eventually got parole for good behaviour.
"I tell (schoolkids) that when you are good, good comes to you. What I needed was someone to tell me, 'Don't do this, do that.' Instead of, 'Don't do that – whack!'"
- Sports & Recreation
- Society & Culture