Scolari, now in his second spell as Brazil boss, returns to Stamford Bridge on Monday night when his side tackle Fabio Capello's Russia in a friendly.
Chelsea were fourth in the Barclays Premier League and still in both the FA Cup and Champions League when Scolari was sacked in February 2009 after just seven months in the hot-seat.
"I believe my time at Chelsea was not as bad as people have been talking about," Scolari said. "The numbers show that, the games we won. I don't think it was that bad."
Asked whether he was given enough time, Scolari added: "I don't think so but it was a decision from the management. Everywhere I work I am just a member of staff and if somebody thinks I am not doing a good job, they can sack me.
"That happens everywhere. I accept that. I did enjoy my time here at Chelsea and I was happy while I was here. It was one of the great experiences I have had in my life."
Scolari was sacked by Russian owner Roman Abramovich so the club could "maintain a challenge for the trophies we are still competing for".
Scolari could not answer whether he would have delivered those trophies. But Chelsea's Brazil defender David Luiz said Scolari is still held in high regard at the club - and suggested he could have a point to prove against Russia.
"Mr Scolari is an excellent professional. He has been a coach in many clubs all over the world and also national teams," Luiz said. "I'm sure he is happy to come back here because regardless of what happened here he has very good memories.
In many ways, Chelsea are in a worse position now than when Scolari left, having become the first defending champions to be eliminated in the group stages of the Champions League. The club are 19 points behind Manchester United in the Barclays Premier League and it has been a winter of discontent at Stamford Bridge, with interim manager Rafael Benitez a hate figure with some supporters.
- Sports & Recreation
- Barclays Premier League
- Champions League