Coe, 56, who exuded calm and confidence as the chairman of the committee that organised the hugely successful Games in July and August, tells of his concerns that kept him awake at night.
"I was less concerned about a Twin Towers scenario because there were missiles on tower blocks and Navy Seals in exercise boarding boats on the Thames," he writes in extracts published by The Times on Saturday.
He says though, he was haunted by a "lone wolf" terrorist attack.
"I was haunted by the fear of that when people were celebrating having street parties or watching TV screens in pubs ten deep. That fear never left me.
"I counted off the days, counted off the hours."
In his book he describes meeting Chris Allison, the assistant commissioner of the Metroplitan Police a few days before the Games ended.
"We didn't even need to say what we were thinking. We just looked at each other and I said: 'This isn't done until we've got everybody home safely.' And he said, 'Yeah, I know'."
In an interview with The Times, Coe says he was sitting with his sons in the Olympic Stadium on the last night of the Paralympic Games when it suddenly occurred to him that the venture that had dominated his life for the last 10 years was over.
After the closing ceremony had ended, his sons asked if they could go down on to the track. Thinking of lots of bureaucratic reasons why they could not he was just about to say "no," when he realised it did not matter any more if they did.
He went down to the track with them and said: "As we stood there, it was instinct. I just pulled them both into a hudddle and thanked them for the last 10 years."
Asked if he felt guilty at not being around for his young family, he said, "Yes absolutely it was an obsession."
Coe, who won the 1,500 metres gold medal at the 1980 Olympics in Moscow and the 1984 Games in Los Angeles is unopposed in the election for chairman of the British Olympic Association on Nov.7.
His book, "Running My Life - The Autobiography" is due to be published the following day.
- Sports & Recreation
- Books & Publishing