The Prime Minister insisted he was “thinking actively” about fighting the next two general elections to become the longest-serving post-war leader.
And he insisted questions over his leadership were now settled after the loss of Wakefield and former stronghold Tiverton and Honiton.
But the attacks kept on coming from his own backbenches on Saturday night, with Damian Green, who chairs the One Nation caucus of Tory MPs, warning the Government “needs to alter both its style and content” and calling on Cabinet members with leadership hopes to show their stripes.
Former minister David Davis also lashed out at the PM’s claim the only argument of “substance” for a change of direction he had heard from his critics was for the UK to return to the EU single market, arguing this is “plainly not true of me, or many others”.
But on Sunday, Northern Ireland Secretary Brandon Lewis threw his support behind the Tory leader, telling Sky's Sophy Ridge on Sunday programme: "We often get criticised in politics when we look short-term, at just the next day, the next election, the next vote.
"Actually we've got somebody as Prime Minister who wants to be looking long-term at how we structurally improve our country for generations to come. That has got to be a good thing."
Mr Johnson earlier insisted the “endless churn” of allegations was “driving people nuts”, as he pushed on with his Rwanda trip despite suggestions further ministerial resignations could follow.
Oliver Dowden resigned as Tory party co-chairman, saying he and Conservative supporters were “distressed and disappointed by recent events” and telling the Prime Minister that “someone must take responsibility”.
But Mr Johnson set his sights on being in office in the “mid 2030s”, in a run that would see him outlast Margaret Thatcher’s reign.
Asked by journalists at the British high commissioner’s residence in Kigali if he would lead his party into the next election, he said: “Will I win? Yes.”
In a buoyant mood, the Prime Minister added: “At the moment I’m actively thinking about the third term and what could happen then, but I will review that when I get to it.”
Labour, meanwhile, challenged the Tories to call an early election, with leader Sir Keir Starmer telling Mr Johnson: “Bring it on.”
Further controversy loomed on Saturday evening as The Telegraph reported the PM intends to impose sweeping new steel tariffs in a drive to win back support in traditional Labour heartlands.
The newspaper said the changes were the same as those cited by Lord Geidt when he resigned as Mr Johnson’s ethics chief, as he claimed he had been forced into an “impossible and odious” position by the Prime Minister.
Former Conservative leader Michael Howard has urged Mr Johnson to step down for the good of the party and the nation, and called on the Cabinet to consider resigning to force him out.
But the PM said in Rwanda: “I love my colleagues and of course I would urge them respectfully: golden rule of politics, the more we focus on Westminster politics the more irritating it is to voters.”
He argued that “what’s driving people nuts is this endless churn of stuff about things that I’m meant to have stuffed up or whatever about my colleagues, their views of me, my character, the leadership, Tory blah blah…”
There are suggestions of a challenge to change the rules of the 1922 Committee of Conservative MPs in order to allow another vote of confidence in Mr Johnson within the next year.
Asked if he believes questions over his leadership are settled, Mr Johnson replied: “Yes.”
Conservative veteran Sir Geoffrey Clifton-Brown, who retained his Cotswolds seat with a majority of 20,000 at the last vote, is among the Tories to have expressed fears they could lose their jobs at the next general election.
In the by-election in the Devonshire constituency of Tiverton and Honiton, a dramatic swing of almost 30% from the Conservatives saw their 24,000 majority overturned by the Liberal Democrats.
In West Yorkshire, Labour seized back Wakefield with a majority of 4,925 on a swing of 12.7% from the Tories.
In an article for The Telegraph, Mr Green said “improvements are needed to restore trust in the Government”.
“The fashion of finding ‘wedge issues’ where you divide the population and leave your political opponents on the wrong side of an argument only works when most people trust what you are saying to them. Without trust, we are left with bombast and rhetoric,” he said.
He added: “It is not a secret that a significant proportion of the Cabinet think they could do a better job of leading the country than the current incumbent. Now would be a good time to demonstrate those leadership qualities.”
Also writing for the newspaper, Mr Davis said: “Boris likes to say that his critics have only one policy difference with him, namely that they want to rejoin the single market. That is plainly not true of me, or many others.
“The biggest policy difference is that we want our Government to stop talking about tax reductions and actually deliver them, so that we are no longer the highest taxing Tory Government in history.”
In a piece for The Observer, Labour leader Sir Keir challenged Mr Johnson to call an early election, insisting “Labour is back”.
“For months Boris Johnson has been privately claiming he will hold an early election,” he wrote.
“My message to him is simple: bring it on. Because the quicker that election comes, the quicker this country will get a Labour government that delivers the positive change people are crying out for.”