Hooker Adam Milner scored a winning try 13 minutes from time as the Tigers ended a seven-match losing run with an impressive second half show after trailing 26-16 at half-time at the Wish Communications Stadium.
"I am a very relieved, happy man. It was a massive relief to win because it was very tense with two minutes to go," said 34-year-old Orr, who was handed the reins at Castleford after head coach Ian Millward was sacked this week.
"It is something for us to build on. I thought we defended really well in the second half to keep Widnes scoreless."
Orr was delighted with the performances of hookers Milner and Daryl Clark and young winger James Clare.
"Daryl Clark has been ill and has not trained all week. He had to go to hospital," added Orr.
"My idea was to start with Adam Milner and then get half an hour into the game and bring Daryl on to speed us up a bit. And then for the last 20 minutes bring Milner back on and move Clark to loose forward where he can still cause problems. I thought both of them played really well. They really got us going forward.
"James Clare has come in for a bit of criticism this year because he has made a few errors and he knows that, but I thought today he was probably one of our best players. It was an outstanding performance from the young kid so I was really pleased for him."
Widnes coach Denis Betts was gracious in defeat after his side came down to earth following their 58-24 win over Salford the previous week.
Betts said: "Castleford came out with passion and energy and we couldn't match that enthusiasm."
- Sports & Recreation
- Adam Milner