Mike Green scored the game-winner with a power play goal with 5:48 left in the third period on a sizzling slap shot past Rangers goalie Henrik Lundqvist in a see-saw struggle in which New York twice fought back from a goal down.
Cheered on by a red-clad, sold-out crowd in Washington, the determined Capitals made the last goal stand despite the frantic efforts of the Rangers in the dying minutes.
"It's a good feeling," Green said after registering the first playoff game-winning goal of his career. "Good to finally get one on the board. I haven't scored in a while."
Washington netminder Braden Holtby, who is 4-0 following playoff losses this postseason, saved 18 of 20 shots. Lundqvist turned aside 23 shots in a tension-packed game that followed a Game Three thriller won by the Rangers in triple overtime.
"We were down 2-1 in the series and it's a big game here. You don't want to go down 3-1," Washington coach Dale Hunter said. "The guys knew what was at stake.
"They came out in the first (period) with a big push and we had a lot of offense and that kind of got us going."
The Capitals came out firing, putting Lundqvist under intense pressure from the start, but the Rangers goalie held firm at the net until Alex Ovechkin fired a shot through the Swede's glove to make it 1-0 at 12:43 of the opening period.
Artem Anisimov equalised just 70 seconds into the second period from in front of the net.
Washington's Nicklas Backstrom used his strength and skill to give the Caps a 2-1 lead at 11:54 of the period.
Backstrom used a shoulder to dump Anisimov to the ice as he protected the puck in the corner and then glided in front of the net to receive a pass from Jason Chimera and wrist a shot over Lundqren's right shoulder.
Anisimov set up the next equaliser about five minutes later helped by some defensive confusion by the Caps, who gave up on a puck heading behind the net they thought would be whistled for icing.
Racing to reach the puck first, Anisimov shoved it to Marian Gaborik, who put it past Holtby to tie it 2-2.
In the end, Holtby was helped by a rousing defensive effort that produced a 26-7 Washington edge in blocked shots by the defense.
"The guys are committed," Caps coach Hunter said. "We know we have to block shots. It's a big part of our game."
Defenseman Karl Azner said his team was determined to put their Game Three disappointment behind them.
"A loss is a loss," said Azner. "We have to forget about it right away. You can't worry about it. The entire season we had so many ups and downs. We're a lot stronger mentally now."
The series returns to New York's Madison Square Garden on Monday for Game Five.