The Rangers gave up two early goals but threatened to steal an improbable victory after leveling the score at 2-2 midway through the final period.
However, the Capitals, who upset defending champions Boston in a seven-game thriller to make the Eastern Conference semi-finals, regained their composure and snatched the win with a power-play goal from Russian Alexander Ovechkin after the Rangers gave away two successive penalties.
"When you battle back as hard we did the time, you can't take four minutes of penalties," said Rangers coach John Tortorella.
The Rangers also hit the post twice before Ovechkin's goal, his third of the play-offs, left the best-of-seven series tied at 1-1 with games three and four moving from Madison Square Garden to Washington DC.
"We hit two posts last game and they hit two this time, that's how close it is," said Caps netminder Braden Holtby.
"The hockey gods were on our side tonight."
Mike Knuble and Jason Chimera got the visitors off to the perfect start with goals in the first period.
The Caps caught the home team napping when Knuble, a former Rangers winger, opened the scoring and the visitors doubled their lead when Jay Beagle beat New York goalie Henrik Lundqvist, one of the finalists for the NHL's MVP award.
Chimera was credited with the goal although television replays showed the puck came off the skate of the Rangers' Ryan McDonagh.
The Rangers pulled one back before the first intermission when Brad Richards buried the puck in the back of the net but the East's number one seeds were unable to score again until the third period.
New York captain Ryan Callahan tied the game at 2-2 with a power-play goal as the Rangers peppered the Caps' goal, but the celebrations were shortlived as Ovechkin's slapshot proved decisive.
"We showed some character coming back and tying the game but we just got into some penalty trouble," Callahan said.
"All year we've responded after losses and this is no different."
In the West, the giant-killing Los Angeles Kings continued their fairytale play-off run to take a 2-0 lead over the St Louis Blues .
The Kings set up dominant 5-2 road victory with a stunning four-goal blitz in the first period inspired by Anze Kopitar to suck the life out of the St Louis crowd.
"We were moving the puck early and shooting," Kopitar said after his two-goal effort in the first period.
"Any time you get the chance, you want to put the puck on the net. It gave us a jump start for sure."
The eighth-seeded Kings, having already upset top seed Vancouver in the first round, now head home for Games Three and Four and a chance to sweep their more-fancied opponents. Game Three is on Thursday.
Mike Richards and Jeff Carter also scored goals in the first period rout, which allowed the Kings to cruise to the finish.
Andy McDonald pegged one back for St Louis early in the second period but Justin Williams responded with another for the Kings. Blues winger Matt D'Agostini netted St Louis' second in the third.
St. Louis are now 0-for-12 on power-plays for the series and were jeered by sections of the crowd after failing to convert a 5-on-3 chance in the third period.