With Columbus controlling the puck for much of the third period, Zherdev took a deflection off of Markus Naslund's pass to Chris Drury and lifted a shot high into the net for his third goal of the season and a 3-1 lead with 6:12 remaining.
It was Zherdev's first three-point game with the Rangers.
"It's not easy going back to the team that traded you," Drury said. "He seemed real focused. He just had a great night."
Voracek poked the puck away from defenseman Dmitri Kalinin near mid-ice and then swooped in on goaltender Henrik Lundqvist, who stopped his first shot but the rebound came right back to Voracek for the putback. It was the rookie's third goal of the season.
"The key for us was that we didn't change anything even though they got that goal," said Lundqvist, who had 25 saves. "We just kept playing the same way and for us it was a perfect road game."
Zherdev assisted on both second-period goals, including Dubinsky's controversial score.
Girardi tied the score with a shot from the right point, beating Pascal Leclaire with a hard shot through congestion.
At the 12:11 mark, the Rangers took the lead while on a 4-on-3 advantage. Zherdev slid a pass into the slot. Dubinsky swiveled the heel of his right skate to deflect the puck past Leclaire. The goal was allowed after a video review.
NHL Rule 49.2 states "a puck that is directed into the net by an attacking player's skate shall be a legitimate goal as long as no distinct kicking motion is evident."
Leclaire had 29 saves for the Blue Jackets, who lost for the first time after winning their first two home games.
Zherdev said it was a relief to get the Columbus trip out of the way.
"Yes. Because tomorrow is next game and a new game," he said.
Columbus C Michael Peca played his first game after a five-game suspension for making contact with an official in the season-opener in Dallas. ... The Rangers are now 2-4 at Columbus, their only previous win coming on Feb. 12, 2001. ... The Rangers were 1-for-8 on the power play and the Blue Jackets 1-for-6. ... Jason Chimera and Rick Nash had three-game goal streaks end for Columbus.