The Rangers (42-17-7) grabbed a 1-0 lead on Ryan Callahan's first-period goal, but dropped to 30-2-3 in games in which they scored first. New York, which lost 4-1 at New Jersey on Tuesday, hadn't lost two straight in regulation since Dec. 13 and 15.
"We didn't do a good enough job in a lot of areas," defenseman Ryan McDonagh said. "We weren't good defending around our net, and they took advantage."
Martin Biron made 18 saves. New York, which began a stretch of three games in four days, is expected to have No. 1 goalie Henrik Lundqvist in net Friday at Chicago.
The Rangers are 1-2-1 in their past four, matching their worst stretch of the season.
"I think we're beating ourselves sometimes in a game and finding ways to lose," coach John Tortorella said. "We're just going to continue to keep working on our game and chip out the mistakes and see if we can get a little traction here."
The Senators are one point behind sixth-place New Jersey in the Eastern Conference.
"I think a game like this just goes to show that we can beat those kinds of teams like Boston and New York," Cowen said. "It's a good confidence booster for playoffs."
With just seven shots through two periods, the Senators rarely tested Biron, but Ottawa took advantage of a number of rebounds.
Turris scored his eighth of the season, jumping on the rebound of Daniel Alfredsson's shot from the point to give the Senators a 3-1 lead early in the third. Spezza made it 4-1 with just over a minute remaining.
Despite being outshot 9-3 in the second, the Senators held their own against the Rangers because of the play of Bishop, who made his second with the Senators. The 25-year-old goalie made a number of big saves, including a great stop on Ruslan Fedotenko late in the period.
Bishop looked a little nervous at the start as the Rangers grabbed the lead 4:36 in. Callahan tipped a pass from Brad Richards into an open right side of the net as he caught Bishop moving.
The lead was short-lived as Smith fired in a slap shot from the top of the left circle at 5:50 for his first goal in 13 games.
"It wasn't good to let in that first one," Bishop said. "I think the team responded great getting a goal right back and we didn't give up many chances and we didn't give up too many rebounds."
With just over a minute left in the period, Foligno banged in Jim O'Brien's rebound to give the Senators a 2-1 lead. It was Foligno's third goal of 2012, and the first that wasn't an empty-netter.
The Senators split the season series with the Rangers at two wins each.