The Senators have opened the season with four wins, including three straight at Scotiabank Place. Ottawa lost its first four home games last season and dropped nine of its first 15 at home before finishing with a 25-13-3 home record.
"We struggled with that early last year and then we turned it around," Alfredsson said. "It's always a work in progress. You always try to tweak some things and I'm sure the coaches will come up with things we can do a lot better, but right now we're feeling pretty good."
Redden put the Senators up 3-1, though only after a video review clearly showed that the puck had crossed the goal line before it was swiped away by New Jersey defenseman Vitaly Vishnevski.
Jason Spezza got his third assist of the game on Alfredsson's fifth goal of the season into an empty net with 48.7 seconds remaining.
Dany Heatley got his fourth goal of the season and Chris Kelly also scored for Ottawa, which is off to its second 4-0 start in the past three seasons.
Martin Gerber made 27 saves for his fourth straight win. Gerber is starting while No. 1 Ray Emery recovers from offseason wrist surgery.
"We've had more consistency this year," said Gerber, who struggled early last season before losing the starting job to Emery in mid-November.
Zach Parise got the Devils within 3-2 shortly after Redden's goal, and under similar circumstances. Parise's shot slid along the goal line and play continued for 10 seconds after Ottawa defenseman Andrej Meszaros cleared the puck.
Another video review showed that the puck had cleared the goal line and the clock was reset to 7:29.
Arron Asham scored his first New Jersey goal 18:17 into the second to draw the Devils within 2-1.
Devils goalie Martin Brodeur stopped 33 shots, including 15 in the first when the Senators held a 17-9 margin.
"There were a lot of shots and a lot of chances, you know, they were all over us, and after that, like the other games, we were OK," Brodeur said. "We played a decent game."
Ottawa held New Jersey to just one power-play opportunity. The Senators have successfully killed 21 penalties to begin the season.
"There weren't a lot of penalties today," Alfredsson said. "The power play came up with the winner, which was nice to see."
New Jersey fell to 1-2 as it continues its season-opening nine-game road trip.
Kelly and Heatley scored first-period goals to stake Ottawa to a 2-0 lead.
"To win in this league -- to win in any league -- you've got to play well for 60 minutes and it's two games in a row where we haven't had a good first period, and three games in a row where we've got one period where we haven't played well because we're not doing what we need to do to be successful," Devils coach Brent Sutter said.
Kelly opened the scoring 4:37 in when he kicked his own rebound out of his skates and beat Brodeur with a shot through traffic for his first goal of the season.
"It just found a way to sneak by him," Kelly said.
Heatley made it a two-goal lead at 15:16 when he poked a loose puck past Brodeur.
Asham got the Devils on the board late in the second when he swiped a shot past Gerber after Dainius Zubrus carried the puck out from behind the net.
The Devils are scheduled to host Ottawa on Oct. 27 for their first game at the 17,625-seat Prudential Center, two nights after a Bon Jovi concert christens the new downtown Newark arena.
The Devils, who lost a five-game playoff series to the Senators last spring, had a 3-1-0 regular-season mark against Ottawa last season. ... Spezza has seven assists this season, at least one in all four games. ... The Senators got off to a team-record 6-0 start in 2005-06 before going on to a franchise-record 53 wins and 113 points.
Copyright by STATS LLC and The Associated Press
Arena: Canadian Tire Centre Location: Ottawa, Ontario
Referees: Rob Martell, Dan O'Halloran Linesmen: Anthony Sericolo, Thor Nelson
Attendance: 18,260 (98.7% full)
Team Stat Comparison
Power Play Goals
Power Play Opportunities
Shots on Goal
Chris Kelly (1)
Assists: Antoine Vermette, Joe Corvo
Dany Heatley (4)
Assists: Jason Spezza, Daniel Alfredsson