But a 4-minute stretch in the fourth quarter made everything all better.
The Magic went on a late 16-0 run, led by the shooting of Ryan Anderson -- who finished with 24 points -- to erase a double-digit deficit and defeat the Raptors 102-96.
Orlando shot 4 for 5 from 3-point range during the burst, with Anderson making two of them, as the Magic took a 94-89 lead with 3:05 left after trailing by as many as 16 points.
"We sort of could've just given up, but we kept pushing through it," Anderson said. "This was huge for us."
Dwight Howard had 19 points and 15 rebounds for the Magic, who have won four straight since a season-opening loss to Oklahoma City. J.J. Redick added 21 points and Hedo Turkoglu had 15 points and seven assists.
"I thought we played a great fourth quarter," Magic coach Stan Van Gundy said. "Our intensity level went way up."
Bargnani paced the Raptors with 28 points, and Jose Calderon had 18 points and 13 assists.
It was the second consecutive late collapse by the Raptors, who also gave up a second-half lead to Dallas on Friday.
"I don't really have an explanation as to why this game got away," Calderon said. "It's tough. It's kind of like the same thing that happened with Dallas. They're two veteran teams. They know how to play in those moments. We've got to learn from that."
Bargnani, who scored only three of his 28 points in the fourth period, was disappointed with his team's showing.
"We really made the game for three quarters; we played good offense and great defense," he said. "But we didn't play that way in the fourth quarter. We played really bad."
The Magic shot 44.4 percent from 3-point range and were 5 for 7 from long range in the fourth quarter.
After the Raptors built another double-digit lead in the second half and looked to be on their way to a road upset, the Magic shots began falling while the Raptors' offense stalled.
Turkoglu made a layup to cut the Raptors' lead to 89-80, then Anderson and Redick made 3-pointers. Howard converted a layup off a pass from Jameer Nelson, followed by 3-pointers from Anderson and Turkoglu to force Toronto into a timeout down 94-89.
The Raptors missed all five of their shots and committed three turnovers during the run. They didn't score between the 7:48 and 2:47 marks of the fourth quarter.
"It was the defensive end of the floor," Van Gundy said. "You're not going to come close to trading baskets and get back in the game, so you obviously have to make plays at both ends. You have no margin for error. But your defense has got to be great, and ours was going down the stretch. To be down 11 going into the fourth and get that done, we played a great game in the fourth quarter."
The Raptors took a double-digit lead late in the first quarter behind some strong shooting from Calderon and Bargnani. The Magic responded, going up 40-39 after a 12-2 run midway through the second quarter that included eight points from Anderson.
Bargnani led all scorers with 19 points after two quarters of play.
Nelson returned for the Magic after missing one game with a neck spasm. He finished with nine points. ... Toronto guard Jerryd Bayless didn't play because of a sprained ankle he sustained Friday against Dallas. There's no definite timetable for his return. ... Calderon surpassed the 3,000-assist mark. ... Howard, who had 24 rebounds in each of the Magic's past two games, was attempting to become the first player with 24-plus rebounds in three consecutive games since Dennis Rodman in November 1993. ... Anderson, who is 21-for-45 (46.6 percent) from deep this season, leads the NBA in 3-pointers made and attempted.