"He's no fun to face," shortstop Derek Jeter said of Halladay. "The only way you're going to beat him is you've got to get a great start from our staff and Andy did that."
Pettitte (9-6), who had not won since July 1, allowed one run and four hits in 6 2/3 innings, walking four and striking out six. He left with runners at first and second in the seventh but Phil Hughes came on and struck out Jose Bautista to end the threat.
"I was getting a little tired," Pettitte said. "It was good that Phil came in and got me out of that."
Hughes gave up two singles to begin the eighth, then struck out two batters before being replaced by Mariano Rivera. Vernon Wells greeted the Yankees closer with a two-run double off the wall in left, cutting it to 4-3, but Rivera got out of it when Alex Rios grounded to short.
Matsui gave New York some insurance by homering to center on Halladay's first pitch of the ninth, his 16th.
It's the second time this season Halladay has allowed three homers to the Yankees. He also gave up three in a 6-5, 12-inning loss at New York on July 4.
"To me, he's still the best pitcher in the league," Damon said. "Fortunately we've gotten him the last two times."
Rivera allowed a pair of singles in the ninth but Aaron Hill flied out to end it, giving the closer his 31st save in 32 chances.
Halladay (11-5) lost back-to-back starts for the first time in 2009, allowing four runs and eight hits in his fifth complete game of the year. The righty, who walked none and struck out five, has won just once in seven starts since coming off the disabled list June 29 after missing two turns with a sore groin.
The subject of intense trade talk leading up to July 31, Halladay got a warm welcome from Toronto fans happy to still see him wearing a Blue Jays uniform beyond the non-waiver trade deadline. For the third straight home start, the righty got a standing ovation as he walked in from the bullpen before the game.
Halladay declined to discuss his feelings about staying with Toronto past the deadline and was blunt when asked about his goals for the remainder of the season.
"Win," Halladay said. "That's the reason you're here. I don't think at any point you can pack it in and work on things. You have to come out every day to try and win."
Halladay came in 8-0 with a 2.10 ERA in his past 10 home starts against the Yankees but was behind quickly in this one as New York scored twice in the first. Damon singled and scored the opening run on a two-out double by Alex Rodriguez. Matsui reached when Halladay couldn't handle a high throw from first baseman Kevin Millar, who was charged with an error. An alert Rodriguez raced around third and was safe at home when he kicked the ball out of catcher Rod Barajas' glove with a feet-first slide.
"Getting those two in the first inning was huge for us because it let Pettitte go out there and just relax and do his thing," Damon said.
Toronto cut the deficit in half in the fourth. Hill led off with a single, extending his hitting streak to 13 games, advanced to third on walks to Millar and Wells and scored on Rios' sacrifice fly.
Damon and Teixeira hit consecutive solo shots to right in the eighth, the 10th time this season the Yankees have hit back-to-back homers. For Damon, the homer was his 18th. Teixeira's blast was his 27th.
"I don't know what to tell you," Halladay said. "I just didn't execute, especially late, and it cost me."
Yankees RHP Ian Kennedy, still recovering from an aneurysm, threw 25 pitches off a mini-mound Tuesday and hopes to play in minor league games before the season ends. Kennedy, who had surgery May 12 to remove an aneurysm from beneath his right biceps, will throw again Friday on a mound half the usual 10-inch height. ... Former Blue Jays second baseman Roberto Alomar attended the game.