It was Nadal's most lopsided defeat in three years.
Roger Federer downed Feliciano Lopez 7-6 (4), 6-4 to set up a matchup against 112th-ranked Nicolas Kiefer, who beat fifth seed Fernando Gonzalez 7-6 (5), 6-2. Lopez, a wild card, committed 34 unforced errors against the top-ranked Federer.
Nadal, who sat out after the U.S. Open because of tendinitis, had five winners and 29 unforced errors against Nalbandian.
"It's hard to play worse, which makes it a lot easier to go forward from this," Nadal said. "I didn't go in with a clear enough idea of the match I wanted to play."
"I took advantage of all of Rafa's bad shots," Nalbandian said. "I think Rafa never felt comfortable out there."
The second-ranked Nadal, coming off a 7-6 (5), 6-4 win Thursday against Andy Murray, was slow to reach Nalbandian's groundstrokes and broken by the third game.
A double-fault in the fifth game set up another break chance for Nalbandian, whose forehand clipped the net to skip beyond Nadal for 4-1. After holding to love, Nalbandian reached Nadal's overhead smash for a crosscourt winner and the set.
The Spaniard's layoff showed as he looped returns out of play or into the net. But Nadal refused to blame fatigue for the loss.
"I don't want to come up with an excuse," he said. "When you lose, you lose. I think the result reflects enough. It wasn't my day. I never felt good at any point."
Djokovic, who reached the semifinals for the 11th time this year, needed treatment on his left ankle before the second set.
Djokovic, the U.S. Open finalist who won Masters titles in Miami and Montreal this year, saved three break points in the opening game despite making six unforced errors. In the 12th game, Ancic reached a volley near the baseline and sent a crosscourt passing shot beyond Djokovic to force the tiebreak.
The 29th-ranked Croat, who missed more than six months of the season because of mononucleosis, couldn't return two straight serves from Djokovic, who went ahead 5-1 in the tiebreak.
At 1-1 in the second set, Ancic twice hit into the net and was broken. Djokovic broke again in the fifth game.
Gonzalez couldn't set up a finals rematch against Federer despite getting out quickly with a break in the third game.
Kiefer immediately broke back as both players held serve until the tiebreaker, where the German's powerful strokes forced Gonzalez into making errors.