Cook squandered a three-stroke lead after 10 holes at Royal Troon but still went to the 72nd leading Vaughan by one. Cook sent his tee shot right into the rough and then played his third shot well short of the green. He decided to putt, but left it 15 feet short for a bogey to finish with an even-par 71 and a 6-under 278 total.
Vaughan made a par for a 70 that forced the Americans back to No. 18 to begin the playoff, the championship's third in four years.
Vaughan pitched to the center of the green and sank his 20-footer. When Cook missed his putt from 12 feet, the former firefighter from Hutchinson, Kansas, won his first tournament on the senior or regular PGA Tour. Vaughan also earned his biggest paycheck, $312,258.
"Money is great, but until you win, you're just another player. It's all about winning. You only get a few of these," said Vaughan, whose mother, Maxine, died in a car crash seven weeks ago after returning from watching him play a tournament in Iowa.
Eduardo Romero finished third after failing to make the playoff by missing an 8-foot putt at the last hole. He had a 70 to finish at 5-under 279, one shot ahead of Bernhard Langer (68). Greg Norman also shot a 68 for a share of fifth place with Gene Jones (68) and three-time champion Tom Watson (70), all at 2-under 282.
The victory gets Vaughan, who has had six operations on his left knee, into the field for next year's British Open at Turnberry.
Cook's collapse recalled his loss in the 1992 British Open at Muirfield, where he missed a short putt and finished second to Nick Faldo.
But Cook defended his decision to putt from at least 10 yards short of the green at the last hole.
"I just felt the putter was a better play and I've been doing that all week," he said.
At one stage, it appeared that Cook was cruising to the title.
He made birdies at Nos. 3-4 and then 7-8 to reach the turn at 10 under, with a three-shot lead over Vaughan. But that slipped to one at the 11th when he drove his ball into some bushes, wound up missing a 5-footer and finished with a double bogey.
"I tried to convince myself for the last 2½ hours, but I couldn't get it done," Cook said. "Bruce played his heart out. He battled yesterday and battled through the first nine today. You know, that's just the way it is."
The 53-year-old Norman nearly became the oldest winner of a regular major last week at the British Open. He maintained that form at Troon, even though he now considers himself a part-time golfer because of his various business interests and recent marriage to tennis great Chris Evert.
Starting at 1-over and seven shots off the lead, Norman birdied Nos. 2-3 and reached the turn in 32, 4 under for the day. When he rolled in a 15-foot birdie at the 14th he was 4 under for the tournament and three off the lead.
"I enjoyed [the two weeks] on a scale of 1 to 10, a 10," said the Australian, who led last week's Open with nine holes to play only to finish tied for third behind Padraig Harrington.
"I enjoyed every step, loved it," Norman said. "From a scoring perspective, I haven't enjoyed that. I've played a lot better than what my scores indicated and just didn't work out."