"You could have swore we just won the Super Bowl the way we were celebrating," Banta-Cain said.
The 49ers are 2-0 against Arizona, 1-8 against everybody else.
The Cardinals (5-6) could have won moments before Warner's fumble, but Neil Rackers missed a 32-yard field goal.
First-year Arizona coach Ken Whisenhunt acknowledged he made several mistakes on the sidelines, probably none more important than on that bizarre field goal sequence.
Rackers initially lined up for a 27-yard try, and kicked it through the uprights. But a delay-of-game penalty nullified the kick and pushed Rackers 5 yards back, where he hooked his attempt just to the left of the upright.
"The clock ran out fast," Whisenhunt said. "That's my fault. I take full responsibility for that. I thought we had time. It wasn't any different than normal. Obviously, I should have called a timeout there."
Moments later, the Cardinals found themselves pinned on their own 3.
Warner dropped back to pass and was hit by Ronald Fields. The quarterback fumbled, then Banta-Cain fell on the ball to give San Francisco (2-9) its first victory since Sept. 16.
"The ball was supposed to be out quick," Whisenhunt said. "We had success with that before, and we couldn't do it. In hindsight, yes, I would have liked to run the ball."
Warner didn't have much to say about the play.
"I didn't see anything open," he said, "and I was just looking for a place to get rid of it at that point, and wasn't able to do so."
Warner, playing with a torn ligament in his non-throwing elbow, completed 34 of 48 passes for 484 yards and two touchdowns, but was intercepted twice early by Walt Harris. His completions included a desperation 48-yarder to Larry Fitzgerald to put the Cardinals (5-6) ahead 21-17 as the half ended.
Trent Dilfer, starting in place of injured Alex Smith for the 49ers, completed 25 of 39 passes for 256 yards and two touchdowns, with no interceptions. Frank Gore, playing like the powerhouse back of last season, rushed for 116 yards and caught 11 passes for 98 yards.
"We just kept pounding the rock," Gore said. "We just kept pounding, kept pounding. We kind of wore them out."
An Arizona player rolled on Gore's sore right ankle early in the game, and Nolan had to help his star running back to his feet. Nolan thought Gore was done for the day.
"I go on the headset and said, 'It doesn't look like you're going to have Frank back,'" Nolan said. "About a series later, I looked out there and the son of a gun was out there."
The victory came five days after the 49ers hired veteran coach Ted Tollner to help embattled first-year offensive coordinator Jim Hostler.
The 49ers entered the game ranked last in the NFL in just about every offensive category. Nolan was asked if Tollner could have had an impact on the team so soon.
"I would like to think that Ted helped us because it's a credit to Ted," Nolan said. "I think he has things to offer, but I don't want to diminish anything and take away anything from what our offensive coaches, including Jim Hostler, did in order to make that happen today."
Arizona outgained San Francisco 552 yards to 374, but the Cardinals had four turnovers to none by the 49ers.
Arizona stopped the 49ers on three overtime possessions, but twice the Cardinals were pinned deep in their territory, with the help of a pair of blocking-in-the-back penalties against rookie Ben Patrick.
Gore ran 35 yards virtually untouched to put San Francisco 31-28 with 1:15 left in regulation, but Warner completed five straight passes for 84 yards in the final 1:11. His last went 30 yards to Bryant Johnson, who slid out of bounds inside the San Francisco 1 with 6 seconds left.
After the Cardinals used their final timeout of regulation, even though the clock had already stopped, Warner threw an incomplete lob to Johnson, then -- with the ball inches from the goal line -- Rackers kicked the tying chip shot.
Fitzgerald caught nine passes for 156 yards and two touchdowns. He broke Larry Centers' franchise record with 30 touchdowns as a Cardinal. For the second time in his career, Fitzgerald has topped 1,000 yards receiving for the season. It was his 12th career 100-yard receiving game, third this season.
Two of Arizona's three first-half turnovers resulted in touchdowns by San Francisco.
The 49ers led 17-10 when Warner scrambled to his left and lofted the ball to the end zone with the final seconds of the half ticking away. The 6-foot-3 Fitzgerald outjumped the crowd, grabbed the ball for the TD, then ran for the tunnel.
Arizona starting cornerback Eric Green left in the first half with an injured right groin. ... The Cardinals had a 5-0 turnover advantage the previous week against Cincinnati. ... San Francisco lost cornerbacks Shawntae Spencer (left quad strain) and Marcus Hudson (left knee sprain), and shifted Mark Roman from safety to fill in.