The 17-year-old Kvitova won only the second WTA Tour match of her career, stunning Williams, the winner of four Wimbledon titles, 2-6, 6-4, 6-3.
"I am very happy," said Kvitova, of the Czech Republic. "It is the biggest [moment] of my life."
After winning the second set, Kvitova took charge midway through the final set. In the fifth game, she had two break points against Williams.
Williams saved the first one with a forehand winner, but Kvitova recovered and followed with a forehand volley winner to go up 3-2.
She closed the match with another break, sending a forehand volley winner past Williams before a stunned crowd.
"She had a good serve and she was left-handed," Williams said. "She mixed it up really well. I'm not happy about the result of the match, but she played well."
Playing in only her 14th career WTA match, Kvitova, ranked 143rd, matched Williams's power with punishing groundstrokes of her own. Williams was hampered by six double faults, including five in the second set.
"I just made some mistakes," Williams said. "I definitely had more errors than I wanted."
In the other upset, Mattek, a 22-year-old ranked 131st, played one of the best matches of her career.
"Honestly, I was really seeing the ball really well and I was there to every shot she hit," Mattek said. "I just stuck to my game plan. My goal was to get on top and get as many points right away as I could. I just came out firing."
In the men's division -- the Regions Morgan Keegan Championships -- fourth seed Radek Stepanek of the Czech Republic beat qualifier Jesse Levine 6-1, 6-4 and fifth seed Sam Querrey survived an upset bid by Michael Russell, winning 3-6, 6-3 7-5.