Chris Bosh had 23 points for the Raptors, who clinched a playoff berth Friday night because Washington beat Miami.
Toronto wasn't celebrating after losing for the 14th time in 20 games.
"The fourth quarter is killing us. We're not closing out games," Ford said. "We just got to be more focused down the stretch, and we just got to find a way to get a win."
The Raptors (38-38) are hardly playing like a team poised to make a strong playoff run. Fans booed the home team at the final buzzer.
The lowly Bobcats finished 2-1 against Toronto this season.
"We have stretches when we play good defense, and then we have stretches when we just don't, and you can't do that trying to make the playoffs or going into the playoffs," Toronto coach Sam Mitchell said.
Richardson became the latest guard to have success against the Raptors, scoring all his points in the first three quarters. He hurt them with penetration and went 4-for-7 from beyond the 3-point arc.
Toronto cut Charlotte's lead to one with 1:15 left, but Boykins' steal on Ford and fast-break layup and free throw gave the Bobcats a 101-97 lead with 41 seconds remaining.
"T.J. relaxed and I went for it," Boykins said.
Ford said the turnover wasn't the reason they lost.
"I don't think that one play dictates the whole game. We just got to do a better job in the fourth quarter," Ford said.
The Bobcats made most of their free throws the rest of the way to seal the victory. Charlotte is trying to avoid a 50-loss season.
"It was a huge win for us," Richardson said. "We know that it's getting close to getting teams eliminated from the playoff but we still got team goals that we want to accomplish and we are going to play hard until the last game of the season."
Rasho Nesterovic added a season-high 23 points for Toronto, but All-Star Bosh didn't get many touches in the final minutes.
"It really wasn't anything about spoiling. We were just trying to get a win," Felton said.
Bobcats forward Gerald Wallace didn't play in the second half because of a strained left groin. ... Toronto decided not to file a protest to the NBA to try and reverse the outcome of an overtime loss against Atlanta on Wednesday. The Raptors argued that Ford got off a game-winning shot at the end of regulation, but officials ruled time expired. ... The Raptors have made the playoffs just five times in their 13-year history.