Dementieva said her back began bothering her Tuesday. She received courtside treatment from a trainer during the match before calling it quits after 58 minutes.
The loss ended an eight-match winning streak for the 10th-seeded Dementieva.
Jankovic, seeded fourth, won her opening match by saving five match points in a third-set tiebreak against Sofia Arvidsson.
The Serb has since swept every set in the past three matches.
"It was a scary feeling, you know, just thinking that I may be over with this tournament," Jankovic said. "But I was really strong mentally and was able to get through that match, which gave me a lot of confidence for the rest of the tournament."
Jankovic had never advanced beyond the third round in five previous appearances at Key Biscayne. She's seeking her first tournament title this year, and on Thursday her opponent will be 19th-seeded Vera Zvonareva, who beat No. 13 Dinara Safina 7-5, 6-4.