The Yellow Jackets (6-1) began the second half on a 23-8 run that ended on Oliver's two free throws, his eighth consecutive point, with 12:02 remaining.
USC (2-4) took its seventh straight road loss and its third consecutive overall after failing to score until Alex Stepheson's layup made it 19-2 at the 12:59 mark of the first half.
The Trojans ended a 20-6 run when Nikola Vucevic's tip-in 7 minutes later cut the lead to 25-22. But Georgia Tech freshman Mfon Adofia hit a pair of free throws to help push the Yellow Jackets ahead by 13 in the final minute of the first half.
Oliver started in place of shooting guard Iman Shumpert, who with miss three to six weeks after undergoing arthroscopic knee surgery. Shumpert, a sophomore, has an 8.3 scoring average and a team-high 28 assists.
Georgia Tech, which scored 22 points off 19 USC turnovers, led by 35 on Lance Storrs' 3-pointer at the 5:50 mark. Freshman Derrick Favors scored with 10 points and 11 rebounds, and D'Andre Bell added 12 points for the Yellow Jackets, who are off until Dec. 14 when they visit Chattanooga.
Stepheson scored a career-high 22, but USC couldn't overcome poor play on the perimeter, going 1 for 12 on 3-point attempts. Donte Smith went 1 for 7 beyond the arc and finished with five points.
Oliver was 4 for 8 on 3-point attempts as Georgia Tech finished 8 of 19 beyond the arc.
Nikola and Vucevic Marcus Johnson each scored 12 points for the Trojans, who host Sacramento State on Tuesday.
In Georgia Tech's first meeting with Southern Cal, James Forrest's 3-pointer with 8-tenths of second gave the Jackets a 79-78 victory in the second round of the 1992 NCAA tournament.
USC tied the series with a 76-57 victory last year in Los Angeles.