With the Hurricanes leading 1-0 early in the third period, Stillman stole the puck from Columbus defenseman Duvie Westcott at the blueline, skated to the top of the left circle and rocketed a slap shot past goaltender Steve Mason.
Stillman also added an empty-netter.
Carolina had a man advantage four times in the opening period but failed to score. The Hurricanes broke through midway through the second period with a power-play goal.
After Columbus and Mason had killed 42 seconds of a 5-on-3, Blue Jackets forward Rick Nash was called for a hook in open ice while going for a loose puck. Just seconds into the two-man advantage, Eric Staal deked a Blue Jacket in the high slot and tossed a back-handed pass to Hamilton who flicked the puck in at the 10:48 mark.
Justin Williams had earlier caught the far post on a breakaway from the right wing, and Staal had also clanged one off the post just seconds before assisting on Hamilton's goal.
The Blue Jackets were on the power play in the final minutes and pulled their goalie and still were unable to beat Grahame.
Carolina finished 1 for 7 on power plays while Columbus was scoreless in five.
The Blue Jackets have two road exhibition games before opening their season Oct. 5 at home against defending Stanley Cup champion Anaheim.
The Hurricanes host Columbus Friday night in their final preseason game then open the season on Oct. 3 at home against Montreal.
Copyright by STATS LLC and The Associated Press
Arena: Nationwide Arena Location: Columbus, Ohio
No scoring this period
Jeff Hamilton (power play) Assist: Eric Staal, Justin Williams