The home crowd were becoming increasingly anxious before Scott Brown's driven cross was diverted into his own goal by Martyn Corrigan.
The three points were vital to the home team as they sought to reduce the deficit on leaders Rangers to one point.
There was no doubt Celtic deserved their narrow victory with Kilmarnock keeper Alan Combe defying a barrage of shots especially in the first half.
Celtic threatened early on when Andreas Hinkel's cut-back found Jan Vennegoor of Hesselink, but Corrigan did well to block the striker's shot.
Scott McDonald went close to putting Celtic ahead when he cut in from the left past two defenders but his shot was brilliantly turned behind by Combe.
Celtic had to make a double substitution after 18 minutes when Hinkel limped off and Vennegoor of Hesselink suffered a bad gash after a clash of heads with Simon Ford.
Combe made a great point-blank save to deny McDonald as the ball bobbed about the Kilmarnock six-yard area, before the keeper was grateful to see another McDonald effort end in the side-netting.
The Killie keeper again did well to rush from his goal to parry McDonald's shot before showing remarkable agility to touch over substitute Derek Riordan's delicate chip.
The visitors somehow survived to half-time with their goal intact as Combe again did well to touch over Shunsuke Nakamura's powerful drive.
At the start of the second half McDonald was booked for complaining at not being awarded a penalty when he went down between two Kilmarnock defenders.
Celtic finally took the lead when Brown fired in a ball which forced Corrigan to lunge at it in the six-yard area before deflecting it into the net with Chris Killen lurking behind.
It was tragic for Combe to lose a goal in such a manner after the keeper had almost single-handedly defied Celtic.
The home team suffered some anxious moments before the points were secure, most notably when only a last-ditch tackle by Lee Naylor on Paul Di Giacomo denied the striker when he looked certain to score from six yards.