As Tim B said, this depends a lot on the situation
I would add that armies were only mobilized during wartime. Having a full-time levy cost a ton of money, and would not be used to guard a castle if they were no danger.
In September 1238, Henry III, his queen Eleanor, and their retinue
were at Woodstock. A man pretending to be insane had thrown himself at
Henry, demanding that Henry give his crown to him. Henry, believing he
was a madman, restrained his servants from beating or killing the man.
That night, however, the same man climbed through a window into the
king's bedchamber, armed with a knife. He meant to stab Henry in his
bed, but Henry was in the queen's chambers with Eleanor. One of the
queen's damsels, Margaret Biset, was reciting her psalter and heard
the man. She alerted the guards, who captured and tortured the
intruder, who confessed that William de Marisco had sent him to murder
the king and queen.
This is a historical example that can show that sneaking into a castle, and even in king's chamber was possible.
Also, a postern could help you to sneak out of the castle. They were used as a hidden, easily defendable passage. Your character can have some trouble finding it if he doesn't know the castle, and it may be locked, as common folk just enter/leave by the gatehouse, but it worth to try.