Expand and fortify. If you can't stop the enemy from shooting, give them more to shoot at. Your homeworld should be your last line of defense, not your first. It's less a question of protecting a planet, and more a question of fortifying a solar system.
As other answers have pointed out, if you've got efficient space travel, you can effectively annihilate a planet comparatively easily. One good asteroid nudged on a collision course, one big enough kamikaze ship (or fleet of kamikaze ships), and wham bam, no more planet (to speak of). It doesn't take much to render a planet uninhabitable. So how do you protect against your comparatively probable doom? You don't let them get close to your planet. You expand, you fortify.
First things first, you do the obvious things to trick out your homeworld. Slap an artificial orbital ring around it covered in guns and starship hangars, give it a network of a couple thousand defense laser satellites, get yourself a couple mile-or-so-long ground-based railguns. Make yourself a network of next-gen fallout shelters miles underground, big enough to house most of your population (or at least those of your population you like—you are are a dictator). If anyone gets all the way to your homeworld, you're already in deep doody, so this should be your last, most drastic line of defense. Also a good place to keep your most complex megatelescopes—this should be the safest place in your empire, and you need as many eyes on the skies as you possibly can. You loose your sensors, and you're blind to any incoming attacks. Space is big and mostly empty, so if you've got telescopes, you'll be able to see pretty much any attack coming.
But that's all just for the worst-case scenario. Fundamentally, you don't want folks even getting to your homeworld. Short of magic hyperspace dimensional popping, people are going to have to get through your solar system to reach your (presumably inner-planet-equivalent) homeworld. The trick is to not let them get far enough. Build bases orbiting every planet, fortify every asteroid, every rock, every speck of space dust. Put guns on everything. It's cheaper and easier to make space habitats than to terraform planets or launch ships out of atmospheres, anyway. Slap outposts everywhere, let your population boom, and give them all guns. You don't just have megatelescopes and laser arrays around your planet—you have them all throughout your system. You have bases all the way out in your Oort Cloud, if you can get them. You want to be able to see everything coming from every direction, and aim a thousand guns in that direction to blast it to bits. There's no stealth in space, so if you have the infrastructure, it becomes easy to sight possible threats.
But as you said, how can you be sure if a ship approaching is a threat or not? Any reasonably fast ship is nearly an apocalypse weapon by itself—how can you protect against that? Sure, millions of defense arrays helps draw the fight away from your people, but what if some loser pretends to be friendly until he's in your atmosphere?
When it comes to this, all your space guns come in handy for a different reason: Intimidation. Creating the illusion of control and dominance is just about as good as actually having it. Technically speaking, anyone could crash their car into any random building and cause a great deal of damage, but no one does, because we're all certain the repercussions from the Powers that Be would be devastating. And, if you've got enough defensive infrastructure, that certainty wouldn't necessarily be misplaced.
As a final bonus, if you really want total military dominance over a solar system, and even its surrounding systems, you can build a Nicoll-Dyson Beam around your sun (effectively an array of super-gigantic mirrors). Such a beam would transform it into a Starkiller-Base-esque laser that could fry even the distant homeworlds of anyone who tries to pull terrorism on your turf.