Mines, exploding space-mines everywhere!
Plagiarizing one of my earlier answers. Note that this answer assumes any and all alien/foreign forces are hostile and also a relatively high tech level for your empire.
There's No Invisibility in Space
While nobody can hear you scream in space, everybody can see you. To the hyperadvanced spacefaring civilizations, even a black-painted intertially-driven scout stands out as a sore thumb, since the defending civ has all the little objects in the system and their trajectories for the next million years calculated and recalculated over and over, so anything NEW, even a few stray photons bouncing off your invading ship from long-range radar sweeps would be ringing alarm bells. Moreover, if you're planning on doing anything in particular in the system you need to slow down. That takes reaction mass (assuming you don't have reactionless drives), so you might as well be announcing your arrival with giant flares. If you don't slow down, well, massive relativistic-speed objects also tend to stick out. If you've got space-warping technology, the warping you induce into space is also going to be easily detectable, since alcubierre warps don't just happen on their own.
Where to defend
You don't want to wait until the enemy gets into your star system. If they're within light-hours of Home, it's already game over. They can blast it with TerraWatt gamma lasers that go through your planet's crust like a bullet through warm butter, or perhaps launch dozens of relativistic kinetic accelerators at it, and believe me, those are a hassle to stop, and your bunkers aren't any good in a molten slag state. No. Your defenses for Earth should probably be far out in the Oort cloud at the very least, preferably a few light-years away from any world you're defending if you have the technology.
How to defend
You will defend in depth. You will have rings and layers, upon rings and layer. The first layer should be your centimeter-sized Remote Telemetry observation satellites. They will serve as an early warning system. Spaceships' heat signatures in space are very hard to disguise. Of course, a 1-meter probe will have an easier time slipping by than some massive colony ship or a warship armored to resist nuclear impacts. Besides optics, your detectors would use gravitational distortions, if far enough out from massive bodies. Since an object's propulsion jet is visible as well as its speed, you can easily tell the mass, so (accelerated) decoys would not work unless identical in mass to the spaceship you're trying to disguise. This confers an advantage to defense, since they don't need to break or accelerate. Your microsatellite sensor grid info could be sent narrowbeam at lightspeed (or darkspeed if you accept FTL and the mess it makes of causality) back to a defense outpost or used to activate some local defenses.
While the micro-satellites are passive detectors, as your next detection layer every cubic AU or so you should have an active scanning station, a radar beacon blasting away into the enemy darkness. This will quickly paint a signature of any approaching target. This information will allow your sectoral defense AIs to order active denial actions, activating your first-line defenses.
Good Old Kinetics
Now what would these defenses be? There's an extended discussion of classical kinetics/rockets/beams elsewhere, and you can read that at your leisure. All good ideas. Probably the best in the class is a straight up directed multiple warhead relativistic impactor, essentially just accelerated mass coming your way really really fast, and spreading out into slivers as it does to cover a vast area. Even with randomized defensive maneuvers, these puppies make it hard to miss -- and even 2 grams travelling a cool .99 c will leave you with $10^{15}J$ to deal with, or about 1 megaton of TNT equivalent. I'm guessing that will leave you with more than just a bad hairday.
But you can do even better.
Rift Weaponry
My other suggestion is rifting. Imagine you have devices with the capacity to bend spacetime itself. Done carefully on a starship and with insane levels of precision, this might effectively shorten long journeys, and maybe provide effective FTL spacetime translation. But if all you care about is damage, than a set chaotic, highly unstructured rifts will play havoc with the delicate innards of a ship (such as humanoids or computers and their minds).
Geometries bend at impossible angles, straight lines twist and intersect with each other, the ship's Combat Information Center and the million Kelvin engine room are suddenly superposed, leaving charred organic residue in the superpurified magnetic convection coils, past and present comingled as you see your mates both alive and and torn to shreds by the rift.
Your defenisve minded civilization has littered space with billions upon billions of such "mines" that quietly store all this energy, like a set of corks in a dam. When activated by the early warning sensor grid, they unplug their carefully designed set of cone-covering spacetime rifts and let it rip the intruding ships apart over a wide area.
Classical AI fleet
Anybody that survives that flashy welcome that the first few dozen layers of kinetics and rift-mines have extended will then be greeted by your local spherical-octant mobile defenses: AI-directed firing platforms who were set on an intercept course as soon as the lightcone (darkcone?) of information reached them.
These massive ships will have all the weaponry of your initial mines, dialed up to 11. Without squishy humans on board, these are capable of pulling 10,000 g accelerations with ease, so they will be on scene toute-suite. Without needing to be stealthy or discreete, these monsters will probably have an utterly obscene power output ...
And just in case they don't suffice, somewhere a few light-days or weeks behind them is the actual Capital Fleet, which is the stuff that mothers in alien civilizations use to scare their disobedient little reptilian brats who won't go to bed.
Inner System
In order to be able to field even such basic defenses as mentioned above, (i.e. nothing that would stop a determined Type II civ with a grudge), your civilization would need to be a high Type I, near Type II at least. Your system's industrial capacity must then be about $10^{26}$ watts, so generating $10^{11}$ (100 billion) kinetic impactors would take about 1 second of the civ's power output, while the rift devices might require more power, (but you'd need fewer of them). That said, a type II civ should be able to reach this basic level of militarization from scratch in about a year without any discernible impact on the inner-system standard of living.
Your innermost defenses, if the visitors actually came for a real fight, will be powered by your Mercurian Solar Grid. While this is normally reserved for the purposes of research and investment banking AIs, in times of emergency the output can be directed into a beamed weapon of near unimaginable strength, with a continuous power output in the range of $10^{25}W$, that would be hitting with the energy of 100 dinosaur-killer asteroids each second. It would take a really determined enemy to get past this one.