“Space is big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist's, but that's just peanuts to space.”
- Douglas Adams, The Hitchiker's Guide to the Galaxy
There are two reasons why your civilization doesn't spot the Dyson Swarm ("DS"), which together make the DS functionally invisible.
Space is really big. We already have problems even finding other countries' secret military satellites, and that's on the scale of a planetary orbit.
Your hypothetical civilization has FTL. Due to the above, one of the only ways for your civilization to find the DS is if one of their ships does a fly-by. Your civilization figured out how to use FTL quite early on, so they tend to go through the outer reaches of the system at superluminal speeds. Such high velocities are good for getting places quickly, but make sightseeing practically impossible. Anyway, why would people want to look at that area of space anyway? It's just a bunch of boring old asteroids.
So, your civilization goes for a few hundred years without noticing the swarm, and everything's fine. However, they still think that 150 times the speed of light is pretty slow, so they put a bunch of money into developing a faster drive. Eventually this investment pays off, and they come up with an Asimovian hyperspace drive. This allows them to get from point A to point B in literally no time at all, skipping the years spent in transit.
There's only one catch. Although the hyperspatial drive is very good at getting you from point A to point B without having to go through the rest of the alphabet, there are some complex calculations involved. So complex, in fact, that you can't do them without the newest in supercomputing technology. Already extremely difficult, these computations become almost impossibly hard when you toss in a gravity well.
Despite having multiple space drives, your society still hasn't managed to prove that P = NP, so hyperspatial drives are only used for the interstellar portion of journeys. The new drive makes 99.99% of a trip go by in the blink of an eye, but the remaining 0.01% has to be made with other means of propulsion.
Unfortunately for aspiring day-trippers, traditional FTL drives interact explosively*explosively* with hyperspatial drives. The only other option is ionic thrusters, which are decidedly non-FTL. This forces ships to go through systems at sightseeing speeds. If only there was something out there to sightsee...
Space may be vast, but so is a Dyson Swarm. Without anything else to do, bored passengers soon spot the swarm.
* Actually, it's a topographically-complex n-dimensional spatial fracture, but the end result is the same.* Actually, it's a topographically complex n-dimensional spatial fracture, but the end result is the same.