What's the minimum safe distance to start building if you don't want any part of the shell or swarm to fall into the event horizon? Could you build a stable structure just short of the Schwarzschild radius or would you need to move farther back to be safe? How much farther? Could you build a shell only a few thousand kilometers in diameter around a relatively small black hole, or would millions of kilometers be the minimum safe distance?
The main use of building a Dyson swarm around a black hole would be to capture the energy released by its accretion disk. This is actually a rather efficient way of producing energy, since it can turn up to 40% of the rest mass of matter in the disk into radiation, much better than nuclear fission or fusion.
You don't want the elements of your swarm colliding with the accretion disk, so they will need to orbit well clear of it, much farther out than the minimum orbital radius. You'll also have to arrange for occasional gaps in the swarm to allow more matter to be fed to the accretion disc, but that is much easier than the other challenges of this project. Those include:
Accelerating your swarm elements to orbital speed, which will probably be at least 0.05c.
Keeping space fairly clear of debris in the area. A relativistic Kessler cascade is going to be a horrible problem.
Converting all those hard X-rays to some form that's easier to transmit or store. If you transmit it, relativity makes that complicated; if you store it, you have to lift it out of the black hole's gravity well.
The minimum safe distance for components of a dyson swarm will be determined by the maximum safe orbital velocity for the individual units of the swarm. This is then dependent of the capacity of this world's technology, For example, if the maximum velocity attainable is half the speed of light, then the dyson swarm will be orbiting the black hole with an orbital velocity of approximately 0.5 c. Now you need to factor in the mass of the black hole and will yield what may be the minimum safe distance (assuming there are no other factors to endanger the dyson swarm).
NOTE: This answer has limited itself to considering a possible minimum safe distance for a dyson swarm and not a dyson shell. Building a dyson shell around a black hole comes with a plethora of additional and probably intractable problems which need to be solved before even beginning to consider what constitutes a minimum safe distance.
There will be a component of velocity due to the black hole's gravitation. If their maximum attainable velocity is one-half lightspeed, the dyson swarm units will be able to boost away from orbit around the back hole, if they need to (just think of it as an additional safety feature). If space vehicles can accelerate to whatever their maximum velocity is, they will also have the capacity to decelerate too.
A Dyson shell is (unfortunately) impossible to build. Nothing with structural strength high enough exists, and it would require continual adjustment to not fall into the star/black hole. However, a Dyson swarm could be built relatively close to the event horizon, albeit at the massive cost of accelerating the individual satellites to near-lightspeed to achieve orbit. Further out, it will be much simpler and easier, but require more materials.
" Dyson shell is (unfortunately) impossible to build. Nothing with structural strength high enough exists, and it would require continual adjustment to not fall into the star/black hole. However, a Dyson swarm could be built relatively close to the event horizon, albeit at the massive cost of accelerating the individual satellites to near-light-speed to achieve orbit. Further out, it will be much simpler and easier, but require more materials" answer to the above:
construct the swarm to be inclusive of a repulsive gravitational field (similar the the astrophysical structure of a pulsar) oscillated/emitted from the swarm itself to sustain around the hole, additionally have each equipped with a laser based projection optical ray to catalyze the accretion disk for higher power achievement through absorption by the swarm.