phosphorous is comparatively rare but is hyper concentrated on Earth due to its importance in biology.
The universe appears to be about 7 parts per million of phosphorous (source). It isn't precisely commonplace, but it is far from rare. It is more common than copper, for example. It may be hyperconcentrated on Earth in biological things, but by the standards of modern technological societies, it is far more economical to exploit mineral sources of the stuff. This will likely also be true for your spacefarers... leave the guano for the pre- and peri-technological societies, because they need less of the stuff.
So outside of Earth, how abundant would these resources be? Where would you best mine them?
Rocky planets with a metallic core around Sunlike stars at similar distances from the galactic center would obviously be a good place to start looking.
You'll find the bulk of Earth's stock of the stuff to be quite deep under the crust and therefore awkward to harvest. Smaller astronomical bodies with a common origin would therefore be much easier to mine. Earth's moon is a particularly special case, as it probably formed from a big impact which stripped off a big chunk of the Earth and as a result it formed vast KREEP terranes where KREEP stands for "Potassium (K), Rare Earth Elements, Phosphorous".
In asteroids, moons, rocky planets, or gas giants?
Forget gas giants. Gravity wells much too deep. it might be practical to extract some kinds of useful light element (eg. helium-3) from a gas-dwarf or Neptunian, but the things you're after are too heavy to form enough of the atmosphere to be practical to exploit and the rocky core is much too hard to access.
Dense metallic asteroids would be a favorite because it takes so much less energy to completely dismantle the things compared to whole planets. You can probably find a reasonable amount of stuff like neodymium and other "rare earth" out there, and though the densities are low they'll still be practical to exploit when you extract the more common metals and minerals.
Lanthanides have been suggested to be more common on Earth than in asteroids, so really your best bet is to strip mine Earthlike worlds and their moons. If you can afford to fly between stars looking for metals, then you can definitely afford vast solar-powered mining facilities that can do the concentration economically for you.
Your ideal target system, perhaps, would be one that had large asteroid belts in Earthlike orbits around Sunlike stars, because there's a good chance they'd have a similar mineral composition to the Earth but without that inconveniently deep gravity well or thick, hard-to-mine mantle. A late-stage protoplanetary disk might also be a good prospect. Neither location is likely to have locals who evolved their naturally, by way of a bonus.