Well, as Andon said, you might want to decrease the spacing of those worms, because your pinnipeds will have almost 10 metres of space between each one. Unless your worms have branching structures.
Barnacles do not have branches, and they would have no reason to do so. Corals have branches because they are colonial organisms that multiply, so they need to expand the limestone core outwards to accommodate more polyps. Trees branch because they are photosynthesizing plants, and that's how plants grow.
The only thing reminiscent of branches that i could see barnacles evolving are external cirri, or feeding legs, but these are made of soft tissue so would not harm the pinnipeds.
The bivalve body plan is also an extremely successful structure in nature, so unless they went a radical change in ecology I doubt that it would change much. Normally, bivalves get nutrition by collecting detritus with their palp proboscidae. Even if they took on the niche of corals, which get energy from bacteria on the polyps that convert sugar into such, they would still not need any kind of structure to fill the 10 metres squared of space between it and the next worm.
Fan worms do have external space-filling structures, but again these are made of soft tissue. So, ultimately, it doesn't look like any of the calcareous animals in your reefs would be able to harm an animal unless you increase the density of them per acre.
Another thing - even if these worms were densely packed, never underestimate the agility of a pinniped.
Otariids are not rotund and have minimal blubber, because they mainly live in warm climates. They are shaped like torpedoes, with large flippers for maneuvres. They can turn on a dime and flip through the air, so they'd be absolutely fine in an underwater thicket.
The Phocids, blubbery though they may seem, are also shockingly agile. They have extremely flexible spines and can perform almost all the underwater maneuvres of an otariids.
The least agile of your pinnipeds are going to be Odobenids, which are much more cumbersome. However, walrus hide can withstand stabs and slashes from ivory tusks, and absorb the full force of a competing bull. In fact, the blubber of these animals is going to help them, not hinder them, in this environment.'
A final word: I think that your date is a bit too extreme. If calcareous reefs evolved 400 million years ago, that could change the course of evolutionary history. Paleozoic fish would evolve to cope with it, the same fish that are going to crawl out of water and become the first tetrapods. Therefore; nearly every chordate species on Earth will have evolved differently.
If early fish evolved to cope with this environment, they might have evolved hard chitinous plating, like some Devonian fish which did that for protection. But an entirely different ocean ecosystem could cause all fish to adapt, so nearly all fish may have hard coverings.
And one of the things that makes amphibians amphibians is their soft skin that requires to be wet frequently, which would not evolve because of the fish' hard skin, meaning that there would never be any amphibians.
And if there were no amphibians, there would be no reptiles. Without reptiles, there would be no mammals or birds. So, I suggest that you wipe out corals far further into the future, at least as recent as the K-Pg extinction. If you can find some natural cause of bleaching that would occur then, there would be a global coral extinction event.
65 million years - especially after a mass extinction - is plenty of time for your annelids, molluscs and arthropods to fill the niche and diversify. Typically, after extinction events, life recovers very quickly, and the survivors "explode" into all available niches.
Thanks for asking, hope the answer helps.