You know how in various fantasy computer games the castles get destroyed by a bunch of little guys running up to it and hitting it with their swords? That doesn't work brilliantly in real life because the sorts of stone you might build a castle with are fairly strong. Pumice, on the other hand, is pretty straightforward to carve with more or less any tool and a little brute force. You could lay siege to it with a bunch of guys with bouyant shields... paddle out to the castle on your shield, then swim up to the wall underneath it and start chipping away.
In fact, given its soft nature and relatively high-friction surface, there's a reasonable chance that a person could scale a pumice wall armed with crampons and steel picks, as if they were an ice climber. That makes your castle very vulnerable to stealthy swimmers at night.
"But!" I imagine you saying. "I could clad my castle with smooth, hard stone blocks, like a real castle!" well, maybe. Only in order to bouy up the weight of a real castle, you'll need a truly colossal foundation of pumice, and the whole enterprise becomes astonishingly unwieldy. You can't have your pumice extending too far down underwater relative to its width or you'll get stability issues (eg. your castle will topple over in poor weather, because floaty-up, sinky-down is a more stable configuration), so you'll really be making a huge, flat pumice island with a little stone tower in the middle. Have a think about how useful that will be. Next, have a think about how you'll move it around, and not be at the mercy of the elements.
Now you're left with a huge artificial island that's going to be extremely hard to manoeuvre, big enough that it will be easily seen coming, and with a tiny fortification in the middle. What on earth is it going to do? It can't realistically protect or attack anything. It might be a cool place for a wizard to live, but that's about it.
There's no better alternative stone, by the way, if you want to use a lighter-than-water construction stone, it is gonna end up looking a lot like pumice because that density has to be shed somewhere.
The obvious answer as suggested elsewhere, is wood, because it is known to work well. If you were wedded to minerals as a fortification material and were prepared to have a slightly more unusual (eg. somewhat implausible) tech level, there's always concrete-hulled boats as a possibility...