As other point out, eventually pumice rafts do sink. They have some other tricky properties.
Wikipedia says they sit about 0.6 metres about water level, suggesting that they are precarious - despite being huge at times, they aren't like icebergs that can be dozens or more metres thick. For comparison sea waves are easily tens of metres high. They also aren't made of huge monolithic lumps like icebergs, but from millions of relatively small fragments, so they can rotate and rub, and have much more surface exposed to the elements. So they are thin, made of millions of little bits of rock, and will deform in overall shape, rotate, jostle, and be swamped, by even a fairly quiet sea.
Despite this, I like the idea and there should be a way to make it work. The flexibility is an advantage if any weight is spread across a raft, and if there's some kind of reason why they can stay a cohesive whole providing support, despite a rough sea.
They also need to have a reason why they don't become waterlogged - a raft can last 20 years (Wikipedia) so its not bad but needs more.
Proposed idea
I suggest that these loose pumice rafts are being modified to be suitable for longer term use. Let's see where that takes us.....
Suppose the pumice is created by active volcanic activity within an area of sea/ocean that's largely enclosed, or a landlocked sea. We could imagine that some kind of oil or other coating is pumped into the sea. It floats, and it coats the pumice then sets or bonds with the rock, perhaps as it passes some narrowed area. (Its tricky to make this environmentally friendly but maybe the gases mean its largely devoid of bird and animal life anyway?)
We now have pumice that is coated in some water repellant surfacing, and which won't easily come off. The sea doesnt need to be deeply covered at all, for your raw pumice to all get coated after its bobbed around for a while. (Oil slicks spread out very thin). So quantities are easy. That's one problem solved.
Suppose the surfacing material, as it ages, is sticky. The pumice raft that tends to form, contains small pieces as small as a marble or smaller. The fringes will break apart but internally within the raft they'll tend to stick and coalesce a bit more. The raft will also get thicker.
I think we can chemically handwave the compound to be sticky, float, and very slowly harden like a waterproof oily glue. Pump it into the water, and your developing raft will emerge into the open sea and every piece will be coated with this stuff. The raft that forms will break away at the edges - but the rafts can be tens of thousands of square miles, so even broken up, they could be huge in human habitation terms.
And now you have your pumice problems solved.