Different coastlines have different ruggedness. They are not all the same.
So assign different fractal dimensions to different segments as you generate it.
E.g. Great Britain is 1.25 but Ireland is 1.22 and varies a bit around the island. Norway most famously is 1.52.
This should work with the geology if you’re doing that level of detail — different kinds of rocks wear differently.
Others commented on your lack of disclosing your projection. Drawing the coastlines on a plane (even if the overall shape is projected first) will not look right!
As for the sea-sized inlet you asked a about, that could very well be a rift. Expect the surrounding geography to be consistent with this. And the larger point is that particular features don’t occur in isolation, but they interact.
I wonder about the islands near the center of the picture associated with the upper-left mass: the whole thing looks like it’s a breakup of a supercontinent, as with our world. So did these flake off as separate plates or what? If sealevel is high the islands would define the shape of the piece that broke away, with shallow seas covering the middle. Your map doesn’t look like that.