Frame challenge
If you mapify a circle then it seems to me that you have little or no control of the final shape. Why not simply draw the outline yourself? *
If you use Windows for example then the original Paint app allows you to hand draw using a mouse. This gives a perfectly adequate coastline appearance IMO. Any simple draw or sketch program should have the same capability.
The following map took me less than 30 seconds using Windows Paint. That includes drawing, colour-filling, saving and uploading.

*Edit
It occurs me that not everyone has the mobility to hand draw a shape. I apologise to anyone that applies to for making the assumption. I will look for other solutions and add them here.
Software
As commented by @Smock, there is,
World Painter TUTORIAL - How To Create Custom Minecraft Maps!
https://youtu.be/c6dULt6oEn0
I'm sure interesting coastlines could be achieved by making use of
Fractal Explorer in Gimp
https://docs.gimp.org/en/plug-in-fractalexplorer.html
https://docs.gimp.org/en/plug-in-ifscompose.html

Information that may be of interest
Fantasy maps and mapmaking tutorials by Jonathan Roberts
http://www.fantasticmaps.com/2013/01/drawing-realistic-coastlines/