So before cannons how would naval warfare work in a medieval fantasy setting? I only know of Greek fire and flaming arrows. Would soldiers still wear armour or use light clothes in case they fell overboard or the ship sank? In my story there is a high presence of magic so maybe that could compensate for a lack of gunpowder. Mainly I need to know what types of weapons would be used ship vs ship. Thank you to all who help!
Well, you might consider the shape and tactics of the Venetian navy. They did very well with archers and infantry and ramming techniques. I don't know if they had greek fire or not, but they certainly rivalled the Byzantine navy and eventually acted quite independently of that Empire. They held sway over the Adriatic and eastern Mediterranean for several hundred years and later kept the Ottoman navy at bay as well. Many of their ship designs seem to evolve from earlier Roman designs. And the Roman navy is another place you can look for inspiration.
Ships wood fire catapults and ballistas at each other for far range.
Up close ships would try ramming each other once one ship rammed another if the other ship didn't sink they would send over borders. Which would take over the ship.
No it's unlikely these guys would wear heavy armor. If your ship was doomed you you could have abandon it set it on fire and then set it on course straight for the enemy ship suicide bomber Style.