BWAHAHAHA!!! YES! Here we go!
So first things first, the fire is a relatively short ranged weapon no matter what. The big draw heavy ballista might be able to deliver accelerants like oil and pitch, but even the handheld bows are going to fire way to hard to stay lit across their flight. The speed of the arrow pulls the burning wick out of the vapor pocket and extinguishes it. Having built a couple flaming arrows myself, it's surprisingly complicated to get right.
But here's the deal, those bows and siege weapons are putting out a HUGE amount of kinetic energy! They might as well be small cannon. Those are the things which kill your crew, so those are the things you focus on defending against.
Let's start with a trireme, that's about right. The physical abilities will come in handy for the rowers below decks. Switching to rowers will be necessary because as others mentioned the sails will be a primary target in a fight. Since we're "peak age of sail" our vessels will still have sophisticated rigging to allow them to move under sail for long voyages then stow them for a fight.
The whole ship is round as possible, built possibly with outriggers below or at the waterline for stability. There is a round reinforced canopy "shell" over the main deck, the masts protrude through this. The sails can be withdrawn into the ship to protect them or cut free and jettisoned over the side if they catch fire. The primary objective of the hull (besides flotation) is to deflect incoming missiles. This also restricts the firing angle, so there may be archery platforms with retractable roofs in parts of the shell.
So it's going to be hard to cripple this ship from long range, the rowers are protected below decks and it's not weather-dependent so it's hard to outmaneuver. The marines on deck are protected by their shell overhead and bulkheads they can use for cover on the sides. It's probably pretty heavy and slow though, so will want to close to engage.
Boarding is going to be awful. These guys can hulk out at will so no one is gonna want to climb aboard one of their vessels to duke it out. If they're fighting an empire of normal people, their primary strategy would be to close and board the non-hulks. Fighting other hulks, yeah not so much. That's where getting the fire on the deck under the shield or into the rower's compartment would be super effective. That's the part a magic flamethrower is PERFECT for.
Ships planning to tangle with these weapons are going to have lots of active and passive fire protection. The rails and top deck are all built to shed water, and have raised ridged walkways designed for traction when oily and wet. Both the prow and stern have adjustable ports which allow water to pour across the deck and sides of the ship when she's turned directly into the waves. Heavy treated and soaked curtains cover the rower's windows, vents, and oar holes during battle. The ship can purposely soak itself and many commanders would likely rather fight in heavy rain or storms.
Rams would be effective weapons. I am still liking the idea of low, missile-resistant ships using outriggers of some sort for stability. You'd have a variety of rams, some traditional and meant to inflict damage below the waterline, others more vertical like an axe blade to try to sever the outriggers of other ships. I think the worst would be a heavily-reinforced cone designed to punch through the side of the oar deck, vents a meter back from the tip to let you fire your magic flamethrower directly into the enemies rowers' compartment.