Your suggestion of using ballistae or catapults presents two issues : Logistics and Damage.
On logistics first :
As others have pointed out, ballistae and catapults are big engines that take up a lot of space. While putting 1 or 2 on a ship isn't a problem (and is a think historical), you're going to have a lot of trouble putting enough of them on a ship's deck to deliver a broadside/of-the-line fire. That's why cannons were so practical : they're essentially metal tubes, and their firepower compared to their size is great.
There are however ideas for traditional weapons you could feasibly puton ships that would help with your space problem :
This is a ballista (there is a more specific name but it escapes me) that was used on battlefields to fire javelin-sized bolts at enemy formations. They are small, operated by 2/3 soldiers, and have a reasonably high rate of fire.
You could put quite a lot of those on the deck of a ship and thus achieve a "rain-down" effect making use of their rate of fire, although they would be less destructive than bigger devices.
But thanks to genius Leonardo da Vinci, we also have catapult designs that are more compact than traditional catapults :
This is da Vinci's catapult concept. As you can see, its main feature is that contrary to a normal ballista, the 'bows' it uses for tension are place vertically, not horizontally. This means that you could place several of these catapults much close together than you could normal ballistas, for the same (theoretical) destructive power.
Now it would seem that we solved the logistics problem. However, the bigger and much more important problem is damage.
On damage :
Cannons, even in their primitive stage, have significantly more potential power against structures than more traditional siege weapons. Even with several of them and magical enhancement, it's highly dubious that a ballista bolt would be able to penetrate or in any case fly much futher than a warship's hull. Ballistae are primarily anti-personel weapons, they do little structural damage.
While catapults are more powerful, they still present the disadvantage of being fairly inaccurate, not least because they fire in an arc. Even with the help of magic, I doubt you could really improve that.
So it would seem that ships in your universe, especially using that reinforced wood, would be mostly safe from catapult/ballistae fire (except for incendiary projectiles). In that sense, of-the-line formations that use broadside firing do not make much sense.
Hence, here is my suggestion in how ship structures should change to account for that :
Your ships are built to be as tough as possible. At the same time, they are built up high or have an elevated part (much like a ship's "castle" in real life") on which as many archers as possible can take place. The ship's tactic, instead of bombarding the enemy ships, is to get close to them, and from their high groud rain down arrows on the enemy crews.
Up close, these ships could wreak havoc on enemy crews and board their ships.
The alternative is that you develop ammunition, maybe magic ammunition, that is primarily fire-based to set fire to enemy ships with your catapults.