I'm writing a fantasy novel and in it I'm designing a huge wooden ship moved by sail and man powered wheel paddle, and I wonder if it would be possible for the ship to carry a ballista inside that fires from the ship's prow? How effective were balistas against ships hull? How powerful was the largest ballista ever known? Sorry for the many questions and thanks in advance.
Is it possible, inside?
Sure, many ships actually did carry ballista, though those were normally on the deck rather than inside, but inside is still a deck, just means it's a covered deck.
But on the top deck rather than a lower deck you potentially have a full three sixty degrees you can target by just swiveling your ballista without having to change course & swing the whole boat about, so it seems a bit silly to put it below decks if you only have the one ballista.
If you're using multiple small ballista it makes more sense to follow the pattern adopted by cannon bearing ships, rank them along the sides so you can bring more of them to bear on a single target simultaneously.
In classical times, Roman ships carried torsion powered catapults as their primary weapon.
Roman warships evolved to become larger and have broader beams in order to become stable fire platforms for these weapons, but this may have been co evolution due to the Roman idea of using Marines to board enemy ships: lager ships could also hold more troops, and would also be more stable if a boarding platform like a Corvus was mounted.
Diagram of a Corvus*
Ancient Greek trireme. The ship is the weapon
Roman "Decres". You can see the huge increase in size and provision for catapults on the deck. The smaller "Liburna" became the mainstay of the fleet after the Romans cleared the seas of pirates, and had no significant opposition
This leads to an important question in your worldbuilding: why does your navy feel the need for such large ships? What advantage do larger, more expensive, ships offer which smaller, cheaper ships do not? As an example, the Venetian "Arsenal" had several large "Galleasses" which were converted into mobile artillery platforms for the Battle of Lepanto. The Venetian merchant fleet had no use for them as the expense of running them ate any profits they might achieve, and the Venetian navy was dubious about their utility due to their size and slowness. Their use as artillery platforms was fortuitous, since there were only six, and indeed they were so slow they only were able to participate in the opening phase of the battle.
Model of a Venetian Galleasses
So while your ship seems to be possible in theory, what practical use does it serve which cannot be carried out by smaller, cheaper ships? The polity which owns the ship or creates the navy will be very keen to find the cheapest, most practical means of carrying out the task, and if there are suitable alternatives, then they will take them over the larger, more costly ship.
How powerful was the largest ballista ever known?
According to Vitrubius (On Architecture, X, 11), the largest Roman ballistae fired stones of 250 pounds.
A catapult this size is an awful idea in a ship: with so much weight it probably took half an hour to fire each shot. It isn't a serious problem in a siege but, since the attack speed of a warship was 8-10 knots, you might want to settle for smaller siege equipment.
I think the standard ballista in a Roman ship shot stones of 40 pounds at distances inferior to 400 yards.
How effective were balistas against ships hull?
They weren't designed for breaching the hull (for that they rammed the ship or broke her oars). They were for killing the sailors on board, which is the reason javelins were also used as ammo for ballistae in ships according to Julius Caesar. There were around 100-120 soldiers on a warship, so killing a score of them with some shots was very effective for morale.