Firepower is the key
The other answer is saying that you can take advantage of upwind speed and manoeuvrability. It is true, but it will not compensate for the enormous firepower of larger monohull warships. Let’s take an example: The galleys were able to move fast even when there was no wind, they were able to go straight upwind if needed. They were much more manoeuvrable. However, in the battle of Cape Celidonia, 5 Spanish galleons (big mono-hull sailing ships) defeated 55 ottomans galleys. This is typical from this period, when large sail-only ships started to replace galleys on the seas.
Basically, during late sail-era, the more fire power you could carry, the more likely you were of winning.
You are not talking about galleys, but light multi-hulls. I believe however that the problem will be similar: your vessels are small. It is difficult to build a large catamaran or outrigger, because of their shape. A monohull warship is basically a big cylinder. Catamaran are two smaller cylinder with pieces attached between them. So the limitation that you have are:
- Less cargo (food, men…) for a similarly sized vessel, so you cannot go so far away without ressuply
- Probably more delicate
- Less guns, very important
- Guns closer to water
- Lower freeboard and bulwarks, making it easier to be boarded. (or harder for them to board bigger ship)
The advantage were already covered in another answer:
- Faster upwind
- Better manoeuvrability
In an open war between well-defined opponents, larger boats, even if they can't catch your catamarans, can come in range of cities and gun them from the sea. They could occupy harbours. They can go further in the open sea. They can carry more men and resources, in a safer way. My guess is that the catamarans are going to lose, not because they are sunk, but because the large ships are going to cut their harbours, resources and so on.
Hull design solution
You ask if anything can be done in terms of hull design. I believe that the only way is making it bigger. But probably, while trying to make them bigger, they are going to lose some of the advantages (think weight and structure fragilization). Maybe the architects are going to reinvent mono-hulls?
Adaptative solution : Guerilla war
So, as these boats won’t stand in front a large mono-hull vessel. Their strategy, in order to exploit their advantage at the maximum, is to play it guerrilla style.
- Approach quickly, discretly
- See if you have a chance
- Attack by surprise, or only smaller boats
- Run away and hide
This strategy is close to what were (and are) doing a lot of pirates.