Sea Castles:
Here is an excellent page on the topic of pre-gunfire naval warfare, and I freely admit to borrowing from it quite a bit (although I was familiar with the ideas, it spells them out very clearly). Here is another good reference about the evolution of naval weaponry.
I suspect what you'll end up having is very tall ships, with sails that can be quickly swung out of the way. Other than that, you'll want very fast, very fire-resistant vessels.
First, Greek fire was a thing in naval warfare well before the advent of gunpowder. Your navies, if they choose to try and sink each other, will use fire to the utmost to damage or destroy rival vessels. Without cannons, this technology will become VERY advanced. I suspect there will be a lot of big flame throwers at sea. If this doesn't match your desired effect, you better introduce fire-proof fabrics and wood treatments as well.
Rams, while one of the few effective ways of sinking ships, will not be practical for these ships. Wind will be powering both vessels in any battle, so one ship is not likely to be going sufficiently fast in a different way with great regularity. Sails are very fragile to impacts, so ships using rams would snap off their own masks frequently.
Boarding is likely to be a prominent strategy. Get on to the enemy ship and kill or capture the other guy. You will prefer to kill him before you get on his deck, so projectiles will also be vital.
Height is a huge advantage if you are shooting arrows, firing ballistae, or just dropping a boarding plank on top of another ship. Before gunpowder, war ships got quite tall. The goal here was to have the advantage of gravity in both boarding and firing projectiles. While this might make them top-heavy (and less sea-worthy), it will help in battle.