Giant mermaids is a nice idea, it seems very plausible that a sea-version of humans would become much bigger with time.
I think the best bet is some poisoned harpoons. Just bows probably don't cut it, they don't look like they have as thick skin as whales, but probably still much thicker than land animals, for isolation in water.
If the medieval people are really smart and lucky, maybe they'll be able to spread some mermaid-diseases, perhaps by spreading body parts of a diseased mermaid they caught (or found after death). Biological warfare was a thing in medieval times, though limited.
The harpoons do assume the ability to chase mermaids down until they come up to breath. If that's not an option (i.e. they are too fast), it's going to be much harder. I don't think they would need to come up to sink the ships - they could probably poke holes in ships using metal/rock/sharp wood, or capsize them using rope stolen from other ships.
Another thing that would help is making bigger boats. If boats are big enough, they can't be capsized by a pod of mermaids, unless the mermaids find some leverage point, which seems unlikely in the ocean. Width is especially helpful, although of course makes them slower. Bigger boats would also naturally have stronger hulls, hopefully preventing the mermaids poking holes. There were no metal ships in medieval times, but maybe the technology for them existed?
If making individual ships bigger doesn't work, then travel is a fleet, and tie the ships together when mermaids are spotted. Maybe you can get advance warning by training birds to scout for you.
The mermaids are big mammals, so a lot of resources and time probably goes into growing each one. Plus they're smart, so I think you don't need to inflict too high a casualty on them to make them give up their piracy. They're probably already apex predators, hunting human ships is not their only means of survival, they'll switch to something easier if ships get too hard.
Some things I don't expect to work:
- I don't think adding spikes to the bottom/side of ship would work well. The mermaids have hands and look pretty nimble, so a spike is probably just something for them to hold on to while capsizing the ship.
- They're probably apex predators, so training other animals to fight probably doesn't do any good. They're most similar to orcas, who are apex predators. These things are smarter, similar sized, and live in similar or bigger pods. Plus mermaids are going to be much better at domesticating sea animals than humans are.
- I think dumping poison in the water would very quickly dilute too much, or be left behind. Similarly with oil. Also it's probably not cheap.
- Solutions with electricity or sound or explosives or such are probably too modern.
- Trying to scare them somehow, like scaring land predators with fire, probably doesn't work since they are very smart.
- I can't easily find anything about medieval methods to detect whale sounds, so I'm assuming it's not possible, but not sure.