I see the monster you describe as a sort of armoured whale. Whaling was already a large industry in the 17th century perhaps modifying a harpoon to be fired from a small cannon is all you need in this case.
Ships
The earlist example of a whaler I could find on wikipedia is the Truelove from 1764 however there are examples of the Dutch sending normal men-of-war after whales in the very early 1600s. They appear to have sent fleets of 11-14 ships after whales.
How to Catch a Sea Monster
They don't go into details on the tactics used but I think it would be reasonable to assume 3 or 4 ships per whale, surrounding it and chasing it until it becomes exhausted and you can pick it off at will. A single ship could do this but remember Moby Dick was based on a real encounter with a whale. Why take the risk? Especially seeing you have a king paying for it.
Harpoons and Cannons
The harpoon cannon wasn't invented until the 1800s but perhaps your enterprising sea monster hunters could load a hand harpoon into a normal cannon and use cloth or linen as a sabot to fire it. Accuracy would be limited but if you have already chased your prey down and it is exhausted you would have plenty of time to experiment. Explosive harpoons became widely used in the early 1700s although these appear to be still handthrown.
You could combine the explosive harpoon with a Chinese rocket firework, these had been used in Europe as instruments of war since the 1650s. Still throwing these by hand would be... risky.
Manpower Required
The Essex, the whaling ship that inspired Moby Dick, had a crew of about 22. This is actually a lot less than I had thought, I would have guessed around 100.
Time Taken
This can vary wildly. A single ship could take years to locate the monster. Assuming you have found it and have begun chasing the monster, that would depend on the stamina of the monster. I can't see this taking longer than a few hours. Then to actually kill it another hour, maybe two if your accuracy is terrible.
Tracking
In response to your comment about tracking the creature. This would have to be done with the human eyeball. Every crewman not involved in the direct management of the ship would have their eyes glued to the horizon looking for the monsters fin. They probably would not use a telescope or binoculars as these devices drastically reduce the field of view and reduce the ability to catch small movements in the preriphary. A telescope might be used to confirm a potential sighting. To increase your chances of finding it you should head to areas where the monster's prey is known to live.
If you have multiple ships you can signal potential sightings with flags or some sort of morse code lamp. Two ships confirming a sighting will give the captains the ability to use parallax to get a distance and heading of the monster.
Basically it is blind luck that a single ship will run into the creature on the open ocean.
References:
