During the 17th century CE, there are many accounts of a giant sea monster living in the sea and the king ordered a search party to destroy it. The sea monster usually shy away from people and feeds on fish, it can reach 50m in length and will only surface to catch breaths. It normally swims quietly in the deep water and can dive up to 1000m depth, it has poor eyesight and hearing but is a good swimmer and top a sailfish while fully submerged for a short distance of 10m. The sea monster took the form of a serpentine with a megamouth head, the scale's armor level is rated type 2A on NIJ standard. How can the people in the 17th century CE hunt the sea monster efficiently without any collateral damage? Answer with the shortest time and least man power wins.
I see the monster you describe as a sort of armoured whale. Whaling was already a large industry in the 17th century perhaps modifying a harpoon to be fired from a small cannon is all you need in this case.
The earliest example of a whaler I could find on Wikipedia is the Truelove from 1764 however there are examples of the Dutch sending normal men-of-war after whales in the very early 1600s. They appear to have sent fleets of 11-14 ships after whales.
How to Catch a Sea Monster
They don't go into details on the tactics used but I think it would be reasonable to assume 3 or 4 ships per whale, surrounding it and chasing it until it becomes exhausted and you can pick it off at will. A single ship could do this but remember Moby Dick was based on a real encounter with a whale. Why take the risk? Especially seeing you have a king paying for it.
Harpoons and Cannons
The harpoon cannon wasn't invented until the 1800s but perhaps your enterprising sea monster hunters could load a hand harpoon into a normal cannon and use cloth or linen as a sabot to fire it. Accuracy would be limited but if you have already chased your prey down and it is exhausted you would have plenty of time to experiment. Explosive harpoons became widely used in the early 1700s although these appear to be still handthrown.
You could combine the explosive harpoon with a Chinese rocket firework, these had been used in Europe as instruments of war since the 1650s. Still throwing these by hand would be... risky.
The Essex, the whaling ship that inspired Moby Dick, had a crew of about 22. This is actually a lot less than I had thought, I would have guessed around 100.
This can vary wildly. A single ship could take years to locate the monster. Assuming you have found it and have begun chasing the monster, that would depend on the stamina of the monster. I can't see this taking longer than a few hours. Then to actually kill it another hour, maybe two if your accuracy is terrible.
In response to your comment about tracking the creature. This would have to be done with the human eyeball. Every crewman not involved in the direct management of the ship would have their eyes glued to the horizon looking for the monsters fin. They probably would not use a telescope or binoculars as these devices drastically reduce the field of view and reduce the ability to catch small movements in the periphery. A telescope might be used to confirm a potential sighting. To increase your chances of finding it you should head to areas where the monster's prey is known to live.
If you have multiple ships you can signal potential sightings with flags or some sort of morse code lamp. Two ships confirming a sighting will give the captains the ability to use parallax to get a distance and heading of the monster.
Basically it is blind luck that a single ship will run into the creature on the open ocean.
Note on SONAR
Wikipedia notes that da Vinci used a sort of passive SONAR. It basically boils down to a tube or drainpipe type arrangement with one end in the water and the other attached to the ear. I am really not sure how effective it would be at locating submarine animals, I'm fairly certain the noise of the waves splashing against the boat's hull would deafen the user.
"...use by humans in the water is initially recorded by Leonardo da Vinci in 1490: a tube inserted into the water was said to be used to detect vessels by placing an ear to the tube."
I hate to give the boring, but practical, answer. There is no reason to fight this creature.
traps and poison are a far more logical approach. Figure out where the creature hunts and drop a ton of poisoned, or just spoiled, food for him to gobble up, wait for him to get ill and die on his own.
A few approaches similar to this could be used, to trap, poison, or weaken the monster. I won't delineate them all because, lets be honest, no one wants to write such a boring solution into the story, but it IS the most logical if you just want to get rid of it.