I would argue that the modern day military would have to monumentally screw up in order to not wipe the floor with the 1600s era military, no matter how great the numerical disparity.
Chances are they would not even have to meet the enemy in the field. All it would take is a show of force with a flight of aircraft bombing their capitalforts and government buildings- that should be more than sufficient to make it clear to the leaders of the continent that it's in their best interest to not resist. While they may not care about the lives of their citizens, the fact that you can put a 1,000lb bomb on a warlord / king's heads anywhere in the continent would make them bend their knee quick enough.
The enemy military would likely be armed with smooth bore muskets, with a range of maybe 100 yards at a pinch, and a rate of fire of maybe a couple of rounds per minute. Their weapons are less accurate, shorter ranged, much less reliable and much slower to fire. Any large formation of men marching towards them is going to be spotted days in advance by aerial reconnaissance, giving them ample time to be attacked with air support and artillery.
Consider how deadly crossing no-mans land in WW1 was. Now imagine that one side has modern weaponry, and the other side has muskets. All it would take is a couple of incredibly bloody battles for the peasants to learn that trying to fight is a death sentence, and the 1600s era nation now longer can field an army.
Most 1600s era rulers did not do a great deal for their peasantry- simply proving food and medical care to the locals would likely be more than enough to encourage them to accept the modern nation as their new rulers, especially when the alternative is to be forced in to a suicidal attack.
Edit: based on the update to the question that all citizens will fight to the death (which is rather unlikely), your only real solution is to wipe out the entire continent. The simplest approach is to destroy their food supply from afar. Invasive species being introduced to ecosystem, crop diseases would do the trick. Poisoning major rivers and contaminating the groundwater could cripple the population if the advanced nation was willing to do it. Introducing diseases in to the populace that they have no natural immunity to would also be very effective.
After a few years of famine and disease, you could probably cut the population down to a fraction of its initial size, and disrupt what industrial/military capacity it would have had. Then, steadily going from village to village and mopping up anyone left is really the only answer if every peasant insists on fighting to the death (assuming you can actually convince your military to conduct a lengthy campaign of genocide)