It's not the ships that are the problem, it's the cannons.
The weight and recoil of the cannnon being fired isn't actually that big of a problem, especially if you're looking at 'mortar' style weapons like the Meg where much of the force is pushing the ship down into the water as opposed to standard cannons where almost all of the force is being applied parallel to the ship's deck.
The HMS Queen for example carried 100 32-pounder and 6 68 pounder cannons with a total weight of around 170 tons just for the cannons. The Mons Meg weighed 3 1/2 tons as opposed to 2 1/2 tons for the 68-pounder and 1 1/2 tons for the much more common 32-pounder.
So, it wouldn't be a huge problem to mount ten or twelve or even twenty Meg-sized guns on a ship like that instead of the massive broadsides that they actually had.
The main problem you're facing is that Age of Sail metallurgy just didn't allow for cannons to be reliable at those sizes, which is why the Mons Meg burst its barrel. 19th century steelmaking just doesn't have any way to reliably contain the kind of pressures involved once you start getting above the ~25lb charges that the 68 pounder used.
The other main problem you're going to have is just hitting the target. Your ships made of magically hardened wood are going to be stronger AND FASTER than the actual age of sail ships were because they can be lighter, more bouyant, and carry more sail without risking the masts. They'd probably be larger too, because you get efficiencies with size of sailing ships. Take a fast frigate and make it 20% larger and it's even faster, as long as you're not running up against the physical limits of what you're building it out of.
What that means is that not only are these ships going to be harder to damage, they're also going to be harder to HIT, which means that even if you could build reliable weapons like the Mons Meg, you'd have a really hard time landing the shots.