There's horses and horses and horses and horses (and horses and even more horses). What ones are your centaurs based off of?
If someone is talking about how fragile a horse's legs are and how easily they get broken, they're implicitly talking about a modern breed of racehorse.
Racehorses are bred specifically to accelerate and gallop quickly. The biggest limitation on how fast a racehorse can run is how fast it can move its legs back and forth, which in turn is determined by the muscle mass vs "dead weight" of bones. If you want to breed a faster horse, you're necessarily breeding one with a lower bone-mass-to-weight ratio. But why would your centaurs have the bodies of a modern thoroughbred, when horses didn't start having those bodies until the 1700s? A centaur that couldn't gallop over slightly rough terrain seems unlikely.
The horse in the video Ash linked is a modern draft horse.
Draft horses are bred to slowly pull a lot of weight, the opposite of racehorses. As you can see from the video, they have massive legs and lots of muscle. Draft horses will happily pull a ton or two across the dirt on a sled, and will not break their legs from such normal low-impact exertion as a tug of war. But again, why would a "naturally"-occurring centaur have the body of a draft horse? Draft horses are tools, and centaurs aren't breeding themselves to pull things for humans.
The progenitors of modern draft horses were late-medieval war horses
A good one to look at is the Andalusian, but a lot of medieval war horses were in the same vein. War horses were trained to jump into the air and kick both forward and backward simultaneously. While wearing mail barding and carrying an adult man who's wearing plate armor. The people they were kicking probably also had steel plate armor. Suffice to say, their limbs were durable, and they probably wouldn't break them by accident in a tug of war, even against another horse. If your centaurs are militaristic, they might have developed a warhorse body. In this case, they'd be plenty muscular and durable for athletic pursuits, and a 500kg centaur would handily beat 500kg of humans in tug of war. But the ancient Greeks who came up with the centaur never saw anything as large as an Andalusian, so it's probably not what they were envisioning.
Some people believe that the centaur myths were created from encounters with early cavalry
In this case, the horses involved would be large enough to be ridden (obviously), but not necessarily by a man in heavy armor. In many depictions, the early riding horses were small enough that the rider's feet hung roughly at the horse's hock. Still, they're depicted as being reasonably well built, and the whole point of cavalry was that they could charge across terrain too rough for chariots. They're probably not going to break bones, but they also wouldn't be able to effortlessly pull ten men off their feet. They'd likely be better than an equal weight of men, but not by too much (and they'd weigh closer to 350kg than to 500kg).
But when the centaur myths started, most Greeks hearing them would have been thinking chariot horses.
For the first half of their history, domestic horses were too small to carry a rider (at least, at a gallop). Early horses were bred for food and for pulling chariots. While this is still "pulling things," chariots were only used on flat, even terrain. There's a huge difference between pulling a wheeled object on a dirt road and pulling a resisting competitor who's digging into the dirt. Depictions of early chariots suggest the horses involved were about as tall as men (MUCH smaller than modern breeds, and were lean with thinish legs. So it's questionable whether a 300kg centaur based on such horses could win against 4 adult men. It might depend on how well the men worked as a team.
But what about wild horses?
In my world, centaurs were created before the domestication of horses, and resemble wild horses with elven upper bodies. Wild horses aren't even as tall as humans, and they're also more stocky in the body (in comparison to their size). They wouldn't get broken bones easily, but they probably wouldn't excel at pulling things. A few humans would probably be able to compete against one centaur.