The trip hammer was historically used for this
First, this process is called comminution. In the most ancient of days, to break up an ore into small chunks to work it in a forge, you simply hit it with a rock hammer. Eventually, the process improved until some sort of mechanical power could be applied: enter the trip hammer.
A water-powered trip hammer is essentially a very large hammer, mounted on a fulcrum, with one end (the helve) moved by a cam. The cam is in turn driven by a water wheel. Alternative, you could have a donkey or ox provide the motive force, though there isn't a lot of evidence that this happened. There is significant mechanical loss in older mechanical systems, lubricated with animal fat and made of wood. A donkey powered trip hammer probably provided no mechanical advantage to a slave with a sledgehammer.
The trip hammer was used for exactly the case you describe, pulverizing rocks to find gravel. The gravel could then be reduced in a furnace to extract valuable materials. This is historically how copper, tin, silver, and gold (along with rarer metals like antimony) were extracted and refined from ore.
As far as hardness goes; don't even worry about it. You can break up rocks using other rocks. Find a polish the hardest rock you can get, and use it as your hammer head. When it breaks, replace. In Roman times in the 4th century AD in Britain, there is an find of large iron hammer heads used for pounding ore.
The water-powered trip hammer is certainly in use in Han China by the first century BC. It is potentially in use in China several hundred years earlier, and there is sporadic evidence that it was used in the Roman empire from the 1st to 4th century AD. It returned to Europe in the 12th century where it achieved its widest use; and the mechanical principles behind its operation were crucial to the Industrial Revolution.
An alternative, used often in the Islamic world with limited access to waterpower, was the stamp mill, though this was not attested until the 10th century in Central Asia. However, creating a stamp mill is feasible with a Roman/Classical-era technology level.