Let's go more with the scenario you are describing, than the question you start out by asking (which appears to be only peripherally related).
Let's assume that a "pebble" is approximately 1 cm3 in size.
Let's also assume that one of these adult men can carry somewhere on the order of 75 kg.
Let's also ignore how four men are able to simultaneously hold on to an object that is 1 cm3 in size, which is going to be a serious challenge, but not unsolvable, in itself.
With these assumptions, we can estimate the pebble's weight to be on the order of $4 \times 75~\text{kg} = 300~\text{kg}$.
As a consequence, the density of the material is something like $300 \times 10^6 = 3 \times 10^8$ kg/m3 (because $\frac{\text{cm}^3}{\text{m}^3} = 10^6$).
Apparently, osmium is the densest naturally occuring element on Earth, at 22.59 g/cm3 = $2.259 \times 10^4$ kg/m3. A pebble-sized ball of osmium would weigh a few tens of grams.
For back-of-the-envelope calculations, it's common to just look at the exponent. Your material is $10^{8-4} = 10^4$ times as dense as osmium. The actual result is that the material these men found is somewhere around 13,000 times as dense as osmium, but this figure could easily be anywhere from 10,000 to 15,000 times depending on how strong these men are.
For comparison, as pointed out by Chinu, a neutron star has a density on the order of $10^{17}$ kg/m3 (actually, several times that; Wikipedia states $3.7 \times 10^{17}$ to $5.9 \times 10^{17}$ kg/m3), which is another nine orders of magnitude (a billion times) more dense than the material you envision. A pebble-sized portion of a neutron star, assuming it stayed together (which it wouldn't, as IndigoFenix already pointed out), would weigh not 300 kg, but more like 300,000,000,000 ($3 \times 10^{11}$) kg.
In conclusion, basically and unfortunately, this question is another example of a lack of sense of scale in space.
I agree with AndyD273: It would probably be better, and less likely to risk loss of suspension of disbelief, to just have these men come up with a super-strong alloy instead. You don't even have to name or describe the parts to such an alloy unless you want to (but if you do, beware of falling into the same trap again by misestimating things).