If they live on the same island as the dragons, then what if they are protected by the dragons? They would need to be able to defend themselves against day-to-day stuff like animal attacks but maybe the dragons view them as pets and protect them from outside invasion.
Regarding their skin, what if they are mammals that grow mutated hair (wider, flatter, shorter and MUCH harder) that acts as a shield/tough skin (think rhino horn or even dragon-scale). It could be a symptom of their diet too - perhaps they have evolved to consume a diet high in metals/minerals and the hard-hair-scale-skin-thing is how their bodies eject the metals/minerals to avoid toxicity from them.
If you wanted to get really crazy their "hair-plates" could be involuntarily activated from being somewhat soft (although stiff) and turn into a full-body shield when they feel frightened or aggressive (hardens and expands to cover all/most of their flesh). [This style of "change" happens somewhat in reptiles (such as bearded dragons and chameleons) - when some reptiles feel threatened or scared (or want to BE scary) they change colour]
Maybe their hair-plates ARE their weapons. If you allow them to involuntarily control the shield aspect of it, then adding on a "sword"/"blade" to that wouldn't be too big of a stretch. Perhaps the hair on their hands grows into giant-claw-like plates at the same time their shield plates active.
I'm picturing a creature a bit like a dwarf - short and squat, long life span, few children born, long gestation period (accounts for low population).
They could eat... rocks. Or how about some sort of native vegetation that has far better nutritional properties than actual rocks but also contains a lot of rock-components? Like a rock-carrot: grows in caves, looks like a greyish carrot, and tastes like a carrot that got sand embedded in it.