You could start by relating the "magic" actual physical properties.
Gems
Gems are very hard, most gems are harder than steel by a wide margin. Assume magic causes "erosion" to the gem used to store it proportional to the magnitude of the desired effect. At that point the Mohs hardness of the material can be used to determine what magic it can hold.
You can add the limitation that the process of enchantment is light based so that the storage medium must be transparent to some degree to light of color related to the type of magic. Or alternately to colors other than that related to magic type.
You can make the "erosion" effect also specific to the process of enchantment, so that you can use any solid material to store any spell, but your chances of succeeding are proportional to the hardness of the material. Enchantment is generally assumed to be a time consuming process requiring specialized skills. It might be cost effective to pay more on materials to reduce the chance of failure.
Metals
If gems store the magic, the metal should shape it when released. Precious metals have some notable properties: conductivity, malleability, and corrosion resistance. They can efficiently conduct the magical energies without huge amounts of energy lost to potentially dangerous side effects. They can be shaped to the precise shapes needed for the runes. And the relase of magical energy does not cause them to corrode and lose magical potency as the runes get corroded from the surface.
With these proviso the optimal material would be gold plated silver or copper alloy. This would depend on the properties of the magical energy, silver has higher electric conductivity, copper thermal. (IIRC, not really important). Stone and organic materials would still be usable for lower "magic density" effects like permanent charms. Base metals could have corrosion resistan coating.
Again I think it is best to link this to the process of enchantment. If labor of shaping the runes is expensive, and it would, using a malleable, easy to work, precious metal makes sense. And a high conductivity metal might allow performing enchantment faster. And the corrosion resistance might drastically reduce the chance of failure and having to start the process from the beginning.
Overall
I guess the general idea is that if constructing magic items is time comsuming and requires specialized and expensive labor then using expensive materials that make work easier does make sense. And precious stones and metals actually do have good properties for this scenario.
Additionally, if magic items are expensive to start with, they would be commissioned by wealthy clients, who fully expect their expensive new toys to also look good with their expensive imported silks.
Also, related to ease of working the material, the tools fo working precious metals are much more enchanter friendly than a forge is.