The building materials you mentioned are all sedimentary (rocks either eroded from volcanic rock and then reformed by action of water) or metamorphic rocks (sedimentary rocks subsequently modified by exposure to subsequent high temperature and/or pressure).
Meanwhile the minerals you mentioned (especially gold, diamonds, platinum, etc.) are typically (but not exclusively) found in certain kinds of igneous rocks.
So the general case is:
However, gold especially has occasionally been found in large quantities within sedimentary rocks (mostly sandstones) and/or sandbars of rivers draining mountainous regions. These rivers contain detritus from the erosion of quartz veins. Gold solidifies around the same temperature and pressure as these quartz veins and, therefore, is often found embedded in quartz veins. It would tend to collect in sandbars of the rivers containing these quartz particles. This is what happened at Sutter's Mill in California.
Since gold (and other noble/precious metals) can be found in sand bars, it isn't a very big stretch to imagine we could possibly find large quantities of gold in certain sandstone deposits. These placer deposits can contain nuggets of platinum group metals and gemstones too.
The concentrations will not be very high compared to what we normally think of as high concentrations but they will be high in comparison to the normal concentration of such materials on the surface of the Earth.
So, in special circumstances, the answer can be