As a participant in the building industry I can vouch that clay is a very versatile building material.
You can create all sorts of structures - houses, towers, public buildings etc. You are just restricted on the form, as, being a mass material, you can only go straight up or tapered in. Clay works well in compression, but not in tension, so you can't do major cantilevers or large roofs. As an example:
The other issue with clay is roofing - it has no spanning capacity so you would need another material to span to support, for instance, clay tiles. If not possible, then you may need to investigate roofing methods using mass clay, for instance domes:
The material is ancient, and there are many uses in history. It has thermal properties as well - with high thermal mass but low insulative value. Thus it works well in arid dry desert environments, where there is a high diurnal range (frigid nights, hot days) as it takes time for the heat transfer to travel through the material, if done right it could be cool during the day and heat during the night.
In terms of the economy clay would spawn a whole industry of those that can manipulate it expertly. Although you say anyone can create intricacy and manipulate, that does not mean they are experts at it. They still need to be experts in their industry. I would imagine:
Artists - if you have no physical impediment to create intricate sculptures, one would imagine a whole new industry in exhibiting, exporting highly sculptural clay objects and statues.
Infrastructure - clay could be used for roads, pipes, water piping and sewerage - these networks will require expertise to design, locate and install them with sufficient drainage gradients. You would still need an engineering industry.
Grand Monuments - if it is plentiful, and no physical impediments, one would imaging massive towers of clay, much larger than seen on Earth in the past, if it requires little physical effort.
Efficiency - if it is rare, clay must be used sparingly, so an industry would be established to preserve it, reuse it, and optimise it (perhaps firing it to increase hardiness, mixing it with other plentiful materials to obtain other attributes).
Research - If it is the only worthy material, significant effort must be made to research the material to enable it to perform tasks we currently expect of others (such as glazing for windows, wood for carts or structure, metal for fixings, mineral wool for insulation). This would be a major priority for the economy.