We're already looking to the cask's big brother to store hydrogen today.
Salt caverns are artificial cavities which are created in geological salt deposits. Future caverns are generally located at a depth of 500 to 1,500 metres. They are three times higher than the Arc de Triomphe. To create such a cavern, it is first necessary to drill into the salt. The second stage consists in injecting water into the salt to dissolve it. The resulting brine (water mixed with salt) is extracted and leaves room for a large, tight cavern where hydrogen can be stored under pressure. (Source)
It all started when humanity needed a place to store vast amounts of excess natural gas.
The first use of a solution-mined salt cavern for storing natural gas was in 1961 when Southeastern Michigan Gas Co. leased, from the Morton Salt Co., and converted an abandoned salt cavern formed by routine brine production. ...
The first such cavern created specifically for the storage of natural gas was constructed by the Saskatchewan Power Corp. and became operational in 1961 The 290,000-bbl cavern, located in Melville, Sask., was constructed in the Prairie Evaporite salt formation at a depth of approximately 3,700 ft.
In the U.S., the first solution-mined salt cavern constructed specifically for the storage of natural gas was completed in the Eminence Salt Dome in Covington County, Miss., in 1970 by Transcontinental Gas Pipe Line Corp. (Source)
But humanity's reason for not doing it until 1961 had nothing to do with the technology needed to form the caverns. Humans have been mining salt underground for 7,000 years.
The history that salt mining in Hallstatt looks back on, is absolutely unique worldwide. During the Neolithic Age – an almost incomprehensible 7,000 years ago – people were already busy here mining for salt. That makes Hallstatt the oldest salt mine in the world! (Source)
In other words, we didn't put two and two together until we had an excess of gas we needed to store. After we figured that out, the leap to hydrogen was logical. But your people have a reason to deal with the need to store gas at a much earlier time! Luckily, the tech for creating the cavern is literally neolithic.
The problem is the tech sealing the entrance.
But I think there's a logical jump to be made here, too, because your earliest alchemists — not knowing yet what the consequences of their efforts could be — lost a few limbs before realizing they needed to contain the effluvia of their efforts (and they were tired if sounding like chipmunks, anyway). But like all good innovators, they weren't trying to move mountains first. They were trying to do small things requiring small capture.
Thus was born the salt cask. A bottle or box a quarter to a half cubic meter in size for which someone discovered that a concoction of (oh, yeah!) tree sap, frog guts, shredded hemp, and a pinch of fine silica could seal in place the openings with small metal disks and simple valves.
Once you have it working for a cask, the move to a cave is just a matter of logistics! Well... and bricks. Lots of bricks.