Trying to construct a material that could survive the Big Bang has many problems. For one, tracing the history of our universe to very early times leads extreme densities, temperature, and energetic particles. The forces themselves are predicted to unify at early times, changing properties as symmetries are restored. At some point the equations themselves break down as the Standard Model and General Relativity are unable to be used in the regime of the extremely early universe.
Rotating Black Hole
Any such solution would be extremely speculative, but the closest I can think of would be for your civilization to attempt to avoid the Big Bang by using a Kerr (rotating) black hole as a 'wormhole'. The theoretical underpinning for this solution is obtained using a coordinate representation of the rotating black hole, that extends into the interior of the event horizon (the Kerr solution breaks down at the event horizon). This new coordinate system shows another interesting surface known as a Cauchy surface. A person diving into the black hole's event horizon (assuming they survive the other negative effects of the strong gravitational field) would continue to fall inwards until they passed the Cauchy horizon. Shockingly, they would then be able to move freely, and, due to a reflection symmetry in this interior region, could exit back through the Cauchy horizon, out through the event horizon and out of the black hole. This new exterior region is sometimes described as an alternate universe. In the case of your story, the civilization would make their rotating black hole, send their container through, and pray that it would survive the journey and exit into an alternate universe that would be safe.
This process is entirely theoretical. Perfect Kerr black holes would be extremely unlikely to form, and the interior solution is itself unstable (meaning your presence falling in might cause the Cauchy horizon to disappear, or change properties). However, if you are determined to have a civilization survive, you are free to hand-wave the details.