Make your universe highly anisotropic.
First, I should start with the disclaimer that general relativity is a very complicated subject, and while I have a cursory familiarity with it I am in no way an expert, so take this with a grain of salt. With that being said, the idea behind this is that in general relativity, the coordinates that someone measures depend on the path they take through space and time.
This is because in general relativity, spacetime is described by a manifold, which is a fancy way of saying that it seems normal and flat close up, but globally it can have some funky structure. The classic analogy is how the Earth looks flat when you're standing in west Texas, but if you walk long enough in one direction you'll end up where you started which is a decidedly not flat thing for geometry to do*. The difference here is that while the Earth is embedded in 3 spatial dimensions, spacetime isn't embedded in any higher dimensional, flat space. This may not seem like a problem at first, but it actually causes issues if you want to define coordinates that everyone can use.
To see why, say you were a two dimensional creature who lived on a spherical manifold like the Earth, only it wasn't embedded in a flat 3D space you could interact with. Like any good, scientifically minded 2D creature, one day you have the thought that life for everyone would go so much smoother if everyone on your world could have some common way of giving directions. So, you grab a stick, point it in a direction, and secure it firmly to the ground. You then declare this to be the standard direction everyone should use for giving directions from now on (let's call it 'weast').
Of course, a standardized direction like this is pretty useless if it's only in one spot, so you need a way for others to 'sync' their directional sticks with yours and then take them to new places across the world. The syncing part is pretty easy-- they just need to bring their stick next to yours and point it in the same direction. They need to be careful transporting it though. Clearly, if they travel back to their hometowns doing doughnuts in their 2D mustangs, the stick is gonna be completely disoriented by the time they arrive. Being the smart scientist you are, you come up with a solution: any time they're walking in a straight line, they should keep the stick so that it has a constant angle with respect to the direction they're walking, and any time they make a turn, they should rotate the stick in the opposite direction they turn by the same amount. The math for this is straightforward, it's just some 2D Euclidean geometry after all! Content, you lean back and wait for the thank you letters to start rolling in.
Much to your dismay, several days later agitated letters start pouring in from around the world. They complain that the different messengers they sent with sticks came back with them pointing in completely different directions. But how can this be?! Well, since we have the luxury of living in a 3D world, let's look at this from the viewpoint of a sphere imbedded in 3D space. Say both the messengers start on the equator and are headed to the north pole. If one of the messengers travels directly to the north pole along a meridian while the other goes a quarter of the way around the equator before heading up a meridian, when they get to their destination, their sticks will be pointing perpendicular to one another! Here's a picture to help illustrate in a less confusing way:
It seems our flat friend has made a fatal error-- he was doing math for a flat world, when he lived in a curved one. In fact, once space is curved it is impossible to come up with a way of translating a coordinate system in a way that doesn't depend on path-- this is the subject of parallel transport and connections in differential geometry. It may be tempting to say that there is a consistent way of transporting coordinates by considering the flat 3D space the sphere is sitting in, but remember that manifolds aren't required to be imbedded in higher dimensional space. Even if it were, the 2D aliens have no way to access this space to make measurements, so it's a moot point.
Finally, getting to the point: general relativity works in a very similar way, only it complicates things by getting time into the mix as well. In relativity, time is not an absolute coordinate. It can get mixed up with the spatial coordinates and the end result is that much like the 2D messengers, two astronauts carrying clocks can start with them synced and end up out of sync when they arrive at the same place. So how is it that cosmologists are always talking about the age of the universe? Well, they use something called the FLRW metric which describes how coordinates evolve in an homogenous, isotropic, expanding universe (basically all those words mean that the universe is modeled to be the same everywhere). If you look at the sphere picture, you might be able to work out that the amount the arrows are out of sync is positively correlated to the percentage of the sphere's surface area the two paths contain. What this means functionally is that if the curvature of spacetime is not that great, or our two astronauts don't diverge far in their paths, then the proper times they measure will not be significantly different.
Since the FLRW metric describes a homogenous universe, it is "easy" (at least by GR standards) to tease out a time coordinate that is useful for everyone as long as a few conditions like low comoving velocity are met. It just so happens that the assumptions of the FLRW metric are a good model of our universe, but this doesn't necessarily have to be the case. One could imagine a highly anisotropic universe with no CMB, many very dense collections of matter in some parts and huge voids in others, all moving at relativistic speeds to one another. In such a universe, giving a single useful age of the universe would be difficult because there would be so much path dependence and the conventions taken would be so abstract and unintuitive. Of course, this kind of universe might not be very conducive to life, but I'll let someone else sort that out as I've written enough already.
* Ok, technically this isn't true because tori exhibit the same behavior and have no curvature but you know what I mean