DISCLAIMER: I am an (intensely interested) layman, not a scientist. Please take my summation of the following concepts accordingly.
I'm currently trying to worldbuild a plausible alien species down to the level of basic biochemistry, and at the moment I'm looking for alternative systems to RNA/DNA that can pass on genetic information. I'm not talking about stuff that essentially just uses the central principle and structure of DNA with some of the specifics changed, like XNAs which just use a different sugar backbone, or Hachimoji DNA which just adds on extra base pairs, I'm talking about plausible hypothesised systems that are radically different to what we use now.
One of the more out-there proposals I've come across is Graham Cairns-Smith's clay hypothesis, which posits that clay crystals were the first genetic material. The idea here is that crystal growth is a form of self-replication that "reproduces" its arrangement, and can even transmit defects. The pattern is then "passed along" when the crystal breaks (scission) and continues to grow independently from the original crystal. Eventually, a "genetic takeover" of sorts happens, where clay crystals that trap certain forms of molecules to their surface improve their replication and catalyse the formation of increasingly complex proto-organic molecules that eventually take over the original genetic substrate as the new genetic material.
Schulman, Yurke and Winfree in their paper "Robust self-replication of combinatorial information via crystal growth and scission" use the same principles to create a set of DNA "tiles" that replicate its sequence of tiles through crystal growth. Each tile has "sticky ends" that hybridise with each other, and under appropriate growth conditions, complementary sticky ends hybridise, while non-complementary sticky ends are unlikely to interact. The interaction of these sticky ends allows for accurate sequence replication during growth, and once crystal growth has propagated the sequence, these additional layers are then "cleaved" off through mechanical scission.
Another early Earth candidate for genetic material, advanced by C. P. J. Maury, is amyloid structures - a "β-sheet peptide-based prion-like amyloid system". It's posited that under the harsh prebiotic conditions of early Earth, beta-sheets would have been a plausible candidate for genetic material, since it is resistant to both UV and ionising radiation, and to high temperatures. Amyloids, in similar fashion to prions, can transfer conformational information, the changed three-dimensional architecture, to daughter molecules. (Would this essentially make early evolution Lamarckian?) In a paper, Maury, Liljeström and Zhao demonstrate the plausibility of the amyloid world hypothesis by creating a peptide called "EGGSVVAAD" that spontaneously forms amyloid fibrils.
However, there are only a handful of novel ideas surrounding this I am aware of, and I'm not entirely sure if and how any of these systems of replication could achieve a significant level of biological complexity (save for something like a "genetic takeover" that effectively replaces the original system), because for complexity the information needs to not only self-replicate but also be interpreted as instructions for building things. I'm not aware of any paper for now that confidently advances a novel system explaining how an alternative replicator mechanism can be translated in this manner. The way DNA/RNA is translated is a fairly convoluted multi-step process and building such a system for any hypothetical replicator is probably very difficult. Most of the papers I come across are at the very basic level of "how can a sequence of information robustly self-reproduce and transmit its characteristics in a way that selection can operate on it", that additional layer of complexity surrounding translation is unfortunately not touched on (either because it's not part of their intention to create a general purpose replicator, or because they can't propose one).
