Yes, we have some
There are jellyfish, hydra and other relatively simple creatures which have evolved immortality, so the basic answer is yes they can evolve.
But it's not the best evolutionary strategy
Evolution is based on selection, and selection generally requires death. Creatures that do not automatically die are going to be prone to overpopulation problems every time there is a rise and then fall in the amount of food available, and the older an organism is, the more likely it is to meet a pathogen to which it is vulnerable.
So it makes sense for this to work for simple creatures, for whom death is most likely to happen at the hands of the environment or predation, rather than needing their own capacity to survive indefinitely. But for more complex creatures like us, the chances are that a mortal species would out-evolve an immortal one, making it a weakness rather than a strength.
Longevity as a benefit
Having said that, we know that it is preferable to have a long natural lifespan in certain circumstances; when there are few natural predators or the creature has good defenses (tortoise), when there can be long periods without any available food (scorpion, many insects, desert animals). This is because only those species which can make it through a long drought have previously survived these events, so they are necessarily the only ones which are still around.
The Defensive Individual
It is conceivable that a species living on a planet or large moon with a specifically periodic toxicity (e.g. a periodic tidal event between moons, or orbiting a binary system with occasional eclipses, or even just very infrequent floods as we have in the desert) could develop both a lack of mortality (per Earth species) and also a periodic lifecycle, in which the main mode is pure survival with no opportunity to reproduce, and occasionally a period of plenty.
Perhaps a good model for this is the scorpion; it lives in deserts that are inhospitable, is well protected and capable of withstanding even nuclear fallout. It still occasionally reproduces, presumably when there is enough food around for the gestation.
We also see the phenomenon of an apparently bare desert exploding into life after the rains, capturing water and germinating as rapidly as possible before the water disappears. So the life of a scorpion-like creature would be mostly a defensive and slow-moving one, seeking preservation over all else, with brief periods of plenty when everything changes and reproduction is possible.
If you want to include an adaptation phase, it could be conceived that a highly effective immune system which defends the creatures during droughts from pathogens (perhaps it is windy and pathogens spread easily?) is then able to share antibodies during the social/reproductive phase via bodily fluid transfer. This would be like a mother provides protection to her unborn child; these creatures could swap pellets of 'blood' or other fluid, thus providing protection within the species for the bearer of your children.
No man is an island, but a man-o-war is a floating ecosystem
No animal can survive entirely alone; it always has a chance of dying so reproducing is the only sensible protection against being in the wrong place at the wrong time.
But what about a colony-type organism, like a man-o-war jellyfish? It is a system of millions of organisms, providing different things. Collectively, I can envision something where individual organisms live and die and evolve, and the strongest get to stay on the mother body; this would be adaptable but without partner, and survival would be everything.
I can't imagine that any organism would be entirely alone, but if reproduction events were very rare, and this gestalt existence was the norm, it would very much feel like the organism was alone. If the life cycle involved the sentient part dying before meeting any progeny, then each consciousness would believe it was a god; born from nothing, living alone, and being unaware of death.
Until it meets some oracle which its ancestors left for it to learn the long and lonely story of their history.
