Let's consider what issues we'll need to overcome if we want the singularity to happen.
1. We need to develop a general-purpose AI.
Existing AIs are trained to solve specific problems, and would be entirely useless outside of those domains. Most current AI research isn't even trying to do this. Rather, we develop models that can effectively make predictions/decisions for the task at hand, and train it with data specific to the task.
Developing models makes heavy use of domain-knowledge; even if you have a fixed task and a fixed set of features, the quality of a model can be massively affected by changing the representation of those features. The structure of the model itself will also depend on what sort of relation we want to model: e.g. does the prediction depend on only the most recently observed features, or does it care about which ones were observed previously? Are there cases where the prediction should depend on global tendencies in the input? Cases where the input has an implicit internal structure that must be modeled? Cases where some features in the sequence matter more than others? In my own area of research (natural language processing), the answer to all of those is yes, and failing to take those into account will make for a worse model.
You could train the best model in the world for a given task, change the input representation slightly, and the model would be completely worthless. It would eventually become useful again if you continued training on the new format, but it would basically be retraining from scratch (and from a poor starting position, too; it would need to unlearn what it had learned before it could make any real progress). If you tried to take a model from one domain and use it in another domain entirely, e.g. taking a machine translation system and trying to use it to control a self-driving car, you would definitely need to retrain from scratch to get any results, and the structure of the model would be all wrong, so you'd probably get terrible results even then.
2. It needs to be able to design other AIs
This may seem like it follows from the first part, but it doesn't.
Even if it possesses generalizable problem solving abilities, that doesn't mean that it can do everything. Case in point, humans are general-purpose intelligences, and most of them don't know how to design AIs.
Being self-improving isn't enough. All AIs are self improving; that's what training is all about. If you give any model new training data, and let it train for longer (and avoid overfitting and various other issues that I'm glossing over for the sake of brevity), any AI will improve. Eventually.
But for every model, there's a theoretical limit to how well it can model the process we're interested in. If we want the kind of unbounded exponential growth that the singularity people talk about, our AI needs to be able to design new models, not just tune the existing ones. Which means it needs domain expertise on designing AIs. The good news is, the people designing the AIs possess that knowledge pretty much by definition. The bad news is, that doesn't mean we can explain it well enough to program that knowledge into an AI: there are a lot of things where we just develop an intuition for what sort of things work, through experience. There are plenty of things we do understand well enough to explain, but the frontiers of research are always something of a black art.
Most of these issues are practical, rather than theoretical, and given enough time, data, and hardware, you could probably have a general-purpose AI figure out the domain knowledge on its own.
3. The improvement must be unbounded by physical limitations (for awhile, at least)
Making an AI that's twice as powerful as the previous one doesn't help if it uses so many resources that it's running at half the speed. AIs are resource-intensive; even the single-domain ones we have now can make full use of just about any hardware we throw at them, up to and including supercomputers. You can certainly make a lot of progress by designing better, more efficient models that run on the current hardware, but eventually you'll need to stop and wait for better machines to be designed and built. And even then, we eventually run up against physical limits: information is limited by light speed delay, and component density is limited by the Schwarzschild radius of the processor, if nothing else (presumably other hard limits would kick in earlier; consult your local physicist for details). Maybe the AIs get good enough before we run into any fundamental limits, maybe not. But the fact that we need to stop and build physical machines at any step of the process means we don't get to stay on the exponential improvement curve; the best case scenario is that the time to develop a new AI goes to 0 and the construction time becomes the dominant factor.
So in conclusion, developing general intelligences in the first place is hard, making them capable of unbounded self improvement is even harder, and even if we manage both of those things, the improvement cannot stay exponential indefinitely. AIs of all varieties will have improved massively by the 2200s, and your world building should take that into account. But the Singularity isn't science; it's prophecy. Instead of bothering to explaining why the prophecy didn't come true, instead extrapolate some of the things that AI could do by that point, based on the progress we've seen in the last 30 years, and show that. The reader will hopefully be too busy exploring all the cool new things in your believable future to worry about the magical elements that aren't there.