The energy that powers magic exists outside of time. That's why it seems to be able to defy time - showing visions of the future, transporting objects instantaneously, creating light or heat or cold out of nowhere (that part of space was or will be much hotter or colder at some time - the heat just gets moved in time, thermodynamics are preserved).
The problem is that because it does not move temporally, once magical power is used up, it is no longer available at any time - even in the past, at the time it was used. It does not change through time, and is no longer present, and so it is as if it had never existed.
That makes it devilishly difficult to remember that a spell had even been cast. Thousands of years ago, it was a plentiful resource. Somehow, we remember that. Over the years, Earth's supply has been reduced to practically nothing by magic users, whose spells have been forgotten. Because the power behind them never existed, the results of the spells have become how things always were.
Perhaps there was a catastrophic magical event - the equivalent of the Atom Bomb - which used up all the supplies at once. Perhaps it wiped out an entire continent, which has now never been.
How would we discover this? Perhaps we land on another planet, only to find fresh stocks of magical energy. Perhaps an astrophysicist theorizes it must have existed in order to explain cosmic inflation or baryon asymmetry. How would we work it out? How, in the age of science, would we discover how to use it again? What would we do with it?