It takes an enormous amount of energy to manufacture a computer, and less to create a ceramic food bowl, but only a small amount to grow a strawberry in Earth soil. Could we be setting product & service prices by measuring the entire amount of energy required to make or do something, including e.g. mining energy, purification energy, delivery energy, pollution clean-up energy etc? Can that be meaningfully computed? This is not quite the same as using the calories as a currency. This is using the energy input to set the base price of a thing.
What you're talking about is the idea of Embodied Energy. For the sake of ease, think of embodied energy as:
the energy it takes to get a product to a consumer; including the harvesting or gathering of raw materials, the refinement of these goods, transportation, and any manufacturing processes.
In a way, the cost of most goods are already linked to their embodied energy. After all, the cost of gathering materials, manufacturing a product, and shipping that product (both to the manufacturer and the consumer) is real, and this is basic economics.
Forcing governments to tax goods for embodied energy that the company does not pay for could be a way to mitigate climate or energy issues, but this is not trading energy directly!
You can't just bust out energy like you can coins. Energy is not a tangible thing. At the most, if there is a universal energy particle1, this energy-particle is more ethereal than dark matter, and at the least, energy is just an idea scientists use to keep track of how much work a thing can to. You cannot hand someone some unit of energy, but you can give them something which possesses energy.
Trading work, or the promise of work, for goods has happened. The Inca Empire hat mit'a, but they also used barter-systems. Other historic economies used a type of work-in-place-of-money system called "corvee."
The real problem with trading energy is the non-physicality of it: that energy must be carried by something. If I had to pay you 20,000 kcal for something, how do you want that energy? Do you want the promise of 20,000 kcal of future mechanical work? What if I gave you some amount of a chemical which, when burned, releases 20,000 kcal of heat energy? What if I turn on a laser that gives you 20,000 kcal in a very short time- essentially burning whatever the laser hits? Maybe I'll expose you to some nuclear radiation which will give you the 20,000 kcal!
Wouldn't this be easier if we just agreed that some thing- be it currency or another good- would be more useful to you?
Trading energy without some middleman- something that holds the energy- is very, very, veeeery impractical, potentially destructive, and also not helpful.
1. According to the standard model, there are particles which act as energy carriers which are sometimes described as energy itself. It appears these particles are not interchangeable- a photon does not do the things a gluon does. These particles are also described as having energy, so the issue of if these particles are energy or not is a matter of contention. Also, we're not sure if describing them as particles is a good idea, too. In any case, if these are particles, all of them can be in your hand, but not held by your hand!
Not for a very long time
Energy is not a good form of currency because it is everywhere and it is very easy to get more. Aluminum used to be more valuable than gold, now it's used as foil and thrown out by the ton. Its abundance has made it nearly worthless (compared to gold). In order for energy to have significant value, it would need to be very scarce, especially since a kilocalorie is not much energy at all.
Energy may be a currency if a society survives until the Heat Death of the universe. In the far far far future energy in the universe will become very scarce. The expanding universe will rob the universe of energy and what remains will be spread so far and thin as to be essentially nonexistent. In this time there will be no stars that shine, nor radioactive elements to derive energy from. The only source of energy will come from the Hawking Radiation from supermassive black holes. Their energy will be the most valuable thing left in the cold dark universe because without it nothing could occur and thuse its value preiceless.