If you cool a rock to exactly one degree below freezing, then drop it in a nice beverage, the brick will cool the beverage... a little. But not very much, in the grand scene of things. However, if you drop a simple ice cube into the beverage, it will lower the temperature considerably! Same mass of object, but a lot more energy. The reason is the state change - the brick stays a brick, but the ice cube melts, and it takes a lot of energy to go from solid to liquid (and from liquid to gas), and vice versa. Incidentally, this is why those drink-rocks just don't work as well as regular ice.
Granted, that's just water. It can hold a significant amount of energy, but you're not going to get it as small as an urn and still be able to do anything but act as a personal cooling device for a few hours, at best. What you really want, therefore, it a new supermaterial. Something that takes a lot of energy to go from solid to liquid. Then, all you need to do is find a way to get the energy into the substance (to make it warmer), or let the energy out (to make it cooler).
The awesome part is that the temperature is exact. Not "vaguely room temperature", but a specific temperature with 0% fluctuation.
Ok, so I lied. There's a little bit more to controlling temperature than the material. However, this works to our advantage! You see, if you increase pressure, water no longer freezes at the same temperature. Increase it enough, and water starts freezing at room temperature (or hotter). Decrease it enough, and water boils at room temperature (or colder).
Why is this to our advantage? Simple! Your supermaterial is at the solid/liquid state change at room temperature and atmospheric pressure. But, by increasing or decreasing the pressure, you can easily change the melting/freezing point by a few degrees up or down. Which means your supermaterial can be regulated at the twist of a dial! As you crank it one way, it increases pressure, the other way, decreases. Turn it up or down to regulate how much energy is released.
As energy-dense as this material is, it will take significant energy to recharge - exactly the energy it lets out, actually. However, with the right material and a good source of energy, a chunk of material could heat an insulated house for a full day, or longer! And since it isn't wasting energy trying to make a house colder than it should be (like a giant block of ice), or warmer than it should be (like a vat of molten salt), it will regulate much more easily.