As other answers suggest, the limit is a few atoms (if we're talking about actual LEDs), or potentially a single atom if we expand the definition to include light emitted by phosphorescence or fluorescence.
If you're talking about a large 2D matrix of these components, then you can't neglect the space needed for supporting machinery.
- LEDs need wires to supply them with current, and some means of switching the current to address individual pixels. These wires can potentially be as narrow as a single atom, and since each pixel has an area of multiple atoms (see below), I think it is reasonable to assume that you can fit whatever circuitry you need for each pixel behind the display.
- Fluorescent pixels need to be excited with (probably ultraviolet) light. This could come from behind the display, but that would mean that the fluorescent pixels have to sit on top of an LED display, so there's no space saving. Alternatively, you could have a plane of fluorescent atoms and illuminate them with an external source, like a laser. The problem then, though, is that if the pixels are smaller than the wavelength of the external light source, you will not be able to activate just one pixel at a time. This limits the pixel size to perhaps 350nm. It's also worth noting that when a fluorescent atom or molecule is excited, it will sometimes (often) return to the relaxed state without emitting a photon, so even if you could have single-atom fluorescent pixels, each one would only work some of the time.
- Phosphorescent atoms can be excited in other ways, like a chemical reaction or a free electron (as in a CRT). In theory the wavelength of an electron is much shorter than that of UV light, so you could illuminate a single-atom pixel with a very precise electron beam. But there are the same yield issues as with fluorescent pixels, plus phosphorescent emission works over longer time scales, so I don't think there is any advantage here.
- Quantum dots, for this purpose, function as giant fluorescent molecules. They are typically made of hundreds of atoms.
- For completeness, I'll rule out incandescent lights. A single-atom light bulb filament would need a lot of space to isolate it from neighboring pixels, and would almost certainly burn out in a tiny fraction of a microsecond.
So, I think the most plausible candidate is LEDs. At least, that doesn't have any obvious roadblocks. I can imagine the front layer of a very hand-waving RGB nano-display looking like this:
I am guessing that you'd need more than one semiconductor atom in the front layer in order for the light to be emitted at the surface, rather than inside the display where you wouldn't see it. And the LEDs need to be electrically isolated in order to operate independently. So, pretending that atoms are all the same size (which they aren't), an RGB pixel has an area of 24 "atoms".
The important thing here is that each sub-pixel can easily be smaller than the wavelength of light it emits. In the way we normally think about resolution, the resolution couldn't be any higher than this anyway.
However, at this scale, it becomes theoretically possible for the display to work as a phased array. If you can get the LED elements to emit light with a single, controlled phase (which seems plausible(?) since we're emitting single photons to order), then you can basically make the whole display work as a programmable, full-color true hologram, so you can display 3D objects just as if the display were a glass window. Of course this would require enormous bandwidth and processing power.
Heat would definitely be an issue – you're doing a lot of switching in a very small volume – but I don't know of any fundamental reason why such a display couldn't work if it were built on, say, a liquid-cooled copper plate. So I'd accept that if I read it in a book.