Given current technology, could TV display screens double as video camera sensors?

Question

This was inspired by this YouTube video titled "Why all solar panels are secretly LEDs (and all LEDs are secretly solar panels)" and the fact that speakers and microphones are similarly related.

So thinking about a contemporary-technology dystopian surveillance society where LED TVs are built so that the behavior of

• the display can be switched between display mode and camera mode, and
• the speakers can be switched between speaker mode and microphone mode

The smart TVs would do this switching rapidly from the OS running on the TVs while the television is "in use".

In camera mode the TV screen would function as a (infra red?) camera sensor and record what it sees in the room while the speakers record the audio. All the data would be sent somewhere for further processing.

To what degree would this be feasible given current technology, and considering also aspects such as that the flat TV screen would lack a lens and possible aperture control?

Answer 1

Yes, with interferometry based synthetic aperture imagery. But probably not in your lifetime, or the lifetimes of anyone that your great grandchildren would know.

It would take several orders of magnitude improvement in digital signal processing technology for it to work at the scale of a video screen. The sensors will need to record at a very high data rate, on the order of petahertz.

Every single pixel will be receiving all of the light that is in its field of view, and none of the light will be focused. Fortunately, light isn't just photons, it's also electromagnetic waves. The sensors will behave like antennas, rather than eyes, passively collecting the radio waves at the wavelength of visible light.

Radio signal sources can be tracked quite easily, by measuring the timing of a wave reaching multiple antennas. The same applies to all sorts of waves; an earthquake's epicenter is calculated by measuring the timing when the waves arrive at seismometers.

We even have the same types of sensors built into our own human bodies. : Our ears can taking a wave and determine the direction that it came from, based on changes of intensity and timing.

We have already proven the concept using vast antenna arrays to collect unfocused photons, to create a focused image. We imaged the supermassive black hole M87* using the Event Horizon Telescope.

Of course, when astronomers use the term "photon" they don't just mean visible light; they mean any coherent electromagnetic wave. This image represents the peak of current engineering feasibility for synthetic aperture imagery. The EHT uses a Very Long Baseline Interferometry, which works in 450 GHz, using very narrowly calibrated equipment designed specifically to tease out the glow of the accretion disk at the wavelengths to detect event horizons around black holes.

In order to get meaningful data, though, your sample rate needs to be at least twice the frequency of the signal rate, preferably more than 4 times the signal rate, or you start getting downsampling errors called aliasing.

In order to record visible light, which has frequencies between 405 THz to 790 THz, you will need a sample rate that is at least 1.58 PHz.

Due to limitations from the speed of light, and the time that it takes electrons to pass through silicon and copper in computers, this is just past the fastest speed that we can record data meaningfully. We would have to pass the data from several sensors into our storage in order to build up a meaningful synthetic aperture image from interferometry. This needs specialized recording technology that we just don't have yet.

And, there's also the problem that LEDs aren't designed to collect light, even though they're capable of doing it... just as sound coming out of a microphone would sound terrible, and sound recorded from speakers is also low quality.

It would take several generations of iterating on the current science in order to use an LED-based computer monitor to record what's happening it a room, and it will always require specialized systems to just record the data in a meaningful way, much less process it into an image. It took several petabytes of data and 3 years of processing in order to build the EHT's image of M87*. It was faster to hand-carry the hard drives from the telescopes around the world to the datacenter, than it would have been to send the data over high speed internet links.

It would take a lot of iterative work to miniaturize the chips necessary to do the calculations, but the technology just barely exists. Such a screen would be prohibitively expensive, as it's much easier to just put a lense in front of a cluster of photodiodes (i.e., a webcam) and hide that in the corner of the screen, but those leave physical evidence... you can see the lense if you take the screen apart.

It could be possible, with non-digital interferometry, to construct such an image of the room in real time... but that equipment barely fits in the basements of large telescope observatories, and requires cryogenic cooling. You wouldn't be able to collect it clandestinely, and would be much cheaper (and higher quality) to use a webcam.

Answer 2

No

Screens these days are sheets of LED lights so that might work as a solar panel but not as a camera. At best you might measure ambient light levels perhaps but no camera.

That said TV these days are coming out with built in cameras and microphones and will get worse as time goes by. Things like Google Assistant and Alexa will come built into a lot of household devices.

Answer 3

Even IF - Why would you want to take a photo with zero (or near few mm) field of vision and a lot of blur?
Because TV screen would (could) act like a photosensitive paper (or if you are old enough camera film). There would be no lens to focus, no apertures to set the amount of light, no pinholes that would allow you to point to certain area.
Screen would take light from everywhere that is not behind the TV. And what would be behind the TV would (could) still have ambient light. So a photo taken with such screen would be very bright (white) on border going to greyish in the middle.

Just put a normal camera in the TV. Speakers can work as microphones anyway.

Answer 4

Yes! ...But not a very good one.

Cameras have some combination of mirrors and lenses to focus the incoming light into something recognizable, and without that, you're not going to see much.

An LED will return the light level it can see; unfortunately, the light level an LED on the bottom left corner and the light level of an LED on the top right corner are going to be mostly the same, unless you are pressing your face against the screen. A television on the wall across the room won't be able to see anything useful.

However, a simple fix can improve the view considerably! Modern televisions have the LEDs right at the screen, or behind little bubbles that spread the light out, so as to produce the maximum view-able angle. But, there exist privacy screens that essentially limit the view angle to a point directly in front of the screen, at a set distance. These screens work by blocking light from escaping in any direction except a very small angle. Beyond that, the screen will look black, because the light has been blocked. And lucky(?) for us, that effect works in reverse! The LEDs will only see the light in a narrow band, instead of the whole room at once; the image will still be fairly blurry, but as long as someone is sitting in the "sweet spot" to view the monitor, the monitor will be able to see them, too!

Even without that, your monitor will need a lot of custom hardware to read the pixels; it's a lot easier and cheaper just to squeeze a tiny, fully functioning camera into the TV's frame - say, right next to the infrared sensor?

Answer 5

Yes, Samsung produced the SUR40 Surface Table for Microsoft.

The panel contained imaging sensors used to produce a picture of anything placed on the table.

Note that without a lens of any sort the range of images it can take is very limited. Also it suffers terribly in certain lighting condition especially involving overhead flourescent lights.

This was from 2011

https://www.engadget.com/2011/01/06/microsoft-and-samsung-unveil-sur40-the-surface-2-0-experience/

Answer 6

A camera needs a lens to work. Screen has no lens in front of it so cannot be used as a camera. This is a single, important thing. The rest would not matter that much. A proper electronics can pick external brigthness reading from the LED no problem.

If to put a tiny lens in front of each LED, it may work as a camera at the focus distance of this lens, given enough LEDs to compose a picture ("insect eye"). Some discrete ("individual") LEDs may have lenses in front of the light emitting diode, made on purpose or happening by chance. Hence low resolution banner has more potential for this kind of application. I do not think that IC leds have anything similar.

An interesting thing for the story, a water droplet on the screen may act as a lens, suddenly allowing the part of screen under the droplet to "see". I am not sure about the focus distance however.

Answer 7

Given enough time, a very low resolution image could be recovered by changing the displayed image such that the liquid crystal layer allows light in through only a single pixel (or a small group of pixels). This would correspond to a black screen with a small white spot. By moving this white spot around the black screen and measuring the voltage at the backlight LEDs, some amount of spacial detail could be recovered (depending on the physical dimensions of the screen and the depth of the liquid crystal layer). The size and thickness of the spot will cause light rays to be rejected by angle based on the spot position.

The minimum transmission area will be limited by the total amount of light transmitted by the black pixels, so your resolution will effectively be limited by the screen's contrast ratio as the white spot must be large enough to transmit more light than the black pixels over most of the screen.

This would only be suitable for taking still images due to the time required to capture one frame, but the necessary mask frames could be hidden between frames during the display of a normal video on the screen.

Answer 8

Somewhat.

What is marketed as "LED TV" is LCD with a LED backlight. There are very few elements and they're placed behind (or to the sides of) a diffuser, so they won't be able to sense anything more than the room's general illumination level.

Some high-end LCD TVs do have a full-array backlight with enough elements to potentially make out a bit more information than just general illumination. These are generally even more rare.

OLED TVs have individual pixel elements, and they already have some means for recording the aging of individual elements as it occurs. Still, any image you get would have very little fidelity due to the lack of any means of focusing the light.

You wouldn't be getting a video. But, if every current from the backlight was recorded, it's probably enough data to forensically distinguish between the patterns of incoming light when someone is in the room watching the TV, and when they're not. After all, isn't that something the dystopian society would care about?

Of course, current TVs could already be recording you with good resolution by simply replacing the IR sensor, used for the remote, with a pinhole IR camera. It would keep working as an IR sensor, so no one would know the difference unless they disassembled the TV and closely inspected the components, specifically looking for a surveillance device.

Answer 9

Actually two questions here:

First of the microphone:

Yes you can, but as an electrical engineer this needs a lot more electronic in the background to accomplish that, in a dystopian setting this can also be hidden in the chips so that it is not obvious when you have a look at the PCB.

Second the screen:

Given current level of Signal processing: No. Explained here in detail.

Give it a few years to a group of signal processing experts and machine learning experts and you may be able to detect how many people are in front of the TV, but do not forget the polarization so you would have a rather small angle of view. It would be easier to use a micro camera that is installed and officially used for video calls on your TV or a hidden camera underneath a one way see through material.

Answer 10

Making this work technically would be a Manhattan project; there's no reason to make such extreme effort when there's a simpler way to get it done.

Selfie-grade cameras like the one on the face of your phone cost less than a dollar in million quantity. Microphones are even cheaper.

Simply incubate a social movement that creates some sort of reason some people might want one on their TV. Encourage every TV maker to put it on every TV, all in the name of populism.

We pretty much already have this with several social media habits, like cell phones tracking your location. For instance Google Maps tracks your location even in the background, nominally to monitor traffic flow and present local attractions, but also usable by police. Those are things you can turn off if you apply both technical skill and diligence, however you need to apply both pretty liberally, and when you do, and the police have reason to notice, it makes you look guilty and prejudices juries.

Eventually it becomes a thing that is on by default, like Google Maps' tracking; you must go to technical extremes to turn it off, and very few people bother. Those who do, the fact that they do is interesting in and of itself.

Answer 11

Many microcontrollers have analog-capable pins that can be configured as either analog inputs (for reading analog signals via an Analog-to-Digital Converter, ADC) or analog outputs (for generating analog signals via a Digital-to-Analog Converter, DAC). If the speaker is connected to such a pin without much of additional electronics, the idea of secretly converting it to the microphone may possibly work.

Of course, this would require hacking through some security features to gain low-level programming access to the chip.

Answer 12

A couple years back, I had read that cell phones were going to start using there glass plates as speakers & I believe it had mentioned car windshields as well. Not sure how well the TV screen will work to spy on you visibly, other than to maybe tell when you may have entered the room or left it. But I can see today's tech being good enough to use your tv screen as a mic to spy on you. Don't get hacked!