10
$\begingroup$

This was inspired by this YouTube video titled "Why all solar panels are secretly LEDs (and all LEDs are secretly solar panels)" and the fact that speakers and microphones are similarly related.

So thinking about a contemporary-technology dystopian surveillance society where LED TVs are built so that the behavior of

  • the display can be switched between display mode and camera mode, and
  • the speakers can be switched between speaker mode and microphone mode

The smart TVs would do this switching rapidly from the OS running on the TVs while the television is "in use".

In camera mode the TV screen would function as a (infra red?) camera sensor and record what it sees in the room while the speakers record the audio. All the data would be sent somewhere for further processing.

To what degree would this be feasible given current technology, and considering also aspects such as that the flat TV screen would lack a lens and possible aperture control?

$\endgroup$
  • 9
    $\begingroup$ Optics doesn't work this way. Before being concerned with a sensor, you must first arrange to create an image. (What the sensor does is convert an image into a chemical or electrical signal.) In order to create an image you need one or more pinholes or lenses. Without one/many pinholes or lenses all you can sense is the general level of illumination. And while it is indeed true that electrical machines are reversible, a speaker makes a terrible microphone, and a microphone makes a terrible speaker. $\endgroup$ – AlexP Aug 26 at 8:04
  • 8
    $\begingroup$ Ummm.... have you read "1984" ? $\endgroup$ – Carl Witthoft Aug 26 at 15:21
  • 6
    $\begingroup$ If you want to justify cameras on TVs besides what we currently have, you could promote the use of video calls (like Skype) but integrated in TVs to use in the living rooms. As I see it, this is more of a marketing problem (with the nefarious spying baked in) than a worldbuilding one. $\endgroup$ – Arturo Torres Sánchez Aug 26 at 17:29
  • 2
    $\begingroup$ This is probably easier $\endgroup$ – Stop Harming Monica Aug 26 at 22:14
  • 3
    $\begingroup$ Given that it's contemporary technology, the question that needs to be asked is whether your "LED TV" screen actually has millions of individual RGB LEDs for each display pixel, or whether it's a bunch of white LEDs illuminating an LCD panel that selectively filters the light colors. While there are OLED screens available, Google says they tend to be pricy and most displays are actually LED-backlight LCDs. $\endgroup$ – jamesqf Aug 27 at 3:28

10 Answers 10

11
$\begingroup$

Yes, with interferometry based synthetic aperture imagery, but...

It would take several orders of magnitude improvement in digital signal processing technology for it to work at the scale of a video screen. The sensors will need to record at a very high data rate, on the order of petahertz.

Every single pixel will be receiving all of the light that is in its field of view, and none of the light will be focused. Fortunately, light isn't just photons, it's also electromagnetic waves. The sensors will behave like antennas, rather than eyes, passively collecting the radio waves at the wavelength of visible light.

Radio signal sources can be tracked quite easily, by measuring the timing of a wave reaching multiple antennas. The same applies to all sorts of waves; an earthquake's epicenter is calculated by measuring the timing when the waves arrive at seismometers.

We even have the same types of sensors built into our own human bodies. : Our ears can taking a wave and determine the direction that it came from, based on changes of intensity and timing.

We have already proven the concept using vast antenna arrays to collect unfocused photons, to create a focused image. We imaged the supermassive black hole M87* using the Event Horizon Telescope.

EHT image of M87*

Of course, when astronomers use the term "photon" they don't just mean visible light; they mean any coherent electromagnetic wave. This image represents the peak of current engineering feasibility for synthetic aperture imagery. The EHT uses a Very Long Baseline Interferometry, which works in 450 GHz, using very narrowly calibrated equipment designed specifically to tease out the glow of the accretion disk at the wavelengths to detect event horizons around black holes.

In order to get meaningful data, though, your sample rate needs to be at least twice the frequency of the signal rate, preferably more than 4 times the signal rate, or you start getting downsampling errors called aliasing.

In order to record visible light, which has frequencies between 405 THz to 790 THz, you will need a sample rate that is at least 1.58 PHz.

Due to limitations from the speed of light, and the time that it takes electrons to pass through silicon and copper in computers, this is just past the fastest speed that we can record data meaningfully. We would have to pass the data from several sensors in order to build up a meaningful synthetic aperture image from interferometry. We would need specialized recording technology that we just don't have yet.

And, there's also the problem that LEDs aren't designed to collect light, even though they're capable of doing it... just as sound coming out of a microphone would sound terrible, and sound recorded from speakers is also low quality.

It would take several generations of iterating on the current science in order to use an LED-based computer monitor to record what's happening it a room, and it will always require specialized systems to just record the data in a meaningful way, much less process it into an image. It took several petabytes of data and 3 years of processing in order to build the EHT's image of M87*. It was faster to hand-carry the hard drives from the telescopes around the world to the datacenter, than it would have been to send the data over high speed internet links.

It would take a lot of iterative work to miniaturize the chips necessary to do the calculations, but the technology just barely exists. Such a screen would be prohibitively expensive, as it's much easier to just put a lense in front of a cluster of photodiodes (i.e., a webcam) and hide that in the corner of the screen, but those leave physical evidence... you can see the lense if you take the screen apart.

It could be possible, with non-digital interferometry, to construct such an image of the room in real time... but that equipment barely fits in the basements of large telescope observatories, and requires cryogenic cooling. You wouldn't be able to collect it clandestinely, and would be much cheaper (and higher quality) to use a webcam.

$\endgroup$
27
$\begingroup$

No

Screens these days are sheets of LED lights so that might work as a solar panel but not as a camera. At best you might measure ambient light levels perhaps but no camera.

That said TV these days are coming out with built in cameras and microphones and will get worse as time goes by. Things like Google Assistant and Alexa will come built into a lot of household devices.

$\endgroup$
  • 4
    $\begingroup$ As the Answer said, you do not need this, because TVs can have cameras and microphones (and other sensors) equipped and can have a connection to the internet. Additionally, even in western democraties in our real world, people are buying devices that can spy on them. They literally pay for that! So it would be no problem for your evil opressian dictatorship to plaster the streets, houses and homes with microphones, cameras and other sensors. Any smartphone can be used to spy on the wearer. You think much to much in terms of 1984, but thats more that 30 years ago - today even more is possible $\endgroup$ – Julian Egner Aug 26 at 9:09
  • $\begingroup$ Already now, software can hack your phone and turn it into a bug you carry around with you. Better than any TV spying system. $\endgroup$ – Thorne Aug 26 at 11:30
  • 2
    $\begingroup$ @JulianEgner Actually it's in terms of George Orwell's Nineteen Eighty-four (1949), where telescreens were two-way TV's. The novel was published seventy years ago. Amusingly large-scale surveillance has expanded & flourished once we passed the witching year of 1984. $\endgroup$ – a4android Aug 26 at 13:01
  • $\begingroup$ That's a scary thought. Its bad enough that its hard to find a non 'smart' tv these days. I imagine soon it will be just as difficult to find one without one of the various 'assistants' who will monitor your living room 24x7 $\endgroup$ – GrandmasterB Aug 28 at 15:23
  • $\begingroup$ @a4android I know - read it. And I find my own findings very scary, because real-life surveillance in democratic countries if much worse that it war in the fictional repressionist county from 1984. and thats only what we know $\endgroup$ – Julian Egner Aug 29 at 7:29
7
$\begingroup$

Even IF - Why would you want to take a photo with zero (or near few mm) field of vision and a lot of blur?
Because TV screen would (could) act like a photosensitive paper (or if you are old enough camera film). There would be no lens to focus, no apertures to set the amount of light, no pinholes that would allow you to point to certain area.
Screen would take light from everywhere that is not behind the TV. And what would be behind the TV would (could) still have ambient light. So a photo taken with such screen would be very bright (white) on border going to greyish in the middle.

Just put a normal camera in the TV. Speakers can work as microphones anyway.

$\endgroup$
4
$\begingroup$

Yes! ...But not a very good one.

Cameras have some combination of mirrors and lenses to focus the incoming light into something recognizable, and without that, you're not going to see much.

An LED will return the light level it can see; unfortunately, the light level an LED on the bottom left corner and the light level of an LED on the top right corner are going to be mostly the same, unless you are pressing your face against the screen. A television on the wall across the room won't be able to see anything useful.

However, a simple fix can improve the view considerably! Modern televisions have the LEDs right at the screen, or behind little bubbles that spread the light out, so as to produce the maximum view-able angle. But, there exist privacy screens that essentially limit the view angle to a point directly in front of the screen, at a set distance. These screens work by blocking light from escaping in any direction except a very small angle. Beyond that, the screen will look black, because the light has been blocked. And lucky(?) for us, that effect works in reverse! The LEDs will only see the light in a narrow band, instead of the whole room at once; the image will still be fairly blurry, but as long as someone is sitting in the "sweet spot" to view the monitor, the monitor will be able to see them, too!

Even without that, your monitor will need a lot of custom hardware to read the pixels; it's a lot easier and cheaper just to squeeze a tiny, fully functioning camera into the TV's frame - say, right next to the infrared sensor?

$\endgroup$
3
$\begingroup$

Yes, Samsung produced the SUR40 Surface Table for Microsoft.

The panel contained imaging sensors used to produce a picture of anything placed on the table.

Note that without a lens of any sort the range of images it can take is very limited. Also it suffers terribly in certain lighting condition especially involving overhead flourescent lights.

This was from 2011

https://www.engadget.com/2011/01/06/microsoft-and-samsung-unveil-sur40-the-surface-2-0-experience/

$\endgroup$
2
$\begingroup$

Given enough time, a very low resolution image could be recovered by changing the displayed image such that the liquid crystal layer allows light in through only a single pixel (or a small group of pixels). This would correspond to a black screen with a small white spot. By moving this white spot around the black screen and measuring the voltage at the backlight LEDs, some amount of spacial detail could be recovered (depending on the physical dimensions of the screen and the depth of the liquid crystal layer). The size and thickness of the spot will cause light rays to be rejected by angle based on the spot position.

The minimum transmission area will be limited by the total amount of light transmitted by the black pixels, so your resolution will effectively be limited by the screen's contrast ratio as the white spot must be large enough to transmit more light than the black pixels over most of the screen.

This would only be suitable for taking still images due to the time required to capture one frame, but the necessary mask frames could be hidden between frames during the display of a normal video on the screen.

$\endgroup$
  • $\begingroup$ "The necessary mask frames could be hidden between frames during the display of a normal video on the screen." I'm not sure if the overall technique would work, but if your display is using Black Frame Insertion (look it up), that would make it really easy to disguise frames that are black except for one white pixel... you'd be hard-pressed to spot the difference from normal BFI. $\endgroup$ – Matthew Aug 26 at 18:27
1
$\begingroup$

A camera needs a lens to work. Screen has no lens in front of it so cannot be used as a camera. This is a single, important thing. The rest would not matter that much. A proper electronics can pick external brigthness reading from the LED no problem.

If to put a tiny lens in front of each LED, it may work as a camera at the focus distance of this lens, given enough LEDs to compose a picture ("insect eye"). Some discrete ("individual") LEDs may have lenses in front of the light emitting diode, made on purpose or happening by chance. Hence low resolution banner has more potential for this kind of application. I do not think that IC leds have anything similar.

An interesting thing for the story, a water droplet on the screen may act as a lens, suddenly allowing the part of screen under the droplet to "see". I am not sure about the focus distance however.

$\endgroup$
1
$\begingroup$

Somewhat.

What is marketed as "LED TV" is LCD with a LED backlight. There are very few elements and they're placed behind (or to the sides of) a diffuser, so they won't be able to sense anything more than the room's general illumination level.

Some high-end LCD TVs do have a full-array backlight with enough elements to potentially make out a bit more information than just general illumination. These are generally even more rare.

OLED TVs have individual pixel elements, and they already have some means for recording the aging of individual elements as it occurs. Still, any image you get would have very little fidelity due to the lack of any means of focusing the light.

You wouldn't be getting a video. But, if every current from the backlight was recorded, it's probably enough data to forensically distinguish between the patterns of incoming light when someone is in the room watching the TV, and when they're not. After all, isn't that something the dystopian society would care about?

Of course, current TVs could already be recording you with good resolution by simply replacing the IR sensor, used for the remote, with a pinhole IR camera. It would keep working as an IR sensor, so no one would know the difference unless they disassembled the TV and closely inspected the components, specifically looking for a surveillance device.

$\endgroup$
0
$\begingroup$

Actually two questions here:

First of the microphone:

Yes you can, but as an electrical engineer this needs a lot more electronic in the background to accomplish that, in a dystopian setting this can also be hidden in the chips so that it is not obvious when you have a look at the PCB.

Second the screen:

Given current level of Signal processing: No. Explained here in detail.

Give it a few years to a group of signal processing experts and machine learning experts and you may be able to detect how many people are in front of the TV, but do not forget the polarization so you would have a rather small angle of view. It would be easier to use a micro camera that is installed and officially used for video calls on your TV or a hidden camera underneath a one way see through material.

$\endgroup$
0
$\begingroup$

Making this work technically would be a Manhattan project; there's no reason to make such extreme effort when there's a simpler way to get it done.

Selfie-grade cameras like the one on the face of your phone cost less than a dollar in million quantity. Microphones are even cheaper.

Simply incubate a social movement that creates some sort of reason some people might want one on their TV. Encourage every TV maker to put it on every TV, all in the name of populism.

We pretty much already have this with several social media habits, like cell phones tracking your location. For instance Google Maps tracks your location even in the background, nominally to monitor traffic flow and present local attractions, but also usable by police. Those are things you can turn off if you apply both technical skill and diligence, however you need to apply both pretty liberally, and when you do, and the police have reason to notice, it makes you look guilty and prejudices juries.

Eventually it becomes a thing that is on by default, like Google Maps' tracking; you must go to technical extremes to turn it off, and very few people bother. Those who do, the fact that they do is interesting in and of itself.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.