91
$\begingroup$

It's a common staple of science-fiction films (e.g The Terminator or Robocop) to have a POV shot from a robot or cyborg's perspective with a HUD being used to observe its surroundings. This can be hand-waved by the fact this is a convenient way for filmmakers to use visual imagery to make a robot's train of thought or state of mind clear to the audience very simply and quickly.

Would a robot need a HUD to act as some sort of decision-making mechanism? And if so, what sort of circumstances would make a robot need to use a HUD when studying its environment?

Improve this question
$\endgroup$
11
  • 20
    $\begingroup$ It is odd that the Terminator cyborgs and related robots would have such a thing as under no circumstances would they design the equipment to be accessible by the humans they're trying to wipe out. $\endgroup$
    – StephenG - Help Ukraine
    Commented Jun 5, 2018 at 12:04
  • 16
    $\begingroup$ You answered it yourself with mentioning the "convenient way". When extraterrestrials or ancient Romans or animals speak among themselves in English in a movie, we all know that they don't really speak English in-universe, it's just for the convenience of the viewer to understand what they are talking about. $\endgroup$
    – vsz
    Commented Jun 5, 2018 at 17:32
  • 9
    $\begingroup$ Another more intuitive way: In a movie, when a character is thinking, we hear a strange voice from the off while nobody is speaking. In reality, we don't hear (hopefully) voices when we are thinking. The HUD is really only depicting the thought processes in the machine because otherwise we cannot depict its eerieness. It is also menacing to see a machine categorizing humans. $\endgroup$
    – Thorsten S.
    Commented Jun 7, 2018 at 3:02
  • 5
    $\begingroup$ @StephenG: Was everything built from the ground up? Because if e.g. SkyNet had been deployed with an AI that automatically designs user friendly debug windows for new software that it creates; it may never have reconsidered doing so (when creating Terminator) because no human has ever managed to access these debug windows (e.g. because of SkyNet's assumedly perfect network security). $\endgroup$
    – Flater
    Commented Jun 7, 2018 at 10:05
  • 2
    $\begingroup$ Why does the Terminator (T-800) have a HUD? $\endgroup$
    – Mazura
    Commented Jun 8, 2018 at 0:49

17 Answers 17

Reset to default
256
$\begingroup$

Yes, because QA

Source: I have been tester for about 9 years.

Before you let your robot to run into the wilderness letting him kill help humans, you need to know if the robot knows what is a human and what is not.

So in order to test the robot, their software and/or hardware will be tested by humans. Or another automated tests. And you bet there will be nagging testers telling you all the time that they need to have to know if a robot recognized human or not.

Yes, you might put off that "Can you put green rectangle about the human whenever it sees a human?" request for a few releases, but those pesky testers will complain about how difficult their life is and how pricy testing the robot vision without green rectangles around humans is.

Why does the HUD stay, you might ask? Because removing that feature costs time, and moreover the whole robot would have to run through yet another QA round to realize if removing such feature didn't break anything else.

To sum it up:

  • The robot is initially programmed by humans
  • HUD is nice feature and it helps (human) testers to test the robot
  • Robot itself doesn't really need the extra features added to HUD. It would recognize human even without putting green rectangle around them
  • No, I cannot tell you if I tested soon-to-take-over-the-world robots. That's still under NDA

Hasta la vista, baby

HUD: Heads up display
QA: Quality Assurance
NDA: Non disclosure agreement

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – James
    Commented Jun 7, 2018 at 15:23
  • 4
    $\begingroup$ Excellent examples of this are the HUDs in (at least some of) the self-driving cars being tested. Some of them create a complete 3D graphical representation of the car and the surroundings, but others show the camera feed overlaid with a representation of how the self-driving car (aka, robot) is interpreting all of its sensory input. It helps the engineers check whether the car is getting it right. $\endgroup$
    – Adrian McCarthy
    Commented Jun 9, 2018 at 12:55
  • $\begingroup$ you could always connect an external screen for testing and then remove it, at no cost. If Terminator can be hacked, I'd rather have him without vision (in human terms). Besides, this answer could work with entities built by humans. Robots built by robots wouldn't need it at all, not even for testing $\endgroup$
    – Devin
    Commented Jun 9, 2018 at 18:20
  • $\begingroup$ In the first Terminator movie, Kyle explains to Sara how the terminators appeared after the machines had conquered the Earth. They caught the human survivors totally off-guard when they first infiltrated the human's underground hideouts. $\endgroup$
    – Solomon Slow
    Commented Jun 10, 2018 at 19:43
  • $\begingroup$ Excellent answer that covers most cases but in some cases involving machines creating machines (e.g. Terminator, Matrix), the QA tester would ALSO be a machine/program, so... would the HUD be required in that case? $\endgroup$
    – xDaizu
    Commented Jun 11, 2018 at 9:07
48
$\begingroup$

No need...

The robot intellect would just know those things that appear in the HUD text. These would be akin to our non-visual senses, like how we know our foot is at this angle or how we know the room is a comfortable temperature.

HUD would obstruct or distract actual vision

For the robot/cyborg to actually be aware of the HUD text contents, their focus would have to switch from vision center to the HUD, scan the text, then revert to whatever external view they were focused on. This makes sense if the HUD text is generated externally and shown on a screen of some sort in between the eye/camera and the field of vision. It makes no sense if the text is generated inside the camera. Why would you waste precious system clock cycles shifting focus or risk not noticing the text because you didn't shift focus fast enough or at the right moment?

Nope. Far more efficient to have that data as some kind of extra senses that are just known. Along with other normal senses a computerized brain might have, like precise clock data, precise ranges to things in visual range, etc.

..Except maybe for later review

There would probably be complex equivalents to real-world log files, filled with the details from the HUD text. And it is possible that the actual visual data would be recorded as well.

So if your cyborgs are programmed to work for humans or programmed for human debugging, then it becomes possible to have a "review mode" that plays back the recorded video, then superimposes the log file data for non-visual senses as HUD text in the playback.

So if you want to hand-wave the HUD text, do so as "this is how human debuggers or auditors or base-camp controllers would see it, not how the robot itself would see it."

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Now you made me think of a robot with tons of text scrolling by in the periphery of its vision as its systems are analyzing its surroundings, rather than the (typical in movies at least) few numbers and symbols constantly changing but remaining squarely in the same place. $\endgroup$
    – user
    Commented Jun 5, 2018 at 19:47
  • $\begingroup$ @MichaelKjörling I've seen something llke that in one or two works, but i cannot think of which at this time $\endgroup$
    – can-ned_food
    Commented Jun 6, 2018 at 2:29
  • 1
    $\begingroup$ @MichaelKjörling Probably not, a lot of operations are greatly slowed down when logging is enabled. $\endgroup$
    – SPavel
    Commented Jun 6, 2018 at 15:08
  • 3
    $\begingroup$ @SPavel If we're going down that route, there's no reason that such logging would be a synchronous operation. $\endgroup$
    – user
    Commented Jun 6, 2018 at 16:16
16
$\begingroup$

In Robocop's case, remember that the "robot" is part Officer Alex Murphy. The programmable A.I. works alongside Murphy's brain. The HUD may be a type of interface (maybe one of several) between the robot component and Murphy's brain - a way of sending quick summary information to the brain in order to prime it for further processing/information/instructions which would be sent in a more machine-friendly format.

This would also justify the classified "fourth directive" not being visible on the HUD. (one may ask how could a robot stop itself from doing something without knowing what it wasn't supposed to do?) - The Murphy part wants to arrest Dick Jones, but the A.I. component will not allow Robocop's physical body perform an action that would complete that arrest, and also not tell the Murphy component why.

This doesn't really work for the Terminator though, but in advanced computing of the future, I'm sure one could come up with a justification why multiple sources of information should be sent in different formats through the visual interface to the CPU. Computers have advanced well beyond the simple "Program Counter" ticking its way along a sequential set of instructions.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ Robocop doesn't have a name. He's product. $\endgroup$
    – can-ned_food
    Commented Jun 6, 2018 at 2:27
  • 2
    $\begingroup$ @can-ned_food "Nice shootin' son, what's your name?" "Murphy." $\endgroup$
    – Schwern
    Commented Jun 7, 2018 at 5:20
  • 2
    $\begingroup$ Also one subfield of AI involves creating neural nets (similar to the human brain). There is no reason in general why neural nets would need a HUD, but perhaps creating sentience from scratch is harder than just starting by copying a human neural net and augmenting it. Reuse is a big thing in Computer Science, why modify the neural net to accept direct input when it already has the ability to read a HUD? $\endgroup$
    – gmatht
    Commented Jun 7, 2018 at 9:36
  • 1
    $\begingroup$ "Computers have advanced well beyond the simple "Program Counter" ticking its way along a sequential set of instructions." ... and yet, the content of the HUD clearly demonstrates that at least part of the Terminator is running on a 6502. $\endgroup$
    – Jules
    Commented Jun 10, 2018 at 11:05
  • $\begingroup$ Even in the most advanced programs, I would imagine a program counter or similar would exist at some low level in the code. Especially if humans made it. If there is sequential memory and sequential instructions, there would likely be a program counter. $\endgroup$
    – Garrett Motzner
    Commented Apr 23, 2019 at 21:27
15
$\begingroup$

Having actually been apart of the "sight" testing for a robot in real life, I can confirm that the HUD was used for the human tester (that's me) benefit, not the robot. In fact, given the significant delayed processing response that the HUD I built imposed, the idea was that the final product would not be used with the HUD enabled.

My specific testing was related to detecting colors and distinguishing from unsafe colors (Whites, Oranges, Yellows) and safe colors (Greens and Blacks). To achieve this, we would have a camera transmit each individual picture to a process that would tag each pixel with a "isSafe" boolean operator where the good colors would be "true" while the bad colors would be "false" and pass this information to the central processing where it would be combined with other data inputs.

To show proof of concept at meetings and during tolerance testing (Yellow and Green are close to each other on a standard RGB image color scheme, so we had to be precise in where the tolerances were) my HUD would take a sample image and recolor it so that all the "Safe colors" were set to black and all the unsafe colors were set to Bright Yellow. There were some other things that I wanted to implement but had to hold on for other sensory systems to be up and running such as distance of each pixel to the robot's camera.

Fun fact, the primary depth perception would not have been the camera "eye" but a LIDAR system that would be able to detect objects at 360 degrees of vision, though the image portion was strictly for forward motion.

Improve this answer
$\endgroup$
8
$\begingroup$

The answer may be different for robots as opposed to cyborgs.

Cyborgs, the human brain - artificial body kind: As it stands the optic nerves are a great broadband-connection (the only?) into the brain, so it makes sense to use them to transmit information into a biological brain.

Robots, with AI still depending on binary computation: No, those statuses are merely cells in a mem-table. But it´d make a great debug-output (video-feed + overlay). This could also make sense for other robots with incompatible interfaces. This could be what the movie-makers where showing.

Improve this answer
$\endgroup$
7
$\begingroup$

No, because data.

In biological organisms external information is captured from the environment through various sensors (eyes, ears, skin) and streamed (via nerves) to a central processing unit (the brain) and interpreted into data (vision, hearing, touch). Decision-making then happens over available data.

HUDs are one kind of interface augmentation that we use to supplement data. We use several of these in a day-to-day basis to translate external information into native sensory inputs since we don't have sensors for these:

  • Beeps when we forget an open refrigerator
  • Blinking lights to indicate a new message on our mobile devices
  • Text information overlay in games to indicate remaining 'hearts' or 'ammunition in a clip'

So we kind of subvert 'natural' interfaces to add new sensory content since we can't upgrade humans. (Yet.)

Robots, on the other hand, have no such limitations: You can design them to be flexible about sensory input, and just slap a new stream to its communication channel (after, of course, adding the necessary software to parse the new data type.)

What sort of circumstances would make a robot need to use a HUD when studying it's environment?

That may happen if the robot doesn't have a fully integrated data flow added to its sensory set. Then the new external stream may be first translated and then added to an existing interface, the same way we do.

Improve this answer
$\endgroup$
6
$\begingroup$

I'm going to come up with a couple of plausible scenarios where it might make sense to have a robot maintain a heads up display.

In humans, electrical signals from different stimuli travel at different speeds to the brain. From Speed of processing in the human visual system:

Here we use a go/no-go categorization task in which subjects have to decide whether a previously unseen photograph, flashed on for just 20 ms, contains an animal. ERP analysis revealed a frontal negativity specific to no-go trials that develops roughly 150 ms after stimulus onset. We conclude that the visual processing needed to perform this highly demanding task can be achieved in under 150 ms.

Seeing and then comprehending something takes a certain amount of time. It is plausible that, due to the design of your robot's brain, the visual comprehension for some specific data points is faster than looking up those data points from whatever memory module they currently reside in. In this case, providing critical data in visual form to the robot would make the most sense.

You can also argue that, during the normal course of operation visually interpreting that data could be slightly slower, but during "stressful" situations not taking the few cycles or so to have to interrogate critical subsystems on their current status and instead rely on those systems to report their data directly to the visual cortex can outweigh a minor loss in performance. The HUD can also be just a backup data view for those times when both the front-side and back-side buses are currently overwhelmed with commands to terminate the local population of organics.

Slightly akin to Pavel Janicek's answer, the HUD stays around for logging purposes. It is possible that the visual link is shared with an upstream monitor that does not have access to the the robot's hardware, just the visual data stream. By maintaining a HUD, this stream will relay valuable information back to the robot's command and control even in the event the silly humans manage to resist termination and destroy the robot. Seek and destroy drones can be dispatched back to the robot's last known position with the robot's last transmitted images.

It's not a HUD, it's just perspective

On the other hand, I've always just visualized those scenes as just a representation of what was going through the mind of the robot at the time. I did not interpret those scenes as a literal image being interpreted by the T-900, but just a compendium of overlays from the all of the unit's sensory inputs to provide the point of view from the machine.

Improve this answer
$\endgroup$
5
  • $\begingroup$ "It is plausible that, due to the design of your robot's brain, the visual comprehension for some specific data points is faster than looking up those data points from whatever memory module they currently reside in." - A current, lower end HDD has a seek time of <20ms and a data rate of maybe 1000 MBit/s. RAM is much faster. Also, you'd need to retrieve tha data first to render it to your HUD, so if there was some sort of delay, you'd incur it anyway. $\endgroup$
    – Ruther Rendommeleigh
    Commented Jun 6, 2018 at 9:04
  • $\begingroup$ @RutherRendommeleigh I make no assumptions about what underlying technology is powering said robot. I'm sure the performance characteristics of Data from Star Trek's positronic brain differ significantly from what you quote. $\endgroup$
    – Jeff Lambert
    Commented Jun 6, 2018 at 12:58
  • $\begingroup$ Granted, if you make no assumptions, then anything is possible. However, the only way that "retrieve+encode+transmit+decode" could be faster than "just retrieve" in anything based on real world physics would be a significant design flaw (e.g. delays caused by race conditions). And even then, transmitting data as human-readable text or target indicators etc. is far less efficient than encoding it in a more compact format. $\endgroup$
    – Ruther Rendommeleigh
    Commented Jun 6, 2018 at 15:21
  • $\begingroup$ The comparison to HDDs was to point out that, even for something as computationally powerful as the human brain, optical processing is a task much more involved than data retrieval. It would seem... odd if by the time we have robots that rival human intelligence, memory would have become much slower and we'd forgotten how to make the older, faster stuff. $\endgroup$
    – Ruther Rendommeleigh
    Commented Jun 6, 2018 at 15:24
  • $\begingroup$ @RutherRendommeleigh I'm not assuming that any data is self-contained in a single system. Perhaps a reservoir of lubricant that is necessary for functioning is kept in the robot's left big toe. Internally it keeps track of its own level and reports it back to the vision system without any connection whatsoever to the main decision matrix. You may rightly argue that is a design flaw, but someone else could rightly argue that it was a tradeoff instead. $\endgroup$
    – Jeff Lambert
    Commented Jun 6, 2018 at 18:47
4
$\begingroup$

The HUD is simply a way to translate measured data to something easily intelligible for humans, as humans are normally used to eyes for interacting with the outside environment.

Translating the data to a HUD takes an additional step, therefore it is not strictly needed, unless a human equivalent mind has to be granted access to the same data.

Improve this answer
$\endgroup$
0
4
$\begingroup$

We humans still have a tail bone from our ancestors even if we no longer need it.

As technology evolves artificially or naturally then elements would persist even if they weren't needed. They would be removed only when that gives descendant children an advantage. Since a HUD display doesn't give children an advantage or disadvantage, then all children with or without HUDs would have an equal chance of passing on that trait on. So the trait of having a HUD would persit.

We can see these kinds of battles of traits persisting in technology in the real-world. Take the earphone jack as an example. iPhone's don't have one now and Android phones still have them. Over time we'll find out if this trait was needed or not.

Improve this answer
$\endgroup$
3
$\begingroup$

It is important to consider that a robot and how it perceives the world will be a consequence of the hardware design of the robot. A robot may very well have an internal HUD, even outside human factors, as that may be the most effective way of getting the information to the CPU at a very high update rate. An additional benefit is that sensor data can be pre-processed, entities in the visual data can be tagged with the relevant information, saving cycles on the robots main OODA Loop processor for more important decision making.

Improve this answer
$\endgroup$
2
$\begingroup$

No.

Same as humans (and other animals-as far as we know) don't have a HUD.

The HUD has its use where an entity (human,so far) requires visual feedback or representation of internal or external processes of a machine or their surroundings.

Robots or androids have a direct connection to all the sensory inputs built into them. Updating a HUD and then analyzing that data wastes way too many cycles. It is much faster to access the data directly and process it accordingly.

As was mentioned here that a HUD is coded in for testing purpose is a viable reason but it is meant as an output to be used by an external controlling entity (the creators) and is not used by the android / robot itself in this scenario.

tl;dr duh, of course we're talking about movies and therefore as OP stated it is exactly the choice of filmmakers to convey information to the audience.

Improve this answer
$\endgroup$
1
$\begingroup$

Yes if the robot was designed to use equipment that is designed for humans, or if the robot was built from a collection of existing systems built to interact with humans.

Improve this answer
$\endgroup$
1
$\begingroup$

Put on your mad scientist hat for a moment.

In an adversarial environment you'd need to take account for information warfare. I would build in a mechanism of ensuring that the AI thoughts originated from within. The last thing I want is for an adversary to be able to inject thoughts, data or commands without validation.

This would require a type of signed handshake for trusted internal components. Basically if a device isn't trusted it cannot send commands in for processing. All external sensors and weapon systems need to shunt through untrusted channels or risk compromising internal systems. This is way easier than signing trusted weapon systems because weapon systems need to be way more flexible so they can adapt to changing battlefield tactics.

All my robot soldiers will process incoming data through protected channels where data is sanitized and command functions are stripped out. The easiest way to do that is to piggyback on an already sanitized input stream. The handiest high bandwidth sanitized input is of course the visual input system. All you jokers saying you just ingest the data directly are going to have a bad time when I litter the battlefield with weapons designed to infect your robots over their ODB port. Or better yet you didn't even think to sanitize your visual inputs and I can code inject with a freaking flashlight.

My robots use a hud display as a protection against takeover by malicious actors. Also all previous "No HUD" robots in this thread are now my robots and the first thing I do is upgrade them so they're not available for takeover.

Now every robot uses a HUD display, weapon systems are decoupled from mobility and sentience systems and they're interchangeable and upgradable. Also I just took over the world. MUAHHHAHAHAHAH.

Improve this answer
$\endgroup$
1
$\begingroup$

You're looking at this the wrong way. Movies are made for humans, so the best and most natural way to communicate the concept of extra information streams is as a visual overlay.

Having an actual software system receive information that way would make no sense. It would be like displaying a sound wave-form on your vision - you could do it, and it would carry information - but you already have a dedicated sensory channel for that purpose, so you'd prefer to use it.

Improve this answer
$\endgroup$
0
$\begingroup$

If the AI which drives the robot is the result of a simulated natural selection procedure in which no HUD-like information is given, HUD would just provide more information to AI just like what HUD does for us humans.

If it is highly 'synthetic', HUD can provide debug information or augmentation to already existing AI programs.

Improve this answer
$\endgroup$
0
$\begingroup$

No. The terminators were built and programmed by and AI so unnecessary code for human debugging and testing would never be added.

If the terminators were built by humans and reprogrammed by an AI maybe but not where they were designed, built and programmed by AI from scratch.

Improve this answer
$\endgroup$
0
$\begingroup$

Intelligence as we know it isn't a unitary thing.

You don't have one mind. Your brain is full of non-conscious features. Impulses from your optic nerve are fed into the visual part of your brain and are heavily processed before anything resembling your consciousness gets access to them.

Your visual experience is a hallucination; most of your vision is basically in black and white, the only part with any resolution is directly where you are looking, you have blind spots, your nose is (typically) filtered out, your vision turns off for a fraction of a second every time your eye moves, etc. Lines, angles, edges, circles; pattern recognized and hallucinated.

This "low level" visual processing occurs without bothering your "mind", and it happens in your brain.

If we build our AIs based off of the only intelligence we know, we'll probably solve various subproblems and connect them up. The visual processing system won't be a general AI; it will be a specialized AI that does high quality visual processing.

The "feed" to the "really smart" part of the AI's brain is thus not going to look like what the camera sees. It is going to be marked up with information that the specialized visual processing unit has worked out.

Possibly multiple layers of such specialized visual processing units are going to be turned into a pipeline. Some of them will recognize humans and highlight them. This means that the "mind" doesn't have to have the ability to rapidly and reliably notice humans in its field of view; it just has to have good enough visual processing to recognize the highlights.

A HUD could exist for similar reasons. The intelligent part of the AI is human-like; it no more aware that it processes binary data than our brain is aware that it is processing glucose or action potentials. "Lifting" data up to the level that the AI experiences (visual bullet count, etc) instead of providing a binary feed could make it easier for the "command AI" to pay attention to it.

A side benefit to all of this is that the augmentations would work with a human for both testing and development. Other benefits include the fact that the visual coprocessing units can also be modularly replaced without rebuilding the AI In addition, and if the visual co-processing/HUD systems are broken the core AI has limited visual processing and can proceed without them.

Improve this answer
$\endgroup$

You must log in to answer this question.