Throughout science-fiction, from Star Wars to sundry others, we see characters manipulate holographic images as if they were solid objects; shaping them, tapping them, manipulating them like some kind of 3 dimensional touch-screen, even though all they are are projections of light. How would this work in reality? Is there a way to use a hologram like a touch-screen?
$\begingroup$
$\endgroup$
10
-
2$\begingroup$ Star Wars doesn't have holographic interface technology and relies on tactile/analog consoles. As part of the aesthetic, Modern Star Wars, especially those set between episode 3-7 (and even the sequels) will replicate the look and feel of episodes 3-6, which are "future computers" by late 70s- early 80s belief. Additionally, Star Wars was one of the first sci-fi films to popularize the "Used Future" ascetic, so most tech looked worn. The Falcon is not the Rolls Royce of space ships. It's a rusty school bus (modded to have a Bugatti engine under the hood, but you don't see that.). + $\endgroup$– hszmvJan 3 at 15:00
-
2$\begingroup$ +Star Wars uses holograms in communications, but it's merely 3D Skype at best. The projection might be manipulated, but almost always from a analog interface (as opposed to directly interacting with the hologram, like Tony Stark's dragging of hologram displays directly. $\endgroup$– hszmvJan 3 at 15:03
-
1$\begingroup$ I should think before you get a holographic keyboard working in a manner that most people would find pleasant, various neuralink type interfaces will be working. You would do your I/O through that without having to wave your hands around like a crazy person. Just be careful about the direction you apply that "delete ." command. $\endgroup$– Boba FitJan 3 at 15:29
-
3$\begingroup$ If you can emit arbitrary (visible wavelength) photons from arbitrary xyz coordinates, one might also be able to sense ordinary matter as it occupied arbitrary xyz coordinates by any number of pseudo-physics methods. The same sort of drawbacks apply that have been discovered by touchscreen technology (lack of tactile feedback chief among them). $\endgroup$– John OJan 3 at 15:55
-
2$\begingroup$ The HID (human interface device) technology has existed for 10+ years already, see e.g. Leap Motion. All you need in addition to that is the actual holographic projection/display tech. $\endgroup$– Jivan PalJan 3 at 20:41
|
Show 5 more comments
13 Answers
$\begingroup$
$\endgroup$
13
Gesture recording
There's a quite workable solution, without beams crossing the room. You project the hologram as usual, in front of the person. Then you'd have video cameras on the other end, recording the person's gestures. A vision software application is connected to two cameras, correcting for perspective and calibrated using certain objects. It can pass the touched coordinates to the program running the hologram and here you are, a non-solid touch screen.
The invention belongs to the technical field of human-computer interaction, and is a visual, operable and non-solid touch screen system which comprises a computer, two network cameras, a head-mounteddisplay and a calibration reference object, wherein the calibration reference object calibrates a virtual touch screen.
https://patents.google.com/patent/CN101673161A/en
3d cursors
The above invention actually implements a 2D interface, hanging in 3D space. A quite "normal" touch panel, in fact.. An interesting challenge: a hologram actually has depth. Maybe, a 3D mouse be possible, allowing the person (eg: a surgeon) to reach into the projected holographic scenery.
-
5
-
2$\begingroup$ We can see exactly this sort of thing already working in VR, too. Instead of "holograms", it's virtual reality, but from the user's perspective that's the same thing -- they must manipulate a 3-d display laid out in front of them. Hand gestures and/or controllers with buttons take care of it, though precisely how the gestures work is up to individual developers to figure out. (But they are pretty similar to phone gestures. Hands together: zoom out. Hands apart: zoom in. Grab and pull. Grab and rotate. Etc.) $\endgroup$– JamieBJan 2 at 20:44
-
$\begingroup$ Alternatively, one could use electronic gloves to reduce the chance of an interpretation error on the part of the software. The tradeoff is that one has to wear the gloves to interact with the hologram, much like using a mouse, so not just anyone could interact with the hologram. $\endgroup$– JafegoJan 3 at 9:41
-
$\begingroup$ I recently saw this done in 2D at an art exhibit. An image is projected on a normal blank wall, and you can interact with it by placing your hand on the wall. I assume there is a camera tracking people's movements which than affect the image being displayed. $\endgroup$– David KJan 3 at 13:28
-
2$\begingroup$ I feel a need to point out that "project the hologram as usual" is mutually exclusive to "in reality." We see holograms so often in the movies that we don't realize that such things don't exist and, based on our current knowledge of physics, can't exist. All current forms of hologram are limited to a volume that you can't reach your hand into. $\endgroup$ Jan 3 at 21:23
$\begingroup$
$\endgroup$
11
Update: There are a few conversations that I've had to repeat, so I figured I'd include the responses here.
A hologram projector isn't a real thing. We see this so much in movies and television that we forget that it has the same real-world viability as artificial gravity. You need a reflective surface to exist at the spot that you're making glow, which means it has to occur inside of a display space. This makes it a holographic display, not a projector.
You cannot reach into a holographic display. This would involve your hand going through and disrupting whatever the hologram is being projected onto.
The way around this is to use augmented reality glasses. The glasses are required to ensure that different images are shown to each of a person's eyes. For proper AR, they also need to allow light from the outside world in, otherwise they're actually just VR.
There are many implementations of hand-tracking machine vision projects. It takes less processing power if you wear a clearly marked motion-capture glove, otherwise you need to ensure that the hands are well lit.
When you combine these, you have holographic input and output devices. You need to set a command structure that differentiates between selecting, grasping, and pushing gestures, but at that point, it's just a matter of defining the protocol.
answered Jan 2 at 20:47
-
1$\begingroup$ Yep, weirdly enough, the touch-part of the "hologram touchscreen" is already possible and working $\endgroup$– HobbamokJan 3 at 15:53
-
$\begingroup$ Glasses.. this answer is not using a holographic projection ? Then, your "touch screen" would not be visible for other people.. unless they are also wearing these glasses and running the same augmented reality program, from a slightly different perspective.. In science fiction, these holographic devices are always visible (in real light ) $\endgroup$– GoodiesJan 3 at 20:24
-
$\begingroup$ @Goodies, Indeed, this is true. Our science currently insists that there is no known way to get a random spot in a room to emit light, and that's what holograms are. In optical science, this is known as the "Princess Leia Effect." I could go down the list of things that have been tried. Either you have to make the air reflective or emissive (and not have your fingers in the way), or you have to do something to show different images to every eye in the room. It's almost as intractable as artificial gravity. $\endgroup$ Jan 3 at 21:12
-
1$\begingroup$ The touchless Japanese interfaces use an optical illusion. The one in the link is convincing, but it uses a cellulose dust mote as an ultra-tiny projection screen. The trick of moving the mote around is notably impressive, but you still couldn't reach into the volume. The gusts of air from your moving hand would push the particle out of the volume. $\endgroup$ Jan 4 at 23:53
-
1$\begingroup$ For what it's worth, I like this answer much better post-edit. $\endgroup$– BobsonJan 10 at 23:37
$\begingroup$
$\endgroup$
1
Visual theremin
The theremin is a no-contact musical instrument which is "played" by moving hands within an oscillating electrical field. This electrical "pitch field" is picked up by separate pitch and volume antennae, which are used to modulate a musical tone. As the hands of the performer, which act as capacitors in the circuit, move in relation to the antennae, the field is perturbed. In the case of the theremin, this produces alterations in the pitch or volume of an otherwise constant tone. In your case, the onward processing can be a movement on a 2d or 3d space mapped to the hologram, and certain gestures can be mapped to an "interaction" (swiping, tapping etc).
The advantage of this design is that the operator's movements directly act upon the signal, as opposed to requiring an entire separate apparatus (cameras and machine vision processor) to detect and digitise the user's movements. The drawback is that it's somewhat more temperamental and constrained. Think of it as the difference between using a radio app on your phone vs an in-built FM receiver.
You could have high-end systems designed to work with expensive multichannel cameras, as per others' replies, with fancy proprietary iGloves that guarantee a smooth, interference-free detection. And you could have rugged, industrial use devices that are little more than an emitter and two antennae, cranky and difficult to work with except for skilled mechanics and suspiciously talented protagonists, but far more durable and portable.
-
$\begingroup$ An interesting idea.. If you had several Theramin-like devices, they'd produce different pitches and volumes depending on their relative positions to your hand. So you could triangulate to produce rough coordinates. It wouldn't be particularly precise though. $\endgroup$– RuadhanJan 4 at 9:58
$\begingroup$
$\endgroup$
2
Detect the heat signature of fingers
Humanity is already developing 3D thermal imagine, though inelegant compared to what you're trying to do, the premise is simple.
Detect the tips of people's fingers.
I can easily imagine a device that detects the moment a finger/thumb (hereafter "fingers") enters the holographic space. Think of it from a 2D perspective. That first moment would look lime one or more infinitely think thermal discs. As the hand pushes further into the holographic space, your computer begins to detect an elongation of those thermal signatures. It's beyond plausible that it would recognize these as fingers, and finally see all those fingers connected to a hand.
The real trick is something along the lines of a double-mouse-click. How do you know when the finger has done something that would cause an expected result? I believe buttons would be simple. The moment the tip of the thermal image crosses the known location of the button, the button's callback function is activated.
A little more complex would be spinning a wheel. In this case, the tool could detect multiple finger crossings, allowing the wheel to be turned at the rate of motion of the fingers. If a single finger touches it, nothing happens. Part of the problem here is training the user to stop their fingers near the outer edges of the wheel (overcoming the lack of touch will take training, but it's not impossible, there's an art form with Japanese Katana that fights choreographed imaginary battles — and the goal is to respond realistically to the non-existent combatant).
Even more complex would be grasping a holographic object. In this case, it's all about how you program the response to those fingers. In a high-sensitivity mode, tapping the object would cause it to move. In a low sensitivity mode, grasping the object with a brief pause (the delay between those blasted mouse clicks) to let the system know you really intended to move the object.
So, yeah, I think this is not only believable, but plausible, and that we're working toward the technology already.
$\begingroup$
$\endgroup$
3
Sonic resistance
In addition to the hologram, have a sound machine nearby that can make waves of sound. It carefully constructs waves of sound to resist people's hands when they touch them, like a light wind. As such you have tactile feedback from touching the screen, and a mixture of light sensors and sound sensors can say exactly where your hands are.
AI interpretation
You'd have an AI which maintains a profile of each person which predicts what they want the screen to do. This helps adjust for people with reduced mobility or unusual motions. Every screen would be calibrated for a standard set of motions, but if you had time you could train a screen to respond to any motion you wanted, from blinking to tapping fingers.
-
$\begingroup$ To get enough air moving fast enough to let you feel a response similar to a keyboard, you are going to have a disagreeable amount of noise. It will be like ten little jet engines trying to blow your hands off. $\endgroup$– Boba FitJan 3 at 15:27
-
$\begingroup$ I said light wind, not like a keyboard. Fans and similar things can be fairly quiet. $\endgroup$ Jan 3 at 15:28
-
$\begingroup$ +1 interesting idea.. I guess 24kHz is convenient, 100kHz feasable, you'll need high frequency / short wavelength to provide enough spacial resolution (detail) $\endgroup$– GoodiesJan 3 at 20:39
$\begingroup$
$\endgroup$
We actually have tech like a " projected laser Keyboard" and sight based key selection (Done by having a camera that notes where your eye is looking to manipulate the interaction). However, the reason these technologies aren't wide spread is because they are difficult to manipulate with any consistency. In the case of a laser keyboard, one problem is that there is no tactile response... that is, when you press a key, the depress and return is a mental signal that the key has been interacted with and you don't need to press it again. This makes typing faster and more efficient and most people can type without looking at their keyboard (the reason why the F and J keys have a raised bump is so you can find them without looking. Knowing where they are located can help guide your hands to all letters in the alphabet in an efficient manner. Because the projected keyboard is non tactile, these features are lost without any net benefit to the typist.
Additionally, the reason both of these keys work is that the interface is watching your movements, which isn't as reliable a method as button pressing, which is a passive system.
This isn't to say holograms don't have a place as 3D projection has it's benefits (especially in a Sci-fi space opera, since space is a 3D environment, and a 2D map of space routes might not take into account that a planet 10 light years on an XY projection is 1000 light years below that position if you keep on the same XYZ plane of Earth.
$\begingroup$
$\endgroup$
1
Eye focus tracking
The system tracks the user's eyes and can tell exactly where they are focusing. The hand motions are simply an aid to this.
e.g. the eyes focus on the object you wish to manipulate, then your hand gesture does what you want to do with it. (Move, close, embiggen, etc.)
-
$\begingroup$ A touch panel should be able to run MS-Paint, doesn't it ? When handling painting programs you'll notice your mouse does not always point at the location you are looking at. How do you prevent mouse location interpretation errors, when only taking into account the eye focus ? Don't get me wrong I think eye tracking is a usefull addition to the setup, but it can't provide mouse coords, unless your user is prepared to accept the result. The panel is visible, the finger touches a specific position. That is how a touch panel works. $\endgroup$– GoodiesJan 3 at 20:32
$\begingroup$
$\endgroup$
a technology like that already exist, all be it it... not working well yet.see laser keyboard
basically you can get a "touch" detection with an IR censor that tell you where the screen has been touched, assuming where are talking about screen that are fully non physical. the only issue with that technique will be that you won't have any touch feeling.
a cool idea i do have for those screen would be them to not be laser related but have a tiny swarm of robot who just form everything you want and emit the light themself. therefore you can actualy touch them
another thing that might work, and who can be fun is: there are no actual projection on the screen, you need an eye implant and to connect to the screen, then everything is projected in AR in the eyes of people who look at those, those tablet would just therefore be an AR enchor point
$\begingroup$
$\endgroup$
2
Ultrasonic haptic feedback
A phased array of ultrasound emitters can produce tactile sensations (vibrating a point or area on your fingers/hand). There's ongoing research in this area, and there appears to be at least one commercial product available. It's not going to feel like you're touching or grasping a solid object, but it can provide useful feedback.
-
$\begingroup$ I think your answer is very similar to Nepene Nep's answer. $\endgroup$– GoodiesJan 3 at 20:42
-
$\begingroup$ Possibly. The comparison of the effect to "a light wind" misses the mark for this sort of ultrasonic approach, though. There's no bulk movement of air. (Others have attempted using air jets, but it's a separate approach, with different tradeoffs.) $\endgroup$– jeffBJan 4 at 17:50
$\begingroup$
$\endgroup$
The keyboard and other input devices are physical. They provide the physical feedback, possibly even some sound for the clicks.
But they are not connected to anything, not even through Blue Tooth or similar means. A holographic overlay gives you visual information about what you are doing. And various visual sensors (lasers, infrared sensors, what-have-you) detect what you have clicked.
Since the keyboard etc. would not have any electronics in them, they could be relatively cheap and light. All they have is comfortably springy keys. The mouse has a little thumb-wheel that rotates with an agreeable amount of stiffness. And so on. You could mount them on any convenient mounting device. And the screen would be holographic, requiring no additional desk space.
Straightforward customization would be possible. You select the size and physical arrangement of keyboard etc. that you prefer. Bigger or smaller keys. Or qwerty versus dvorak. Or even those freaky two-part keyboards. Then you overlay the hologram to match the physical config of the device. You could even have things such as a keyboard over here for when you are sitting, another for when you are standing at some equipment, even one for the bathtub. The hologram over the keyboard and showing the screen (screens?) would jump from keyboard to keyboard.
Combine with additional gestures such as waving, swiping, pinching, etc., as various tablet devices have. It becomes as simple or complicated as your needs and desires require.
$\begingroup$
$\endgroup$
It is possible to track a small object in 3D space with around mm precision with current technology. The example below was implemented about 35 years ago!. This is usually done with wiring linked targets - which is no problem for the application that you envisage, but wholly unattached is possible.
One method of several is to have 3 very small coils orthogonally positioned (one each in X Y Z axis relative to eg the wearers fingertip, and have the space of interest scanned by 3 orthogonal fields.
Mounting two coil sets a small distance apart (see example below) allows position and full orientation to be determined.
Here is an example - tracking a blow-fly in 3D space.
2 orthogonal coils of about 100 turns (using 10 um diameter wire !!!) of 2mm diameter or less are mounted on a blow fly and tracked in 3D space. They use a halter with wired connection without impeding the blowfly's flight significantly.
Position accuracy of 1mm with a 1 kHz update rate was achieved.
Photos from paper here:
This is a related follow-up paper.
Not content with the stunning initial result they mounted a second coil triplet on the blowfly's head so they could tell where it was looking while in flight.
BLOWFLY FLIGHT AND OPTIC FLOW II. HEAD MOVEMENTS DURING FLIGHT
Usefully related.
AC magnetic field detection system applied to motion tracking
$\begingroup$
$\endgroup$
This was mentioned in a couple comments but I think it deserves an answer, the Meta Quest is already doing this quite nicely.
It identifies the location of your hands without controllers and brings them into VR allowing you to see them and use them to interact with other objects. It's all done with the existing tracking cameras built into the headset, no external devices required.
There are two levels of hand usage now, used to be you could only use gestures to select and move items and access a few menu buttons--but a new update has allowed you to actually see your hands and fingers represented in their actual positions, this is only supported in a few apps right now. Netiher mode is really great for playing games because they don't have buttons or joysticks but this should improve over time as better gestures are developed.
To take it one step further, some keyboards can communicate with the quest and can be brought into VR. Combining this with the ability to see your hands is supposed to let you type quite well with the headset on.
The one issue I've heard of is when you place one hand in front of the other, the headset may lose track of the blocked hand because the only cameras are in the headset.
There are also haptic feedback gloves coming to market, when you touch your hand on an object it should feel like the object pushes back against you, or when you try to squeeze something it can prevent your fingers from closing. I even saw a demo of one that used low level electric shocks to directly manipulate your muscles as well giving you the feeling of something pushing back.
The exact same technology should work for holograms (They are nothing more than VR projections wihtout a headset--the interactions would be identical). The only issues I can imagine are hardware design, like projecting into a space where your light could be blocked by a hand.
$\begingroup$
$\endgroup$
In current technology a computer generated hologram would be able to monitor it's own output using a stereoscopic camera or by using unseen infrared or UV light bar pattern sweeps of the region looking for fingers approaching or interacting with the display region.
Imagine something like a current technology 3D scanner such as an Einscan combined with a holographic projection. One can use pre-exposed and predrawn holographic images currently and simply choose which ones to display in reaction to the input.
Making it seem real and less fake would require lots of pre drawn and exposed holographic film. If there is a currently available computer generated hologram display then one can add the scanning feature to it and you have only the limits of software and your imagination. One should also have a complete lack of any worry about contributing nothing of worth to society.
Sliding displays like in The Minority Report would be annoying because there would be someone next to you on the bus with their tablet gesticulating wildly, and when they elbow you in the chest you will wish you never wished for interactive holograms.
One can simply don 3D goggles for computer generated virtual reality, or transparent virtual heads up display glasses or direct to the eye projectors and be presented with mixed reality/virtual reality, the requirement to be able to manipulate the results only requires that is be able to scan the motions of your hands instead of measuring the motions of the hands by wearing vr gloves.
I would Not want to be continually driving over freshly fallen people wearing VR goggles whose lagging internet connection caused them to miss the curb and fall into the street. Commute times are already long enough. I've nearly hit several people when I borrowed my wife's prius IV and those people simply had earbuds in and were peering too intently at their smartphones. There should also be a small and separate fenced-in area in parks for people larping with sticks and swords, pretending to be dogs, wearing fursuits, using VR goggles, or ALL of the above. A place well apart from the rest of humanity who can then use the park like me- jogging and carrying a large dual cassette boombox.
