In short: 3D, immersive version of a mouse needed. How does it work?

Call it whatever you want, Virtual Reality, Mixed Reality, Augmented Reality. They're all slightly different, and yet they share the same immersive quality and the 3D feel (and perhaps make use of things such as redirected walking (where a 90 degree turn appears to turn you 80 degrees) and redirected touching (look it up, it's pretty cool).

Now, what this nice immersive place is missing is a good mouse. Yes, the clicking, annoying thing that we use everyday, and what I inevitably go back to after trying and failing to do any sort of precise design work on a tablet with my short and stubby Trump-like fingers.

It's what graduated us from consoles to using a 2D GUI. Now, all of my futureworld stories (past 2020 or so) just HAVE to feature AR (my favorite flavor of mixed-virtual world). Currently, I think most 3D design is done in CAD programs, with folks staring at 2d screens and clicking and typing away. Seems like we're missing something, doesn't it?

Specifically, I want to have a 12 year old girl, let's call her Alice, summoning into existence from scratch (and a few libraries) an entire (populated) palace, with the environment half-listening to her (mouse-like-object)-given directions, half using deep neural nets to conjure up the rest based on these clues and Alice's past history in the environment.

To review, I want my story to feature this 3D-virtuality-optimized-mouse-like-object(?) - I put object with a question mark because I am open to the idea of there being no physically interactive artifact, but something else altogether. It has to meet a few requirements:

  • it has to accept user input (duh).
  • it has to enable commands to start, flow through a potential forest of paths and end, clearly and intuitively.
  • a 12 year old should be able to use it safely.
  • it has to work in a 3d environment, so allow for more degrees of freedom across a wider angle-set that anything we would do with a regular screen/mouse combo.
  • it should involve not much more work/strain than using a mouse (i.e. it should not involve doing cartwheels).
  • 1
    $\begingroup$ Although the current answerers are right in that we might not use a physical controller in the future, a current example is the HTC Vive's controller. Here's an example of someone using them to build something, to show one way things like menus and 3d selection are currently implemented. $\endgroup$ Commented Apr 22, 2016 at 1:35
  • 1
    $\begingroup$ Why wouldn't your hands work properly for this? Have a designated action for bringing up a "menu" persay, and then it just works like a touch screen on a tablet. $\endgroup$
    – Aify
    Commented Apr 22, 2016 at 5:25
  • $\begingroup$ Have you read the Mortality Doctrine series by James Dashner? It delves into great depth with virtual reality, AI's, and what you are asking about, $\endgroup$ Commented Apr 22, 2016 at 21:40

4 Answers 4


Interaction is in the eyes of the beholder

... and in the brainwaves too.

There is no better pointer than the focus of the user's eyes. The old-fashioned way would be by tracking eye movement, the modern one takes signals directly from the brain through the Datajack equivalent in your world.

Interaction is done by brainwave activity. You think about the object and what you want to happen to it and your brainwave interpreter (a neural network application specifically trained to your brain over many years) translates it to commands the AR system understands.

As an example: Alice is building a tower for her palace. She looks at the empty space where the wall she just formed ends and thinks of a square tower that looks just like the wall. The interpreter recognizes the intent to create a new object from the focus on an empty space, the cube shape and finds a match between what is imagined and what is seen (i.e. the wall's color, texture, etc). The AR system then takes over and inserts a complete tower with windows, doors, stairs, etc, as mashed together from a humongous library of objects.

The tower is not quite too taste, so in quick succession, Alice looks at the windows, thinks a gothic arch shape into them, removes all doors by looking at them and thinking blank wall, then adds banners hanging from the sides.

Finally she adds another floor to the tower by focusing on the middle floor and then imagining the tower cut in two just above it, then the entire section again.

  • $\begingroup$ Neat, I love it, but would a 12 year old be able to use it -- I know it currently takes years to train interpreters to read the most basic of brain signals, never mind deciphering complex thoughts like "put a moderately flowery Ionian set of 12 columns over there." $\endgroup$ Commented Apr 21, 2016 at 22:26
  • $\begingroup$ It'd be a bit ambitious for 2020, that's true. But 2030? I'd expect that by then we have enough computing power. Still, a fully trained personal interpreter would be an incredibly valuable possession and probably make the difference between the Gods of AR and the blind fools struggling to hit the "Send" button. Getting a good one for your child could be more important than finding a good school... $\endgroup$
    – Cyrus
    Commented Apr 21, 2016 at 22:31
  • $\begingroup$ How will you deal with misclicks? $\endgroup$
    – enkryptor
    Commented Apr 22, 2016 at 9:37

Your own hands.

Why use a separate peripheral at all? Using motion tracking (perhaps involving a glove with inertial sensors, IR LEDs, or just reflective spots, depending on just how advanced your tech is), your hands become a very intuitive, extraordinarily powerful HID (Human Input Device, the computer jargon for any device that converts human input into computer signals, including mouse and keyboard). By twisting your wrist you can easily flick things around or invoke menus. The analogue to "clicking" could be touching a fingertip to your thumb, thus raising the potential number of "mouse buttons" from 2-3 to 4. Per hand!

In fact, the XBox 360's Kinect already uses your hands, in combination with arm gestures, such as standing there and mimicking the letter 'Y' to access the system menu from within any game. It's trivial, given a little time and effort, to go from there to a more precise tracking system that can accurately follow your individual fingers, and do so with greater precision than the now almost decade-old Kinect.

As alluded to already, using your hands, and given a system with enough precision to track fingers, gives you a lot of power. Certain hand gestures can invoke different functions. There's really no limit to how involved these could be: Maybe a single hand gesture, such as the Vulcan salute, accesses a function, or maybe it's a quick series of gestures akin to a fraternity's secret handshake (and analogous to keyboard command sequences like Ctrl-C or Ctrl-Shift-V).

Combine these hand gestures with voice input, and you have yourself a ridiculously powerful human input system!

But can a 12-year-old use a system like this?

Absolutely! I'm a 30-something programmer-cum-systems administrator. I've been a computer nerd since I was 6, and I've been programming them since I was 8. My 4th-grade Science Fair project was a computer program I wrote to test one's knowledge of molecules; when I was 12, I was creating adventure games in HyperCard. And you know what? I've seen today's 12-year-olds leave me in the dust, using keyboard shortcuts I didn't even know existed to accomplish in a fraction of the time what my practiced mousing would otherwise take!

So 12-year-olds certainly have the faculties to use complex systems, and they have the coordination as well.

That being said, a complex system like this would certainly have different "modes" for the user. Similar to how many programs today hide a lot of configuration options behind a "I'm an advanced user" setting, newcomers to the system could use the "basic" controls, which for many operations would require opening and navigating menus; as users gain skill, they can enable progressively more advanced command sets, gaining with each one a new set of gesture-based shortcuts. By progressively opening up new options like this, you significantly reduce the risk of accidentally invoking the wrong command with a careless gesture, while learning the system in a more deliberately-paced rate.

And, of course, just like today's advanced rendering software, you'd always have the option to undo the last few actions you've done -- so even if you accidentally erase an entire family of cute fluffy little bunnies, there's no need to cry; just hit the Ctrl-Z equivalent, and POOF! bunnies are back!

  • $\begingroup$ Well, I like the idea, but I don't want Alice to accidentally trigger menus and erase the occasional rabbit or peasant (she'd be heart-broken). $\endgroup$ Commented Apr 21, 2016 at 22:23
  • $\begingroup$ Well, no solution is 100% error proof. So you need to have saveguards in any case. $\endgroup$
    – user6415
    Commented Apr 22, 2016 at 10:12
  • $\begingroup$ @SerbanTanasa I've edited my answer to more fully address the question of whether or not a 12-year-old could use an advanced system like this. $\endgroup$
    – Kromey
    Commented Apr 22, 2016 at 16:29

We currently have things like that. There are 3D 'mice' that sit on a desktop and allow panning and rotating an object in 3 dimensions, such as the SpaceMouse. They are used in combination with a regular mouse.

However, these mice chain you to a desktop, and upcoming AR technologies such as the MS HoloLens headset allow the user to walk about and manipulate virtual objects using their hands.

The problem with hand-motion-capture is that the user must keep their hands and the object being manipulated in the device's field of view, and the user can't feel what they are doing. However, these limitations can be overcome with a device like the Novint Xio, which attaches to the arm and both registers the user's motions directly and provides haptic feedback.

Since the Novint Xio is intended as a gaming device, there shouldn't be any problems with teaching a twelve-year-old how to use one, provided that it fits.

Of course, this is early days for AR/VR, and there are many devices already in development, such as the teslasuit, that are touted to be able to provide full-body haptic feedback and motion capture.

Additional degrees of freedom in the capabilities of the control/feedback suit of the VR/AR environment could be a combination of specific gestures and spoken commands.

Alice wouldn't have to worry about accidentally deleting a bunny that she'd created, since to do so, she'd have to grab the rabbit, open the recycle bin and toss it in, and even then, she'd have the option of pulling it back out if she changed her mind, as long as she hadn't emptied the trash, of course.


The Power Glove's great-great-grandchild.

You pull it on and it fits to your elbow. The position of fingers and thumb can be used to conjour menus and there are buttons and screens running up the length of the glove (either holographic, AR or maybe plain-old flexible OLED touchscreens) that allow you to change the function of your hand gestures and to read information on the library you're accessing.

Add a micro-computer (or a nanotech/femtotech processor) and the gauntlet (gauntlet sounds so much cooler than glove) can even have a limited AI personality to advise Alice on her actions.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .