Portable Alternatives to Traditional Keyboard/Mouse Input

Question

In a near-future alternate reality where smartphones were never invented, people rely heavily on augmented reality and highly mobile computing. High-tech clothing and accessories are everywhere. Heads-up display in your motorcycle helmet, audio-transcribing glasses and text-to-voice speakers for the deaf and mute, and extra robotic appendages are ubiquitous. And don't forget shoes! Spring-loaded jumping, retractable wheels or ice cleats, and electromagnetic Wall Walkers^TM are all fair game!

The problem that needs solving is a matter of giving complex, potentially well-timed input. Sitting down at a screen, keyboard and mouse give you all you need, but what about when you're out and about? What if one hand, or neither, is available?

The meta-problem is that I'm looking for fun and flavorful solutions that would be viable enough, even if something dead-practical would realistically beat it out. Video game controllers, arm bracers with a menagerie of programmable buttons and knobs, gesture-sensing jewelry- flex your creative brain muscle and go wild! Obscure real-life tech is certainly welcome as well.

Answer 1

Touch-sensitive gloves with chorded typing input schemes

The user dons a glove which has sensors built in the fingertips. Then by touching two or more fingertips together for a specified time several different input combinations can be sent to the device in question. 20 or so signals are definitely possible with just one hand - adding timing constraints, touch-sensitive points on phalanges other than the fingertips, double taps or a second glove easily increases this to cover the whole alphabet.

The resulting input scheme is discreet (both when worn and when used), pretty much universal, quite robust and needs no new inventions. Some training and nifty UX design is certainly required for effective usage though.

Answer 2

Direct Neuro Input

You've kind of painted yourself into a corner of inevitability with this particular detail:

and extra robotic appendages are ubiquitous.

This fact means that the ability to hook a person's nervous system into a machine that can do stuff that our biological nervous system is not already designed to do has become ubiquitous. So just like you can give someone a 3rd robotic arm, there is nothing left preventing you from attaching a computer to a person's nervous system and allow it to respond in detail to our thoughts.

The reason I say this is inevitable when you reach this level of tech is because you no longer get any lag time or inaccuracy between your thought and your body's somatic response to that thought. You can literally "type" at the speed of thought (maybe even faster with some AI autocomplete features). You can perform pixel perfect interactions with your software because you don't need your crummy eyes and limited motor skills to tell the computer the exact path you want to draw.

Heck, you won't even need the idea of buttons or voice to speak because you will be able to just think about what your options are and make them happen. In this way, checking your email or doing a Google search is going to feel like remembering something you already know or a lucid dream where things you need are just kind of there when the need arises, and learning to do complex tasks that perform dozens of parallel operations could be as simple learning to control all the muscles in your hand to work together to hold your morning coffee.

The question then remains, not if this is the best tech for your civilization, but how to make it flashy:

Cyberjacking

Probably the most interesting version of direct Direct Neuro Input is cyberjacking. This is where you basically install a bunch of plugs into a person (usually depicted on the neck or head somewhere) and you literally plug in. It creates a visually interesting interface even if you can't see what is happening on the inside of the person's head.

The problem of course with cyberjacking is that so much tech is wireless these days that it raises the question about why you would not just have an internal wireless device installed in you. So if go this route, I'd suggest some deep cultural problem should exist for not going wireless like wide spread brain hacking or oversaturation of wireless frequencies or needing data connections that are too complex and data intensive for wireless to be practical.

Non-Perceptual Experiences

This may be hard to pull off in certain mediums, but if this is for a written story or you have a narrator/inner voice, then the lack of sights and sounds can become interesting unto itself. Explaining how your interactions with your computer are literally thoughts, you could make things very interesting.

Responsive Interfaces

One of the biggest software interface trends in the past 10 years has been the idea of responsive design. The way a program looks on a phone can be very different than on a touch screen tablet, which is also very different than for a PC. Likewise, software in this world may need to be responsive to thinking styles.

Sometimes people think in words, sometimes in vague concepts, sometimes in pictures, and sometimes our thoughts are purely subconscious or instinctive... so the user experience of a program might actually adapt to a person's mentality which could lead to some very interesting experiences like the software changing how it looks, feels, sounds, and responds along with a person's emotional state or level of focus. It also means that two people using the same program may have 2 very different experiences.

Behavioral/Cultural Anomalies

Ever see a musician listen to a song they like and they mime out the song in the air like they are playing it? Well people who were born before neuro-jacking was commonplace may do the same thing. Your elderly may still mime out mouse and keyboard or cellphone gestures while controlling thier computers, while younger people may just stand with their hands to thier side without any bias that thier hands are needed to do anything. This will become an "old folk behavior" that may get ageistly mocked. "Ugh! grandpa, you don't need to envision a keyboard to message people."

You may also notice something similar to the echo chamber effect where people who think one way will get software that adapts to thier thought style which then reinforces that thinking style for them. Visual thinkers become radical visual thinkers, verbal thinkers become radical verbal thinkers, etc. This could cause a general increase in neurodivergence or intensify certain psychological disorders.

Answer 3

You already have AR and voice recognition. Throw in gesture control like in the movie "The Minority Report" and you have no need for a keyboard and mouse.

https://www.youtube.com/watch?v=-gjU42OSFgM

The only extra you need is a personal AI assistant so you can tell the computer what you want in plain English and it can work it out and just do it for you.

Think JARVIS from Ironman

If you want next level, then you need implants. Elon's Neuralink is looing to implant a chip in a person's brain allowing a direct computer interface so the blind can see or the paralysed to walk. You also have the possibility of the computer being able to read minds. See Mind Reading AI

Answer 4

Unless you directly neuro-jack the person, what you're looking for is a set of muscles on a human that can be monitored and translated into computing instructions.

The easy way to go about this is to monitor the flex state of the muscles in the hands. Keyboards basically do that indirectly by detecting where the fingertips are, but that part isn't technically necessary.

Another incredibly agile set of muscles exist in the human tongue. We don't think of it that way, but that's what makes human speech possible. Again, if you can monitor the flexure of those muscles, you don't need to actually say anything.

Beyond that, you're looking at using a set of muscles in a way that we aren't in the habit of using them, and you might not even have all of the muscle control in all of the people. A trivial example would be facial muscles, where some people can raise an eyebrow independently, but not necessarily the other, or some of us can wiggle our ears or nose. You'd have to find a set of universally available facial expressions to map your input to, but it might be worth it in an emergency.

Our eye muscles are excellent mouse/joystick inputs, with blinks effective as clicks, but that requires us to stop using our eyes for looking at non-input things. Anyone who has tried to keep up with action flicks with subtitles can tell you the flaws in that.

From there, you can get creative. We've theorized controlling battle suits with full-body muscular monitoring, of course, but you could use the roll of your shoulders to control the throttle of a vehicle while you used your hands for gunnery. With 600 muscles in the human body, you have a lot of options.

Answer 5

Heads-up display in your motorcycle helmet, audio-transcribing glasses and text-to-voice speakers

You have part of your answer here already, just don't limit those devices to deaf or mute people.

You don't need a keyboard when using eyes, voice, gestures you can give commands to an interface which is made in such a way that can gather and understand those input. You have probably heard of Siri or Alexa, they get instructions without needing a keyboard. And augmented reality devices like the Hololens allow the users to interact with a virtual console, without needing a keyboard.

Answer 6

Retinal cursor

Similar I believe to the system used by some sufferers of ALS - using your Eye movement as the tracking point, then using facial and hand gestures to provide additional inputs.

Answer 7

Now I have an unorthodox option. (Main idea is not invented myself, it comes from a rather controversial Russian Sci-Fi author, who thinks that communism married with radio would save the planet.)

Morse code

Yes, this old thing with dots and dashes and a single key input. (Improvements and better versions, based on Huffman code or similar might be also used, but for the non-specialists it's the same idea.)

The point is that a trained operator can both transmit and receive Morse code at a quite high speed. Basically, everything else, barring the real keyboard (unsure, even, in this case) or a direct neural link (too far in the future), is slower.

Summarizing,

• Morse code a simple and elegant solution;
• it can be worn concealed or made into a glove or a clothing part;
• Morse code offers high speeds at low costs and technical development level;
• Morse code requires training, but radio operators were taught in quite the masses before, so it's doable.

What to do with input?

But, you would say, any text input is just text. What with Minority Report styled AR (as mentioned in above answers) or with commands to the OS / applications?

Well, the good old UNIX-style console does wonders today and I am not convinced it could not continue doing wonders tomorrow.

So, now following on the Morse code idea, it's the audible interface (might be Morse as well) for the output and Morse key for input in a dialogue mode with a typical command-line UNIX shell.

Answer 8

Neural implants

It might seem like sci-fi, but we can actually do a lot with the brain. Iirc a study with EEG measured the electrical activity in the motor cortex. For example, the movement of the arm. This was then translated to movement of a mouse on a screen.

The most important discoveries were that over time people got accurate control over the mouse. It was nearly as easy as controlling the arm. If this is extended to daily use it might be as easy or easier. Another was that where they initially needed to move the arm for effect, at a certain point the movement of the mouse and movement of the arm were separated. So activity in the motor area of the arm would not result in movement of the arm and only the moyse. The other way around, mouse not moving but arm is, would require more sophisticated software.

We can extrapolate this to a plausible technology in the future. With neural implants directly in/on the brain we get a much more detailed picture of the brain activity. The brain activity is measured and certain learned patterns are used for software control purposes, which is translated to the action you want. From zooming out and changing your destination on your motorcycle helmet, to controlling both your own and a robot arm simultaneously, to controlling a quadcopter drone with a grappling hook, or your spring loaded shoes.

Humans have shown a great ability to learn these things with extra translation steps. Fingers moving controller buttons and sticks. You've essentially removed the middle man, or finger.

Keep in mind that the motor cortex was probably chosen for the easy access (lots is close to the surface) and the easy human control over the patterns inside these areas. You might be able to add it to other parts of the brain as well, which can learn the required inputs. Also keep in mind that even to this date the true neural implants are horrible spiked things that destroy brain matter when implanted. It would be a good thing to solve that before subjecting people to mass neural damage.

Answer 9

Brainscanner cap

For the input devices, most users prefer as immediate experience as possible. The less delay and more intuitive interface, the better. A direct mind-machine interface is a common sci-fi option. There are, however, valid objections against invasive data jacks, that plug directly into user's neural system. Many people are reluctant to permanently modify their bodies. A surgery is needed, whenever the hardware breaks. And imagine forgetting to seal your data jack before jumping into a swimming pool!

If you want a safe solution, that doesn't make your people look like cyborgs, try a brainscanner cap. Electrodes on the inside monitor user's brain activity, while sophisticated electronics translate those signals to a stream of data, commonly known as "intrapersonal communication" or "inner voice". First use requires a fair amount of calibration, but then the device remembers its user's mental profile.

Of course the cap shouldn't record everything the user thinks - only the parts, that the user wants to be the device input. So to start typing, the user has to think some rare keyword. Once the cap finds that word in the captured inner voice, it starts recording. When the keyword appears again, the recording is stopped.

The cap can also accept some preset instructions to figure out, what to do with the text. For example: "...hey, this is the street, where Mike lives - let's pay him a quick visit TRANSCRIBE START Mike, are you at home? TRANSCRIBE END SEND TO MIKE SMITH MACRO END oh he really should repaint his fence, this green is getting so..."

Answer 10

arm bracers with a menagerie of programmable buttons and knobs

Obscure real-life tech

On aircraft carrier flight decks, large drones such as the Boeing MQ-25 refueling tanker are operated by deck operators using a "Control Display Unit".

Answer 11

A modification of @zovits's answer. The gloves + AR could cover the mouse input, but the keyboard is missing.

Lean into AR and add binary

You can count in binary on your fingers, by either having fingers straight vs bent or the finger tip touching vs not touching something (like your on palm). Using gloves with sensors in the fingers (or hand trackers) create combination of binary. This gives you 1024 possible combinations. ASCII tables only use 128 values (some of which aren't characters). What about the other 896? Macros.

Macros are used to streamline or automate tasks that have to be done over and over. Want to open your email. Don't open it by "clicking" your browser in AR, get to your email and login. Just program a weird, 8 finger combination that does all that instantly. These macros can be context specific too. If your your phone app is open an entire block of macros are your speed dial options.

Answer 12

A natural addition to your existing tech stack would be some sort of projected/laser keyboard. They project the image of a keyboard (or other interface) onto a surface and track your hands as you "press" the buttons. They would have a number of advantages:

• Having heads-up displays means you already have the tech for projecting an interface, just add an infrared or optical sensor to watch for "keystrokes".
• A piece of equipment can project an interface when needed, or a user can have a body-mounted projector that wirelessly connects to nearby equipment.
• The user interface is flexible and dynamic. Switch between a standard keyboard, a series of knobs and sliders, or whatever else makes sense for the use case at hand.
• For extra security and fanciness, the interface could be projected inside the user's smart glasses so that only they can see what's being typed.

Answer 13

Contact lenses with augmented reality. If you place hands in a position for text input, AR keyboard appears under them, if you are grabbing an imaginary mouse in the air, AR makes one for you.

Also works with glasses, audio-transcribing or not and heads-up displays in motorcycle helmets.

Answer 14

Rings on every segment of each finger, with a sensor on the thumb. Gives you three Glyphs per Finger, makes 12 Glyphs per Hand. Sort by usage, meaning EIAOU go on the fingertips.

Now the 2nd hand as alternator. Thumb to first Fingertip the other 2 letters. Thumb to to first mid. Big Letters. And so on and so forth. Primary Hand is the letter chooser. 2ndary hand is the alternator.

Alternatively, all fingers selectors typing away at water surface is at armslength down the body, most likely thighs. Then pants with touch markings to help find the keys.