How would a user interact with a direct neural interface?

Question

Assume humanity has developed a cybernetic implant technology which allowed humans to be "advanced". The technology would be surgically implanted and provide a range of functionality (I'm not interested in cybernetics to replace damaged limbs/senses which the brain already has 'wiring' for).

An example would be an implant which allowed the user to have a voice call with a friend. How would the user:

• Initiate the call
• Receive feedback on successful connections/rejected or failed calls
• Speak to their friend

My initial reaction was that this sort of technology would be accessed via your inner monologue... instead of saying "Hey Siri" or "Ok Glass" the user would think it. Is this likely or is it more realistic that the brain would learn these devices much like we learn to walk and talk? Are there other options?

Answer 1

Using such an implant successfully would require a lot of training. When you make a phonecall with a mental command, it becomes hard to differentiate between thinking about calling someone and actually doing it.

Using uncommon pictures to control the actions of the implant might not be a good idea either. It is just too easy that mental stimuli from your environment trigger involuntary actions. Let's say your trigger for making a call to your boss is thinking of a pink elephant. See, you just thought about one and made an involuntary call.

A possible safeguard could be when the implant tells the user that they triggered an action and then monitors the mental reaction to the announcement before executing it. It should be easy to detect a stress reaction when announcing an action the user didn't actually want to do and then cancel it. The downside would be that it might make it difficult to get the implant to perform actions you want to perform even though they cause stress, like calling your boss to tell them you made a bad mistake or asking someone out to a first date.

And then during the telepathic conversation it might be hard for the implant to differentiate between those thoughts which are intended to be sent and those which are meant to be private. The conversation could become a lot more honest than intended. I will sell you this part for just 2000 credits. Heh, you idiot will sure fall for that. I am so glad you don't know that this piece of junk is not even worth 100. Wait, did I just think that or telethink that? Why do you want to strangle me?

However, we are perfectly mentally capable of making this differentiation when it comes to controlling our physical bodies. We can think about moving an arm without actually doing so. That's because we learned how to send signals from our brain to our limbs. This is a skill we were not born with. A baby is born with absolutely no motor control. It takes months until we learn even the most basic controls of our own body and years until we master it.

Getting a neural implant which adds new abilities to the body would be like getting a new bodypart. The user would have to learn how to utilize it, just like a toddler needs to learn how to utilize its arms and legs. Unfortunately an adult brain learns far more slowly than that of a toddler. It might be very difficult for most adults to learn how to use it. The situation is similar to that of people who lose their motor control due to brain damage. Some people are able to relearn how to use their limbs, but not all are successful and very few get close to the level of control they had before their accident.

On the other hand, when the implant is implanted during early childhood while the motoric abilities of the human develop, using it might become just as intuitive as any other bodypart.

Just like we currently have the generation of the "digital natives" who learned how to intuitively use the Internet during their childhood, your world might experience a generation of "neural natives" who intuitively learned how to control neural implants.

Answer 2

Just like you can lift your arm by simply willing your arm to lift, there is no need to come up with obscure real world things like pink elephants that you are to think about when you want to e.g. place a call.

It will take some training, but eventually you would just be willing the call to be made. Not "thinking about calling" and definitely not about pink elephants.

In the same way, it would be possible to easily have a electro-telepathic conversation with someone while simultaneously thinking about that conversation, just like you can control your voice separately from your thoughts. Instead of saying words you just have to learn how to send them. This will probably take even more training, but there is no reason why it shouldn't be possible.

Answer 3

Each person would develop their own way of using the interface

Each person's brain develops differently. That is one of the joys of humanity. However, we have to interact with each other, so clearly we need to develop a common ground. We develop language for this. However language is not fixed in stone, and that is what I base my answer around.

The closer one gets to the "core" of an individual, the more customized and fluid language becomes. This is because, as you get closer to an individual, you relate to them more, and have more common ground to work from. Consider how you would ask someone to take out the trash for you, given different relationships:

• Stranger: Hey, how are you doing? Look, can you do me a HUGE favor? I'm running really late for a big meeting at our city hall, and I forgot to take the trash out. If I don't get the trash out, my wife will kill him. You're married right? You understand? Do you think you could help me out and take the trash out? Thank you!
• Friend: Blast! I forgot the trash today. Do you think you could take it out for me? I'd owe you big time!
• Close Friend: Hey, think you could take the trash out for me?
• My Wife: Before I can possibly forget the trash, she throws just one glance at me, to remind me that I need to take the trash out right away. This glance is different from the "time to clean the catbox" glance and very different from the "take the trash out at halftime" glance. I wonder if I can pretend I saw the halftime glance instead. Nope, she saw my glance, and responded with another one... that glance was mean... I better take it out real quick, and buy her flowers tomorrow. Yep, that glance says its a good plan.

It is reasonable to extend this to how we handle our muscles. We develop a decidedly personal language with which to express what we want the muscles of our body to do. It is part of why personal training after an injury is so hard: our muscles and nerves are no longer speaking the same language they used to. We have to develop a new language.

It is also worth noting that we speak the language of our body so fluently, and there are so few misinterpretations, that we often even forget that it is reasonable to model our mind-body interactions this way. We usually get surprised when someone does something to our body to disrupt this communication (like nerve pinches).

Accordingly, each and every neural implant owner will develop their own personalized way of interacting. However, there will be standard ways of learning.

An infant learns to handle its muscles the hard way: trial and error. A child learns to speak their native tongue the hard way: trial and error. However, when learning a second language, whether its a second spoken language, or a different way to handle muscles (say, how to swing a hammer), the rules change. We can start by describing what we want to do in our native tongue, then slowly translating that into the second language. At some point, we find we are no longer translating... we have indeed learned a new language.

This occurs with people learning martial arts, or yoga all the time. They try to do the forms using the body-language they know, until one day someone shows them there is a new muscle that they'd had the entire time and never used it to its full potential. At that time, the practitioner has to begin learning a new language!

How can we adapt this to neural interfaces? If the neural interface is implanted from birth, we will learn to use it intuitively, like a muscle. Calling someone will not have an image, it will simply be a decision to call someone with exactly the same flair as the decision to pick up a glass of water.

If the neural interface is implanted later, we have to learn to use it. This could be done in the infant style, but it isn't as efficient. We're smarter than that. What companies will do instead is provide common visuals for calling. They might be visual or auditory. They might even be the physical act of mimicking picking up a phone. Anything which allows the brain to have the control needed to make a call is sufficient.

After that, the plasticity of the brain will take over, hopefully aided by a plastic neural interface. The brain will begin developing a new language to communicate, exactly how we transition from total strangers, to friends, and for some, husband and wife. Someone whose initial interface calls for visualizing the phone number or IP address of a person to make a call, will watch it shift to something more organic: perhaps the name of the individual. Eventually it will be the face of the individual. Eventually they will just think about calling the individual, and the neural link will respond.

There will be an interesting phase where it gets hard to explain the visualization you are using. Someone may claim "I picture their face, and it calls for me," but when faced with the puzzler, "if you pictured someone's face right now, without wanting to call them, would it call anyways?" the answer would be a very frustrated "no, it wouldn't, but I have no idea why... it just wouldn't. It knows better." The visual would be only part of the communication... the rest would be in the tiny subtleties the brain develops (the equivalent of the married couple's glance communication).

Note: The more complicated the interface (your 3 element system is nice and simple), the more complicated the training would need to be to gain control quickly. For something very difficult, like a neural link to control a powered exoskeleton, there might actually be schools to teach you how to learn to talk with the implant (similar to physical training classes to teach you to use your muscles again).

Note: such neural interfaces would likely aim to be as plastic as the brain, because its much faster to develop a language between two plastic things than it is to develop it all on the brain side. This has huge positive implications when talking about dealing with viruses and other highly negative aspects of neural interfaces in science fiction.

Answer 4

There is loads of good and bad cyberpunk stories which try to imagine how would such implant work. I will try to invent my own idea:

When you think about something specific, your brain creates specific "pattern" which we are able to read even with today's tech. So it would be plausible to assume that such technology will get miniaturized good enough to be implanted into your brain:

To accept the call, think of smell of grass after rain

To decline call, think of having cold shower

To place call, think of ... shiny phone? :)

And so on.

Also, some implants work on the base, that you "just hear naturally" (to some degree of quality of hearing), so another plausible idea is, that once you accept/place the call, you just start naturally talking.

The selection of your friends then could be connected with another vision nerve implant, where you could select your friends just by looking at them. Something like Google glass interface, but internalized

Answer 5

At present, any complex devices that we use rely upon having a user interface that we perceive through our existing senses and manipulate through our existing appendages. Learning to use these devices requires training - in many cases, not much training is required since we already know many of the pre-requisite skills.

However, when we are talking about a direct neural interface, we could consider the brain to be both a computer and a collection of software. I would anticipate that in the early stages of neural interfacing technology, output from the device would be overlaid onto our existing senses, and input would piggyback on our existing motor centres or tap into the internal stream of consciousness and listen for commands. Think of this as training a robot.

However, I would also anticipate that in a technologically mature neural interface, this piggy-backing onto existing senses and brain centres would no longer occur. Instead, it would be as if an entirely new limb or organ was grafted onto the body, along with the skills to use it. Think of this as actually reprogramming the robot's software to include new features.

To use the example of an implanted mobile phone:

In an early neural interface, the phone would be implanted and hooked into the brain. When activated, it would display visual output in the user's field of view, and audio output would be heard as sound. It would be controlled by thinking about it and then issuing commands. Communicating using it would require subvocalisation. It would require some training to use such a device.

In a mature neural interface (henceforth, a Mature-NI), as soon as the device was attached, the user would know how to use it, as if it had always been there. It would have no visual or audible output, nor would it require a subvocalized input. Instead, the user would simply send a message to the intended recipient, as naturally as speaking, gesturing or thinking, and would just know if the message was received, ignored, or not received. These messages could be words, images, scents, tastes, other sensory data, or even emotions. Messages from others would simply arrive unless the user was ignoring that person, and the user would just know who had sent it.

Such a device would probably allow communication with more than one other person, and may allow multiple conversations to take place simultaneously, including several "communicated" and one vocal conversation. If more senders were attempting to contact the user than the device could handle, the user would probably know who wanted their attention (or the most important 'who's if there were a lot), just not what they wanted. It would probably become common to switch rapidly from one caller to another just as if they were all standing in the same room talking about different things.

If a Mature-NI device was removed, the user may remember using it, but would not be able to say how they did so. In effect, all the processing power involved in manipulating the UI would have been located on the device, not in the user's brain. A similar effect could be gained if a person's left arm was amputated, and all the neurons in the brain that had anything to do with moving or sensing the left arm were also removed. It would be as if the left arm had never existed - the user would remember having one, and some skills would just not work any more because they relied on the left arm, but there would be no sense of something missing as is the case with simple limb amputation.

Conversely, adding a Mature-NI device would be like putting the amputee's left arm and associated brain tissue back just as it was before it was removed - except that the first time such a device was added, there would be no incomplete skills that would work once again. If such a device was added and used, then removed and then replaced, the removal and replacement would be just like a person's arm and associated brain tissue being removed then replaced.

A mature neural interface would most likely be a device similar to a Neural Lace from Ian M Banks' Culture novels, which grows throughout an organic brain, interfacing with every neuron.

Answer 6

Good question! And I really like all the answers but want to add something.

With our physical body it is easy to distinguish between thinking and actually doing something because actually doing something takes much energy (comparing to just thinking of it), and in order to direct that energy along our nerves we have to apply focus.

This is, in particular, how all the learning to master our body is done. Everyone has to apply conscious efforts and keep up the focus to exploit certain neural circuitry to do things and to learn to do them well. As the skill is gained, the action itself may be delegated to the automatic control, but the initiation and allowance for the action still requires conscious commandment (in sober healthy individuals at least).

So, concerning such a neural interface, it would be logical to indeed make it feel like an extension of the body and interface with the brain in the same circuitry. It may not necessarily be a sensation of a limb, but what, for example, if in order to control such an interface, you'd have to focus on an object like 1 meter in front of you, or inside of you, in your imagination, and apply the correct conscious activation commands. I bet with such a technology, it'd be easy to distinguish whether the person did really guide the "movements" with his focus, or these movemetns are parts of involuntary imagination activity - the EM patterns will be different.

The present evidence that psychoactive drugs can induce these kinds of sensations and allow to control them (to some extent) makes sure that we are quite capable of that. The brains neuroplasticity will do.

It will be rather easy to learn to handle it. Not any harder than a car or an airplane. As we gain the initial control over our body in childhood, other types of such a control become increasingly easier to grasp. For example, a car driver feels the car as his body at the moment he drives. This comes to no surprise to us and we also know, that the basic/average car driving skill can be acquired relatively easily.

To summarize this, one could control such a device with his focus of attention and persistance. This cannot usually result in involuntary device operation since it requires energy redistribution under a conscious effort and a mentally healthy person should have no problems with that. Caution must be excersised when considering allowance for such devices for people with OCD or Turrets Syndrome, though, a neurologist is required to tell if this will really interfere with such an interface (although I'm pretty sure OCD really might).

Answer 7

Rather than executing commands with your thoughts which while a cool concept has a whole host of complications and issues why not just use sub-dermal displays?

Think of it as a simple LED panel/touch sensor implanted under the skin. It could work like the wearables we are seeing come out now. Simply touch the sub-dermal implant to answer or disconnect, heck it could even be a watch. The whole apparatus could be invisible unless activated and aside from a power source, which I am thinking you could use some sort of body heat converter or something.

Answer 8

I'm going to assume the neural interface is quite advanced and sensitive enough to understand whatever you're thinking.

Have the person visualize doing the things that they want the neural interface to do: pick up a phone and see a friend (or walk up to him) to place a call, speak to him, set down the phone (or walk away from him) to hang up. The neural interface giving you information (including your friend's voice) could feed into your visual and auditory processing parts of your brain directly, so you see and hear things that aren't really there.

I think that actively visualizing these things is quite different from the random, internal chatter that I would not want the neural net to pick up on, so it seems like a plausible explanation. However, some people can't visualize things at all, so they might have to use alternative neural interface options or physical interfaces. These alternatives would likely be considered inferior to the way that "visualizers" can interact with their interfaces (if it weren't inferior, everyone would use it); these people might even be marginalized in society because of this, if you'd like your story to go that way.

Answer 9

There should be 2 modes: Train mode and work mode.

Step 1 - Training

The interface asks you to think about the number 1. After thinking about it for a few seconds, it goes on the next numbers 2,3,4... That way you train it to differ between a few numbers.

Step 2 - Working example

You are using a calculator. You allow the program to use the neural interface to input numbers. You think of 4. If the neural interface is trained well by you, it sends a 4 to the network. Otherwise it might send a wrong number to the program. You see a 3 on the screen. You tell your neural interface that this assumption was wrong, because you clearly thought of a 4, not a 3. Now it asks you the number that you thought of. You enter 4 and the interface saves a tuple (3,4) for later training processes (Step 1 again!), because it fails to disambiguate between those numbers.

Similar to a child learning words and numbers, you and the interface learn to communicate with each other. The interface will become very adapted to you. My example merely covered differentiating between small numbers. Sooner or later, the interface will understand you thinking of more complex words/tasks/numbers.

There even might be a pre-trained interface. A company might ask 1000 people to train it a specific set. But nonetheless you will have to train it too, because every human is different. At least such a preliminary training might save you, the customer, a lot of time.

Answer 10

I imagine such an interface would feel like an extra limb or two - say, an extra pair of hands. Or eyes. Or a whole new body, depending on how far you want to go.

The only real problem I see here, is where exactly these "limbs" will be "attached" to your physical body, especially without feeling like they'd bump into things around you. My guess here is going along a "fourth dimension" or something. (Which is what the whole internet thing is already, anyway.)

As for feedback from these extra "limbs", it'll be like moving your arms about with your eyes closed; you can still feel how the different joints are stretched out, and this will tell you where your hands are without you having to look. And this is before you get the skin's feedback as you start bumping into things.

Similar with these implants. The most sensible (to me) option, is to have them feel like arms/hands/fingers with a set number of joints, and so moving those joints would trigger events; kinda like moving your fingers across a keyboard. Get some proper practice with this, and only then do you assign which events go with each "keypress".

One advantage of this, is you can "key" and "mouse" indefinitely with these extra "limbs" without suffering (physical) fatigue or carpal tunnel or anything, while fiddling with that drawing/song/whatever.

Not that this would exclude composing by "thought-to-MIDI" or anything; tapping directly into the thought process sounds like a nice, quick way of writing/composing/etc, while the extra "limbs" deal with commands like "dial this number". These are two options, and it'll be possible to have both.

Answer 11

Note that such devices already exist, although still experimental, and capable only of a very narrow range of input.

You can use a device to guide a mouse cursor over the screen. Functioning prototypes already exist. This device actually makes EEG images and looks for patterns the user trained it about. It's still quite rudimentary, but its speed and accuracy might be improved in the future.

You have to train yourself to use it, and the system itself has to learn you. You train it (and yourself) by actually wanting the cursor to move in a direction, and not just by thinking about that direction. So by reading the word "left" somewhere and thinking about the direction "left" is not the same as you wanting the cursor to move to the left.

So it doesn't "understand" your thoughts, and it definitely doesn't parse your thoughts as if it was a text. This makes a full textual conversation impossible with our current technology, but if in the future it could be expanded to full textual parsing of your thought (for your example with the voice call), accidental calling could still be avoided with the above methods. You'll train the system to respond to your desire to make a call, so it won't respond to you reading the sentence "I want to make a call" or you just randomly thinking about it. This strong desire leads to completely different images in your EEG than just the thoughts themselves.

Answer 12

Similar to James' answer, but a little different.

Keep the neural interface but make the UI look and feel like a heads-up display with visualized and subvocalized commands.

Because it is all inside the brain it doesn't need actual optical feeds to the eyes. It just writes the visual information directly into the brain, or the optic nerve. Then the user might visualize a button lighting up, or mentally focus on part of the UI to bring up more details.

A subvocalized command would be like talking to himself inside his head, "Computer, call James." and it would open a phone connection which would act exactly like a cell phone except the audio input and the output are entirely in the user's mind.

Answer 13

ill try another perspective. since it's user interface. ill approach it from a user point of view.

I want the interface to properly filter between my thoughts and my dreams. I don't want to be dreaming I'm calling my boss and my phone calls him in the middle of the night.

I want a lock-in mode for communal devices like the TV. I don't want the channel to change when my 3 year old decides he wants to watch spongebob.

I want visual display of who is using a communal device or who last used it. I'm don't want to be blamed for leaving the kitchen lights on just because everyone knows I like the occasional midnight sandwich.

I don't want devices that automatically sync to my brainwaves without my permission. I don't want the pretty pharmacist to know the anti-itch cream is for me.