Can people learn precise hand-eye coordination for prosthetic or external mechanical limbs that don't have normal proportions?

For example, someone is controlling a small mech via a suit or exoskeleton that takes in their real body movements (like pacific rim but one person and smaller). But that mech has very long hefty arms compared to its legs, like a gorilla. The controls would still work intuitively so long as it had the same joints as a human, but would the pilot be able to learn proficient robohand-to-eye coordination in a reasonable time frame (at most a few years of deliberate training)?

Or say a prosthetic controlled neurally with the same situation, a very large/long arm (given the rest of the body could support it). Could they become as proficient with it as their lost real arm, despite its proportion and size difference?

If so, how long would it realistically take? Would any special training methods work faster? Would it be practical?

New contributor
Silent Bastion is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • 2
    $\begingroup$ Children seem to do just fine as they grow from 2 feet to 6 feet. $\endgroup$ – DKNguyen 2 days ago
  • 2
    $\begingroup$ The answer to "how long" and "special training methods" would depend on whether the link is mechanical or neural, and if neural, whether proprioception is available. $\endgroup$ – Alexander 2 days ago
  • 4
    $\begingroup$ Incidentally, the difficulty of learning to use limbs of new proportions is why pubescent teenagers are stereotypically clumsy. $\endgroup$ – Nuclear Hoagie 2 days ago
  • 3
    $\begingroup$ Our brains are reprogrammable. In one experiment, people were fitted with glasses that reversed their vision (I think up to down). It took about a month before they were suddenly fine with it. Afterwards, it took the same amount of time until they were able to work without the glasses. $\endgroup$ – NomadMaker 2 days ago
  • 1
    $\begingroup$ @DKNguyen It was a common stereotype when I was growing up - a recent Italian study has confirmed some truth to this. See this BBC article $\endgroup$ – Alan 2 days ago

A month to a year

The human mind is incredibly flexible. Via association we can learn a great deal. When you start riding a bike or car you can see this very well. Moving your arms and legs in certain ways creates desired movement. We haven't evolved with these movements, yet we can learn them. Even better is controlling a computer for example. Moving your fingers on a keyboard or a mouse makes the computer do stuff. You can look around in a computer game, or type and select whatever you want in a word document. Finally, you can learn to fly an FPV drone. You put glasses on and get feedback of the camera. Moving your hands you can fly any direction you want. At the start it might be difficult, but with extensive training you'll be able to control it very well in a month, or for slow learners about six. These show that you can learn new abilities and movement by moving your body in a certain way in specific situations.

Even more impressive is doing something that flips it all upside down and left side right, going against what you've learned. A man made special glasses that flip and mirror the light coming in. After three months he was able to drive a car without problems. Interestingly he had trouble adjusting when he removed the glasses. The more he started switching, the easier it got.

For your questions Starfish Prime already has a great example. These monkeys receive a neural implant, either EEG or some evil spiked things into the brain. These can measure the electrical impulses of multiple neuron. They first try to match some signals in the brain, like moving up with your arm should move the arm up. Then the rest is just the monkey learning. A problem is noise of the signals, but we're getting better at it. But this noise is visible in the gif of the monkey moving the arm. It isn't fast and controlled yet. We're getting better at it though.

Interestingly EEG headsets exist that give you control over a mouse on your computer. Again, the movement of the mouse is mapped to a certain function of your cortex, like moving your arm. At the start people often need to move the arm in conjunction with the mouse. Interestingly at a certain moment these people start to think differently, and are able to move the mouse without moving the arm. That despite it being the same brain patterns on the cortical level. This shows that we're able to learn that some brain signals for an arm can be used for something completely different than moving an arm.

The conclusion is simple. Via association it will be relatively easy to learn a new body plan, like longer arms. That is very simple compared to flipping your world upside down and mirroring it, or moving a mouse along a screen. Even better. Most people already experienced getting longer arms during puberty! Although it might be a temporary and gradual thing, it does happen and is learned.

Now the kicker. EEG and many of the current brain pattern recognition is still in the starting phases of development. They are crude, relatively inaccurate and prone to noise. A suit is more like a keyboard and mouse, so you're able to control it without noise. If you then see through the eyes of the mech, you'll certainly be able to learn this new body plan quite quickly. Especially if there's bio feedback (you feel what the controlled item is feeling). With the mech and an intensive week you can probably get ok in controlling it. In an intensive month quite proficient and in two you'll think of it as a second skin.

  • 1
    $\begingroup$ Two examples of learning the reverse action of already known activities: The Backwards Brain Bicycle - Smarter Every Day 133 (2015-04) and Reverse Steering Bike on National Geographic Brain Games (2017-01) $\endgroup$ – Makyen 2 days ago
  • 1
    $\begingroup$ Look at some of the things military helicopter pilots get up to with subconscious control of a control interface that is completely alien to the thing under control which is completely alien to any human body part. The human brain is really fluid: sciencedirect.com/science/article/pii/S0960982209013244 $\endgroup$ – user1937198 2 days ago
  • 1
    $\begingroup$ It probably took me a year or so to be able to play the drums with my eyes closed. And a decade to play the game Left for Dead using echolocation. All you need is feedback and zero latency. Half a second or more of lag is the only way that game is hard for me anymore because it messes with my muscle memory and timing. When I play that game, I'm a robotic zombie 'in the zone'. $\endgroup$ – Mazura 2 days ago
  • 1
    $\begingroup$ @Mazura, consistent latency shouldn't be a problem either—your body alone has 0.1–0.2s latency and many machines humans control have seconds or even minutes of latency. The brain just learns to anticipate it and give the commands sufficiently in advance. Jitter is a problem though because you can't plan for that. $\endgroup$ – Jan Hudec 2 days ago

Humans routinely learn and master how to control an excavator: a very long arm with the body moving on way more compact base, which is the same situation you are describing.

It doesn't take that long to achieve a good coordination with moving the excavator and operating its arm: I would say that a few months of routine usage are enough to become proficient. (But I am not an expert, if anybody can provide better figures out of first hand knowledge, it's better).

I have received training on how to use a crane with a remote controller: already after a shift of work my coordination was dramatically better than it was at the beginning of the day.

  • 1
    $\begingroup$ Here is a video of a Liebherr excavator with a very proficient pilot at the controls. $\endgroup$ – AlexP Feb 22 at 13:09
  • $\begingroup$ To extend this a little further: you could compare some video game interfaces to very complex machinery, with new control mechanisms and functions. We often get used to those quite quickly. $\endgroup$ – Whitehot 2 days ago

Here's a clip of a monkey feeding itself with a simple robotic arm controlled via a direct brain interface, taken in the implausible scifi future of 2008:

Monkey with brain-controlled robotic arm feeding itself

(Nature article on the subject)

This work was done by these folks who have since done some work on humans (PDF article). They aren't saying how long it took to train her (or the earlier monkeys), at least not anywhere that I spotted, though maybe you'll have more luck.

The issue here does not seem to be the learning, but more the implantation and maintenance of the neural interfaces which is still very much cutting edge stuff... the issue is making a safe, reliable, commercialisable neural interface, not training people to use it.

What I can infer from what I've read is that it a) apparently takes less than a year, b) doesn't require functioning limbs but c) does require suitable bits of motor cortex to hook up to and d) works even for arms that aren't necessarily the same size or geometry or speed as the operator's usual limbs, even in the absense of proprioceptive feedback.


Your Answer

Silent Bastion is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.