Can humans learn unique robotic hand-eye coordination?

Question

Can people learn precise hand-eye coordination for prosthetic or external mechanical limbs that don't have normal proportions?

For example, someone is controlling a small mech via a suit or exoskeleton that takes in their real body movements (like pacific rim but one person and smaller). But that mech has very long hefty arms compared to its legs, like a gorilla. The controls would still work intuitively so long as it had the same joints as a human, but would the pilot be able to learn proficient robohand-to-eye coordination in a reasonable time frame (at most a few years of deliberate training)?

Or say a prosthetic controlled neurally with the same situation, a very large/long arm (given the rest of the body could support it). Could they become as proficient with it as their lost real arm, despite its proportion and size difference?

If so, how long would it realistically take? Would any special training methods work faster? Would it be practical?

Answer 1

A month to a year

The human mind is incredibly flexible. Via association we can learn a great deal. When you start riding a bike or car you can see this very well. Moving your arms and legs in certain ways creates desired movement. We haven't evolved with these movements, yet we can learn them. Even better is controlling a computer for example. Moving your fingers on a keyboard or a mouse makes the computer do stuff. You can look around in a computer game, or type and select whatever you want in a word document. Finally, you can learn to fly an FPV drone. You put glasses on and get feedback of the camera. Moving your hands you can fly any direction you want. At the start it might be difficult, but with extensive training you'll be able to control it very well in a month, or for slow learners about six. These show that you can learn new abilities and movement by moving your body in a certain way in specific situations.

Even more impressive is doing something that flips it all upside down and left side right, going against what you've learned. A man made special glasses that flip and mirror the light coming in. After three months he was able to drive a car without problems. Interestingly he had trouble adjusting when he removed the glasses. The more he started switching, the easier it got.

For your questions Starfish Prime already has a great example. These monkeys receive a neural implant, either EEG or some evil spiked things into the brain. These can measure the electrical impulses of multiple neuron. They first try to match some signals in the brain, like moving up with your arm should move the arm up. Then the rest is just the monkey learning. A problem is noise of the signals, but we're getting better at it. But this noise is visible in the gif of the monkey moving the arm. It isn't fast and controlled yet. We're getting better at it though.

Interestingly EEG headsets exist that give you control over a mouse on your computer. Again, the movement of the mouse is mapped to a certain function of your cortex, like moving your arm. At the start people often need to move the arm in conjunction with the mouse. Interestingly at a certain moment these people start to think differently, and are able to move the mouse without moving the arm. That despite it being the same brain patterns on the cortical level. This shows that we're able to learn that some brain signals for an arm can be used for something completely different than moving an arm.

The conclusion is simple. Via association it will be relatively easy to learn a new body plan, like longer arms. That is very simple compared to flipping your world upside down and mirroring it, or moving a mouse along a screen. Even better. Most people already experienced getting longer arms during puberty! Although it might be a temporary and gradual thing, it does happen and is learned.

Now the kicker. EEG and many of the current brain pattern recognition is still in the starting phases of development. They are crude, relatively inaccurate and prone to noise. A suit is more like a keyboard and mouse, so you're able to control it without noise. If you then see through the eyes of the mech, you'll certainly be able to learn this new body plan quite quickly. Especially if there's bio feedback (you feel what the controlled item is feeling). With the mech and an intensive week you can probably get ok in controlling it. In an intensive month quite proficient and in two you'll think of it as a second skin.

Answer 2

Humans routinely learn and master how to control an excavator: a very long arm with the body moving on way more compact base, which is the same situation you are describing.

It doesn't take that long to achieve a good coordination with moving the excavator and operating its arm: I would say that a few months of routine usage are enough to become proficient. (But I am not an expert, if anybody can provide better figures out of first hand knowledge, it's better).

I have received training on how to use a crane with a remote controller: already after a shift of work my coordination was dramatically better than it was at the beginning of the day.

Answer 3

Here's a clip of a monkey feeding itself with a simple robotic arm controlled via a direct brain interface, taken in the implausible scifi future of 2008:

(Nature article on the subject)

This work was done by these folks who have since done some work on humans (PDF article). They aren't saying how long it took to train her (or the earlier monkeys), at least not anywhere that I spotted, though maybe you'll have more luck.

The issue here does not seem to be the learning, but more the implantation and maintenance of the neural interfaces which is still very much cutting edge stuff... the issue is making a safe, reliable, commercialisable neural interface, not training people to use it.

What I can infer from what I've read is that it a) apparently takes less than a year, b) doesn't require functioning limbs but c) does require suitable bits of motor cortex to hook up to and d) works even for arms that aren't necessarily the same size or geometry or speed as the operator's usual limbs, even in the absense of proprioceptive feedback.