MIT tech controls robotic hand with enough dexterity to play piano

By Stephen Beech

A high-tech wristband enables wearers to control a robotic hand with such dexterity that it can play the piano.

By moving their own hands and fingers, users can direct a robot to play keyboards or even play basketball.

The state-of-the-art device also allows people to manipulate objects in a virtual environment.

American engineers designed the ultrasound wristband that precisely tracks a wearer’s hand movements in real time.

The human hand contains 34 muscles, 27 joints, and more than 100 tendons and ligaments.

The wristband produces ultrasound images of the wrist’s muscles, tendons, and ligaments as the hand moves.

It is paired with an AI algorithm that continuously translates the images into the corresponding positions of the five fingers and palm.

The researchers can also train the wristband to learn a wearer’s hand motions, which the device can communicate in real time to a robot or a virtual environment.

In demonstrations, the Massachusetts Institute of Technology (MIT) team showed that a person wearing the wristband can wirelessly control a robotic hand.

As the person gestures or points, the robot does the same.

The wearer can manipulate the robot to play a simple tune on the piano and shoot a small basketball into a desktop hoop.

Wearers can also manipulate objects on a computer screen, for example pinching their fingers together to enlarge and minimise a virtual object.

The MIT team is using the wristband to gather hand motion data from many more users with different hand sizes, finger shapes, and gestures.

They envision building a large dataset of hand motions that can be plumbed to train humanoid robots in dexterity tasks, such as performing certain surgical procedures.

The team say the ultrasound band could also be used to grasp, manipulate, and interact with objects in video games, design applications, or other virtual settings.

MIT team leader Professor Xuanhe Zhao said: “We think this work has immediate impact in potentially replacing hand tracking techniques with wearable ultrasound bands in virtual and augmented reality.

"It could also provide huge amounts of training data for dexterous humanoid robots."

Zhao’s team has been developing various forms of ultrasound stickers - miniaturised versions of the transducers used in doctor’s offices that are paired with hydrogel material that can safely stick to skin.

For the new study, published in the journal Nature Electronics, the team incorporated the ultrasound sticker design into a wearable wristband to continuously image the muscles and tendons in the wrist.

Study co-author Gengxi Lu said: “The tendons and muscles in your wrist are like strings pulling on puppets, which are your fingers.

“So the idea is, each time you take a picture of the state of the strings, you’ll know the state of the hand.”

More from this section

The world’s great fish migrations are collapsing – that’s a problem for millions of people

+3

Research warns having bladder condition may raise risk of dementia

+5

Bug’s ‘mind-boggling’ ability to create heat to avoid freezing to death

The team designed a wristband with an ultrasound sticker the size of a smartwatch, and added onboard electronics as small as a mobile phone.

The researchers attached the wristband to a volunteer’s wrist, and confirmed that the device produced "clear and continuous" images of the wrist as the volunteer moved their fingers in various gestures.

The team found that they could identify specific regions in ultrasound images of the wrist that correlate to each of the 22 degrees of freedom.

For example, changes in one region relate to thumb extension, while changes in another region correlate with movements of the index finger.

By matching changes in certain regions of the ultrasound images with hand positions recorded by the cameras, the team could label wrist image regions with the corresponding degree of freedom in the hand.

But to do the translation continuously, and in real time, would be an impossible task for humans.

The team used an AI algorithm that can be trained to recognize image patterns and correlate them with specific labels and the hand’s various degrees of freedom.

The team trained the algorithm with ultrasound images that they labelled, annotating the image regions associated with a specific degree of freedom.

They tested the algorithm on a new set of ultrasound images and found it correctly predicted the corresponding hand gestures.

Once the team successfully paired the AI algorithm with the wristband, they tested the device on more volunteers.

For the new study, eight volunteers with different hand and wrist sizes wore the wristband while they formed various hand gestures and grasps.

The volunteers also held objects including a tennis ball, a plastic bottle, a pair of scissors, and a pencil.

In each case, the wristband precisely tracked and predicted the position of the hand.

To demonstrate potential applications, the team developed a simple computer program that they wirelessly paired with the wristband.

As a wearer went through the motions of pinching and grasping, the gestures corresponded to zooming in and out on an object on the computer screen, and virtually moving and manipulating it in a smooth and continuous fashion.

The team also tested the wristband as a wireless controller of a simple commercial robotic hand.

While wearing the wristband, a volunteer went through the motions of playing a keyboard.

The robot in turn mimicked the motions in real time to play a simple tune on a piano.

The same robot was also able to mimic a person’s finger taps to play a desktop basketball game.

Zhao plans to further miniaturize the wristband’s hardware, as well as train the AI software on more gestures and movements from volunteers with various hand sizes and shapes.

The team is working towards a wearable hand tracker that can be worn by anyone, to wirelessly manipulate humanoid robots or virtual objects with high dexterity.

Zhao added: "We believe this is the most advanced way to track dexterous hand motion, through wearable imaging of the wrist.

“We think these wearable ultrasound bands can provide intuitive and versatile controls for virtual reality and robotic hands.”