A brain implant has enabled people with paralysis to speak while gesturing, such as nodding or waving. These sorts of exchanges, done via an avatar, could help them communicate with more nuance.
“In our everyday life, gestures play an important role in communication,” says Samantha Brosler at the University of California, San Francisco. “Saying ‘maybe’ and nodding your head has a very different connotation than if you shake your head.”
Read more‘Fusogenic’ neurosurgery let paralysed pigs walk again – are we next?
Experimental brain implants have boosted the ability to either speak or move among people with paralysis, but attempts to do both simultaneously haven’t been very successful. To achieve this, implants need to cover a large part of the sensorimotor cortex, a brain region involved in speech and movement, says Brosler.
Advertisement
Now, Brosler and her colleagues have tested an iPhone-sized implant in this part of the brain in two people with paralysis. The first participant, who the researchers call Bravo-1r, became paralysed after a stroke, while the second, Bravo-6, had amyotrophic lateral sclerosis, the most common form of motor neuron disease.
Both participants had lost almost all movement in their limbs. They also couldn’t speak and could only make unintelligible sounds.
Free newsletter
Sign up to Health Check
Sign up to newsletter
First, the team recorded the participants’ brain activity while they repeatedly attempted to say 10 phrases – such as “hello”, “nice to meet you” and “how’s it going?” – or perform 10 gestures, like waving, clapping and shaking their head. In some cases, the participants attempted to do these separately, while in others they tried both simultaneously.
The team then used each of the participants’ brain recordings to train customised machine learning models to predict their intended phrases and gestures.
To put the models to the test, the participants attempted to repeat various pairs of phrases and gestures that were displayed on a screen, which the models weren’t aware of. Their predictions were used to control the movement of an avatar resembling the participants, while the speech predictions flashed up on the screen.
The team chose to use avatars, rather than attempting for the individuals themselves to speak and gesture, because this offered a relatively simple way to prove that both actions could be simultaneously decoded from brain activity, says Brosler.
Paralysed man can feel objects through another person’s handKeith Thomas, a man in his 40s with no sensation or movement in his hands, is able to feel and move objects by controlling another person’s hand via a brain implant. The technique might one day even allow us to experience another person’s body over long distances.
Bravo-1r’s intended gestures and speech were predicted with 88 and 84 per cent accuracy, respectively, while Bravo-6’s were predicted with 66 and 70 per cent accuracy. “If those models were operating on complete chance, the accuracy would be like 9 per cent,” says Brosler. “It was really exciting.”
“It’s a very nice piece of work, with robust results,” says Henri Lorach at the University of Lausanne in Switzerland. But it’s based on a very limited set of phrases and gestures, and these still need improving, he says. “For daily use without frustration, there needs to be higher accuracy.”
Brosler says the team is working to address this by further training the models and tweaking the algorithms. She also says that future models could move a person’s own body – if their joints and muscles are in a good enough condition – and produce a synthetic voice, rather than having to do this via an avatar.
Journal Reference:Nature Neuroscience DOI: 10.1038/s41593-026-02446-2
[Image text:] Decoded gesture from brain activity BRAVO:'hello Decoded speech from brain activity