ABOVE: F. WILLETT ET AL./NATURE 2021/ERIKAWOODRUM
A brain-implant system trained to decode the neural signals for handwriting from a paralyzed man enabled a computer to type up to 90 characters per minute with 94 percent accuracy, researchers report yesterday (May 12) in Nature. The study’s authors say this brain-computer interface (BCI) is a considerable improvement over other experimental devices aimed at facilitating communication for people who cannot speak or move, but many steps remain before it might be used clinically.
“There are so many aspects of [the study] that are great,” says Emily Oby, who works on BCIs at the University of Pittsburgh and was not involved in the work. “It’s a really good demonstration of human BCI that is working towards clinical viability,” and also contributes to understanding why the handwriting-based system seems to work better than BCIs based on translating the neural signals for more straightforward physical motions such as pointing at letters on a display.
The study came out of a long-term clinical trial called BrainGate2 in which participants who are paralyzed have sensors implanted in the motor cortex of their brains and work with researchers who aim to use the sensors’ data to develop BCIs. “Because of the animal model heritage and the history of the [BCI] field, a lot of the early stuff is focused on this point-and-click typing method where you move a cursor on a screen, and you type on keys individually,” explains Frank Willett, a member of the Neural Prosthetics Translational Laboratory (NPTL) at Stanford University and a Howard Hughes Medical Institute research specialist. “We’re interested in kind of pushing the boundaries and looking at other ways to let people communicate.”
See “Speech Decoded from Brain Activity in Area for Hand Control”
Willett and his colleagues worked with a BrainGate2 participant nicknamed T5 who has a spinal injury, is able to talk, and has a sensor in an area of the brain known as the hand knob that is associated with hand movement. In several sessions, they asked T5 to pretend he was holding a pen and writing hundreds of sentences they showed him on a screen. They then used the activity detected by T5’s sensor to train a neural network to identify the letters T5 was writing, and tested the program’s ability to generate text in real time based on brain signals generated as he imagined writing new sentences.
The researchers report that the trained network enabled T5 to “type” at a speed of up to 90 characters per minute and had 94.1 percent accuracy in deciphering the letters he wrote. That’s a considerable improvement on a previous BCI the group developed that was based on having participants control a computer mouse with their brain signals and click on letters, which achieved about 40 characters per minute. In fact , the authors write, to their knowledge, it’s the fastest typing rate for any BCI so far.
Speed is critical for people who need BCIs to communicate, notes Oby, because “the faster and more efficiently that they can communicate the better, in terms of increasing their quality of life, and just making interactions more easy and smooth and less stressful.”
See “The Rise of BCI Enables Advances in Neuroscience”
To see what accounts for this superior performance, the authors analyzed the neural patterns corresponding to letters and to the straight reaching movements used in the point-and-click BCI. They found that the patterns for the letters are more distinct from one another, making them easier for a neural network to decipher. They also devised their own 26-letter alphabet, replete with curvy lines, that their simulations indicate would enable an even more accurate BCI by eschewing letters that are written similarly to one another.
“[It] makes a lot of sense . . . that having more complex movement dynamics can really help improve the communication rate, the accuracy of the decoding,” says Edward Chang, a neurosurgeon at the University of California, San Francisco, who has worked informally with the NPTL group but was not involved in the current study. “They’re really exploiting a new dimension of features that help make the signals more discriminable.”
To the best of the authors’ knowledge, their BCI enables the fastest typing rate of any developed so far.
There are several improvements that would be needed to make the BCI ready for clinical use. Those include tweaks to the brain implant itself, such as making it smaller and capable of wireless signal transmission, says study coauthor Jaimie Henderson, a neurosurgeon in the NPTL who consults for the BCI company Neuralink and is on the medical advisory board for Enspire, a company exploring deep-brain stimulation for stroke recovery. In addition, in the study the researchers needed to regularly calibrate the BCI to account for minute shifts in the positions of the sensors that alter what neural activity they pick up; ideally, Henderson and Willett say, this process, as well as the initial training of the neural network, would be automated.
Henderson, Willett, and senior author Krishna Shenoy, another NPTL member and a Howard Hughes Medical Institute investigator who consults for or serves on the advisory boards of several BCI-related companies, have filed a patent application for the neural decoding method they used and are talking with companies about the possibility of licensing it, Shenoy says. Ultimately, Willett and Henderson say, they’re interested in exploring neural signals for speech as a way to enable even faster communication than with handwriting. The rate of speech is about 150–200 words per minute, Henderson notes, and decoding it is an interesting scientific endeavor because it’s uniquely human and because it’s not fully understood how speech is produced in the brain. “We feel like that’s a very rich area of exploration, and so one of our big goals over the next five to ten years is to really tackle the problem of understanding speech and decoding it into both text and spoken word.”
F.R. Willett et al., “High-performance brain-to-text communication via handwriting,” Nature, 593:249–54, 2021.
Correction (May 13): The new BCI was able to type at 90 characters per minute, not 90 words per minute as originally stated in the article. The Scientist regrets the error.
