Novel prototype technology translates brain signals into speech

July 26, 2021

The field of communication neuroprosthetics has so far focused on restoring some degree of speech to patients who have lost the ability due to stroke, disease, or an accident; it typically involves spelling-based approaches to type out letters one-by-one in text – a slow and laborious task. Fortunately, University of California, San Francisco (UCSF) neurosurgeon Dr. Edward Chang has seen fit to develop a novel technology that instead “decodes full words from the brain activity of someone who is paralysed and cannot speak.”

Dr. Chang’s prototype system translates signals intended to control muscles of the vocal system for speaking words, rather than signals to move the arm or hand to enable typing. According to the Joan and Sanford Weill Chair of Neurological Surgery at UCSF, this approach not only taps into the natural and fluid aspects of speech, but also promises more rapid and organic communication.

Previously, Dr. Chang and colleagues in the UCSF Weill Institute for Neurosciences mapped brain activity patterns associated with vocal tract movements that produce each consonant and vowel, facilitated by patients with normal speech. Postdoctoral engineer David Moses then developed new methods for real-time decoding of those patterns into speech recognition of more accurate, full words.

However, the team’s success in decoding speech in participants who were able to speak didn’t guarantee that the system would work in patients with a paralysed vocal tract. “Our [custom neural network] models needed to learn the mapping between complex brain activity patterns and intended speech,” said Moses. “The best way to find out whether this could work was to try it.”

This prompted the launch of the “BRAVO” (Brain-Computer Interface Restoration of Arm and Voice) Study, with participant BRAVO1 – a 30-year old male suffering from a devastating brainstem stroke that severely damaged the connection between his brain and his vocal tract and limbs. Since his injury, he has had extremely limited head, neck, and limb movements, and can only communicate by using a pointer attached to a baseball cap to poke letters on a screen.

For the study, Dr. Chang surgically implanted a high-density electrode array over BRAVO1’s speech motor cortex. After the participant’s full recovery, Dr. Chang’s team recorded 22 hours of neural activity in this brain region over 48 sessions and several months. In each session, BRAVO1 attempted to say specific vocabulary words while electrodes recorded brain signals from his speech cortex; at the same time, the custom models worked to distinguish subtle patterns in brain activity to detect speech attempts and identify which words he was trying to say.

The team found that the system was able to decode words from brain activity at a rate of up to 18 words per minute with up to 93% accuracy (75% median), helped in part by a language model that implemented an “auto-correct” function, similar to what is used by consumer texting and speech recognition software today.

“This is an important technological milestone for a person who cannot communicate naturally,” said Moses, “and it demonstrates the potential for this approach to give a voice to people with severe paralysis and speech loss.”

Dr. Chang and Moses are also working to increase the number of words in the system’s available vocabulary, as well as improve its rate of speech.

Tags: brain signals, disease, Novel prototype technology translates brain signals into speech, stroke