Artificial intelligence turns brain activity into speech for most people who are frozen and unable to speak, the signals for what they want to say hide in their brains. No one could interpret those signs directly. But three research groups have recently made progress in transforming data from electrodes placed on the brain into computer-generated speech. Using computational models called neural networks, they have, in some cases, reconstructed words and sentences that are understandable to human listeners.

1. 8/29/2019 Artificial intelligence turns brain activity into speech
https://medium.com/@usmsystems23/artificial-intelligence-turns-brain-activity-into-speech-53bd82b52abb 1/3
Arti cial intelligence turns brain
activity into speech
usm systems
Aug 29 · 4 min read
Artificial intelligence turns brain activity into speech for most people who are frozen and
unable to speak, the signals for what they want to say hide in their brains. No one could
interpret those signs directly. But three research groups have recently made progress in
transforming data from electrodes placed on the brain into computer-generated speech.
Using computational models called neural networks, they have, in some cases,
reconstructed words and sentences that are understandable to human listeners.
None of the efforts described in recent months papers on the PrintPrint Server
BioReceive, people have just been unable to recreate the expected speech. Instead, the
researchers monitored parts of the brain that people read aloud, quietly, or listen to
recordings. But it is understandable to show the reconstructed speech is “definitely
exciting,” said Stephanie Martin, a neural engineer at the University of Geneva,
Switzerland, who was not involved in the new projects.
People who have lost their ability to speak after a stroke or disease can use their eyes or
make other small movements to control the cursor or select letters on the screen. If the
brain-computer interface can directly recreate their speech, they can retrieve it again:
control over tone and image, for example, or the ability to interfere with fast-moving
speech.
The barriers are high. “We are trying to create a pattern of neurons that turn on and off
at different time points and inflate speech sound,” says Nima Mesgarani, a computer
scientist at Columbia University. “Mapping from one to the next is not very
straightforward.” How these codes translate into speech sounds varies from person to
person, so computer models must be “trained” on each person. And models do best with
very accurate data, which requires skull opening.

2. 8/29/2019 Artificial intelligence turns brain activity into speech
https://medium.com/@usmsystems23/artificial-intelligence-turns-brain-activity-into-speech-53bd82b52abb 2/3
Researchers can make such invasive recording only in rare cases. When one removes a
brain tumor, electrical readouts from the exposed brain help surgeons identify and avoid
key speech and motor areas. Another is when a person with epilepsy is implanted with
electrodes for several days to determine the source of the seizure before surgery. “We
have a maximum of 20 minutes, maybe 30,” for data collection, Martin says. “We are
limited.”
The groups behind the new papers produce highly valuable data by putting information
into neural networks that process complex patterns by passing information through
layers of computational “nodes”. Networks learn by adjusting the connections between
nodes. In the experiments, the networks were subjected to recordings of speech
produced and heard by a person and simultaneous brain activity data.
The Mesgarani team relied on data from five people with epilepsy. Their network
analyzed recordings from the auditory cortex because they heard recordings of stories
and people naming numbers from zero to nine. The computer then reconstructs the
spoken numbers from the neural data alone; When “talking” computer numbers, a group
of listeners named them with 75% accuracy.
Another group, led by computer scientist Tanja Schultz at University Bremen, Germany,
is based on data from six people who underwent brain tumor surgery. The microphone
captures their voices while reading single-letter words aloud. Meanwhile, electrodes are
recorded from the brain’s speech planning areas and motor areas, which send
commands to the vocal tract to pronounce words. Computer scientists Miguel Angrick
and Christian Herf, now at Maastricht University, trained a network that mapped
electrode readouts to audio recordings and then reconstructed words from invisible
brain data. According to the computerized scoring system, 40% of computer-generated
words are intelligible.
Computer audio of each word based on actual audio from the study participants,
followed by speech planning and activity in motor areas of the brain.
Finally, neurosurgeon Edward Chang and his team at the University of California, San
Francisco reconstructed whole sentences from speech and motor activity extracted from
the brain, and three epilepsy patients read aloud. In the online test, 166 people heard
one of the sentences and had to choose from 10 written choices. Some sentences are

3. 8/29/2019 Artificial intelligence turns brain activity into speech
https://medium.com/@usmsystems23/artificial-intelligence-turns-brain-activity-into-speech-53bd82b52abb 3/3
correctly identified by more than 80% of the time. Researchers have pushed this
paradigm further: They used it to recreate sentences from recorded data when people
quietly cried words. This is an important result, says Herff, “one step closer to the speech
prosthesis we all have in mind.”
However, “What we’re waiting for is how [these methods] are going to do when the
patients can’t speak,” says Stephanie Riès, a neuroscientist at San Diego State University
in California who studies language production. The brain signals when a person silently
“speaks” or “hears” their voice in their head aren’t identical to signals of speech or
hearing. Without external sound to match brain activity, it may be hard for a computer
even to sort out where inner speech starts and ends.
Decoding imagined speech will require “a huge jump,” says Gerwin Schalk, a neuro
engineers at the National Center for Adaptive Neurotechnologies at the New York State
Department of Health in Albany. “It’s unclear how to do that at all.”
One approach, Herff says, might be to give feedback to the user of the brain-computer
interface: If they can hear the computer’s speech interpretation in real-time, they may be
able to adjust their thoughts to get the result they want. With enough training of both
users and neural networks, brain and computer might meet in the middle.
Arti cial Intelligence Machine Learning Ai Services Ai Solution Ml Services
About Help Legal