Can you hear me?

Perhaps it will not be too long before we can hear a completely paralysed person saying, “oh it’s a bit cold outside”, the moment he/she relaxingly walks out of the house on a quiet winter morning, wearing an exoskeleton suit.

With immense concentration, Anlu Lin slowly lifted her right hand and, after a brief pause, let it sink gently onto a key of a grand piano, thus starting an emotional duet with the renowned virtuoso pianist Lang Lang in November last year. It was a beautiful piece of music, but the spotlight had always been on Lin herself, who technically was not using her ‘own’ hand.

Anlu Lin’s performance with Lang Lang

Despite losing her right arm at the age of two in an accident, Lin never gave up on the dream of being able to play the piano, with both hands. Twenty years later, it was the BrainCo team, founded by Harvard University graduate Bicheng Han, who fulfilled Lin’s dream by building her a prosthetic arm that can be operated through her mind.

Brain-computer interface (BCI) – telekinesis comes true

BrainCo is one of the pioneers in brain-computer interface (BCI), a technology that allows bidirectional communication between the brain and an external device, a process known as the ‘brain control’.

In a BCI system, brain signals are acquired through various means, including electrocorticography, which involves poking a number of electrodes into the brain.

The collected brain signals are decoded by a computer, which extracts the useful information, e.g. signals that control arm movement. This information is then translated into kinetic commands that control the movement of a robotic arm.

Brain-computer interface (BCI) controlling a robotic arm
(diagram by Kelvin Kwok, photos from wikimedia.org and pixabay.com)

Restoring motion and the sense of touch

As noted by Daniel Dennett (1), the first human experiment on brain control probably dated back to 1963, when Grey Walter presented his pioneering work in a meeting at Oxford University.

In his experiment, Walter asked his patients to press a button to advance slides on a slide projector at their own pace, while he continuously recorded their brain motor cortex activity that preceded hand movement. He then established an information transmission link between their motor cortex and the projector, and repeated the experiment. To the patients’ surprise, the projector now responded to their will to advance slides even before their hand movement was physically initiated.

This unpublished study was later followed by the seminal work by Miguel Nicolelis’s team (2-5) that laid the foundation for robotic prosthesis control, with which Lin played the piano. They also built a BCI exoskeleton suit for Juliano Pinto, a 29-year-old paraplegic, who made the kick off in the 2014 World Cup.

In 2016, researchers at the University of Pittsburgh helped Nathan Copeland to restore the sense of touch, as he shook hands with Barack Obama using a prosthetic arm equipped with pressure sensors that were able to relay signals back to Copeland’s brain.

Hope is now regained for those who suffer from physical disability.

Communication tool for the disabled

But BCI is not stopping there.

As the Ice Bucket Challenge went viral on Facebook and Twitter between 2014 and 2015, the BCI field has already progressed from assisting physical movements to providing means of communications for those who suffer from devastating motor neuron diseases, including amyotrophic lateral sclerosis (ALS).

Before the widespread use of mature BCI technology, Stephen Hawking, a physicist with ALS and famous for his popular science bestseller A Brief History of Time, relied on the infrared detector mounted on his spectacles that catches the slightest twitches in his cheek muscles, which in turn instruct his speech-generating device in choosing alphabets for the words that he wanted to say.

It was not until 2016 when Hanneke de Bruijne, another late-stage ALS patient, could independently instruct a computer cursor to spell out words using brain signals alone. This BCI system allows her to type out approximately two words every minute, and is more reliable than her previous system, which types slightly quicker by tracking her eye movement but depended heavily on light intensity of the environment.

Hanneke de Bruijne’s BCI device.
Electrodes implanted into Hanneke de Bruijne’s brain collect brain signals and communicate with a tablet connected to her wheelchair. A cursor (red) on the tablet screen moves slowly across the screen and hovers on each alphabet. When de Bruijne wants to choose the letter, she imagines pressing her thumb against her forefinger. The brain signal generated is then detected by the electrode, which ‘clicks’ on the letter.

(diagram by Kelvin Kwok)

However, despite the great improvement in the quality of life in disabled patients, the limited communication speed is still a major obstacle for their well-being. Indeed, 150 words are easily spoken every minute in a natural speech, while most patients using BCI struggle to transmit even 10 words within the same time frame.

Speaking smoothly again

To provide an alternative to the spelling approach of current BCI technology, Edward Chang’s team at the University of California San Francisco recently attempted to synthesise speech directly by translating brain signals (7). They used a three-step approach to accomplish this task, which was published earlier this year.

Ventral sensorimotor contex in the brain
(diagram by Kelvin Kwok, photo from pixabay.com)

Chang and colleagues view natural speech as a continuous flow of overlapping, multi-articulatory movement of the vocal tract. With this in mind, they first sought to deduce how the brain controls these movements (8), focusing on a specific region of the brain called the ventral sensorimotor cortex (vSMC).

This seemingly unattractive region on the side of your brain is probably controlling the most complex motor behavior in humans, as it orchestrates the coordinated movements of more than 100 muscles to produce fluent speech.

When participants in the experiment spoke sentences aloud or silently as instructed, Chang’s team simultaneously recorded the neural activity in their vSMC, together with the gross movement of their facial parts that are phonetically important. This allowed them to decode brain signals into corresponding articulatory movements of the vocal tract (8). 

Unfortunately, this correlation is not quite yet sufficient to generate any speech, because there is still an information gap between articulatory movement and actual speech.

To fill this gap, Chang’s team went on and found public databases that contain articulatory movement data together with the corresponding audio recordings. Based on these data, they trained the computer to establish a correlation between articulatory movements and audio tracks.

With this extra information, the computer can finally translate the decoded articulatory movements into acoustic features.

The three-step approach of Chang’s team.
1) Brain signals were measured and decoded into articulatory movements. 2) Using public databases, articulatory movements were correlated with corresponding speech waveform. 3) This in turn trained the computer to translate acoustic features from the decoded articulatory movements based on brain signals.These acoustic features were then used to synthesise speech waveform.
(diagram by Kelvin Kwok, photos from wikimedia.org and pixabay.com)

After optimizing this complex training network, the computer successfully synthesized acoustic waveforms that were recognizable to an independent listener, who was able to transcribe the synthesized speeches with up to more than 70% accuracy compared to the original speech.

Chang’s team further compared the decoded neural-articulatory signals from different participants, and found a high correlation among them. This means that the speech synthesis system can probably be applied universally without individualised training, including in those who cannot speak.

Bright future ahead with BCI

Undoubtedly, there is still plenty of room for improvement in Chang’s speech synthesis tool, and it is still restricted to English language.

Nevertheless, this seemingly primitive attempt to generate speech from pure brain activity represents a big step forward for those whose minds are trapped inside due to motor neuron diseases.

Perhaps it will not be too long before we can hear someone with ALS saying, “oh it’s a bit cold outside”, the moment he or she relaxingly walks out of the house on a quiet winter morning, wearing an exoskeleton suit.

References:
  1. Dennett DC. Consciousness Explained. Boston: Little, Brown, 1991, p. xiii.
  2. Nature. 408 (6810): 361–5.
  3. PLoS Biology. 1 (2): E42.
  4. Journal of Neuroscience. 25 (19): 4681–93.
  5. Nature. 479(7372): 228–231.
  6. Neuropraxis, 22(3), 85–91.
  7. Nature. 568(7753):493-498
  8. Neuron. 6;98(5):1042-1054.e4

All photos are licensed under Creative Commons license

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s