Developing a New Speech Perception in Noise Training for Autism.pdf

DEVELOPING A NEW SPEECH
PERCEPTION IN NOISE
TRAINING FOR AUTISM
Barbara Fernandes, M.S CCC-SLP

Smarty Ears CEO
Julia Irwin, PhD
Child and Developmental Psychology

Professor and Senior Scientist

Haskins Laboratories

New Haven, CT, United States
Jessica Sullivan, PhD

Assistant Professor

Hampton University

Hampton, Virginia, United States
Texas Speech and Hearing Association Convention 2020
Smarty Ears Founder & CEO

Speech-Language-Pathologist

Dallas, TX, United States

THE PRESENTER
- Speech Language Pathologist (TCU graduate)

- Smarty Ears Founder and CEO since 2009

- Published over 60 apps for speech, language and
communication

- Smarty Symbols Founder

- Principal Investigator at a National Institute on
Deafness and Other Communication Disorders
(NIDCD) SBIR - Phase 1 Grant (2019/2020)

- i-Corp Awardee 2020
DEVELOPING A NEW SPEECH PERCEPTION IN NOISE TRAINING FOR AUTISM

Barbara Fernandes M.S CCC-SLP


DISCLOSURES
- Barbara Fernandes is not receiving any
fees for this presentation

- Barbara Fernandes has ownership in
Smarty Ears, LLC

- Barbara Fernandes has ownership in
Smarty Symbols, LLC

- Principal Investigator at a National Institute
on Deafness and Other Communication
Disorders (NIDCD) SBIR - Phase 1 Grant
(2019/2020)

- i-Corp Awardee 2020




WHAT IS THIS PRESENTATION ABOUT?


Relationships Learning Persistence
Collaboration
Product Development Problem Solving
Research

WHAT DO WE KNOW?
Let’s refresh our memories on basic concepts


Audiology Autism
Speech to Noise Ratio
Sensory Integration
60 dB
40 dB
+20dB
Speech Noise
“The process of the brain organizing and
interpreting this information is called Sensory
Integration.”
https://occupationaltherapychildren.com.au/our-team/

WHAT DO WE DO IN NOISY SITUATIONS


Typical Response in Noisy Conditions
“The process of the brain organizing and
interpreting this information is called Sensory
Integration.”

WE HEAR BETTER WHEN WE SEE
Sound and Mouth
When one both hears and sees a
speaker’s utterance, speech
perception is more accurate
(Sumby and Pollack, 1954; Ross
et al., 2007a; Stevenson and
James, 2009; Ross et al., 2011;
Stevenson et al., 2014e; Altieri et
al., 2015; Stevenson et al., 2015)
and less e
ff
ortful (Fraser et al.,
2010) than when only auditory
information is available.



NOISE
What do we know about speech in noise?
High-functioning individuals with
autism (HFA) or Asperger's syndrome
(AS) commonly report difficulties
understanding speech in situations
where there is background speech or
noise. Alcántara JI1, Weisblatt EJ, Moore BC, Bolton PF ( 2004)
It is generally known that children have
a more di
ffi
cult time understanding
speech in background noise than do
adults (development of average adult
performance occurs around the age of
ten; i.e. Elliott & Katz, 1980, Elliott et
al., 1981, Johnson, 2000).



THE SCIENCE
Three di
ff
erence evidences from past studies:
Smith EG, Bennetto L. Audiovisual speech integration and lipreading in autism. J
Child Psychol Psychiatry. 2007;48:813–821. [PubMed] [Google Scholar]
Methods: In the present study, audiovisual speech integration was investigated in 18 adolescents with high‐functioning autism and 19 well‐
matched adolescents with typical development using a speech in noise paradigm. Speech reception thresholds were calculated for auditory
only and audiovisual matched speech, and lipreading ability was measured.
1st Evidence:
Results: Compared to individuals with typical development, individuals with autism showed less bene
fi
t from the addition of visual
information in audiovisual speech perception. We also found that individuals with autism were signi
fi
cantly worse than those in the
comparison group at lipreading.
Conclusions: Combined de
fi
cits in audiovisual speech integration and lipreading in individuals with autism are likely to contribute to
ongoing dif
fi
culties in speech comprehension, and may also be related to delays in early language development.



THE SCIENCE
Three di
ff


Irwin JR, Tornatore LA, Brancazio L, Whalen DH. Can children with autism
spectrum disorders “hear” a speaking face? Child development. 2011;82:1397–1403.
[PMC free article] [PubMed] [Google Scholar]
Methods: This study used eye‐tracking methodology to assess audiovisual speech perception in 26 children ranging in age from 5 to 15
years, half with autism spectrum disorders (ASD) and half with typical development.
2nd Evidence:
Results: Responses were compared on a set of auditory, visual, and audiovisual speech perception tasks. Even when
fi
xated on the face
of the speaker, children with ASD were less visually in
fl
uenced than typical development controls.
Conclusions: This indicates fundamental differences in the processing of audiovisual speech in children with ASD, which may contribute
to their language and communication impairments.

THE SCIENCE
Three di
ff


Foxe JJ, Molholm S, Del Bene VA, Frey H-P, Russo NN, Blanco D, Saint-Amour D,
Ross LA. Severe multisensory speech integration deficits in high-functioning school-
aged children with autism spectrum disorder (asd) and their resolution during early
adolescence. Cerebral Cortex. 2013:bht213.
Methods: The primary aim here was to track the development of these abilities in children with autism, since multisensory de
fi
cits are
increasingly recognized as a component of the autism spectrum disorder (ASD) phenotype. The abilities of high-functioning ASD
children (n = 84) to integrate seen and heard speech were assessed cross-sectionally, while environmental noise levels were
systematically manipulated, comparing them with age-matched neurotypical children (n = 142).
3rd Evidence:
Results: Severe integration de
fi
cits were uncovered in ASD, which were increasingly pronounced as background noise increased. These
de
fi
cits were evident in school-aged ASD children (5–12 year olds), but were fully ameliorated in ASD children entering adolescence (13–
15 year olds).
Conclusions: The severity of multisensory de
fi
cits uncovered has important implications for educators and clinicians working in ASD.
We consider the observation that the multisensory speech system recovers substantially in adolescence as an indication that it is likely
amenable to intervention during earlier childhood, with potentially profound implications for the development of social communication
abilities in ASD children.

IN SUMMARY HERE IS WHAT WE KNOW:
TD Children perform worse than adults


ASD children have Severe integration de
fi
cits which were increasingly pronounced as
background noise increased.
Audio-Visual helps TD children integrate and understand speech in noisy conditions
ASD children do not appear to bene
fi
t from seeing the speaker face (multisensory integration
di
ffi
culty).
ASD adolescents demonstrate improved abilities indicating the importance and potential for
early intervention.
The louder the noise the more di
ffi
cult it gets for ASD children to understand

speech.

IN NOISY CONDITIONS:


TYPICAL DEVELOPING
CHILDREN ARE ABLE TO
USE VISUAL INFORMATION
TO ENHANCE PERCEPTION
ASD KIDS (EVEN WHEN LOOKING
AT THE FACE) ARE NOT ABLE TO
USE VISUAL INFORMATION TO
ENHANCE SPEECH PERCEPTION

CAN WE DEVELOP A TRAINING SYSTEM TO HELP CHILDREN
WITH AUTISM INTEGRATE VISUAL INFORMATION AVAILABLE
TO IMPROVE SPEECH PERCEPTION IN NOISY CONDITIONS?


YES NO

WHAT ARE THE RISKS FOR A PRIVATE COMPANY
TO DEVELOP SUCH NOVEL TECHNOLOGY?


- ASD as a spectrum

- Possibility that the limitations observed in ASD children can’t be corrected with
therapy.

- How do you go about developing such a novel tool?

- How long should the treatment last to be e
ff
ective?

- New technology must have its e
ff
ectiveness evaluated

- Cost of running clinical trials

- Clinical trials could prove to have minimal e
ff
ectiveness

- previous research shows that adolescents with ASD have normal speech to noise
integration

CREATING NOVEL THERAPEUTIC
TOOLS IS VERY RISKY



SMALL BUSINESS INNOVATION
RESEARCH


“The Small Business Innovation Research (SBIR) program is a
highly competitive program that encourages domestic small
businesses to engage in Federal Research/Research and
Development (R/R&D) that has the potential for commercialization.
By including quali
fi
ed small businesses in the nation's R&D arena,
high-tech innovation is stimulated and the United States gains
entrepreneurial spirit as it meets its speci
fi
c research and
development needs”
https://www.sbir.gov/about/about-sbir



SMARTY EARS WAS
AWARDED AN SBIR IN 2019
Title: Listening 2 Faces: Improving Speech Perception of
Speech in Noise in Children With Communication Disorders
Awarding Agency: National Institute on Deafness and Other Communication
Disorders
Principal Investigator: Barbara Fernandes, M.S CCC-SLP
Co- Investigators: Dr. Julia Irwin & Dr. Jessica Sullivan

INTRODUCING: LISTENING 2 FACES - THE APP
Listening 2 Faces was developed to serve as a
listening training protocol for improving listening
skills in children with di
ffi
culty understanding
speech in the presence of varying levels of
background noise.


Development time: 6 to 7 months.

DEVELOPING LISTENING 2 FACES


THE APP CONTENT SELECTION
- Develop a library of monosyllabic words including
all phonemes of standard American English.

- Organize the information of each word in order
for it to be used on the app and easily display the
evidence (analyze if word characteristics can
create di
ff
erences in speech perception)

- Balance the word lists for each actor so that
sounds aren’t too unevenly weighted per actor

- The output: 900 monosyllabic words that can be
visually identi
fi
ed



THE APP CONTENT DEVELOPMENT



THE VIDEO RECORDINGS
Barbara Fernandes & Darla Gardner
- Recruiting the participants

- Diverse participants: age,
gender, ethnicity

- Video standards

- Dialectal considerations



RECORDING STANDARD
Black t-shirt
No blinking
Head and shoulders
White background
Natural but pleasant
emotion
A
ffi
rmative tone



RECORDING STANDARD



THE FACES OF LISTENING 2 FACES
Age Range: 5 - 70 years old
Gender: 7 females & 6 males
Race/Ethnicity: Asian-
American,

African American, Hispanic-
American, Caucasian
Exclusion: accented foreign
speech, articulation

errors.



VIDEO EDITING AND MASTERING
- The Problem: What if One Person Speaker Is
Louder Than the Other? What if One Word Is
Louder Than the Other?

- Each Video Was Mastered by an Audio
Engineer to Level to an Approximate -16LUFS

- LUFS =Loudness Unit Full Scale

- 1 to 1 Correspondence to dBa

- Decibel = Air Pressure

- LUFS = Perceived Loudness Over a Period of
Time

“In practical applications, LUs are equal to decibels. Amplifying audio by
2dB is the same as raising its volume by 2 Loudness Units. The same
applies for LUFS units’.”
https://backtracks.fm/blog/whats-the-di
ff
erence-between-decibels-and-lufs/

LET’S TALK AND LISTEN TO NOISE!


“Objective: The objective of this investigation was to assess whether auditory
training in interrupted noise would improve speech recognition in noise for children
with HI and perhaps enhance their utilization of glimpsing skills.”
WHAT DOES INTERRUPED NOISE SOUND LIKE?
"Children who participated in auditory training in interrupted noise demonstrated a signi
fi
cantly
greater improvement in speech recognition compared to those who trained in continuous noise.
Those who trained in interrupted noise demonstrated similar improvements in both noise
conditions while those who trained in continuous noise only showed modest improvements in the
interrupted noise condition.”
Sullivan, J. R., Thibodeau, L. M., & Assmann, P. F. (2013). Auditory training of
speech recognition with interrupted and continuous noise maskers by
children with hearing impairment. The Journal of the Acoustical Society of
America, 133(1), 495–501. https://doi.org/10.1121/1.4770247
CLASSROOM INTERRUPED

LET’S TALK AND LISTEN TO NOISE!


WILL CHILDREN
WITH ASD BENEFIT
AS MUCH AS
CHILDREN WITH HL
IF WE UTILIZE
INTERRUPTED
NOISE TRAINING?

EXPLORING THE LISTENING 2 FACES APP


- Listening 2 Faces uses a space
theme

- Very clean and intuitive interface

- All the data will be locally stored
on the user’s iPad

- Clinicians can add an unlimited
number of children

- Easy to use for home training

- Customizations are based on
answers from the child’s pro
fi
le.

THE CALIBRATION


- Listening 2 Faces utilizes the
microphone of the iPad to
calibrate an initial sound
level (-16LUFS)

- Children can select a volume
that is comfortable to them

- This calibration is repeated
every time they start the app

AUDITORY NOISE ASSESSMENT


- Listening 2 Faces includes a pre and
a post noise screener ( ANA Test).

- The ANA includes easy to follow
steps

- Classroom Noise is presented at
either a 0SNR or a -5SNR

- If selected “yes” for ampli
fi
cation =
0SNR

- 15 monosyllabic words

- Include 15 di
ff
erent initial
consonants

AUDITORY NOISE ASSESSMENT


- The app automatically generates data
from the auditory noise assessment

- Children are prompted to complete an
existing ANA test after completing the
L2F training

- Data from Initial and Exits are
compared

- A
fi
nal written report is generated

- Initial and
fi
nal share same target word
but have di
ff
erent speakers

- ANA speakers are also di
ff
erent from
training speakers ( generalization)

- Noise generalization: Interruped/
Classroom

MILION DOLLAR QUESTION #3:


HOW RELIABLE IS
THE L2F ANA TEST
IN COMPARISON TO
OTHER STANDARD
MEASURES FOR
SPEECH IN NOISE?

THE TRAINING


- A step by step demo
familiarizes the child with
the training set up

- The space theme is
carried throughout the app

- Children press play to start
the videos

- Tap to select word choice

- The training adapts based
on the child’s performance

THE TRAINING


Training Protocol:
1. Five days a week training for 4
weeks.

2. 3 blocks of 8 minutes practice
daily

3. Always starting at -15 Speech to
Noise Ratio (SNR)

4. A matching game can be played
between each 8 minute blocks

5. Children are exposed to all
speakers in each 24 minutes
block

THE TRAINING



THE TRAINING


Training Protocol:
1. Children practice a total
of 480 minutes on L2F

2. 80% accuracy rule -
Increase SNR level in
3db Steps

3. Noise increases only up
to max comfortable level
selected by child

4. Speech is decreased the
remainder to a max of
-12dB

HOW DOES L2F KEEP CHILDREN MOTIVATED?


1. Rocket shows
progress minute by
minute

2. Matching game

3. Gems for consistency

4. Badges

5. Graphics change
daily

6. Final certi
fi
cate

THE DATA ANALYSIS ON L2F



THE CLINICAL TRIAL


- Children are currently being recruited in two sites:
Hampton University and Southern Connecticut State
University

- 3 arms: children with an autism spectrum disorder
(ASD), children with hearing loss (HL) and children
with no developmental disabilities, or typical
development (TD).

- 30 children

- Age: 8 to 12 years old

- 4 weeks in home program

- Battery of pre and post test

- IRB: Southern Connecticut State University
Institutional Review Board.

I-CORPS PROGRAM


- Smarty Ears was awarded a
supplemental grant to participate in an
NIH funded program called i-corps

- We are currently seeking to conduct 100
interviews to gather feedback for the
commercialization of Listening 2 Faces

- If you would like to be a part of it and
receive early access to L2F contact us at
bfernandes@smartyearsapps.com

Complete form: http://bit.ly/39S8y4r

CLEVER SOLUTIONS TO REAL PROBLEMS


- How are you going to make sure we are
not training speech in noise in an already
noisy environment?



- What happens if the child
wants to raise the volume in
the middle of a training?



- What happens if the child just
sits in front of the iPad just to
run the timer to move across
the levels?

FINAL CONSIDERATIONS


- There is potential for a signi
fi
cant impact for
children in the Autism spectrum

- Other potential uses of Listening 2 Faces

- Early release date

- L2F is a good example where the bene
fi
ts of
screen time outweighs the possible negative
e
ff
ects (if any).

FINAL CONSIDERATIONS


“The tests we face in
life's journey are not to
reveal our weaknesses
but to help us discover
our inner strengths. We
can only know how
strong we are when we
strive and thrive beyond
the challenges we face.”

― Kemi Sogunle

REFERENCES
Dunlop, William A et al. “Speech Discrimination Di
ffi
culties in High-Functioning Autism Spectrum Disorder Are Likely Independent
of Auditory Hypersensitivity.” Frontiers in human neuroscience vol. 10 401. 9 Aug. 2016, doi:10.3389/fnhum.2016.00401

Irwin JR, Tornatore LA, Brancazio L, Whalen DH. Can children with autism spectrum disorders “hear” a speaking face? Child
development. 2011;82:1397–1403

Stevenson, Ryan A et al. “Multisensory speech perception in autism spectrum disorder: From phoneme to whole-word
perception.” Autism research : o
ffi
cial journal of the International Society for Autism Research vol. 10,7 (2017): 1280-1290.
doi:10.1002/aur.1776

Smith EG, Bennetto L. Audiovisual speech integration and lipreading in autism. J Child Psychol Psychiatry. 2007;48:813–821.
[PubMed] [Google Scholar]

Sullivan, J. R., Thibodeau, L. M., & Assmann, P. F. (2013). Auditory training of speech recognition with interrupted and continuous
noise maskers by children with hearing impairment. The Journal of the Acoustical Society of America, 133(1), 495–501. https://
doi.org/10.1121/1.4770247

Schaaf, R.C., Benevides, T., Mailloux, Z. et al. An Intervention for Sensory Di
ffi
culties in Children with Autism: A Randomized
Trial. J Autism Dev Disord 44, 1493–1506 (2014). https://doi.org/10.1007/s10803-013-1983-8



Developing a New Speech Perception in Noise Training for Autism.pdf

Recommended

Recommended

More Related Content

Similar to Developing a New Speech Perception in Noise Training for Autism.pdf

Similar to Developing a New Speech Perception in Noise Training for Autism.pdf (20)

More from Barbara Fernandes

More from Barbara Fernandes (11)

Recently uploaded

Recently uploaded (20)

Developing a New Speech Perception in Noise Training for Autism.pdf