MirrorMirror: A Mobile Application to Improve Speechreading Acquisition, presentation at CHI 2018, Montreal, Canada.

1. MirrorMirror:
A Mobile Application to Improve
Speechreading Acquisition
Benjamin M. Gorman
David R. Flatla
Digitally-Augmented Perception Research

2. Benjamin M. Gorman
David R. Flatla
Digitally-Augmented Perception Research
MirrorMirror:
A Mobile Application to Improve
Speechreading Acquisition

3. Million People
360
~5% World’s population
have hearing loss
World Health Organization 2015

4. ? ?
COMMUNIC ATION
Barriers

5. Cece Bell, El Deafo (2014)
Speechreading (lipreading) =

6. To Many To ManyOne One
VISEMENEMES PHONEMES
(A 'Lipshape') (A 'speech sound')

7. Cece Bell, El Deafo (2014)

8. Speechreading Classes
• Speechreading classes primarily
focus on teaching people how to
speechread more effectively
• However, classes are only provided
in a handful of countries around
the world

9. In our 2017 CHI Paper we interviewed speechreading tutors to
determine how new Speechreading Acquisition Tools could help:

10. • Currently, there are a limited
number of Speechreading
Acquisition Tools (SATs) that
support speechreading practice
outside of class
• It is unclear how often these are
used by speechreaders
Current SATs

11. lipreading.orglipreadingpractice.co.uk
Current SATs

12. • A limited selection of content
• A limited selection of speakers
• The user cannot customise the
content with particular words,
people or situations they
encounter daily.
Current SATs

13. • We conducted a postal
questionnaire with speechreading
students
• So we could begin to understand
how students practiced and used
speechreading outside of class
Postal Questionnaire

14. Participants
• 59 Participants completed the postal
questionnaire.
• Sourced from four speechreading
tutors.
• Aged between 45-93 years old
(M=73.9, SD=10.1)
• Around 3/4 (76%) were Female

15. Hearing Loss Classi!cation
Profound
Severe
Moderate
Mild
5
19
28
5
No. of Participants
All participants reported
having a hearing loss
Participants:
Hearing Loss

16. Number of Years
0-5 Years
5-10 Years
10-15 Years
15-20 Years
20 Years +
Not Given
8
10
5
10
22
4
No. of Participants
Participants:
Hearing Loss
Majority of participants had a
hearing loss for over 10 years

17. • 56 participants reported using
hearing aids
• One participant used cochlear
implants
• Two stated they did not use any
assistive technology
Participants:
Assistive Technology

18. Questionnaire Findings:
Speechreading
• Over 50% of questionnaire
participants reported attending
speechreading classes for over two
years …
• … yet less than a third (32%) of
participants rated their
speechreading ability as "Good"
and nobody rated it "Excellent".

19. • 66% of participants report that
they practice outside of class
• When we interviewed tutors, they
reported that students practiced
using techniques from classes …
Questionnaire Summary:
Do students practice outside of class?

20. • However, participants reported
primarily practicing by using:
Observation and watching Television
Questionnaire Summary:
How do they practice?

21. Frequency
Daily
Once a Week
2-3 Times a Week
1-2 Times per Month
1-2 Times per Year
Never
Not Given
9
4
4
1
2
19
20
8
5
2
7
2
20
15
Subtitles O! Subtitles On
Questionnaire Findings :
Video & TV Practice
Frequency of practice with subtitles on/off

22. • Both of these techniques provide an
endless supply of practice material
• When using television and
observation, it is difficult for them
to assess whether they are
understanding the speaker correctly
Questionnaire Summary:
How do they practice?

23. Frequency
Daily
Once a Week
2-3 Times a Week
1-2 Times per Month
1-2 Times per Year
Never
Not Given
0
4
0
5
4
17
29
No. of Participants
Questionnaire Findings :
Mirror Practice
Frequency of mirror practice

24. Likes:
• Helps them to learn lipshapes
• Compare lipshaps between
speakers
• Show visual differences between
words
Questionnaire Summary:
Mirror Practice

25. Dislikes:
• Watching themselves in the
mirror
• That they have full knowledge of
what they are saying
• That they over-emphasise words
Questionnaire Summary:
Mirror Practice

26. • Tutors reported that students
often use some of the SATs
discussed earlier
• However, none of the
questionnaire participants
reported using any current SATs
Questionnaire Summary:
Technology use?

27. Device
iPad
iPhone
Amazon Kindle
Amazon Kindle Fire
Tablet
Android Smartphone
Mobile Phone
Windows Phone
Laptop
20
9
14
2
5
6
21
1
2
No. of Participants
Questionnaire Findings :
Mobile Device Usage

28. Situations
Group Conversations
Restaurants
Co!ee Shops
Transport
Dentist
Shopping
Doctors
Opticians
Home
54
46
32
33
26
23
17
16
10
General 4
Noisy Places 3
Work 2
Classes 1
No. of Participants
Questionnaire Findings :
Situations

29. Challenge
People turning away
People talking quickly
People covering mouths
Quiet Speakers
Words looking the same on the lips
Accents
Fatigue
Beards/Facial Hair
Concentration
54
53
49
44
43
34
33
31
25
Other 6
No. of Participants
Questionnaire Findings :
Challenges

30. Questionnaire Summary:
• 1) Current SATs (e.g., lipreading.org,
lipreadingpractice.co.uk) have
limited content
• 2) TV and observation provide
limited feedback
• 3) Mirror practice provides too
much feedback

31. Questionnaire Summary:
• 1) Current SATs (e.g., lipreading.org,
lipreadingpractice.co.uk) have
limited content
• 2) TV and observation provide
limited feedback
• 3) Mirror practice provides too
much feedback

32. Questionnaire Summary:
• 1) Current SATs (e.g., lipreading.org,
lipreadingpractice.co.uk) have
limited content
• 2) TV and observation provide
limited feedback
• 3) Mirror practice provides too
much feedback

33. MirrorMirror

34. Speaker Tab

35. Speaker Tab - Adding Speaker
Benjamin
Gorman

36. Library Tab

37. Library Tab

38. Word List - Adding Words

39. Record Video

41. Practice Tab

42. Lipshape Practice

43. Lipshape Practice

45. Evaluation: Case Study
• Stage 1: Briefing & Tutorial Session
• Stage 2: In-The-Wild-Deployment
• Stage 3: Post-Deployment Discussion Session

46. Participants
Participant
P1 (Male, 74)
P2 (Female, 54)
P3 (Male, 75)
Moderate-to-severe
Profound
Severe
Hearing Loss
1 Year - Fair
2 Years - Fair
2 Years - Good
Speechreading Ability
3 participants, recruited from
speechreading classes

47. Stage 1: Briefing & Tutorial
Session
• We met with each participant and
explained the study
• The participant then completed a
similar questionnaire discussed earlier
• The participants were given a tutorial
on how to use the features of
MirrorMirror

48. Stage 2: In-The-Wild-
Deployment
• Participants were given an Android
tablet (Samsung Galaxy Tab 3)
with MirrorMirror pre-installed.
• Participants were asked to use the
app for daily speechreading
practice for around a week.

49. Task Results: Speakers
• P1 added two speakers (his wife
and his son),
• P2 added four speakers (herself,
her husband, her daughter, and a
colleague), and
• P3 added two speakers (himself
and his wife).

50. Lipshape
P/B/M
S/D/T
K/G/N
Ch/Sh/J
L/N/K
Z/T/S
3
3
3
3
3
3
3
3
4
3
3
3
P1 P2
6
6
6
6
6
6
P3
Task Results: Words
Words added to each Lipshape

51. Speaker
P1
Author
Wife
Son
Total
0
1*
12
6
19
No. Videos Recorded
Task Results: P1 Videos
P1 added 19 videos overall
*added during tutorial

52. Speaker
P2
Daughter
Husband
Colleague
Total
36
36
36
37
145
No. Videos Recorded
Task Results: P2 Videos
P2 added 145 videos overall

53. Speaker
P3
Wife
Total
54
17
71
No. Videos Recorded
Task Results: P3 Videos
P3 added 71 videos overall

54. Task Results:
Lipshape Practice Sessions
Participants made great use of Lipshape Practice, using it
multiple times per day during the evaluation.
Participant
P1
P2
P3
14
72
43
No. Sessions
76
706
367
No. Trials
62
462
234
No. Correct
14
244
133
No. Incorrect

55. Stage 3: Post-deployment
Discussion
• After the deployment, we met
with each participant to discuss
MirrorMirror in a questionnaire-
based structured interview
• Interviews were audio recorded
for later transcription

56. Post-Deployment
Discussion: P1
"Quite easy, straightforward…and quite helpful"
and felt it was
"interesting to be able to try [and] work out which
word was being used"

57. Post-Deployment
Discussion: P1
And when asked if he had practiced
speechreading with his family before he said:
"No I maybe tried once with my wife…maybe
lasted a minute or two but really no."

58. Post-Deployment
Discussion: P1
"Yes … because I wasn't practicing often
enough but [MirrorMirror] would be a nice
easy way to practice particularly if there were
videos of people I didn't know how they
spoke."

59. Post-Deployment
Discussion: P2
"It was interesting seeing my daughter, I found
her lipreading quite easy...I think I got more of her
ones right than I did my husband."

60. Post-Deployment
Discussion: P2
"I think if there was something in particular that
you were going to, an event or something and
you knew you were going to be asked certain
questions [MirrorMirror] would be really handy."

61. Post-Deployment
Discussion: P3
"…found it very interesting."
adding that he
"liked the basic idea of it. It's different from
what [his] training in lipreading has been."

62. Post-Deployment
Discussion: P3
"…interesting with [MirrorMirror] that words
like parked and packed and I was surprised
I could actually tell the difference…
I got the 'r'…and that made me think that this
was useful".

63. Next Steps
• Our results indicate that MirrorMirror
could improve participants' ability to
practice outside of class
• Our evaluation does not evaluate
MirrorMirror's effectiveness in
improving overall speechreading ability

64. Next Steps
• We would like to next look at deploying
MirrorMirror in speechreading classes
over a semester/term
• It would also be interesting to deploy to
individuals who have hearing loss but
have not taken speechreading classes

65. Generalisations &
Extensions
MirrorMirror could be extended in two ways:
• Adding a 'Sentence Practice' mode to the
'Practice' tab
• Adding a share video library feature would
allow students to share their recorded
videos with other class members

66. Generalisations &
Extensions
• By updating the lipshape categories,
MirrorMirror could be extended to support
speechreading practice in other languages
• MirrorMirror could also be adapted to help
people learn a foreign language. This could
help with learning pronunciation or region-
specific dialects

67. Summary
• Current Speechreading Acquisition Tools
(SATs) are not adaptable to individual students
• To address the limitations we conducted a
postal questionnaire with speechreading
students to gather requirements for a new SAT
• And used our findings to develop MirrorMirror
• We evaluated MirrorMirror and found that it
improved participants' ability to practice
outside of classes

68. Thanks!
Benjamin Gorman
b.gorman@dundee.ac.uk
Digitally-Augmented Perception Research