Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MirrorMirror: A Mobile Application to Improve Speechreading Acquisition

56 views

Published on

MirrorMirror: A Mobile Application to Improve Speechreading Acquisition, presentation at CHI 2018, Montreal, Canada.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

MirrorMirror: A Mobile Application to Improve Speechreading Acquisition

  1. 1. MirrorMirror: A Mobile Application to Improve Speechreading Acquisition Benjamin M. Gorman David R. Flatla Digitally-Augmented Perception Research
  2. 2. Benjamin M. Gorman David R. Flatla Digitally-Augmented Perception Research MirrorMirror: A Mobile Application to Improve Speechreading Acquisition
  3. 3. Million People 360 ~5% World’s population have hearing loss World Health Organization 2015
  4. 4. ? ? COMMUNIC ATION Barriers
  5. 5. Cece Bell, El Deafo (2014) Speechreading (lipreading) =
  6. 6. To Many To ManyOne One VISEMENEMES PHONEMES (A 'Lipshape') (A 'speech sound')
  7. 7. Cece Bell, El Deafo (2014)
  8. 8. Speechreading Classes • Speechreading classes primarily focus on teaching people how to speechread more effectively • However, classes are only provided in a handful of countries around the world
  9. 9. In our 2017 CHI Paper we interviewed speechreading tutors to determine how new Speechreading Acquisition Tools could help:
  10. 10. • Currently, there are a limited number of Speechreading Acquisition Tools (SATs) that support speechreading practice outside of class • It is unclear how often these are used by speechreaders Current SATs
  11. 11. lipreading.orglipreadingpractice.co.uk Current SATs
  12. 12. • A limited selection of content • A limited selection of speakers • The user cannot customise the content with particular words, people or situations they encounter daily. Current SATs
  13. 13. • We conducted a postal questionnaire with speechreading students • So we could begin to understand how students practiced and used speechreading outside of class Postal Questionnaire
  14. 14. Participants • 59 Participants completed the postal questionnaire. • Sourced from four speechreading tutors. • Aged between 45-93 years old (M=73.9, SD=10.1) • Around 3/4 (76%) were Female
  15. 15. Hearing Loss Classi!cation Profound Severe Moderate Mild 5 19 28 5 No. of Participants All participants reported having a hearing loss Participants: Hearing Loss
  16. 16. Number of Years 0-5 Years 5-10 Years 10-15 Years 15-20 Years 20 Years + Not Given 8 10 5 10 22 4 No. of Participants Participants: Hearing Loss Majority of participants had a hearing loss for over 10 years
  17. 17. • 56 participants reported using hearing aids • One participant used cochlear implants • Two stated they did not use any assistive technology Participants: Assistive Technology
  18. 18. Questionnaire Findings: Speechreading • Over 50% of questionnaire participants reported attending speechreading classes for over two years … • … yet less than a third (32%) of participants rated their speechreading ability as "Good" and nobody rated it "Excellent".
  19. 19. • 66% of participants report that they practice outside of class • When we interviewed tutors, they reported that students practiced using techniques from classes … Questionnaire Summary: Do students practice outside of class?
  20. 20. • However, participants reported primarily practicing by using: Observation and watching Television Questionnaire Summary: How do they practice?
  21. 21. Frequency Daily Once a Week 2-3 Times a Week 1-2 Times per Month 1-2 Times per Year Never Not Given 9 4 4 1 2 19 20 8 5 2 7 2 20 15 Subtitles O! Subtitles On Questionnaire Findings : Video & TV Practice Frequency of practice with subtitles on/off
  22. 22. • Both of these techniques provide an endless supply of practice material • When using television and observation, it is difficult for them to assess whether they are understanding the speaker correctly Questionnaire Summary: How do they practice?
  23. 23. Frequency Daily Once a Week 2-3 Times a Week 1-2 Times per Month 1-2 Times per Year Never Not Given 0 4 0 5 4 17 29 No. of Participants Questionnaire Findings : Mirror Practice Frequency of mirror practice
  24. 24. Likes: • Helps them to learn lipshapes • Compare lipshaps between speakers • Show visual differences between words Questionnaire Summary: Mirror Practice
  25. 25. Dislikes: • Watching themselves in the mirror • That they have full knowledge of what they are saying • That they over-emphasise words Questionnaire Summary: Mirror Practice
  26. 26. • Tutors reported that students often use some of the SATs discussed earlier • However, none of the questionnaire participants reported using any current SATs Questionnaire Summary: Technology use?
  27. 27. Device iPad iPhone Amazon Kindle Amazon Kindle Fire Tablet Android Smartphone Mobile Phone Windows Phone Laptop 20 9 14 2 5 6 21 1 2 No. of Participants Questionnaire Findings : Mobile Device Usage
  28. 28. Situations Group Conversations Restaurants Co!ee Shops Transport Dentist Shopping Doctors Opticians Home 54 46 32 33 26 23 17 16 10 General 4 Noisy Places 3 Work 2 Classes 1 No. of Participants Questionnaire Findings : Situations
  29. 29. Challenge People turning away People talking quickly People covering mouths Quiet Speakers Words looking the same on the lips Accents Fatigue Beards/Facial Hair Concentration 54 53 49 44 43 34 33 31 25 Other 6 No. of Participants Questionnaire Findings : Challenges
  30. 30. Questionnaire Summary: • 1) Current SATs (e.g., lipreading.org, lipreadingpractice.co.uk) have limited content • 2) TV and observation provide limited feedback • 3) Mirror practice provides too much feedback
  31. 31. Questionnaire Summary: • 1) Current SATs (e.g., lipreading.org, lipreadingpractice.co.uk) have limited content • 2) TV and observation provide limited feedback • 3) Mirror practice provides too much feedback
  32. 32. Questionnaire Summary: • 1) Current SATs (e.g., lipreading.org, lipreadingpractice.co.uk) have limited content • 2) TV and observation provide limited feedback • 3) Mirror practice provides too much feedback
  33. 33. MirrorMirror
  34. 34. Speaker Tab
  35. 35. Speaker Tab - Adding Speaker Benjamin Gorman
  36. 36. Library Tab
  37. 37. Library Tab
  38. 38. Word List - Adding Words
  39. 39. Record Video
  40. 40. Practice Tab
  41. 41. Lipshape Practice
  42. 42. Lipshape Practice
  43. 43. Evaluation: Case Study • Stage 1: Briefing & Tutorial Session • Stage 2: In-The-Wild-Deployment • Stage 3: Post-Deployment Discussion Session
  44. 44. Participants Participant P1 (Male, 74) P2 (Female, 54) P3 (Male, 75) Moderate-to-severe Profound Severe Hearing Loss 1 Year - Fair 2 Years - Fair 2 Years - Good Speechreading Ability 3 participants, recruited from speechreading classes
  45. 45. Stage 1: Briefing & Tutorial Session • We met with each participant and explained the study • The participant then completed a similar questionnaire discussed earlier • The participants were given a tutorial on how to use the features of MirrorMirror
  46. 46. Stage 2: In-The-Wild- Deployment • Participants were given an Android tablet (Samsung Galaxy Tab 3) with MirrorMirror pre-installed. • Participants were asked to use the app for daily speechreading practice for around a week.
  47. 47. Task Results: Speakers • P1 added two speakers (his wife and his son), • P2 added four speakers (herself, her husband, her daughter, and a colleague), and • P3 added two speakers (himself and his wife).
  48. 48. Lipshape P/B/M S/D/T K/G/N Ch/Sh/J L/N/K Z/T/S 3 3 3 3 3 3 3 3 4 3 3 3 P1 P2 6 6 6 6 6 6 P3 Task Results: Words Words added to each Lipshape
  49. 49. Speaker P1 Author Wife Son Total 0 1* 12 6 19 No. Videos Recorded Task Results: P1 Videos P1 added 19 videos overall *added during tutorial
  50. 50. Speaker P2 Daughter Husband Colleague Total 36 36 36 37 145 No. Videos Recorded Task Results: P2 Videos P2 added 145 videos overall
  51. 51. Speaker P3 Wife Total 54 17 71 No. Videos Recorded Task Results: P3 Videos P3 added 71 videos overall
  52. 52. Task Results: Lipshape Practice Sessions Participants made great use of Lipshape Practice, using it multiple times per day during the evaluation. Participant P1 P2 P3 14 72 43 No. Sessions 76 706 367 No. Trials 62 462 234 No. Correct 14 244 133 No. Incorrect
  53. 53. Stage 3: Post-deployment Discussion • After the deployment, we met with each participant to discuss MirrorMirror in a questionnaire- based structured interview • Interviews were audio recorded for later transcription
  54. 54. Post-Deployment Discussion: P1 "Quite easy, straightforward…and quite helpful" and felt it was "interesting to be able to try [and] work out which word was being used"
  55. 55. Post-Deployment Discussion: P1 And when asked if he had practiced speechreading with his family before he said: "No I maybe tried once with my wife…maybe lasted a minute or two but really no."
  56. 56. Post-Deployment Discussion: P1 "Yes … because I wasn't practicing often enough but [MirrorMirror] would be a nice easy way to practice particularly if there were videos of people I didn't know how they spoke."
  57. 57. Post-Deployment Discussion: P2 "It was interesting seeing my daughter, I found her lipreading quite easy...I think I got more of her ones right than I did my husband."
  58. 58. Post-Deployment Discussion: P2 "I think if there was something in particular that you were going to, an event or something and you knew you were going to be asked certain questions [MirrorMirror] would be really handy."
  59. 59. Post-Deployment Discussion: P3 "…found it very interesting." adding that he "liked the basic idea of it. It's different from what [his] training in lipreading has been."
  60. 60. Post-Deployment Discussion: P3 "…interesting with [MirrorMirror] that words like parked and packed and I was surprised I could actually tell the difference… I got the 'r'…and that made me think that this was useful".
  61. 61. Next Steps • Our results indicate that MirrorMirror could improve participants' ability to practice outside of class • Our evaluation does not evaluate MirrorMirror's effectiveness in improving overall speechreading ability
  62. 62. Next Steps • We would like to next look at deploying MirrorMirror in speechreading classes over a semester/term • It would also be interesting to deploy to individuals who have hearing loss but have not taken speechreading classes
  63. 63. Generalisations & Extensions MirrorMirror could be extended in two ways: • Adding a 'Sentence Practice' mode to the 'Practice' tab • Adding a share video library feature would allow students to share their recorded videos with other class members
  64. 64. Generalisations & Extensions • By updating the lipshape categories, MirrorMirror could be extended to support speechreading practice in other languages • MirrorMirror could also be adapted to help people learn a foreign language. This could help with learning pronunciation or region- specific dialects
  65. 65. Summary • Current Speechreading Acquisition Tools (SATs) are not adaptable to individual students • To address the limitations we conducted a postal questionnaire with speechreading students to gather requirements for a new SAT • And used our findings to develop MirrorMirror • We evaluated MirrorMirror and found that it improved participants' ability to practice outside of classes
  66. 66. Thanks! Benjamin Gorman b.gorman@dundee.ac.uk Digitally-Augmented Perception Research

×