Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Chi 2017 presentation

131 views

Published on

At least 360 million people worldwide have disabling hearing loss that frequently causes difficulties in day-to-day conversations. Traditional technology (e.g., hearing aids) often fails to offer enough value, has low adoption rates, and can result in social stigma. Speechreading can dramatically improve conversational understanding, but speechreading is a skill that can be challenging to learn. To address this, we developed a novel speechreading acquisition framework that can be used to design Speechreading Acquisition Tools (SATs) – a new type of technology to improve speechreading acquisition. We interviewed seven speechreading tutors and used thematic analysis to identify and organise the key elements of our framework. We then evaluated our framework by using it to: 1) categorise every tutor-identified speechreading teaching technique, 2) critically evaluate existing conversational aids, and 3) design three new SATs. Through the use of SATs designed using our framework, the speechreading abilities of people with hearing loss around the world should be enhanced, thereby improving the conversational foundation of their day-to-day lives.

Full Presentation: https://www.dropbox.com/s/qts1g2nguj6xixv/CHI_2017_Presentation.key?dl=0

Teaser Video: https://vimeo.com/196870070

Paper: http://dl.acm.org/citation.cfm?doid=3025453.3025560

  • Be the first to comment

Chi 2017 presentation

  1. 1. Benjamin M. Gorman AFramework for Speechreading AcquisitionTools David R. Flatla
  2. 2. Benjamin M. Gorman AFramework for Speechreading AcquisitionTools David R. Flatla
  3. 3. CAN YOU READ MY LIPS? from Little Moving Pictures
  4. 4. CAN YOU READ MY LIPS? from Little Moving Pictures
  5. 5. 1 in 6 People Action on Hearing Loss Information 2011 (UK)
  6. 6. Million People 360 ~5% World’s population have disabling hearing loss World Health Organization 2015
  7. 7. ? COMMUNICATION Barriers
  8. 8. Only of people with hearing loss regulary use hearing aids 14% Action on Hearing Loss Information,2011 (UK)
  9. 9. SPEECHREADING can break barriers and improve understanding
  10. 10. SPEECHREADING can break barriers and improve understanding
  11. 11. Teeth Nasal Cavity Lips Vocal Chords Tongue
  12. 12. Teeth Nasal Cavity Lips Vocal Chords Tongue
  13. 13. Cece Bell, El Deafo (2014)
  14. 14. Speechreading can improve: Educational outcomes Employability Social isolation And restore conversational con!dence
  15. 15. To ManyOne VISEME PHONEMES
  16. 16. Cece Bell, El Deafo (2014)
  17. 17. Cece Bell, El Deafo (2014)
  18. 18. SPEECHREADING CLA SSES
  19. 19. 50Classes Available Across Scotland
  20. 20. 275More Classes To Meet Demand There needs to be
  21. 21. Speechreading can dramatically improve conversational understanding. However, speechreading is a skill that can be challenging to learn. We want to design technology to help address this problem.
  22. 22. Speechreading can dramatically improve conversational understanding. However, speechreading is a skill that can be challenging to learn. We want to design technology to help address this problem.
  23. 23. Speechreading can dramatically improve conversational understanding. However, speechreading is a skill that can be challenging to learn. We want to design technology to help address this problem.
  24. 24. §We developed a framework that can be used to design – a new type of technology to improve speechreading acquisition Speechreading Acquisition Tools
  25. 25. §We developed a framework that can be used to design – a new type of technology to improve speechreading acquisition Speechreading Acquisition Tools
  26. 26. RESEARCH QUESTIONS 1. Do tutors teach analytic or synthetic training methods? 2. What do tutors consider to be the teachable unit of lipreading? 3. What technology if any is currently used in teaching? 4. Could lipreading as a technique be improved with new technology or training approaches? 5. How do students practice when they are not in class?
  27. 27. RESEARCH QUESTIONS 1. Do tutors teach analytic or synthetic training methods? 2. What do tutors consider to be the teachable unit of lipreading? 3. What technology if any is currently used in teaching? 4. Could lipreading as a technique be improved with new technology or training approaches? 5. How do students practice when they are not in class?
  28. 28. RESEARCH QUESTIONS 1. Do tutors teach analytic or synthetic training methods? 2. What do tutors consider to be the teachable unit of lipreading? 3. What technology if any is currently used in teaching? 4. Could lipreading as a technique be improved with new technology or training approaches? 5. How do students practice when they are not in class?
  29. 29. TEACHING METHODS
  30. 30. Synthetic Basic Unit: Sentence (CONTEXT)
  31. 31. Analytic Basic Unit: Syllable or Phoneme (EYE TRAINING)
  32. 32. RESEARCH QUESTIONS 1. Do tutors teach analytic or synthetic training methods? 2. What do tutors consider to be the teachable unit of lipreading? 3. What technology if any is currently used in teaching? 4. Could lipreading as a technique be improved with new technology or training approaches? 5. How do students practice when they are not in class?
  33. 33. RESEARCH QUESTIONS 1. Do tutors teach analytic or synthetic training methods? 2. What do tutors consider to be the teachable unit of lipreading? 3. What technology if any is currently used in teaching? 4. Could lipreading as a technique be improved with new technology or training approaches? 5. How do students practice when they are not in class?
  34. 34. RESEARCH QUESTIONS 1. Do tutors teach analytic or synthetic training methods? 2. What do tutors consider to be the teachable unit of lipreading? 3. What technology if any is currently used in teaching? 4. Could lipreading as a technique be improved with new technology or training approaches? 5. How do students practice when they are not in class?
  35. 35. We interviewed 7 speechreading tutors to see how acquisition could be improved with technology.
  36. 36. All Female 7 Tutors Aged between 42-78 4/7 with hearing loss Teaching Experience: 6 months - 32 years
  37. 37. All Female 7 Tutors Aged between 42-78 4/7 with hearing loss Teaching Experience: 6 months - 32 years
  38. 38. All Female 7 Tutors Aged between 42-78 4/7 with hearing loss Teaching Experience: 6 months - 32 years
  39. 39. All Female 7 Tutors Aged between 42-78 4/7 with hearing loss Teaching Experience: 6 months - 32 years
  40. 40. All Female 7 Tutors Aged between 42-78 4/7 with hearing loss Teaching Experience: 6 months - 32 years
  41. 41. THEMATIC ANALYSIS
  42. 42. Thematic Map Speechreading Classes Speechreading as a Skill Access To Speechreading Continual vs Fixed Attitudes to Technology Open to New Approaches Critical of Current Approaches Teaching Practices Varied Techniques Synthetic vs Analytic
  43. 43. FINDINGS
  44. 44. FRAMEWORK DESIGN
  45. 45. FRAMEWORK DESIGN
  46. 46. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW
  47. 47. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW
  48. 48. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW
  49. 49. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW
  50. 50. FRAMEWORK EVALUATION
  51. 51. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Teaching Techniques
  52. 52. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Mirror Practice QREs Word Quizzes (name the animals) Cue Recognition (body language) Pair Work Framed Sentence Excercises Scenarios Mystery Object Stories Speech Movements Lip Shapes Finger Spelling (initial letter) Teaching Techniques
  53. 53. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Mirror Practice QREs Word Quizzes (name the animals) Cue Recognition (body language) Pair Work Framed Sentence Excercises Scenarios Mystery Object Stories Speech Movements Lip Shapes Finger Spelling (initial letter) Teaching Techniques
  54. 54. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Mirror Practice QREs Word Quizzes (name the animals) Cue Recognition (body language) Pair Work Framed Sentence Excercises Scenarios Mystery Object Stories Speech Movements Lip Shapes Finger Spelling (initial letter) Teaching Techniques
  55. 55. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Mirror Practice QREs Word Quizzes (name the animals) Cue Recognition (body language) Pair Work Framed Sentence Excercises Scenarios Mystery Object Stories Speech Movements Lip Shapes Finger Spelling (initial letter) Teaching Techniques
  56. 56. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Mirror Practice QREs Word Quizzes (name the animals) Cue Recognition (body language) Pair Work Framed Sentence Exercises Scenarios Mystery Object Stories Speech Movements Lip Shapes Finger Spelling (initial letter) Teaching Techniques
  57. 57. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Mirror Practice QREs Word Quizzes (name the animals) Cue Recognition (body language) Pair Work Framed Sentence Exercises Scenarios Mystery Object Stories Speech Movements Lip Shapes Finger Spelling (initial letter) Teaching Techniques
  58. 58. FRAMEWORK EVALUATION
  59. 59. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages LipreadingPractice CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi Related Work
  60. 60. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages LipreadingPractice CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi Related Work
  61. 61. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages LipreadingPractice CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi Related Work
  62. 62. FRAMEWORK EVALUATION
  63. 63. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages LipreadingPractice CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi
  64. 64. PhonemeViz
  65. 65. PhonemeViz
  66. 66. Skill Type Amountof Information LipreadingPractice ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages PhonemeViz MirrorMirror ContextCueView CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi
  67. 67. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages LipreadingPractice CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi
  68. 68. MirrorMirror
  69. 69. Skill Type Amountof Information LipreadingPractice ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages PhonemeViz MirrorMirror ContextCueView CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi
  70. 70. Skill Type Amountof Information LipreadingPractice ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages PhonemeViz MirrorMirror ContextCueView CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi
  71. 71. Speechreading: A Way to Improve Understanding Kaplan et al. (1985) Constellation Diagram
  72. 72. ContextCueView
  73. 73. ContextCueView
  74. 74. Skill Type Amountof Information LipreadingPractice ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages PhonemeViz MirrorMirror ContextCueView CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi
  75. 75. Speechreading can dramatically improve conversational understanding. However, speechreading is a skill that can be challenging to learn. We want to design technology to help address this problem.
  76. 76. Speechreading can dramatically improve conversational understanding. However, speechreading is a skill that can be challenging to learn. We want to design technology to help address this problem.
  77. 77. Speechreading can dramatically improve conversational understanding. However, speechreading is a skill that can be challenging to learn. We want to design technology to help address this problem.
  78. 78. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW SAT Framework
  79. 79. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Mirror Practice QREs Word Quizzes (name the animals) Cue Recognition (body language) Pair Work Framed Sentence Excercises Scenarios Mystery Object Stories Speech Movements Lip Shapes Finger Spelling (initial letter) Teaching Techniques
  80. 80. Skill Type Amountof Information ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages LipreadingPractice CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi Related Work
  81. 81. Skill Type Amountof Information LipreadingPractice ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages PhonemeViz MirrorMirror ContextCueView CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi Designing new SATs
  82. 82. Skill Type Amountof Information LipreadingPractice ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages PhonemeViz MirrorMirror ContextCueView CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi Summary & Conclusion b.gorman@dundee.ac.uk @benjgorman
  83. 83. EXTENSIONS
  84. 84. Skill Type Amountof Information LipreadingPractice ANALYTIC HYBRID SYNTHETIC HIGHMEDLOW Cued Speech, Spectrograms Subtitles, Signed Languages PhonemeViz MirrorMirror ContextCueView CME (C) CME (B) DAVID Upton, Ebrahimi TactileSATs, CME(A) Lip Assistant, iBaldi Summary & Conclusion b.gorman@dundee.ac.uk @benjgorman

×