Auditory training has been shown to promote perceptual learning, i.e. the modification of perception and behaviour following sensory experience and evidences are showing. The efficacy of training often depends on the degree to which the training paradigm is interactive, immersive, and engaging. Our aim is to investigate how auditory perceptual learning, educational technologies and game design can be further combined into an approach of training that is suitable for use by individuals outside the laboratory, e.g. on home computers or mobile devices.

1. Intrinsic Integration and the Design of Games for Auditory Perceptual Learning Interactive Technologies and Games: Education, Health and Disability Nottingham 27 October 2009 Nicolas Van Labeke, Daniel Shub, Mike Sharples National Biomedical Research Unit in HearingLearning Sciences Research Institute – University of Nottingham www.lsri.nottingham.ac.uk/nvl/ http://hearing.nihr.ac.uk/

2. Perceptual Learning & Auditory Training Perceptual learning is the relatively permanent change of perception following sensory experience Auditory training improves listening and language/literacy skills and reduces the handicap associated with hearing impairment A wide range auditory training software exists

3. The Problem Sweetow & Sabes (2006, p555) “Despite the high motivation and willingness of the subjects in this project, over 20% initially enrolled in the project dropped out of the study. It is clear that not all patients will be willing to put forth the time or effort required to complete this, or any, rehabilitation program [...]. This cannot be blamed entirely on patients, however. Some audiologists may be reluctant to require patients to add further effort beyond the purchase of hearing devices.” Potentially the problem arises because the training is derived from auditory testing paradigms based on psychophysics & signal detection theory

4. Auditory Testing/Training (1) STAR package – MRC Institute of Hearing Research, Nottingham http://www.ihr.mrc.ac.uk/

5. Auditory Testing/Training (2) Sound Express – TigerSpeech Technology http://www.tigerspeech.com/tst_soundex.html

6. Auditory Testing/Training (3) Hoop Nut – Scientific Learning®BrainApps™ http://www.scilearn.com/products/brainapps/hoop-nut/

8. Intrinsic Integration (Habgood 2007) How do we design effective learning games? Learning material Motivational factors: challenge, fantasy, curiosity, control, cooperation, competition, recognition (Malone & Lepper 1987) Core mechanics Rules of the game Essential interactions required to create gaming experience Explain individual preferences to games Define different game genre & flow experience

10. Integrate the learning material with the core mechanics and NOT the fantasy “Deliver learning material through the parts of the game that are the most fun to play, riding on the back of the flow experience produced by the game, and not interrupting or diminishing its impact” “Embody the learning material within the structure of the gaming world and the player’s interactions with it, providing an external representation of the learning content that is explored through the core mechanics of the gameplay”

12. Better conditions for realisation of motivational factors

14. Unfavourable conditions for realisation of motivational factors

15. Limited user engagement (e.g. extrinsic motivation) ? ? ? ?

17. Auditory Learning GamesAuditory Bubble & Beyond Future iterations Principles of auditory learning Design guidelines Toolkits, authoring tools ? ? ?

18. ? Auditory Learning Games Performance & Psychophysical procedures Auditory Testing/Training Auditory Learning Game Performance on the game is well defined Performance on auditory task is unknown What to measure? How to measure? How to tailor difficulty Effects of divided attention visual & auditory tasks Performance on the auditory task is well defined allowing: Measurement of auditory learning Tailoring of auditory task difficulty Limited cognitive load ? ?

19. Questions What aspects of the design of educational games & other audio-related computer systems are applicable for auditory learning games? How deep should the intrinsic integration between the auditory material and the core mechanics be? How do we address the wide range of age- and disability-related factors of the hearing impaired?

20. Appendixes

21. Beyond Auditory Learning GamesAn Auditory Learning Framework Casual games Multiple small self-contained auditory solutions Several games for same auditory task (individual preferences), specific games for specific tasks Integrated framework Audio processing components Web-based deployment: online/offline, game recommendation, in-the-wild training (e.g. mobile) Technology-Enhanced Learning Learner Modelling / Cognitive modelling scaffolding, macro & micro adaptation Data integration (from testing, training, self-report, audiologist, …) and visualisation performance, history, progress, … Open Learner Modelling

22. Intrinsic Motivation and Learning Malone, T. W., & Lepper, M. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In Aptitude, Learning and Instruction: Conative and affective process analyses, Snow & Farr (Eds.), pp. 253-223.

23. Auditory Training training Social training Cognitive SpeechIntelligibility training Transfer Skill training PhonemeDiscrimination training Pure toneDiscrimination Time

24. References Amitay, S., Hawkey, D. J., & Moore, D. R. (2005). Auditory frequency discrimination learning is affected by stimulus variability. Perception & Psychophysics, 67(4), 691-698. Moore, D. R., Rosenberg, J. F., & Coleman, J. S. (2005). Discrimination training of phonemic contrasts enhances phonological processing in mainstream school children. Brain and Language, 94(1), 72-85. Smith, G. E., Housen, P., Yaffe, K., Ruff, R., Kennison, R. F., Mahncke, H. W., Zelinski, E. M. (2009). A cognitive training program based on principles of brain plasticity: results from the Improvement in Memory with Plasticity-based Adaptive Cognitive Training (IMPACT) study. Journal of the American Geriatrics Society, 57(4), 594-603. Sweetow, R. W., & Sabes, J. H. (2006). The need for and development of an adaptive Listening and Communication Enhancement (LACE) Program. Journal of the American Academy of Audiology, 17(8), 538-558. de Miranda, E. C., Gil, D., & Iório, M. C. M. (2008). Formal auditory training in elderly hearing aid users. RevistaBrasileira de Otorrinolaringologia, 74(6). Fu, Q., & Galvin, J. J. (2007). Computer-Assisted Speech Training for Cochlear Implant Patients: Feasibility, Outcomes, and Future Directions. Seminars in Hearing, 28(2), 142-150. Moore, D. R., Ferguson, M. A., Halliday, L. F., & Riley, A. (2008). Frequency discrimination in children: Perception, learning and attention. Hearing Research, 238(1-2), 147-154. Malone, T. W., & Lepper, M. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In Aptitude, Learning and Instruction: Conative and affective process analyses, Snow & Farr (Eds.), pp. 253-223. Habgood, M. P. J. (2007). The Effective Integration of Digital Games and Learning Content. PhD thesis, University of Nottingham. Ross, M. (2005). Home-based auditory and speech reading training; A Review of Four Programs. Hearing loss Magazine, November/December, pp. 30-34.