1. Nicolas Van Labeke
Learning Sciences Research Institute
National Biomedical Research Unit in Hearing
www.lsri.nottingham.ac.uk/nvl/
http://hearing.nihr.ac.uk/
Auditory perceptual learning
and game design
An example for Tinnitus management

2. Perceptual Learning & Auditory Training
• Perceptual learning is the relatively
permanent change of perception
following sensory experience
• Auditory training improves listening
and language/literacy skills and
reduces the handicap associated with
hearing impairment
• A wide range auditory training
software exists

3. Auditory Training
Time
Skill
Social
Cognitive
Pure tone
Discrimination
Phoneme
Discrimination
Speech
Intelligibility

4. The Problem
Sweetow & Sabes (2006, p555)
“Despite the high motivation and willingness of the subjects
in this project, over 20% initially enrolled in the project
dropped out of the study.
It is clear that not all patients will be willing to put forth
the time or effort required to complete this, or any,
rehabilitation program [...].
This cannot be blamed entirely on patients, however. Some
audiologists may be reluctant to require patients to add
further effort beyond the purchase of hearing devices.”
Potentially the problem arises because the training is derived
from auditory testing paradigms based on psychophysics &
signal detection theory

5. Auditory Testing/Training (1)
STAR package – MRC Institute of Hearing Research, Nottingham
http://www.ihr.mrc.ac.uk/

6. Auditory Testing/Training (2)
Sound Express – TigerSpeech Technology
http://www.tigerspeech.com/tst_soundex.html

7. Auditory Testing/Training (3)
Hoop Nut – Scientific Learning® BrainApps™
http://www.scilearn.com/products/brainapps/hoop-nut/

8. Psychophysical Procedures
Marvit et al. (2003)

9. The Solution (?)
• Combine auditory perceptual learning,
technology-enhanced learning and game
theories to create auditory learning games
suitable for use outside the lab
– training for learning  testing of learning
– User-Centred approach of design
– Design games that are intrinsically motivating
– Wide range of age- and disability-related factors
of people with hearing loss favours a casual
game approach
– Principles of auditory learning suggest the use of
intrinsic integration (Habgood 2007)

10. ?
Game-based Auditory Learning
Auditory Testing/Training Auditory Learning Game
• Flexible core mechanics
– Interaction with stimuli (doing
something with them)
– Creating purpose and aim for
individual actions
 Better conditions for realisation of
motivational factors
 Engaging flow experience
 Design Space?
• Psychoacoustic-based:
– Interaction limited to stimuli
delivery/detection mechanism
– Repetitive and monotonous
– aimless and purposeless at
short-term
 Unfavourable conditions for
realisation of motivational factors
 Monotonous flow experience
 Limited user engagement
(hence extrinsic motivation)
? ??

11. Game-based Auditory Learning
Auditory Bubble
• Training for learning
 testing of learning
• Used pre-existing game
• First iteration:
– Define elements of
auditory games
– Graft auditory learning
material on core
mechanics
Frozen Bubble
http://www.frozen-bubble.org/

12. ?
Game-based Auditory Learning
Performance & Psychophysical procedures
Auditory Testing/Training Auditory Learning Game
• Performance on the game
is well defined
– Performance on auditory task
is unknown
– What/how to measure? How
to tailor difficulty
• Effects of divided attention
– visual & auditory tasks
• Performance on the
auditory task is well
defined
– Measurement of auditory
learning
– Tailoring of auditory task
difficulty
• Limited cognitive load
? ?
=> Intrinsic Integration as design methodology

13. Intrinsic Integration
• How do we design effective learning games?
• Deploy learning material
• Motivational factors: challenge, fantasy,
curiosity, control, cooperation, competition,
recognition (Malone & Lepper 1987)
• Habgood (2007): Core mechanics
– Rules of the game
– Essential interactions required to create gaming
experience
– Explain individual preferences to games
– Define different game genre & flow experience

14. Intrinsic Integration (Habgood 2007)
• Motivational effects of challenge, control, etc. are
realized through the core mechanics
• Integrate the learning material with the core
mechanics and NOT the fantasy
“Deliver learning material through the parts of the
game that are the most fun to play, riding on the
back of the flow experience produced by the game,
and not interrupting or diminishing its impact”
“Embody the learning material within the structure
of the gaming world and the player’s interactions
with it, providing an external representation of the
learning content that is explored through the core
mechanics of the gameplay”

15. Game-based
Auditory Discrimination
Therapy
for Tinnitus

16. Tinnitus
• Perception of sound in the
absence of any
corresponding external
sound
• may be heard in one or both ears
or in the middle of the head; may
be difficult to pinpoint its exact
location; may be low, medium or
high-pitched; may be a single
noise or two or more
components; may be continuous
or come and go.
• Management use mix of
counselling (self-help) and
sound-based therapies
– Tinnitus Masking (TM): sense of
relief, trying to completely mask
the tinnitus
– Tinnitus Retraining Therapy (TRT):
reaching mixing point with
ambient noise; habituation &
emotional response
• Auditory Discrimination
Therapy ?
230,000 adults in the UK have tinnitus to a degree that
has a severe effect on their ability to lead a normal
life.
There are 470,000 adults with tinnitus that has a severe
effect on their quality of life.
2.3 million adults have tinnitus that makes it difficult for
them to sleep.
2.3 million adults have tinnitus that they find severely
to moderately annoying.
4.7 million adults have experienced tinnitus for longer
than five minutes - this does not include tinnitus
experienced for a short time after exposure to loud noise.
(RNID 2009)

17. Neuroscience of Tinnitus in 25s
tonotopy & neuroplasticity
Adjamian et al. (2009)
Herraiz et al (2009)
König et al. (2006)

18. Moderate quality = Further studies are likely to impact confidence in the estimate of effect and may
change the estimate.
Low quality = Further studies are very likely to change confidence and likely to change the estimate
Hoare et al (2010)

19. Game-based ADT for Tinnitus
Management
• Continuation of NBRUH Tinnitus studies
• Evaluating different game-based paradigms
– STARv2 (baseline)
• Active listening, trial-based tasks
– Auditory Submarine
• Interactive listening (sonar analogy)
• system-induced gameplay (coerced decision-making)
– Auditory Treasure Hunter
• Interactive listening (metal detector analogy)
• listener-induced gameplay (un-coerced, autonomous
decision-making)
• Impact of visual representation in gameplay and
task performance
– Distractor / support / scaffolding
– Effectiveness of auditory learning games

20. Learning Material : Frequency
Discrimination
• Pure tone signal
• Stimuli parameters:
– s : 200ms
– is: 500ms
– Ft : “near” legion-edge
(eg 2.5KHz)
– Fc : 50% Ft , down to
threshold (just noticeable
difference)
• Constant loudness
• Identical signal duration
• ADSR envelope for
“clean” output (usually
AR ramp 20ms)
2I-2AFC
3I-3AFC
oddball

21. Auditory Submarine
Interactive listening
2-Interval discrimination
Coerced gameplay

25. (Theoretical) Difference Limen
=
the smallest change in frequency
that can be detected subjectively
=
Gate size

26. 0
2
4
6
8
10
12
14
16
18
0
200
400
600
800
1000
1200
1400
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Level-by-level adaptive mechanism
Level
Delta (4%)
Delta (10%)
Delta (%20)
Training Frequency:
2.5 KHz
Staircase: 1-down, 1-up
Base: 50%
Step: 4% - 10% - 20%
Theoretical
Difference
Limen (Hz)
Game
Level
Runs

27. Gate 1
Gate 2
Gate 5
. . . . . .
Game Levels
• Level associated with
particular Frequency
Difference Limens
• Level consist of running
through 5 gates
– Initial gates acting as
“training” to current DL
– Final gate acting as
“validation” of training &
access to next level
• “Lives” introduced to
limit repetitive failures
• Time as a performance
marker
– Speed increase
– booster mode

28. Gate 1
Gate 2
VisualAuditory
VisualAuditory
Auditory
Gate 5
. . . . . .
Visual Elements
• Visual elements act
as support to
auditory task
– Gate appearing only
after some time in
the run
• Scaffolding =
decay over time
– by gate (50%)
– by level (staircase)
• Other visual
elements?
– E.g. Submarine
fading out

30. Auditory Treasure Hunter
Interactive listening
3-Interval discrimination
Un-coerced strategic
gameplay

31. 20
60
40
160 160
120
200

33. 160

36. 0
2
4
6
8
10
12
14
16
18
0
200
400
600
800
1000
1200
1400
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Level-by-level adaptive mechanism
Level
Delta (4%)
Delta (10%)
Delta (%20)
Game
Level
Runs
Theoretical Difference
Limen (DELTA)
Level n

37. Game Levels
• Level associated with
particular Frequency
Difference Limens
• Un-coerced goals
– Target as amount of
gold to collect
• Define success and
failure of level
– Limited by number of
“lives” (digging)
– Time not a constraint
(auditory and
cognitive processing)
• Layout of level
(zones, nuggets)
– Increase difficulty?
– Increase playability?
6
4

38. 20
60
40
160 160
120
200

39. Visual Elements
• Visual elements act as
complement to
auditory task
– Missing information
(depth and value of
nuggets)
• How to make sure
they do not replace
the auditory task?
– “Detection” scheme for
left and right
– fading out of
content, depth and
location
• Within level (actions)
• Across level (staircase)
– Memory?

41. By Proximity

42. By Value

43. By Value

44. Study overview
• Cross-over design
– Comparison of treatments
(training phase 1)
– Comparison of game experience
(training phases 1-3)
• Evaluation based on tinnitus-
related outcomes measures AND
desirability & flow experience
– Desirability & semi-structured
interview at Phase 1
– Usability at Phase 2 & 3
– Preference at Phase 3
• Iterative Design process
– Prototype development
– Alpha testing: usability with
NBRUH/LSRI members (mostly
normal hearing)
– Beta testing: piloting and usability
with participants of previous
Tinnitus Study
STARv2
Auditory
Submarine
Auditory
Treasure Hunter
Phase 1 Phase 2 Phase 3

45. Questions
• Game-based training paradigm
– Impact on auditory tasks, stimuli, processes and
performance?
Frequency Discrimination?
Using other dimensions
of auditory stimuli?

46. Questions
• Intrinsic integration of core mechanics and
learning material
– Bespoke design vs. generalisation?
– Design guideline and toolkit?
– General principles?
• Interaction with stimuli (i.e. doing something with them)
 Creating purpose and aim for individual actions
 Engaging flow experience
 Better conditions for realisation of motivational factors
• Compliance vs. engagement
– Time-on-task as an outcome, not a requirement
• “lifelong” commitment?
– Impromptu return to task

47. Questions
• What aspects of the design of
educational games & other audio-
related computer systems are applicable
for auditory learning games?
• How deep should the intrinsic
integration between the auditory
material and the core mechanics be?
• How do we address the wide range of
age- and disability-related factors of
the hearing impaired?

48. References
Amitay, S., Hawkey, D. J., & Moore, D. R.
(2005). Auditory frequency discrimination
learning is affected by stimulus variability.
Perception & Psychophysics, 67(4), 691-
698.
Moore, D. R., Rosenberg, J. F., & Coleman, J.
S. (2005). Discrimination training of
phonemic contrasts enhances phonological
processing in mainstream school children.
Brain and Language, 94(1), 72-85.
Smith, G.
E., Housen, P., Yaffe, K., Ruff, R., Kenniso
n, R. F., Mahncke, H. W., Zelinski, E. M.
(2009). A cognitive training program
based on principles of brain plasticity:
results from the Improvement in Memory
with Plasticity-based Adaptive Cognitive
Training (IMPACT) study. Journal of the
American Geriatrics Society, 57(4), 594-
603.
Sweetow, R. W., & Sabes, J. H. (2006). The
need for and development of an adaptive
Listening and Communication
Enhancement (LACE) Program. Journal of
the American Academy of
Audiology, 17(8), 538-558.
de Miranda, E. C., Gil, D., & Iório, M. C. M.
(2008). Formal auditory training in elderly
hearing aid users. Revista Brasileira de
Otorrinolaringologia, 74(6).
Fu, Q., & Galvin, J. J. (2007). Computer-
Assisted Speech Training for Cochlear
Implant Patients:
Feasibility, Outcomes, and Future
Directions. Seminars in
Hearing, 28(2), 142-150.
Moore, D. R., Ferguson, M. A., Halliday, L.
F., & Riley, A. (2008). Frequency
discrimination in children:
Perception, learning and attention.
Hearing Research, 238(1-2), 147-154.
Malone, T. W., & Lepper, M. (1987). Making
learning fun: A taxonomy of intrinsic
motivations for learning. In
Aptitude, Learning and Instruction:
Conative and affective process
analyses, Snow & Farr (Eds.), pp. 253-
223.
Habgood, M. P. J. (2007). The Effective
Integration of Digital Games and Learning
Content. PhD thesis, University of
Nottingham.
Ross, M. (2005). Home-based auditory and
speech reading training; A Review of Four
Programs. Hearing loss
Magazine, November/December, pp. 30-
34.