Details of the GReAT project examining and testing computer therapy design with people with severa aphasia.

1. Computer Gesture Recognition in
Aphasia Therapy:
findings from The GReAT Project
Abi Roper and Naomi Cocks
Division of Language and Communication Science
City University London

2. Today
•Gesture Therapy
•Existing Computer Therapies
•The GReAT Project
•Designing and Refining a Computer Gesture
Therapy - Gest
•Gest Demonstration
•Delivering a Computer Therapy
•Gest Pilot Study
•Outcomes and Implications

3. Previous Studies of Gesture Therapy
To compensate for speech (e.g. Daumuller &
Goldenberg, 2010)
To facilitate speech (e.g. Boo & Rose, 2011; Marangolo
et al, 2010; Rose & Douglas, 2008)
Findings
• Even people with severe aphasia can improve
gesture production
• Treatments with a gesture component can
enhance naming

4. But …
Gains are often very modest
Gains may relate to the intensity of therapy
(Caute, 2012)

5. Computer Therapy for Aphasia
• Sentence Shaper – Speech production
Exercises sentence level
• “The program is most likely to be effective for people
who are able to produce some speech, and whose
executive function and/or memory allows them to
utilize a software program that requires executing
certain actions in order (for example, turning a sound
recorder on, speaking then turning the recorder off)
and self-monitoring (playing back their utterances in
order to correct and expand them).”

6. SWORD Spoken Word Production Exercises.
Developed for Apraxia of Speech.
Software Design and Useability Strengths
• Mouse only control: All participants reported that they were
able to use the mouse / touchpad (following instruction) to
navigate the programme.
• AP1’s wife: “It’s about making it simple. Easy to read.”
• All participants responded that the programme never once
crashed.
• AOS 29’s Husband: “I could do me back garden!” Reports
being happy that his wife was doing something
independently.
• AP2’s wife: [whilst her husband was using the therapy] “I got a
lot of gardening done!”

7. Computer Therapy for Aphasia
Software Design and Usability Limitations
Mouse Access: AOS 29 reported that it took some time
to learn to use the mouse but she got used to using
it. Reported that a touch screen would have made
the laptop a lot easier to use.
SLT Comments: “Some patients needed prompting to
remind them that they need to click. Some patients
timed out before providing a response and were
awarded a fail. Is it possible to give even longer than
25 seconds?”

8. The GReAT Project
What’s different and novel about the GReAT
project?
• Use of Gesture Recognition Technology
(Gesture Recognition in Aphasia Therapy)
• Software Design and Useability Focus
• Participatory Design Process

9. Project Aims
• To develop an affordable, computer-
based technology that can be used in
therapy at home to help people with
severe aphasia to gesture.
• To evaluate the efficacy of the
technology within a pilot therapy study

10. Project Structure
• Phase 1: Designing a prototype gesture
therapy using participatory design
methods.
• Phase 2: Testing and piloting the
prototype

11. Project Team
Human Computer Interaction Design & Language and Communication Science
Stephanie Wilson Sam Muscroft Julia Galliers Jane Marshall
Naomi Cocks Tim Pring Abi Roper

12. Phase 1
• Designing a prototype gesture
therapy using participatory
design methods.

13. Consultants
Justine Everson Gerald Hartup Carol Watson
Philip Pepper Emma Buswell

14. Consultants
•Role: to test and feedback about relevant technology.
•Person Specifications:
–Expressive aphasia language difficulties.
–Able to attend university once or twice a month for
participatory design sessions.
•Recruited through in house clinic and through links with
the Stroke Association Communication Support
Co-ordinators.
•Employed by City University London as Casual Staff
members.

15. Methods: Participatory Design
Sessions
•Participatory design – engaging end users in design
process
•Sessions explored offline gesture therapy, computer
gesture recognition, interaction within 3D worlds and
computer interfaces.
•Consultants took part in 9 sessions each
•Project team involved in each session
- 1 HCID Researcher
- 1 HCID Developer
- 1 Speech and Language Therapist Researcher
- 2 or 3 Consultants

16. Session Structure
1. Introduction to scheduled
activities
2. Round table gesture activity
3. Demonstration of Technology
4. Trial use of technology by one
consultant - followed by
interview at computer
5. Tea break
6. Trial use of technology by
remaining consultant(s)

17. Participatory Design

18. What did we learn from the
Sessions?
1. Consistency
2. Simplicity
3. Pace
4. Reliability
5. Rewards
6. Individual Differences
7. Potential of ‘gaming’.

19. Key Features of GeST
Separate keyboard Gesture recognition
← →
OK
Gestures presented in
isolation & in context 3D worlds

20. The Prototype
OK
← →

21. Tool Design
3 Levels

22. Tool Design
•User sees gesture demonstrated twice
•Is invited to copy the gesture
•User monitors their
production attempt
on screen
•Receives reward and feedback for each
correctly recognised gesture

23. Demonstration Video

24. Using the Therapy at home
•Pilot study (Coming up next)
•How does this work at home?
Key differences between lab and
home –
User practising independently,
User intending to practise daily.
User practising in non-lab
conditions.

25. Things to consider when setting
up
•Lighting conditions
•Safety and permanence
(negotiate!)
•User comfort and access

26. Things to consider when training
•Develop the user’s confidence in the system.
(Be confident yourself)
Demonstrate:
1. Allow user to observe entirely
2. Allow user to observe and operate
interaction buttons
3. Allow user to operate alone but with
support as needed (confidence)

27. Things to consider when training
•Reinforce how to switch the computer
on and off several times.
•Make an appointment to come back in
one week to review.
•At review appointment, observe and re-
train difficult procedures.

28. Testing GeST
•Pilot study
•Participants received GeST for a total of 6
weeks.
•First three weeks supported by a
therapist with hour long session each
week.
•Second three weeks no support from
therapist.

29. Phase 2
• The Pilot Study

30. Questions
• Will practice with Gest improve participants’ production of
gestures &/or spoken words?
• Will improvements be specific to items that feature in the
programme?
• Will gains occur when Gest is used without ongoing
therapist support?
• Will gains be maintained after Gest is withdrawn?
• What are participants’ views about Gest?
• What are carers’ views about Gest?
(where relevant)
• Is Gest easy and enjoyable to use?

31. Participants
• 9 people with severe aphasia
– Consent to take part
– Fluent pre-stroke users of English
– Naming score <20%
– Able to recognise pictures
– No known dementia or other cognitive impairment

32. Consent
Screening
Phase 1 with weekly
Tests (1)
visits from therapist
3 Weeks
Practice
Phase 2 with no weekly
Tests (2)
visits from therapist
3 Weeks
Practice
Tests (3)
3 weeks
no tool
Total time commitment: about 14 weeks Tests (4)

33. Practice Phases
• Each last 3 weeks
• Each practise 15 gestures with the tool
• Phase 1: Weekly visits from therapist
• Phase 2: Initial but no weekly visits

34. Tests
• 60 items
– Gesture from picture
– Name from picture What is the
name of this?
How would
you gesture
Items: this?
30 practised with Gest
15 familiarised only
15 controls

35. Scoring Gestures
• Gesture tests are filmed
• 4 Scoring videos created
• Each video contains 60 gestures in random
order:
– 15 from test 1
– 15 from test 2
– 15 from test 3
– 15 from test 4

36. Scoring Gestures
• Scores
– Recognition Score
– Rating Score
• Scorers are ‘blind’ to the time of assessment

37. Results
Gesture Recognition

38. Statistical Analysis
• Time x Group interaction
• Planned comparisons:
– Items that were treated with Supported Gest
changed over time, other groups did not
– Supported Gest items improved significantly
following practice (between time 1 and time 2),
but not in the other phases
– Recognition of Supported Gest items was
significantly improved against baseline even at
time 4

39. Interim Conclusions
• Repeated testing without intervention (Control
items) did not improve gesture production
• Independent practice with Gest and familiarisation
did not improve gesture production.
• Using Gest with therapist support improved gesture
production. However:
– Gains were modest
– Differences between the groups were very small.

40. Usage Logs
• Record
– Number of sessions
– Length of sessions
– Levels of programme accessed
– Number of gestures recognised

41. More Interim Conclusions
• Benefits from Supported Gest may reflect
– Therapist input (‘The Abi Factor’)
– Usage

42. More Results
Naming

43. More Conclusions
• Using Gest did not facilitate naming of the
gesture targets
• This was despite the repeated inclusion of
spoken names in the programme

44. Qualitative Observations:
Some ‘Carer’ Comments

45. Independence of Use
• ‘She uses it all on her own, I don’t know how to
operate it’
• The first session I stayed with L, after that I’ve
helped only if she’s found something
particularly frustrating’
• All comment that the participant initiated use
of Gest

46. Enjoyment
• All say that the participant enjoyed Gest
• ‘he likes it when they clapped’
• ‘some of the gestures are particularly fitting
and she enjoyed rainbow’

47. Views about Technology
• ‘I was a technophobe and when they said
‘computer’ I thought it was going to cause
problems. I thought I wouldn’t understand
and he wouldn’t understand it. But it’s so
‘easy’

48. Reservations
• Carry over to real life (1 carer):
• ‘while she works on it here (points to
computer) it doesn’t necessarily translate’
• She wanted a hankie last night and didn’t
make a gesture’

49. Conclusions
• Will practice with Gest improve participants’ production of
gestures?
Yes
• Will improvements be specific to items that feature in the
programme?
Yes
• Will gains occur when Gest is used without ongoing therapist
support?
No
• Will gains be maintained after Gest is withdrawn?
Yes
• Will practice with Gest improve naming of target items
No

50. Conclusions
• What are participants’ views about Gest?
Very Positive
• What are carers’ views about Gest?
Very Positive
• Is Gest easy and enjoyable to use?
Yes

51. Acknowledgements
The Research Councils UK Digital
Economy Programme
The Stroke Association
Consultants and their families
Participants and their families
All our wonderful students who helped to score data
Thank You
GReAT@city.ac.uk
www.soi.city.ac.uk/great

52. Find Out More
Website: www.soi.city.ac.uk/great
Facebook: www.facebook.com/aphasiatech
Vimeo: www.vimeo.com/aphasiatech
Email: GReAT@city.ac.uk