Technology & Games for Stroke Rehabilitation James Burke School of Computing & Information Engineering [email_address] University of Ulster, Coleraine

Overview THE PROBLEM AREA : Stroke is a leading cause of severe physical disability. Effective rehabilitation must be early , intensive and repetitive . Traditional tasks often mundane - difficult to maintain patient motivation . THE PROPOSED SOLUTION : Games and novel technology may be effective in optimising engagement . We have developed several games upper limb post-stroke rehabilitation which use novel technology.

THE PROBLEM AREA :

Stroke is a leading cause of severe physical disability.

Effective rehabilitation must be early , intensive and repetitive .

Traditional tasks often mundane - difficult to maintain patient motivation .

THE PROPOSED SOLUTION :

Games and novel technology may be effective in optimising engagement .

We have developed several games upper limb post-stroke rehabilitation which use novel technology.

Games + Rehabilitation MEANINGFUL PLAY Effective feedback . Conservative response to failure - reward engagement . CHALLENGE Challenge should match player’s abilities . Maintains engagement . “ Designers of rehabilitation tasks can benefit from examining the formulas that commercial game developers use…” [Rizzo, 2005] Inputs choices Player Game Meaningful play Feedback

MEANINGFUL PLAY

Effective feedback .

Conservative response to failure - reward engagement .

CHALLENGE

Challenge should match player’s abilities .

Maintains engagement .

Webcam Games for Upper-Limb Rehabilitation

Webcam Games Simple 2D games which use a webcam image as player input. Games store patient profile . Profile stores reaching ability, play-time, scores, etc. Played with coloured gloves or any single coloured object. No attaching of wires required , untethered movement. Games can be played standing up or seated . Potential for home rehabilitation . No expertise required .

Simple 2D games which use a webcam image as player input.

Games store patient profile .

Profile stores reaching ability, play-time, scores, etc.

Played with coloured gloves or any single coloured object.

No attaching of wires required , untethered movement.

Games can be played standing up or seated .

Potential for home rehabilitation .

No expertise required .

Webcam Games Architecture Player Movement Wearing Gloves Image Captured by USB Webcam Image Processed – Pixels identified for each marker Output to Monitor / Projector

RGB Colour Segmentation Markers identified through calibration. Each frame, each pixel in the camera image is examined: For each pixel p (p r , p g , p b ): Where µ = mean colour vector t = threshold scale value σ = standard deviation L R R G B 0 µ t σ

Markers identified through calibration.

Each frame, each pixel in the camera image is examined:

Webcam Games SINGLE ARM EXERCISE: “ Rabbit Chase” Player must catch a rabbit as it peers out of holes on the screen. “ Bubble Trouble” Player must burst bubbles as they float around the screen. TWO ARM EXERCISE: “ Arrow Attack” Player must touch corresponding arrows with both hands simultaneously as they enter boxes. “ Bubble Trouble” (two handed version) Bubbles colour coded and show arrow.

SINGLE ARM EXERCISE:

“ Rabbit Chase”

Player must catch a rabbit as it peers out of holes on the screen.

“ Bubble Trouble”

Player must burst bubbles as they float around the screen.

TWO ARM EXERCISE:

“ Arrow Attack”

Player must touch corresponding arrows with both hands simultaneously as they enter boxes.

“ Bubble Trouble” (two handed version)

Bubbles colour coded and show arrow.

Webcam Games Initial reach calibration: Player must roll each ball as far as they possibly can. Allows system to determine range of movement and position game elements accordingly. Additional tools: Camera calibration. Profile & data log analyser. View playing time and movement trends.

Initial reach calibration:

Player must roll each ball as far as they possibly can.

Allows system to determine range of movement and position game elements accordingly.

Additional tools:

Camera calibration.

Profile & data log analyser.

View playing time and movement trends.

Webcam Games Demo Video: http://www.vimeo.com/4279126

Design Principles Simple game concepts. Effective feedback : Sound effects, hit & miss messages, colour indicators, particle effects, timer bar, score graph. Clear, simple, friendly user interface. Manual or automatic difficulty adjustment. Help maintain suitable level of challenge for level of ability. No need for computer literacy.

Simple game concepts.

Effective feedback :

Sound effects, hit & miss messages, colour indicators, particle effects, timer bar, score graph.

Clear, simple, friendly user interface.

Manual or automatic difficulty adjustment.

Help maintain suitable level of challenge for level of ability.

No need for computer literacy.

Wii Controller

Wii Remote Features Face & back buttons (1) Infra-red camera (2) Uses sensor bar for position Accelerometer 3 degrees of freedom Vibration feedback Speaker (3) LED indicators (4) Expansion port (5) e.g. Nunchuck, classic controlle (1) (5) (2) (3) (4)

Face & back buttons (1)

Infra-red camera (2)

Uses sensor bar for position

Accelerometer

3 degrees of freedom

Vibration feedback

Speaker (3)

LED indicators (4)

Expansion port (5)

e.g. Nunchuck, classic controlle

Infrared Sensing Camera at tip of controller: PixArt optical sensor. Sensor bar with LED clusters on each side. Clusters seen by Wii controller as two bright dots. Camera can detect 4 clusters in total. Works up to 5 meters. Allows for depth as well as (x,y) 2D plane : Determined by distance between sensor bar dots. Sensor bar powered by Wii or USB/battery.

Camera at tip of controller:

PixArt optical sensor.

Sensor bar with LED clusters on each side.

Clusters seen by Wii controller as two bright dots.

Camera can detect 4 clusters in total.

Works up to 5 meters.

Allows for depth as well as (x,y) 2D plane :

Determined by distance between sensor bar dots.

Sensor bar powered by Wii or USB/battery.

Accelerometer ADXL330 accelerometer. Senses linear acceleration along 3 fixed axis. Can measure pitch & roll angles. Yaw determined by optical sensor. Accelerometer can not determine position in 3D space alone. [Battersby, 2007]

ADXL330 accelerometer.

Senses linear acceleration along 3 fixed axis.

Can measure pitch & roll angles.

Yaw determined by optical sensor.

Accelerometer can not determine position in 3D space alone. [Battersby, 2007]

Wii Remote Application Wii being used with existing games (e.g. Wii Sports ) in clinics and groups . Optical tracking: Immersive environments. [Murgia, 2008] Virtual cane for the blind. [Evett, 2008] Head tracking. [Chung-Lee, 2007] Potential for arm and wrist rehabilitation. Accuracy limitations? Ongoing studies.

Wii being used with existing games (e.g. Wii Sports ) in clinics and groups .

Optical tracking:

Immersive environments. [Murgia, 2008]

Virtual cane for the blind. [Evett, 2008]

Head tracking. [Chung-Lee, 2007]

Potential for arm and wrist rehabilitation.

Accuracy limitations?

Ongoing studies.

Johnny Chung Lee Wii “Head Tracking” Video

Wii Vibraphone Application Application developed as test for Wii controller on PC (Windows) . Wii controller interfaced via Bluetooth . Virtual vibraphone instrument allowing for recording and playing tunes. Potential for wrist rehabilitation .

Application developed as test for Wii controller on PC (Windows) .

Wii controller interfaced via Bluetooth .

Virtual vibraphone instrument allowing for recording and playing tunes.

Potential for wrist rehabilitation .

Wii braphone Demo

Conclusions Games have potential to engage and motivate during rehabilitation. No need for computer or video game experience. Design games which are friendly for all users. New innovative controllers at a low price may offer intuitive control mechanisms suitable for rehabilitation. Low-cost off-the-shelf hardware may offer an opportunity for home rehabilitation in addition to traditional therapy.

Games have potential to engage and motivate during rehabilitation.

No need for computer or video game experience.

Design games which are friendly for all users.

New innovative controllers at a low price may offer intuitive control mechanisms suitable for rehabilitation.

Low-cost off-the-shelf hardware may offer an opportunity for home rehabilitation in addition to traditional therapy.

References Battersby S. (2008) The Nintendo Wii controller as an adaptive assistive device – a technical report . http://www.ics.heacademy.ac.uk/events/presentations/670_SJBmiddlesbrough2.doc Burke J. et al. (2009 ) Serious Games for Upper Limb Rehabilitation Following Stroke. IEEE International Conference in Games and Virtual Worlds for Serious Applications (VS Games). Chung-Lee J. (2007) Head Tracking for Desktop VR Displays using the WiiRemote. http://www.youtube.com/watch?v=Jd3-eiid-Uw Evett L. et al. (2008) Accessible virtual environments for people who are blind – creating an intelligent virtual cane using the Nintendo Wii controller . Proc. 7th ICDVRAT with ArtAbilitation, Maia, Portugal, 2008. Murgia A. et al. (2008) Low-cost optical tracking for immersive collaboration in the CAVE using the Wii Remote . Proc. 7th ICDVRAT with ArtAbilitation, Maia, Portugal, 2008. Rizzo A., Kim G. J. (2005) A SWOT Analysis of the Field of Virtual Reality Rehabilitation and Therapy . Presence: Teleoperators and Virtual Environments 14: 119-146.

Battersby S. (2008) The Nintendo Wii controller as an adaptive assistive device – a technical report . http://www.ics.heacademy.ac.uk/events/presentations/670_SJBmiddlesbrough2.doc

Burke J. et al. (2009 ) Serious Games for Upper Limb Rehabilitation Following Stroke. IEEE International Conference in Games and Virtual Worlds for Serious Applications (VS Games).

Chung-Lee J. (2007) Head Tracking for Desktop VR Displays using the WiiRemote. http://www.youtube.com/watch?v=Jd3-eiid-Uw

Evett L. et al. (2008) Accessible virtual environments for people who are blind – creating an intelligent virtual cane using the Nintendo Wii controller . Proc. 7th ICDVRAT with ArtAbilitation, Maia, Portugal, 2008.

Murgia A. et al. (2008) Low-cost optical tracking for immersive collaboration in the CAVE using the Wii Remote . Proc. 7th ICDVRAT with ArtAbilitation, Maia, Portugal, 2008.

Rizzo A., Kim G. J. (2005) A SWOT Analysis of the Field of Virtual Reality Rehabilitation and Therapy . Presence: Teleoperators and Virtual Environments 14: 119-146.