Real-time Sensory Substitution to Enable Players who are Blind to Play Video games using Whole-Body Gestures

Real-time Sensory Substitution to Enable Players who are Blind to Play Video games using Whole-Body Gestures Tony Morelli, Eelke Folmer - FDG’11, Bordeaux Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Gesture Based Interaction Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Reliance on Visual Cues Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Reliance on Visual Cues dodge Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Reliance on Visual Cues dodge ♫ ♫ Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

visual impairment & obesity 60,000 Blind Children in US More obese Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Barriers to physical activity Rely on sighted guide Safety Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Exercise games ★ Moderate to Vigorous PA (MVPA) ★Independently ★60 min of MVPA daily (CDC) ★Safer Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Research Question Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Gestures w/o visual cues? Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Gestures w/o visual cues? ? Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Sensory Substitution Visual information Haptic Audio Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Sensory Substitution closed captioning Visual information Haptic Audio Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Sensory Substitution closed captioning Visual information Haptic Audio braille speech synthesis Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

SS with Gestures? Wii Sports VI Tennis serve ♫ bounce ♫ Bu zz return Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

User studies vigorous moderate light VI Tennis Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

User studies vigorous moderate Upper Body light VI Tennis Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

User studies vigorous moderate light VI Tennis Whole Body Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Whole Body Exergames Yield Vigorous PA [Biddis & Irwin 2010] Track Gestures using Computer Vision Problems with Sensory Substitution »Systems are Controller less »Access to Source code Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Real Time Sensory Substitution Le*!! Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Video Analysis Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Instrumentation Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Real time Video Analysis <SECTION> <XMIN>100</XMIN> <XMAX>300</XMAX> <YMIN>150</YMIN> <YMAX>300</YMAX> <MINR>230</MINR> <MAXR>999</MAXR> <MING>230</MING> <MAXG>999</MAXG> <MINB>0</MINB> <MAXB>100</MAXB> <ACTION <WIIMOTE> <ID>0</ID> <DURATION>-1</DURATION> </WIIMOTE> <AUDIO> <ID>alert.wav</ID> <LOOP>0</LOOP> </AUDIO> </ACTION> </SECTION> Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Real time Video Analysis Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Evaluate How eﬃcient is RTSS? Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Thursday, June 30, 2011

User Study Anonymously recruited and tested Evaluated 2 versions of the hurlde game »Regular (Audio + Video) »RTSS ( Audio + Haptic) 28 sighted participants Novice players Switch between versions a*er 3 players 12 people played Regular <-- data loss 16 people played RTSS Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Results version N success time (seconds) RTSS 12 0.39 (δ=0.26) 28.67 (δ=8.41) Regular 16 0.37 (δ=0.32) 26.76 (δ=4.76) No Signiﬁcant Diﬀerence p>.05 Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Jump Timing Accuracy Signiﬁcant Diﬀerence p<0.05 Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Observations Game Restarts »False Starts ‣Regular: 2 players started too early ‣RTSS: None »Out of Range ‣Regular: None ‣RTSS: 3 players out of range Successful jump depends on pace Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Limitations of RTSS Kinect Sports Kinect Adventures ★ Predeﬁned gestures ★ Diﬀerent gestures ★ Fixed target ★ Random targets ★ 1 haptic cue ★ (directional) haptic cues? Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Future Work Techniques for keeping players in range and facing the camera. Multiplayer Detecting cues/objects Test with Blind Kids & Measure Energy Expenditure Integrate RTSS in Console OS Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Questions Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Questions ? Player-Game Interaction Research University of Nevada, RenoThursday, June 30, 2011

Real-time Sensory Substitution to Enable Players who are Blind to Play Video games using Whole-Body Gestures

by eelke folmer on Jun 30, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Real-time Sensory Substitution to Enable Players who are Blind to Play Video games using Whole-Body Gestures — Presentation Transcript