Evaluation of a Natural User Interaction Gameplay System Using the Microsoft Kinect Augmented with Non-invasive Brain Computer Interfaces by Peter Mitchell, Dr. Brett Wilkinson, and Dr. Sean Fitzgibbon

1. Evaluating brain signal input
for Kinect-based games
Dr Brett Wilkinson
Presenting the work of:
Mr Peter Mitchell, Dr Brett Wilkinson, Dr Sean Fitzgibbon and
Mr Lawrence Sambrooks

2. Research Overview
• Used Natural User Interfaces (NUIs) through a
combination of the Microsoft Kinect and Emotiv
EPOC to provide a full body Human Computer
Interaction experience (HCI).
• Research completed as a pilot study to determine the
usability of the combination of hardware to explore
whether there is future potential for the combination.
Emotiv 2010, Arizona State University, viewed 9 September 2013, <http://lsrl.lab.asu.edu/site/?p=848>
Microsoft Kinect 2012, Microsoft, viewed 9 September 2013 , <http://www.microsoft.com/en-us/kinectforwindows/>

3. Presentation Overview
• Background
• Testing methodology
• The Games
– Tile Puzzle
– Street Puzzle
– River Puzzle
• Results

4. Background on Existing
Studies
• BrainBasher (van de Laar, 2009)
• Used actual and imagined movement to
have participants attempt to match
actions. (seen top right)
• BacteriaHunt (Bos et al., 2010)
• BCI interaction used to provide a
speed modifier in combination with
keyboard interaction.
• AlphaWoW (Bos et al. 2010)
• Used a variety of BCI methods for
character interaction in the game
World of Warcraft. Inner speech,
association, and mental states.
VAN DE LAAR, B. L. 2009. Actual and imagined movement in BCI gaming.
BOS, D.-O., REUDERINK, B., VAN DE LAAR, B., GURKOK, H., MUHL, C., POEL, M., HEYLEN, D. & NIJHOLT, A. Human-computer
interaction for BCI games: Usability and user experience. Cyberworlds (CW), 2010 International Conference on, 2010. IEEE, 277-281.

5. Goals for Testing
• Does BCI input with Kinect-based
games modify the experience?
• What signals are most appropriate for
gameplay?
• Can individuals maintain control over
their own brain waves?

6. Testing Approach
• 15 participants from Flinders University
– Students and academics
– Primarily male
• Play three puzzle games
• Complete post experiment survey
• Complete post experiment NASA TLX

7. Application Overview: Tile Puzzle

8. Application Overview: Tile Puzzle
• Time-based task
– Freedom to explore the interaction techniques with
the Emotiv and Kinect
• Concentration and Relaxation used as input
– Relax: reveal hidden image
– Concentrate: hide image
• Kinect used to map movements and speech to
interaction
– Control cursor
– Select, place, rotate tiles

9. Original Mock-up Example
Image of initial state
Initial State
Image of moving
squares
Swapping Tiles
Image of Calm view
Relaxed State
Image of complete
Completed Puzzle
Free Sandstone Image 2012, viewed 2/04/2012, http://www.hoskingindustries.com.au/blog/tag/grunge/page/2/
Free Sandstone Image 2012, viewed 2/04/2012, http://www.spiralgraphics.biz/packs/stone_muted/index.htm?36
Hieroglify font, http://www.fontspace.com/download/1123/e17737daec4347e0b3edd50cd5c47df6/barmee_hieroglify.zip

10. Finished Tile Puzzle (Video)

11. Street Puzzle Overview
Street Puzzle

12. Application Overview: Street Puzzle
• Motion – goal – control brain state
• Rail-based task
– Set, randomised path
• Concentration and Relaxation used as input
– Relax: slow down game time
– Concentrate: speed up game time
• Kinect used to map movements to interaction
– Sideway step to jump rail
– Both hands used to halt motion

13. River Puzzle Overview
River Puzzle

14. Application Overview: River Puzzle
• Time-based task
– Selection of appropriate game items within a set
time
– The more collected the higher the score
• Concentration and Relaxation used as input
– Relax: slow down game time
– Concentrate: speed up game time
• Kinect used to map movements to interaction
– Control cursor
– Select treasure and place in inventory

15. Results
General Gameplay Responses

16. Results Continued
Ease of use with specific HCI components

17. Results Continued
Combination of Inputs and Marketability

18. Results Continued
Comparison of Percent Time Spent in EEG States

19. Results Continued
• Technical issues encountered with BCI
equipment:
– Cheap headset resulted in limited
performance
– Calibration difficulties and inconsistencies
– Delay between updates
– Muscle movement heavily contaminated
data.

20. Future Work
• Look at the potential of other BCI devices
• Look at the potential of other platforms
• Extended evaluation to investigate if
training of state can be achieved

21. Conclusion
• Pilot study indicated that the technology
can work together
• Developed a functional test platform
• User evaluation conducted to suggest
the potential for training brain state