Research Methods and data collection technique summary for Computer Science and other disciplines. The content is meant for graduate and post-graduate researchers. This workshop was made in my MSc under the supervisor of my supervisors and with the input of the lab members around me.

1. Workshop by Rina R. Wehbe

2. § Creating a dialogue between humans and computers
§ Easier exchange of information
§ Optimizing interactions and tasks
§ Illustrating the importance and current limitations of the study of Human
Computer Interaction (HCI)
2

3. § Design of everyday things
§ Task-oriented Productivity
§ Expert Use Software
§ Design of Interaction Techniques
§ Entertainment Software
§ Games User Research (GUR)
3

4. 4
Planning
Data
Collection
Prototype
Testing &
Feedback
Requirements

5. 5

6. § Quantitative Measures
§ Metrics
§ Measurements
§ Formulas
§ Physiological Measures
§ EEG
§ Heart Rate
§ Skin Conductance
§ Qualitative Methodology
§ Interview
§ Focus Groups
§ Think aloud
§ AND MORE.
6

7. § Selective attention in video game players (Bavelier, 2012)
§ Learning and video games (Bavelier, 2012)
§ Inter-hemispheric Alpha Asymmetry and User experience (Salminen, 2009)
7

8. § Characterizing and measuring user experience (Ijsselsteijn, 2007)
§ SCI model of Immersion (Ermi, 2005)
§ Sensory
§ challenged based
§ Imaginative
8

9. 9

10. § Real-time data
§ Differential analyses provides us with a interpretation of the same data
§ Combine with other measures for more complete picture
10

11. § Physiological measures allow us to understand the user by investigating the
fundamental physiological reactions
§ Methodologies include:
§ Electroencephalography (EEG)
§ Cardiovascular Measures
§ Skin Conductance
§ Eye Tracking
§ Respiratory Measures
§ Muscle Activation
11

12. § EEG is collected using sensors that sit on scalp
of the participant. As neurons fire the
electrical impulses generated are recorded.
§ Data can yield information about:
§ Cognition (e.g. Cognitive Load, Errors)
§ Information Processing (e.g. Visual, Auditory)
§ Emotions (e.g. Excitement, boredom, frustration)
12
http://www.esa.int/gsp/ACT/images/bio/sleep.jpg

13. 13

14. 14

15. 15

16. 16

17. § Measures of the heart activity
§ Average Heart Rate
§ Heart Rate Variability (HRV)
§ Indicator of Stress, Excitement,
§ Physical exertion
17
http://m.eet.com/media/1115118/c0726-figure1.gif

18. § Also commonly referred to as a measure
of Galvanic Skin Response (GSR) or
Electrodermal Activity (EDA).
§ Measures the conductance of the skin which varies
in proportion to the amount sweat due to the
body’s natural sympathetic response.
§ Measures psychological Arousal (Stress, excitement).
18
http://www.hyperbody.nl/fileadmin/images/imi_51.jpg

19. § Measures what eyes look at
§ Saccades (fast movement)
§ Gaze path
§ Fixations (dwell times)
§ Attention focus
§ Pupil dilation/blink rate
§ Attention precedes gaze (200ms)
§ Used mainly to improve interface
§ Lack of 3D analysis tools
19

20. 20

21. 21

22. 22

23. ÌEasy to use
ÌObjective
ÌCovert
ÌContinuous
ÌQuantifiable
ÌReplicable
ÌEmpirical power
VCan be expensive
VLack of good tools
VTime-consuming
§ PROS § CONS
23

24. 24

25. 25

26. 26
Cat Purrs Pet

27. 27
Cat
Purrs
Pet
Cat
Bites

28. 28
Feedback Corrective
Action
Learning

29. 29

30. 30
FFT
Alpha
System in
Control
User in
Control

31. 31
Fairclough (2009) Csikszentmihalyi (1990)

32. 32
Pope (1994)

33. Implicit and Explicit System Interventions
33

34. § Creating technology that adapt to the users current state
§ Offer Assistance
§ Change the level of challenge
§ Incorporate an emotional display
§ Dynamic Difficulty Adjustment in games
§ Flight and Driving Automation
§ Health and Exergaming
34

35. § Using EEG to feedback into the system
§ Monitors operator engagement
§ To prefect the approach they also test out engagement during videogame
play
§ Pope, A.T., Bogart, E.H., Bartolome, D.S., 1995. Biocybernetic system
evaluates indices of operator engagement in automated task. Biological
Psychology 40, 187–195.
35

36. 36

37. § Relationship between the measures and response dictate how we can
design a system
§ Four possible relationships:
§ One-to-one
§ Many-to-one
§ One-to-Many
§ Many-to-Many
37

38. § Designing valid systems and experiments
§ Selecting stimuli carefully
§ Previous literature
§ Validation of responses
§ Mixed Measures approaches
§ Qualitative Measures
§ Multiple indicators
§ Ecological Validity
§ “In-the-Wild” testing
38

39. 39

40. § Privacy
§ Constant Monitoring Situation
§ Who has access to the information?
§ Anatomy of the User
§ Control of the system
§ When should the system intervene?
§ Consequences of the System’s Actions
40

41. Combination Techniques
41

42. § Tradition of expert reviews and heuristics in the ‘traditional’ usability &
software engineering
§ Games demand specifically designed heuristics
§ Easy implementation in the game design and development process
42

43. Playability
Evaluations
Player Experience Evaluations
Game Development Project
Concept
Design
Implementation
Alpha
Release Beta Release
Release
43

44. PROS
§ Cost-efficient
§ Time-efficient
§ Can be implemented at any stage
of a project
§ identify majority of existing bugs
§ Several sets of heuristics
CONS
§ Diversification of
§ Game genres
§ Input devices
§ Goals Challenges Heuristics
§ Lack of game usability &
playability experts in the game
industry
44

45. § “Typical” playtest
§ Watch people play the game or use the
system
§ Observe their behavior
§ Simulate at-home experience
§ Goals:
§ Understand workflow of the user
§ Errors & Misunderstandings
45

46. PROS
§ + Get a feel for player interaction
with game
§ + Importance of what people do—
not what they say
CONS
§ – Presence of observers can bias
results
§ – Salient event can slant
interpretation
§ – Behavior requires interpretation
46

47. 47

48. § Think-aloud protocol:
§ People describe their actions as they play
§ Unprompted and uncorrected
§ In conjunction with direct observation
48

49. PROS
§ + Enables real-time glimpse into
player thoughts, feelings, and
motivations
§ + Bring up unnoticed details
§ + Effective for ‘why’ questions
CONS
§ – Interferes with gameplay
§ – Creates an artificial experience
§ – Inaccurate and biased
49

50. § Set of standardized questions
§ Forced choice responses
§ Quantify feedback/opinions
§ Player categorization
50

51. 51

52. PROS
§ + Less biased responses
§ + Response validation
§ + Forced choice helpful for
revealing preference
§ + Time-based comparisons
CONS
§ – Eliminate nuance
§ – Difficulty in converting ratings to
meaningful decisions
§ – Limited solution space
52

53. § Data about behavior of players
§ In game environments
§ Anything a game engine can log
§ Examples
§ Player movement
§ Firing weapons
§ Interacting with NPCs
§ Interface interaction
53

54. 54

55. 55

56. Questions? Comments? Thoughts? Feelings?

57. § This slide set included points, slides, and references from both Lennart E.Nacke and Pejman Mirza-Babaei
§ Csikszentlmihalyi, M. (1990). The Psychology of Optimal Experience. New York: Harper and Row.
§ Fairclough, S. H. (2009). Fundamentals of physiological computing. Interacting with Computers, 21(1-2), 133–145.
http://doi.org/10.1016/j.intcom.2008.10.011
§ Fairclough, S. H., Karran, A. J., & Gilleade, K. (2015, April). Classification accuracy from the perspective of the user: Real-time
interaction with physiological computing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing
Systems (pp. 3029-3038). ACM.
§ Frey, J., Daniel, M., Castet, J., Hachet, M., & Lotte, F. (2016). Framework for Electroencephalography-based Evaluation of User
Experience. arXiv preprint arXiv:1601.02768.
§ Pope, A.T., Bogart, E.H., Bartolome, D.S., 1995. Biocybernetic system evaluates indices of operator engagement in automated
task. Biological Psychology 40, 187–195.
§ Pulsipher, L. (n.d.). Why we play. Retrieved March 10, 2016, from http://gamecareerguide.com/features/625/why_we_.php
§ Yuksel, B. F., Oleson, K. B., Harrison, L., Peck, E. M., Afergan, D., Chang, R., & Jacob, R. J. (2016). Learn Piano with BACh: An
Adaptive Learning Interface that Adjusts Task Difficulty based on Brain State. In Proceedings of the SIGCHI conference on Human
Factors in Computing Systems (2016),InPress. DOI: http://dx. doi. org/10.1145/2858036 (Vol. 2858388).
§ Yannakakis, G. N., & Togelius, J. (2011). Experience-driven procedural content generation. Affective Computing, IEEE Transactions
on, 2(3), 147-161.
57

58. § Frey, J., Daniel, M., Castet, J., Hachet, M., & Lotte, F. (2016). Framework for Electroencephalography-based Evaluation of User
Experience. arXiv preprint arXiv:1601.02768.
§ Yuksel, B. F., Oleson, K. B., Harrison, L., Peck, E. M., Afergan, D., Chang, R., & Jacob, R. J. (2016). Learn Piano with BACh: An
Adaptive Learning Interface that Adjusts Task Difficulty based on Brain State. In Proceedings of the SIGCHI conference on Human
Factors in Computing Systems (2016),InPress. DOI: http://dx. doi. org/10.1145/2858036 (Vol. 2858388).
§ Yannakakis, G. N., & Togelius, J. (2011). Experience-driven procedural content generation. Affective Computing, IEEE
Transactions on, 2(3), 147-161.
§ Stellmach, S. (2009). Visual Analysis of Eye Gaze Data in Virtual Environments. Master’s Thesis. University of Magdeburg.
§ Drachen, A., Canossa, A., & Yannakakis, G. N. (2009). Player Modeling using Self-Organization in Tomb Raider: Underworld.
Proc. IEEE CIG2009, Milano, Italy.
58