2016-04-27 research seminar, 2nd presenter

1. Avar Pentel
PhD student, Tallinn University,
School of Digital Technologies
Supervisor: Tobias Ley
• Area of reasearch:
– User profiling
– Detecting users motor behaviour via
standard input devices such as keyboard
and mouse
– and connecting it to users demographic
data, emotions, etc

2. Title of Presentation
Employing Think-Aloud Protocol to
Connect User Emotions and Mouse
Movements*
* Based on the paper (2015) with the same title
available at IEEE Xplore digital library:
http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7387970

3. Goal
To predict user’s emotional state by
analyzing mouse movements logs

4. Outline of the Presentation
• Related work
• Experimental setup
– Data collection procedure
– Associating tasks to emotional states
– Features
– Machine learning
• Results
• Conclusion

5. Related work
• Special equipment
• Small samples
• Specific tasks, no general link between
emotion and mouse movement is studied

6. Data Collection Procedure

7. Idea from Christmas
Calendar

8. Data Collection Procedure

9. Data collection
• The game has collected data about:
–Each mouse click.
–Time when button was clicked.
–All mouse movements with
timestamps

10. Data Collection

11. Example of mouse log
(x,y, timestamp)
70,34,1365354712662,74,34,1365354713453,78,3
6,1365354713488,81,38,1365354713517,85,44,13
65354713537,87,50,1365354713560,89,53,13653
54713573,90,58,1365354713598,91,63,13653547
13622,92,59,1365354713903,95,53,13653547139
27,97,49,1365354713942,100,45,1365354713954,
103,40,1365354713976,106,36,1365354714001,1
10,34,1365354714049,105,33,1365354714390,10
0,30,1365354714414,96,27,1365354714439,93,25
,1365354714475,93,21,1365354714561,95,18,136
5354714598,98,15,1365354714622

12. Data
• 916 game sessions played by 282
individual users. Participants were
between 12 and 52 years old.
• As each game session consisted of 24
searching tasks, we had all together 21984
comparable (standardized session-wise)
records, each of them presenting mouse
movement logs between two button clicks.

13. Connecting Emotions with Tasks
Old and New approach
1) Retrospective feedback
2) Concurrent Think-Aloud protocol

14. Old Approach – retrospective feedback
First pilot:

15. There was no room of variety of
emotions

16. Self-Reports on Russel’s Model

17. Self-Reports on Likert Scale
• Interviews with selected particiapants
(N=44)
• Right after game session
• Still image of the game session was shown
Content Confused
Binary mapping: (1-3) content, (5-7) confused,
(4) neutral

18. Self-Reports on Likert Scale
• Emotion data about 44*24=1056 tasks
• All target finging times standardized session-
wise
• Pearson correlation between self reports and
standardized finding time was found (r = 0.86)
Content Confused

19. Self-Reports on Likert Scale
• Tasks reported as confused had finding
speed 0.5 standard deviation below mean
• Tasks reported as content had finding
speed 0.5 above mean
Content Confused
Binary mapping: (1-3) content, (5-7) confused,
(4) left out a neutral

20. Separation of Classes
Standardized item finding speed
Second half of
each of these
logs counted as
characterizing
non confusion
First half of
each of these
logs counted
as
characterizing
confused state

21. Using Think-Aloud Protocol

22. Think-Aloud Protocol
• Users reported five kinds of emotions -
confusion, frustration, shame, content and flow.
Strongest emotions were confusion and
frustration. Here is an example how users were
expressing themselves during states of
confusion and frustration:
• “Where is number x, where is number x, it is not
there, it is impossible, I looked everywhere, it is
missing. “
• “It can’t be, you hide a button, it is not there.”
• “I saw it before, but now it is not there any
more.”

23. Russel’s Model

24. Think-Aloud protocol

25. Using Think-Aloud Protocol
• 400 sessions (20 users * 20)
• 400*24 = 9600 comparable tasks with
emotion data

26. Separation of Classes

27. Final Datasets
• Confused class with 3170 examples
and all the rest with 18814 examples.
• In the case with separation gap
between classes, the second class had
12381 examples.
• Before applying classification
algorithms, we balanced our datasets.

28. Features

29. Distance (curvature)
1
2
3
4
1
2 3
1
2
3
45
Ratio between the 3-6
movements length and
shortest path between
the beginning and end
point

30. Speed (σ of the speed)
s4
s5
s1
s2
s3
s6
s7
s8
s9s10
Speed is measured for each 10px movement
separately

31. N
S
W O
Direction

32. Angle based features
• Sum of consecutive turns greater than an angle A (A counted by
45-degree step), normalized by number of movements.
• 18 features representing turns from 0 to 180 degrees’ by 10-degree
step. Counted results were normalized by the number of movements.
• Sum of all angles divided by number of movements – 1.
• σ of angles.
A
B
α

33. Feature selection
10

34. Machine learning
• Logistic Regression
• Support Vector Machine
• Random Forest
• C4.5
– Motivation based on literature.
– Java implementations of data analysis
package Weka.
– 10-fold cross validation

35. Results (with separation gap)

36. Results (without separation gap)

37. Conclusion
• Mouse movements reveal users
emotsional states
• But is the confusion and frustration in
current study comparable with
confusion and frustration caused by
solving mathematical equation or some
other cognitively more demanding task?

38. Conclusion
However, if we relay on
2D Circumplex Model of
Emotion, then all kind of
confusion and frustration
is located in the same
place.

39. Thank You!

40. Q&A