Software development is an intellectual activity requiring creativity and problem-solving skills, which are known to be influenced by emotions. Developers experience a wide range of affective states during programming tasks, which may have an impact on their job performance and wellbeing. Early recognition of negative emotions, such as stress or frustration can enable just-in-time intervention for developers and team managers, in order to prevent burnout and undesired turnover. In this talk, I will provide an overview of recent research findings of sentiment analysis in software engineering (SE), address the open challenges, and provide empirically-based guidelines for safe (re)use of SE-specific tools in order to obtain meaningful results.

1. Recognizing Developers' Emotions:
Advances and Open Challenges
@NicoleNovielli nicole.novielli@uniba.it
QUATIC - September 13, 2022
Nicole Novielli
University of Bari, Italy
Collaborative Development Group

2. Faculty
• Filippo Lanubile
• Nicole Novielli
• Fabio Calefato
Visiting Professors
and Researchers
COLLAB - Collaborative Development Group
Research topics
PhD Students
• Luigi Quaranta
• Daniela Grassi
• Giuseppe Colavito
• Serena Versino
Final-year undergrad students

3. My research
• Human Aspects in Software
Engineering
• Affective Computing
• Natural Language Processing

4. Recognizing developers’ emotions
BrainLink
EEG monitors the electrical
activity of the brain

5. Emotions in Software Development

6. Software Engineering involves social interaction
• Programmers cooperate, directly or indirectly
• Massive adoption of social media and rise of the ‘social programmer’ (Storey, ‘12) and
the surrounding ecosystem
• Computer-mediated interaction became prevalent during the pandemic!

7. Studying Emotions in Software Engineering

8. Studying Emotions in Software Engineering

9. Studying Emotions in Software Engineering

10. Collaborative software development and knowledge-sharing
– Correlation of emotions with issue-fixing time (Ortu et al., MSR 2015)
– Early burnout discovery (Mantyla et al. MSR 2015)
– Anger detection (Gachechiladze et al., ICSE-NIER 2017)
– Empirically-driven guidelines for question writings (Calefato et al., IST 2018)
– Confusion in code reviews (Ebert et al., SANER 2019)
– Negative sentiment in SADT (Fucci et al., MSR 2021)
Recommender systems
– Pattern-based mining of opinions in Q&A websites (Lin et al., ICSE 2019)
– Opinion search and summarization for APIs (Uddin and Khomh, ASE 2017)
Requirements engineering
– User feedback (Guzman and Maalej, RE’14; van Oordt and Guzman, RE ’21, Kurtanovic and Maalej, 2020)
– App improvement (Panichella et al., ICSME ‘14)
Actionable insights for

11. Emotions and Wellbeing at the Workplace

12. Developers’ emotions, productivity and wellbeing

13. Developers’ emotions, productivity and wellbeing

14. Developers’ emotions, productivity and wellbeing

15. Developers’ emotions, productivity and wellbeing

16. Our vision

17. Biometrics

18. Acknowledgements
Filippo Lanubile
University of Bari
Davide Fucci
BTH
Alexander Serebrenik
TU/e
Daniela Girardi
University of Bari
Daniela Grassi
University of Bari

20. Electroencephalography (EEG)
• Electrical activity of the brain
• Cerebral waves categorized based on their frequency
• Delta (<4 Hz): recorded during sleep
• Theta (4-7,5 Hz): decrease of vigilance level
• Alpha (4-12,5 Hz): relax
• Beta (13-30 Hz): mental process
• Gamma (>30 Hz): anxiety

21. Galvanic skin response (GSR)
• Electrical activity of the skin
• Changes due to the variation in body sweating
• Electrical changes in the skin could be due to variation in emotions

22. Empatica E4

23. Empirical studies: overview

24. Circumplex Model of Affect
J. A. Russell, A circumplex model of affect, Journal of Personality and Social Psychology 1980, Vol 39, no. 6

25. Lab studies

27. Controlled experiment
Can we use noninvasive, low cost sensors for reliable emotion
recognition?
VS.
https://www.eecs.qmul.ac.uk/mmv/datasets/deap/

28. Controlled experiment
Machine learning - performance comparable with the state of the art

30. What is the minimal set of non-invasive
biometric sensors to recognize
developers’ emotions?

31. Previous studies

32. Instrumentation
BrainLink
EEG monitors the electrical
activity of the brain

33. Instrumentation
• 23 participants
• Java development tasks
• Gold standard: self report and
arousal

34. Self-report of emotions and progress
Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry, 25(1), 49-59.
Valence (Un)Pleasantness of the
emotion stimulus
Arousal Level of activation of the
emotion stimulus

35. Gold standard based on self report
Valence Arousal
Posi
ti
ve Nega
ti
ve High Low
44 (32%) 94 (68%) 85 (62%) 53 (38%)

36. Machine Learning
Preprocessing
signals’ noise
removal
Feature extraction
statistical measures of
signals
Classification
Hold out -90-10 and
Leave-one-subject-out
(LOSO) setting
https://
fi
gshare.com/articles/_/9206474

37. Machine Learning

38. Sensor-based emotion detection
Minimal set of sensors

40. Field study

41. Research questions
RQ1. What is the range of developers’ emotions at the workplace?
RQ2. To what extent are developers emotions related to self-assessed
productivity during the workday?
RQ3. What are the triggers for developers’ emotions at the workplace?
RQ4. Can we predict developers’ emotions at the workplace using
lightweight biometric sensors?

42. Research questions
RQ1. What is the range of developers’ emotions at the workplace?
RQ2. To what extent are developers emotions related to self-assessed
productivity during the workday?
RQ3. What are the triggers for developers’ emotions at the workplace?
RQ4. Can we predict developers’ emotions at the workplace using
lightweight biometric sensors?
Biometrics

43. Link to full paper
IEEE Transactions on Software Engineering, 2021

44. Study protocol 2/3 weeks

46. Productivity
Self-assessed
Values ranging from very low to
very high

47. Triggers for self-reported emotions

48. .
coding
bug
fi
xing
testing
design
meeting
email, helping
networking
learning
administrative task
documentation
just arrived
other
Based on Meyer et al. (TSE, 2019)

49. Participants
Five companies
Dutch software development companies, including
- One startup (1 founder and 2 employees)
- Two SMEs (between 20 and 200 employees)
- Two large companies (> 20.000 employees)
21 professional developers
- 18 men, 3 women
- Average age: 33 years
± 7.2, ranging from 23 to 50
- Average experience in software development: 8 years
± 6.2, ranging from 1 to 25

50. Emotions and Productivity

51. Emotions and Productivity
RQ1. What is the range of developers’ emotions at the workplace?
RQ2. To what extent are developers emotions related to self-assessed
productivity during the workday?

52. Developer’s emotion at the workplace

53. Developer’s emotion at the workplace

54. Valence is positively correlated with perceived
productivity, with stronger correlation in the
afternoon.
Interaction between valence and time (day vs.
afternoon)
Emotions and self-assessed productivity

55. Valence is positively correlated with perceived
productivity
Stronger correlation in the afternoon.
Conversely, the correlation between dominance
and productivity is stronger in the morning.
This could be due to fatigue, which is known to
impair emotion regulation.
Emotions and self-assessed productivity

56. Emotion Triggers

57. Emotion Triggers
RQ3. What are the triggers for developers’ emotions at the workplace?

58. 48
pos: 18 neg: 19
RQ3: triggers of emotions

59. Triggers for positive and negative emotions

60. Triggers for positive and negative emotions

61. Triggers for positive and negative emotions

62. Sensor-based Emotion Recognition

63. Can we predict developers’ emotions at the workplace
using lightweight biometric sensors?

64. Minimal sensor setting for emotion recognition while programming

65. Machine
Learning
Sensor-based Emotion Recognition
Empatica E4

66. Machine Learning

67. Sensor-based emotion recognition
Good: Our classi
fi
ers outperform the baseline

68. Sensor-based emotion recognition
Good: The performance is comparable to the on in the lab

69. Sensor-based emotion recognition
Expected: Hold-out scores are higher than leave-one-subject out

70. Variability between participants
Variability between participants
Also observed in the lab study (Girardi et al., ICSE 2020)

76. • Individual training/
fi
ne-tuning of emo
ti
on classi
fi
ers
• Further valida
ti
on with larger/more diverse pool of par
ti
cipants
from di
ff
erent companies
• Self-disclosure of nega
ti
ve emo
ti
ons
• Analysis of GSR peaks as proxy of stress
Open challenges and future studies

77. • Individual training/
fi
ne-tuning of emo
ti
on classi
fi
ers
• Further valida
ti
on with larger/more diverse pool of par
ti
cipants
from di
ff
erent companies
• Self-disclosure of nega
ti
ve emo
ti
ons
• Analysis of GSR peaks as proxy of stress
Open challenges and future studies

78. • Individual training/
fi
ne-tuning of emo
ti
on classi
fi
ers
• Further valida
ti
on with larger/more diverse pool of par
ti
cipants
from di
ff
erent companies
• Self-disclosure of nega
ti
ve emo
ti
ons
• Analysis of GSR peaks as proxy of stress
Open challenges and future studies

79. • Individual training/
fi
ne-tuning of emo
ti
on classi
fi
ers
• Further valida
ti
on with larger/more diverse pool of par
ti
cipants
from di
ff
erent companies
• Self-disclosure of nega
ti
ve emo
ti
ons
• Analysis of GSR peaks as proxy of stress
Open challenges and future studies

80. Preprint available

81. • Individual training/
fi
ne-tuning of emo
ti
on classi
fi
ers
• Further valida
ti
on with larger/more diverse pool of par
ti
cipants
from di
ff
erent companies
• Self-disclosure of nega
ti
ve emo
ti
ons
• Analysis of GSR peaks as proxy of stress
Open challenges and future studies

82. Emotions as a proxy for engagement in users’ interviews
Wed Sept 2 2020

83. @NicoleNovielli
nicole.novielli@uniba.it
http://collab.di.uniba.it/nicole/
Thanks!
Q&A