Emotion Analysis in Software Ecosystems
•Download as PPTX, PDF•
0 likes•53 views
Software developers are known to experience a wide range of emotions while performing development tasks. Emotions expressed in developer communication might reflect openness of the ecosystem to newcomers, presence of conflicts, problems in the software development process or source code itself. In this talk, based on a recent work with Nicole Novielli, I present an overview of the state-of-the-art research on analysis of emotions in software engineering focusing on the studies of emotion in context of software ecosystems. To encourage further applications of emotion analysis in the industry and research we also discuss currently available emotion analysis tools and datasets as well as outline directions for future research. This is a keynote talk given at the 11th International Workshop on Software Engineering for Systems-of-Systems and Software Ecosystems (SESoS 2023), collocated with ICSE 2023 in Melbourne, Australia.
Recommended
Recommended
More Related Content
Similar to Emotion Analysis in Software Ecosystems
Similar to Emotion Analysis in Software Ecosystems (20)
More from Alexander Serebrenik
More from Alexander Serebrenik (20)
Recently uploaded
Recently uploaded (20)
Emotion Analysis in Software Ecosystems
- 1. EMOTION ANALYSIS IN SOFTWARE ECOSYSTEMS* *BASED ON JOINT WORK WITH NICOLE NOVIELLI Image by Ryan McGuire from Pixabay ALEXANDER SEREBRENIK
- 2. Image by pvproductions on Freepik Image by Freepik
- 4. FEHR, B., & RUSSELL, J. A. (1984). CONCEPT OF EMOTION VIEWED FROM A PROTOTYPE PERSPECTIVE. JOURNAL OF EXPERIMENTAL PSYCHOLOGY: GENERAL, 113(3), 464– 486. “EVERYONE KNOWS WHAT AN EMOTION IS, UNTIL ASKED TO GIVE A DEFINITION. THEN, IT SEEMS, NO ONE KNOWS”
- 5. J. J. Gross and L. F. Barrett, “Emotion generation and emotion regulation: One or two depends on your point of view,” Emot Rev, vol. 3, no. 1, pp. 8–16, Jan. 2011.
- 6. J. J. Gross and L. F. Barrett, “Emotion generation and emotion regulation: One or two depends on your point of view,” Emot Rev, vol. 3, no. 1, pp. 8–16, Jan. 2011.
- 7. J. J. Gross and L. F. Barrett, “Emotion generation and emotion regulation: One or two depends on your point of view,” Emot Rev, vol. 3, no. 1, pp. 8–16, Jan. 2011. H. L. Meiselman, Emotion measurement, 1st ed. Woodhead Publishing is an imprint of Elsevier, 2016.
- 8. J. J. Gross and L. F. Barrett, “Emotion generation and emotion regulation: One or two depends on your point of view,” Emot Rev, vol. 3, no. 1, pp. 8–16, Jan. 2011. H. L. Meiselman, Emotion measurement, 1st ed. Woodhead Publishing is an imprint of Elsevier, 2016.
- 9. Alex Cummaudo, Ulrike Maria Graetsch, Maheswaree Kissoon Curumsing, Rajesh Vasa, Scott Barnett, John Grundy: Emotions in Computer Vision Service Q&A. SEmotion@ICSE 2021: 13-18 Gias Uddin, Foutse Khomh: Automatic Mining of Opinions Expressed About APIs in Stack Overflow. IEEE Trans. Software Eng. 47(3): 522-559 (2021)
- 10. Image by rawpixel.com on Freepik Image by Freepik Image by Freepik Image by master1305 on Freepik
- 11. SO/IEC/IEEE 12207 Bin Lin, Nathan Cassee, Alexander Serebrenik, Gabriele Bavota, Nicole Novielli, Michele Lanza: Opinion Mining for Software Development: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 31(3): 38:1-38:41 (2022)
- 12. SO/IEC/IEEE 12207 assessing technique/API discovering rationale API usage general user satisfaction user-reported issues/requests identifying requirements detecting emotion/sentiment relating emotion/sentiment to performance evaluating trust Bin Lin, Nathan Cassee, Alexander Serebrenik, Gabriele Bavota, Nicole Novielli, Michele Lanza: Opinion Mining for Software Development: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 31(3): 38:1-38:41 (2022)
- 14. Images: Empatica website, Marco Verch Professional Photographer, kues1 on Freepik, Kevin De
- 15. Girardi et al. 2020, Vrzakova et al. 2020, … Most of the research in the field. Novielli et al. 2022 Çalikli et al. 2018, Schneider et al. 2015, … Herrmann and Klünder 2021, … Images: Empatica website, Marco Verch Professional Photographer, kues1 on Freepik, Kevin De
- 17. Jongeling et al.: General-purpose sentiment analysis tools are not reliable when applied to software engineering texts (EMSE 2017) Novielli et al.: Different SE-applications require tool adaptation (MSR 2020)
- 19. Image by Freepik
- 20. Image by pvproductions on Freepik
- 21. Image by Racool_studio on Freepik
- 22. Ecosystem by Nick Youngson CC BY-SA 3.0 Pix4free.org
- 26. Image by pvproductions on Freepik Image by Freepik
- 27. Image by jannoon028 on Freepik
- 28. Reliability of the tools is rarely evaluated threatening conclusions of the studies.
- 31. Guzman et al. Java projects are more negative than projects in other languages ✖️ Guzman et al. There is no difference between other programming languages ✖️ Guzman et al. Comments on Mondays are more negative than on other days ✖️ Guzman et al. Afternoon comments are more positive than the evening ones ✖️ Pletea et al. Security comments/discussions are more often negative than non- security comments/discussions ✔️ Pletea et al. Security discussions are more emotional than non-security ones ✖️ Jongeling et al. 2017 Tool % negative in security % negative in rest SentiStrength-SE 15.46% 12.27% ✔️ DEVA 25.87% 21.10% ✔️ SentiCR 28.93% 19.49% ✔️ Senti4SD 17.58% 11.20% ✔️ Novielli et al. 2021
- 33. Francisco Jurado, Pilar Rodríguez Marín: Sentiment Analysis in monitoring software development processes: An exploratory case study on GitHub's project issues. J. Syst. Softw. 104: 82-89 (2015)
- 34. 0 25 50 75 100 125 Junior Member Senior Positive Neutral Negative Yusuf Sulistyo Nugroho, Syful Islam, Keitaro Nakasai, Ifraz Rehman, Hideaki Hata, Raula Gaikovina Kula, Meiyappan Nagappan, Kenichi Matsumoto: How are project-specific forums utilized? A study of participation, content, and sentiment in the Eclipse ecosystem. Empir. Softw. Eng. 26(6): 132 (2021)
- 35. Image by comp on Freepik
- 36. Future applications of emotion analysis Replication and revalidation of the previous studies Better sentiment analysis tools More, more diverse and better data
- 37. “Goede tijden, slechte tijden”, episode 5492 (2017) Tanja Crijns, Metehan Doyran, Maurits van der Goes, Cecilia Herrera, Heysem Kaya , Osman Semih Kayhan, Rana Klein, Vincent Koops, Cas Laugs, Daan Odijk, Albert Ali Salah, Alexander Serebrenik, Yasemin Tımar, Anja Volk: Multimodal Emotion Recognition for Visualizing Storyline in a TV Series. ICT with Industry 2020
- 38. “Goede tijden, slechte tijden”, episode 5492 (2017) Facial expressions Speech Music Text
- 39. Nicole Novielli, Daniela Grassi, Filippo Lanubile, Alexander Serebrenik:Sensor-Based Emotion Recognition in Software Development: Facial Expressions as Gold Standard. ACII 2022: 1-8
- 40. Image by pvproductions on Freepik Image by Freepik
- 41. Image by pvproductions on Freepik Image by Freepik
- 42. Image by pvproductions on Freepik Image by Freepik
- 43. Image by pvproductions on Freepik Image by Freepik
Editor's Notes
- Software developers are known to experience a wide range of emotions while performing development tasks. Emotions expressed in developer communication might reflect openness of the ecosystem to newcomers, presence of conflicts, problems in the software development process or source code itself. In this talk we present an overview of the state-of-the-art research on analysis of emotions in software engineering focusing on the studies of emotion in context of software ecosystems. In 2003 Dave et al. have coined the term “opinion mining” to refer to “processing a set of search results for a given item, generating a list of product attributes (quality, features, etc.) and aggregating opinions about each of them (poor, mixed, good)”. Nine years later, in 2012 Brooks and Swigger have applied sentiment analysis in the context of software engineering. Today another nine years have passed and it is time to look back: what have we achieved as a research community and where should we go next?
- Before diving into the topic of emotion analysis in software ecosystems, we need to discuss the two terms: “software ecosystem” and “emotion”.
- Let us start with discussing emotion. Emotions have always been in the center of human inquiry with pre-Socratic philosophers being among the first to think about this topic [28]. Emotions have been studied by numerous philosophers [28], historians [29], sociologists [30], psychologists [31], biologists and neurophysiologists [32], economists [33], musicologists [34], literature scientists [35], and computing researchers [36].
- Despite this, or maybe due to this, a definition of emotion proved to be elusive…
- It should come as no surprise then that multiple theories of emotion have been proposed in the literature. Gross and Barrett [39] and Meiselman [40] propose to arrange emotion theories along a continuum ranging from theories of basic emotion through theories of appraisal to psychological construction theories. Emotional appraisal refers to processes by which individuals' cognitions about events predict their emotional reactions to those events.
- If we look at the software engineering then two models have influenced the thinking about emotion: the theory of basic emotion of Ekman and the multidimensional model of Russell.
- Ekman believes emotions to be separate and distinct from each other. He associated emotions with facial expressions, distinctive patterns of activation of the autonomic nervous system, as well as connected emotions in humans to comparable expressions observed in other primates. Ekman further argued that there are more emotional words than actual emotions and that only emotions satisfying specific criteria can be seen as basic emotions. These emotions are anger, surprise, disgust, enjoyment, fear, and sadness; later research suggests that contempt should be seen as a basic emotion too. Starting from a similar list of basic emotions Shaver et al. have proposed a tree-like structure gradually refining these emotions to emotion names. This hierarchy of emotion labels includes basic (primary) emotions, which are further refined into secondary and tertiary ones, e.g., anger is refined to such secondary emotions as envy, rage and exasperation with such tertiary subspecies of rage as outrage, hatred or dislike.
- Russel has observed that the distinction between such emotions as sadness and anger present in English is absent from some African languages, while English misses words for Bengali obhiman, which refers to sorrow caused by the insensitivity of a loved one or German Schadenfreude, which refers to pleasure derived from another’s displeasure. Based on these and similar arguments he argued that emotions can be operationalised along several dimensions such as valence, arousal, and dominance. Using these dimensions Russel states that both excitement and calmness can be characterised by positive valence and feeling in control, with arousal being high for excitement and low for calmness. Anger shares with excitement high arousal and dominance but differs from it by negative valence.
- Here we see two examples of studies that have used a discrete and a continuous models of emotion. Cummaudo et al. have studied StackOverflow posts about computer-vision cloud services. They have observed that while most of the time no emotion has been expressed. Unsurprisingly, Discrepancy-based questions—indicative of the frustrations developers face when the API does something unexpected—had the highest proportion of Anger detected Uddin and Khomh have studied a similar dataset - StackOverflow posts about APIs (but APIs) in general. In this case, the authors focus on valence, also known as sentiment (negative vs positive) and observed that certain aspects of APIs such as Performance triggered more opinionated statements than other aspects of APIs such as security. Among these opinionated statements Security-related ones are predominantly positive while the opinions related to Performance and Portability are much more mixed.
- The previous slide shows that studying emotions allows us to understand what developers think about products they are using such as APIs. However, there is more. Software development has long been recognised as a problem solving activity, and emotions are known to impact problem-solving skills and creativity. Moreover, software development is a collaborative process and such sites as GitHub and Stack Overflow further require communication to facilitate knowledge sharing and co-creation of software.
- ISO/IEC/IEEE 12207 Systems and software engineering — Software life cycle processes. In general, Lin et al have observed that sentiment analysis and related techniques have been used to support various software engineering activities. The yellow spots are the software lifecycle processes where opinion mining techniques have been used.
- When an activity has been considered in the context of opinion mining a series of topics have been considered; some of them are shown on the slide. The examples we have seen before are related to API usage. Mention identifying reqs and rationale
- However, there is a gap between the theoretical models of emotions and results shown on one the previous slides. This gap is related to applying the theoretical models in practical settings.
- Scientific literature on emotion measurement covers a broad spectrum of techniques including psychophysiological signals (e.g., electrodermal skin response, neuroendocrine factors or heart rate), observation of behaviour (e.g., vocal and verbal characteristics or body expressions and postures), measurement of facial expressions, self-reporting questionnaires and text analysis.
- Many of these techniques have applied in context of software engineering as well: e.g., Girardi et al. have analysed psychophysiological signals to recognise emotions of developers; Novielli et al. have argued that facial expressions should be used as a gold standard; self-reporting questionnaires have been used by Çalikli et al. and Schneider et al., respectively. As the communication between software developers is to a large extent text-based, the use of vocal information for emotion detection has not been explored in the software engineering context. The work of Herrmann and Klünder takes the first step in this direction. The authors advocate usage of audio to analyse emotion present in software project meetings. However, their approach starts with converting audio to text and, hence ignore the tone and intonation that can reflect emotion experienced by the meeting participants. Marco Verch Professional Photographer Image by kues1 on Freepik
- The very same dominance of text-based communication has led to lions’ share of emotion measurement techniques to focus on textual artefacts produced by software developers, e.g., code review comments, Stack Overflow questions, commit messages or bug descriptions. Text-based sentiment analysis is of course a very well-known task and numerous tools are available. Many of them have been used in software engineering.
- However, several studies have shown that general purpose tools are not working for software engineering texts, and that different contexts and SE-applications require different tools.
- And indeed many tools have been proposed in the literature! Some of them detect distinct emotions following Ekman’s model: joy, love, sadness, anger, surprise, fear. This is, for example, the case for EmoTxT and an unnamed BERT-based classifier. Other tools are based on the continuos model of Russel and, in particular, on the valence component of this model. These tools would classify sentiment as positive, neutral and negative. This is, for example, the case for almost all tools with the name starting with Senti.
- Summarising the first part of the talk: Emotions influence both cognitive processes such as problem solving and interpersonal interaction, both important elements of software engineering. While multiple theories of emotion can be found in the literature, studies of emotion in the context of software engineering either use theories with a small number of distinct emotions, or those positioning emotions in a continuous one- or multidimensional space. While a broad spectre of emotion measurement techniques can be found in the psychological literature, and many of them have been applied in the software engineering context, text-based techniques (sentiment analysis) remains dominant. Multiple sentiment analysis techniques have been designed specially for software engineering.
- Let us move now to the second part of the talk, namely let us talk about ecosystems. And, of course, in this forum, after a day full of conversations about ecosystems and systems of systems, one might think that discussing the notion of a “software ecosystem” is not necessary.
- However, many definitions of software ecosystems can be found in the literature Bosch and Bosch-Sijtsema: “A software ecosystem consists of a software platform, a set of internal and external developers and a community of domain experts in service to a community of users that compose relevant solution elements to satisfy their needs.” Lungu has approached ecosystems as collections of jointly developed projects… To understand the state-of-the-art research on emotion analysis of software ecosystems, we did not want to focus on a specific definition as not to limit our scope.
- Rather than selecting one of these definitions a priori we have decided to start with adopting a nominalistic approach, i.e., state that an ecosystem is whatever is being called an “ecosystem”.
- Following this approach we conduct a literature review of sentiment and emotion in software ecosystems recording the definitions of the ecosystems used in the primary studies, the ecosystems considered and the insights obtained in the primary studies. While additional articles might have studied ecosystems without using the term “ecosystem”, the goal is not to provide a comprehensive overview of emotion analysis in software ecosystems but rather to identify what kind of artefacts are usually being called “ecosystems” in the literature on opinion mining for software development.
- Luckily, Lin et al. have collected 185 articles as part of their systematic literature review of opinion mining for software development. Opinion mining is a rather broad term but in practice it is used interchangeably with “sentiment analysis”, and even more specifically to classification sentiment into positive, neutral and negative. We check the articles that refer to software ecosystems and keep 28 primary studies. None of them has provided a formal definition of an ecosystem.
- It turns out that in context of the sentiment and emotion studies in software engineering, “ecosystems” are often seen as either platforms or collections of interrelated communication platforms supporting software development (e.g., GitHub, Stack Overflow, Google Play app store) => closer to the definition of Bosch and Bosch-Sijtsema: “A software ecosystem consists of a software platform, a set of internal and external developers and a community of domain experts in service to a community of users that compose relevant solution elements to satisfy their needs.” or as collections of interrelated software projects (e.g., Apache, Eclipse, OW2). => The focus on projects within the ecosystems is shared with the way Lungu has approached ecosystems as collections of jointly developed projects Finally, several papers have used the word “ecosystem” in a very generic sense not necessarily disclosing a particular meaning
- It turns out that in context of the sentiment and emotion studies in software engineering, “ecosystems” are often seen as either platforms or collections of interrelated communication platforms supporting software development (e.g., GitHub, Stack Overflow, Google Play app store) => closer to the definition of Bosch and Bosch-Sijtsema: “A software ecosystem consists of a software platform, a set of internal and external developers and a community of domain experts in service to a community of users that compose relevant solution elements to satisfy their needs.” or as collections of interrelated software projects (e.g., Apache, Eclipse, OW2). => The focus on projects within the ecosystems is shared with the way Lungu has approached ecosystems as collections of jointly developed projects Finally, several papers have used the word “ecosystem” in a very generic sense not necessarily disclosing a particular meaning
- This means that now we are ready to put the two and two together and see what the literature study tells us about emotion analysis in software ecosystems
- A word of caution is in place though: due to use of different datasets and tools conclusions derived by similar studies might appear contradictory. Moreover, as the studies of Jongeling et al. and Novielli et al. show, validity conclusions about texts created by software engineers but derived using general purpose sentiment analysis tools that have not been adjusted to the software engineering domain, should be reassessed as those tools are known to be unreliable in the software engineering context. In particular, this is the case for all results published prior to 2017 as the first software engineering specific sentiment analysis tool has been published in 2017. For example, such insights as “Java GitHub projects tend to have a slightly more negative score than projects implemented in other languages” or that comments on Monday were more negative than comments on the other days could not have been confirmed when a different sentiment analysis tool has been used.
- Hence the main message here: please check the tools before using them. If the tools are broken or unreliable, so will be the results of the paper. Moreover, we need better tools!
- So let us move back to the ecosystems… When ecosystems are treated as communication platforms, sentiment is used to predict the outcome of developer’s activities on these platforms (e.g., whether their Stack Overflow question will be answered or whether their GitHub merge will be broken) or to understand differences in the context where different sentiment is observed: e.g., security-related entities are more negative than the rest of the entities, and that this does not depend on whether one considers as “entities” commits or pull requests, and individual comments or entire discussions; or that comments rather than questions and answers tend to express emotions, and that this can be attributed to the fact that comments do not influence the reputation scores and hence can be seen as a kind of “lawless region” where anything goes. Yes/No Image by 8photo on Freepik Boxes: Image by lifeforstock on Freepik
- Let us zoom in on a couple of studies that have tried to understand the impact of emotion on the variables of interest such as getting good answers on Stack Overflow or getting GitHub pull requests merged. Understanding the impact of emotion allows one to formulate guidelines on how developers should behave in order to achieve their goals. Here we see two examples of such guidelines.
- The second group of studies has investigated the software engineering context where specific emotions can be observed. This is one of the first such studies conducted by Daniel Pletea, Bogdan Vasilescu and myself in 2014 (and this is the slide I have used back then!) What is nice about this result is that it has been replicated several times.
- When ecosystems are treated as collections of projects, the studies focus on differences between different kinds of contributors, different projects and popularity of different emotions. Image by Freepik
- And again, let us zoom in on a couple of papers in this category. Most of the communication is neutral. This is something we observe repeatedly for different studies and different data sources. Joy is the most common emotion among those expressed Disgust is barely present Raspberry Pi has a relatively high % of fear.
- Nughoro et al. [?] have observed that Junior contributors and Members post more positive messages than Senior contributors; moreover Juniors both start positive interactions and receive positive responses, while Seniors post initiate positive, neutral and negative communication. These observations seem to concur with the idea that the relative power of the actor and target affects the extent to which display rules require controlling one’s expressions
- I would like to dedicate the last part of this talk to discussion of future work that we as emotion researchers can consider working on.
- From the emotion analysis perspective we need to consider further applications of emotion analysis - after all, many more applications are possible. For example, using sentiment analysis one can detect setback experienced by developers, and adapt the planning. We have seen that some tools are unreliable meaning that we need better tools, on the one hand, and replications of the previous published studied, on the other hand. How else would we know whether the results of these studies are correct? Finally, since sentiment analysis tools are usually ML-classifiers we need more, more diverse and better data. “More diverse” refers to platforms that have not been considered before such as BountySource, devRant, DEV, HackerNews, and sources going beyond English-speaking sources and English-speaking developers.
- From the emotion analysis perspective we need to consider further applications of emotion analysis - after all, many more applications are possible. For example, using sentiment analysis one can detect setback experienced by developers, and adapt the planning. We have seen that some tools are unreliable meaning that we need better tools, on the one hand, and replications of the previous published studied, on the other hand. How else would we know whether the results of these studies are correct? Finally, since sentiment analysis tools are usually ML-classifiers we need more, more diverse and better data. “More diverse” refers to platforms that have not been considered before such as BountySource, devRant, DEV, HackerNews, and sources going beyond English-speaking sources and English-speaking developers.
- Multimodal emotion analysis
- Multimodal emotion analysis Facial expressions: x axis is the time (number of frame)
- A similar approach is needed for software developers: Along the same dimension, Novielli et al. advocate in favor of the design and implementation of tools combining multiple approaches for emotion assessment, to fully support emotion awareness during software development. Specifically, they envisage the emergence of tools and practices including both self-reporting of emotions through experience sample as well as emotion detection through using biometrics, as they might provide complementary information on the emotional status of an individual.
- To summarise. We have talked about emotion analysis of software ecosystems.
- We started with discussing different theoretical models of emotion available in scientific literature, and shown that these models have been applied in software engineering. The gap between the theoretical model and actual results is related to measurement of emotion. This is a challenging task and while many techniques have been tried, the field is dominated by text-based analysis.
- We have conducted a literature study and observed that sentiment analysis papers referring to ecosystems tend to interpret the word “ecosystem” either as platforms or collections of platforms such as Stack Overflow or GitHub used to develop software, or as collections of projects such as Apache or Eclipse. The first interpretation comes close to the vision of Bosch and Bosch-Sijtsema, the second one - of Lungu. When ecosystems are treated as communication platforms, sentiment is used to predict the outcome of developer’s activities on these platforms (e.g., whether their Stack Overflow question will be answered or whether their GitHub merge will be broken) or to understand differences in the context where different sentiment is observed: e.g., security-related entities are more negative than the rest of the entities. The former can provide guidelines to developers; the latter - further insights in software development practices. When ecosystems are treated as collections of projects, the studies focus on differences between different kinds of contributors, different projects and popularity of different emotions.
- Finally, looking forward we need to seek new applications of sentiment analysis, develop better tools, ensure revalidation of previous results and seek more and more diverse data. To address the inherent limitations of the text-based techniques, multimodel emotion analysis is needed.