User acceptance is key for the adoption of a new technology. In this work we experiment with a novel service for tutors in workplace learning settings. Sentiment analysis is a way to extract feelings and emotions from a text. In a learning setting such a sentiment analysis can be part of learning analytics. It has the potential to foster the understanding of emotions in shared discussions in learning environments, detect group dynamics as well as the impact of certain topics on learners’ sentiments. However, sentiment analysis presents some challenges too, as lived experiences, expectations and ultimately acceptance of this technology varies greatly and can become barriers to adoption. In order to design a system for learning analytics accepted by tutors we experimented with proof-of-concept prototypes and received valuable feedback from tutors regarding the usefulness of the overall sentiment analysis as well as certain features. The qualitative feedback confirms the overall interest of tutors in sentiment analysis and gives important hints towards more detailed analytical elements.

1. www.employid.eu
EmployIDEmployID
ChristianVoigt, Barbara Kieslinger,Teresa Schäfer
Zentrum fuer Soziale Innovation,Vienna,Austria
[voigt, kieslinger, schaefer] @zsi.at
HCI International 2017
Vancouver, Canada, 9 - 14 July 2017
This project has received funding from the European Union’s Seventh Framework Programme
for research, technological development and demonstration under grant agreement no. 619619
User Experiences Around Sentiment Analyses,
Facilitating Workplace Learning

2. www.employid.eu
Conceptual Frame
14th July 2017 ChristianVoigt | voigt@zsi.at 2

3. www.employid.eu
• User acceptance is key for the adoption of a new
technology
• Sentiment analysis is a way to extract feelings and
emotions from a text
• Can be part of learning analytics
• Has the potential to foster the understanding of emotions
in shared discussions, detect group dynamics as well as
the impact of certain topics
• However, lived experience of sentiment analysis,
related expectations and ultimately acceptance of this
technology varies greatly and can become barriers to
adoption.
Challenge
14th July 2017 ChristianVoigt | voigt@zsi.at 3

4. www.employid.eu
• Natural language processing aiming to extract feelings,
affects, emotions or opinions in a text
• Outputs can be simple dichotomies (positive versus
negative expressions), scoring approaches from −5 (very
negative) to +5 (very positive)
• Categorical approaches such as distinguishing basic
emotions such as joy, trust or anger
• Already widely applied in product or movie reviews,
political discourse analysis or spam detection
Short recap of sentiment analysis
14th July 2017 ChristianVoigt | voigt@zsi.at 4

5. www.employid.eu
• Participants of this course were all employees of public
employment services (Work Coaches)
• Learning objectives included developing a set of skills and
competences in areas like
• coaching or
• providing labour market information,
• the use of digital tools.
• Learning for career and labour market transitions occurs across
four domains:
• relational development;
• cognitive development;
• practical development;
• emotional development
The context of workplace learning
14th July 2017 ChristianVoigt | voigt@zsi.at 5

6. www.employid.eu
• Learning analytics is applied to MOOCs (Massive Open Online
Courses
• Text materials (n=1,1170 postings) from 68 participants
• Tutors can not read every posting and monitor the activities of
every learner in order to anticipate a conflict or identifying the
learners who would need additional support
• Number of visualizations (proof-of-concept prototypes) are
triggering further visualization
• e.g. displaying an overly negative tone in week 2 is enhanced by
showing the keywords of the debate, related co-word analyses or
relevant text snippets
• Five exploratory interviews, presenting the scenarios and
prototypes
Method
14th July 2017 ChristianVoigt | voigt@zsi.at 6

7. www.employid.eu
Visuals Used
14th July 2017 ChristianVoigt | voigt@zsi.at 7

8. www.employid.eu
Determinants of positive or negative sentiments
14th July 2017 ChristianVoigt | voigt@zsi.at 8

9. www.employid.eu
Co-word analysis
14th July 2017 ChristianVoigt | voigt@zsi.at 9

10. www.employid.eu
Overview on a level of individual learners
14th July 2017 ChristianVoigt | voigt@zsi.at 10

11. www.employid.eu
Feedback
14th July 2017 ChristianVoigt | voigt@zsi.at 11

12. www.employid.eu
• “when you express emotions there are greater chances of
learning”
• Linked to overall course evaluation, not monitoring
• No agreement on whether learners should see the analysis
• The fact that categorizations are probabilistic and need to be
followed up through ‘in depth’ reading of postings relativized
usefulness of sentiment analysis
• However, the analysis could support required reporting after
the training had been delivered
Evaluations and comments … (1/2)
14th July 2017 ChristianVoigt | voigt@zsi.at 12

13. www.employid.eu
• Further use was seen in ‚Managing visibility‘ in MOOCS where
the number of comments can quickly become overwhelming
(hence comments could be ranked by their emotional content)
• Linking emotions to individual learners was perceived as the the
least valuable feature
• Training the system (i.e. the use of supervised algorithms /
machine learning) was not seen as a cost – effective approach,
but also because sentiment analysis was not perceived as a core
service within their mandate to deliver trainings
Evaluations and comments … (2/2)
14th July 2017 ChristianVoigt | voigt@zsi.at 13

14. www.employid.eu
Thanks !
14th July 2017 ChristianVoigt | voigt@zsi.at 14