Online Forums vs. Social Networks: Two Case Studies to support eGovernment with Topic Opinion Analysis
1. Online Forums vs. Social Networks: Two
Case Studies to support eGovernment
with Topic Opinion Analysis
Timo Wandhöfer, Beccy Allen, Steve
Taylor, Paul Walland, Sergej Sizov
IFIP EGOV13, University of Koblenz
2. Contents
• Background & Context
• Applied Process Model
• Online Forums: The HeadsUp Case
• Social Networks: The Facebook/Twitter Case
• General Findings
• Conclusions & Further Work
4. Context:
WeGov EU-Project
• Allow policy makers to interact directly with
citizens using Social Network Sites
– Use the tools the citizens already use
• Find and understand
people’s opinions
• Become part of the
discussion
• Open dialog
• Respect privacy
• Encourage trust
5. Problem Statement
• In many cases, discussion tracks in social
media become long and complex
• Stakeholders (WeGov) are often interested in
gaining a quick overview of a discussion
• However, completely reading hundreds of
posts is too time-consuming to be practical
• There is thus a huge need to summarize the
discussion tracks…
10. Online Forums
• Civil society groups run forums and blogs to
connect with their members and supporters
• Analysing the themes of discussions is often
beyond the organizations resources
11. HeadsUp
• Online forums for connecting young people with
politics and politicians
• Forums ran for a decade (2003-2013) by the
Hansard Society
• They racked up nearly 10,000 comments from
students in 50 forums
• 330 parliamentarians & decision-makers have
taken part in debates with 11-18 year olds
• HeadsUp users' views have been mentioned in
debates in both the House of Commons and the
House of Lords
12. Methodology
• Evaluating the usefulness (accuracy, reliability) with
regard to forum data using a real world data set
• Independent data (created beforehand)
• Each forum was analyzed manually within a report
• Three forums of different sizes:
– Sex Education – Do you get enough? (36 posts)
– Youth Citizenship Commission: are young peopled
allergic to politics? (317 posts)
– How equal is Britain? (1186 posts)
• Toolbox output was compared to the forum reports
– Themes of the debate, sentiment, user interface
13. Findings
• The toolbox is best at dealing with large quantities of
data, amounts that could not be analyzed effectively
by a human.
• The toolkit performs well on relatively in-depth data -
this lends itself to blogs and forums that encourage
more considered and less immediate responses.
• The toolbox also performed well in showing the
nuances between different elements of a wider
debate. E.g. the subtopics for “How equal is Britain?”
• The toolkit works best when analyzing medium
length comments that focus on one issue and when
spelling and vocabulary are good.
15. Methodology:
Intention
• The intention of this case study was the
validation of usefulness of topic opinion
analysis of social media for politics.
• Therefore we designed use cases how this
technology may support politicians’ everyday
life.
16. Methodology:
Real Life Applications
1. Local Facebook topics:
– Monitoring a sample of at least ten Facebook pages
represent a geographical area like an MP’s constituency.
– Apply topic opinion analysis to extract the topics that people
discuss on the pages.
– Each topic is a combination of words that represents a
theme of the discussion, and comes with key users, and key
comments.
2. Monitoring topics on Twitter:
– Identify subtopics e.g. the general debate on climate change
covers subtopics like green energy, new kinds of
technologies.
– Because the results are already filtered by the search, the
analysis produce subtopics.
17. Methodology:
Stakeholders
• Conducted with a number of governmental
representatives as end-user stakeholders:
– two members of the German Bundestag
– four employees that work directly for a member of
the German Bundestag
– two members of the State Parliament North Rhine-
Westphalia
– one small German city (Kempten)
– one big German city (Cologne)
– and with a German state chancellery (Saarland)
18. Methodology:
Qualitative Workflow
• Input: Crawling of individual data (themes
and constituency)
• Analysis: Topic Opinion Analysis
• Report: Reports covering an individual
analysis
• Questionnaire: Using individual examples
from the report
• Follow-up interview: Conducted to receive
more in-depth assessments about the results
19. Findings
• Sensible and expected topics
– All topics from local Facebook pages that were assessed as
understandable were known beforehand; the reasons being:
WeGov stakeholders are ‘aware of the (online) public area’
– Equal with Twitter
• Quality of topics
– Better results with Twitter
– Less clear topics with local Facebook pages. The reason why
42% of the 110 topics were assessed as understandable topics
is due to the fact that policy makers know what’s happening in
the area of their electorate
• Different meanings for topics
– the combination of five words for one topic could have
multiple meanings
21. Potential end-user
groups
• The toolbox could play an important role in
helping to understand feedback across a range
of communication channels
– small not-for-profit organizations
– larger media organizations
– as well as politicians and policy
(parties, parliaments, bigger city administrations)
• Anyway: Experiences with social media and as
well a social media strategy in general are
necessary!
22. Improvements
• A plain explanation of how the algorithm understands and
processes data -> ensure users trust the results. An explanation
of irregularities. E.g.
– Why the same data sometimes yields different results?
– Why keywords appear in the order and frequency that they do?
• Showing the hidden workings of the toolkit. E.g.
– The relative influence of a greater number of key words e.g. via a tag
cloud.
– Highlighting positive/negative words that contribute to sentiment
scores.
• Implementing more options for the users to refine the data and
customize it to their situation and needs. E.g.
– The ability to exclude certain posts or words from analysis.
– Splitting up long posts into sections that can be analyzed separately
to avoid the conflicting analysis of longer posts.
23. Conclusions
• Two case studies how to apply and validate topic opinion analysis
– HeadsUp focused on accuracy/reliability
– Social network case focused on the usability within the decision-
makers’ everyday life
• Follow the same process model and show added value / boarder-
lines
• Within both cases there was already an understanding of what
the data were about – we call it the „control group“
• Toolbox performed well in showing the nuances between
different elements of a wider debate
• With local Facebook pages the quality of results worsen due to
the fact of the quality of input data and less political
conversations
• Qualitative validating process was very effective but time
consuming
• The Cases protect the rights and privacy of citizens
Editor's Notes
Figure 2 shows the applied process model how stakeholders were engaged, both to determine requirements and to evaluate the toolbox. The idea behind this approach is to identify potential use cases that are in the end users’ daily working lives. These cases are therefore of value to the end user, and can be used for validating the toolbox and its analysis results. Figure 1 shows two examples of such use cases (“HeadsUp” and “social networks”), and these are discussed throughout the paper to illustrate how the topic opinion analysis can be applied in everyday politics.The top row in Figure 2 shows users on the Internet (the digital society) – for instance users of online forums or social networks. The second row shows stakeholders, and how they interact with the users on the Internet. In the use cases, the stakeholders already perform (often manual) analyses on the data they get from citizens on social networks. The results of their existing analyses are shown in the bottom row – here we call these data the “control group”. The control group is compared with the toolbox’s analyses of the same data. For instance the operator of the HeadsUp discussion forum (cp. left) analyses the forum discussions manually to get an insight on the debate. Another example is the policy maker (cp. right) who extracts topics from social networks to get insight into the discussion.