This document summarizes two case studies on using topic opinion analysis to support eGovernment. The first case study analyzed online forums to validate the accuracy of summarizing large amounts of data. It found the toolbox performed well on in-depth forums with nuanced discussions. The second case study analyzed social media like Facebook and Twitter. It found the toolbox identified expected topics that stakeholders were already aware of. Topic quality was better for Twitter than Facebook. Overall, the studies found topic analysis can help understand diverse communications but improvements like explaining algorithms are still needed. Qualitative validation was effective but time-consuming. Topic analysis shows potential to support politicians and organizations if combined with social media strategies.

1. Online Forums vs. Social Networks: Two
Case Studies to support eGovernment
with Topic Opinion Analysis
Timo Wandhöfer, Beccy Allen, Steve
Taylor, Paul Walland, Sergej Sizov
IFIP EGOV13, University of Koblenz

2. Contents
• Background & Context
• Applied Process Model
• Online Forums: The HeadsUp Case
• Social Networks: The Facebook/Twitter Case
• General Findings
• Conclusions & Further Work

3. Background

4. Context:
WeGov EU-Project
• Allow policy makers to interact directly with
citizens using Social Network Sites
– Use the tools the citizens already use
• Find and understand
people’s opinions
• Become part of the
discussion
• Open dialog
• Respect privacy
• Encourage trust

5. Problem Statement
• In many cases, discussion tracks in social
media become long and complex
• Stakeholders (WeGov) are often interested in
gaining a quick overview of a discussion
• However, completely reading hundreds of
posts is too time-consuming to be practical
• There is thus a huge need to summarize the
discussion tracks…

6. WeGov Toolbox: Topic
Opinion Analysis

7. Is this really useful within
everyday life?

8. Applied Process Model

9. Online Forums:
The HeadsUp Case

10. Online Forums
• Civil society groups run forums and blogs to
connect with their members and supporters
• Analysing the themes of discussions is often
beyond the organizations resources

11. HeadsUp
• Online forums for connecting young people with
politics and politicians
• Forums ran for a decade (2003-2013) by the
Hansard Society
• They racked up nearly 10,000 comments from
students in 50 forums
• 330 parliamentarians & decision-makers have
taken part in debates with 11-18 year olds
• HeadsUp users' views have been mentioned in
debates in both the House of Commons and the
House of Lords

12. Methodology
• Evaluating the usefulness (accuracy, reliability) with
regard to forum data using a real world data set
• Independent data (created beforehand)
• Each forum was analyzed manually within a report
• Three forums of different sizes:
– Sex Education – Do you get enough? (36 posts)
– Youth Citizenship Commission: are young peopled
allergic to politics? (317 posts)
– How equal is Britain? (1186 posts)
• Toolbox output was compared to the forum reports
– Themes of the debate, sentiment, user interface

13. Findings
• The toolbox is best at dealing with large quantities of
data, amounts that could not be analyzed effectively
by a human.
• The toolkit performs well on relatively in-depth data -
this lends itself to blogs and forums that encourage
more considered and less immediate responses.
• The toolbox also performed well in showing the
nuances between different elements of a wider
debate. E.g. the subtopics for “How equal is Britain?”
• The toolkit works best when analyzing medium
length comments that focus on one issue and when
spelling and vocabulary are good.

14. Social Networks:
Facebook/Twitter Case

15. Methodology:
Intention
• The intention of this case study was the
validation of usefulness of topic opinion
analysis of social media for politics.
• Therefore we designed use cases how this
technology may support politicians’ everyday
life.

16. Methodology:
Real Life Applications
1. Local Facebook topics:
– Monitoring a sample of at least ten Facebook pages
represent a geographical area like an MP’s constituency.
– Apply topic opinion analysis to extract the topics that people
discuss on the pages.
– Each topic is a combination of words that represents a
theme of the discussion, and comes with key users, and key
comments.
2. Monitoring topics on Twitter:
– Identify subtopics e.g. the general debate on climate change
covers subtopics like green energy, new kinds of
technologies.
– Because the results are already filtered by the search, the
analysis produce subtopics.

17. Methodology:
Stakeholders
• Conducted with a number of governmental
representatives as end-user stakeholders:
– two members of the German Bundestag
– four employees that work directly for a member of
the German Bundestag
– two members of the State Parliament North Rhine-
Westphalia
– one small German city (Kempten)
– one big German city (Cologne)
– and with a German state chancellery (Saarland)

18. Methodology:
Qualitative Workflow
• Input: Crawling of individual data (themes
and constituency)
• Analysis: Topic Opinion Analysis
• Report: Reports covering an individual
analysis
• Questionnaire: Using individual examples
from the report
• Follow-up interview: Conducted to receive
more in-depth assessments about the results

19. Findings
• Sensible and expected topics
– All topics from local Facebook pages that were assessed as
understandable were known beforehand; the reasons being:
WeGov stakeholders are ‘aware of the (online) public area’
– Equal with Twitter
• Quality of topics
– Better results with Twitter
– Less clear topics with local Facebook pages. The reason why
42% of the 110 topics were assessed as understandable topics
is due to the fact that policy makers know what’s happening in
the area of their electorate
• Different meanings for topics
– the combination of five words for one topic could have
multiple meanings

20. General Findings

21. Potential end-user
groups
• The toolbox could play an important role in
helping to understand feedback across a range
of communication channels
– small not-for-profit organizations
– larger media organizations
– as well as politicians and policy
(parties, parliaments, bigger city administrations)
• Anyway: Experiences with social media and as
well a social media strategy in general are
necessary!

22. Improvements
• A plain explanation of how the algorithm understands and
processes data -> ensure users trust the results. An explanation
of irregularities. E.g.
– Why the same data sometimes yields different results?
– Why keywords appear in the order and frequency that they do?
• Showing the hidden workings of the toolkit. E.g.
– The relative influence of a greater number of key words e.g. via a tag
cloud.
– Highlighting positive/negative words that contribute to sentiment
scores.
• Implementing more options for the users to refine the data and
customize it to their situation and needs. E.g.
– The ability to exclude certain posts or words from analysis.
– Splitting up long posts into sections that can be analyzed separately
to avoid the conflicting analysis of longer posts.

23. Conclusions
• Two case studies how to apply and validate topic opinion analysis
– HeadsUp focused on accuracy/reliability
– Social network case focused on the usability within the decision-
makers’ everyday life
• Follow the same process model and show added value / boarder-
lines
• Within both cases there was already an understanding of what
the data were about – we call it the „control group“
• Toolbox performed well in showing the nuances between
different elements of a wider debate
• With local Facebook pages the quality of results worsen due to
the fact of the quality of input data and less political
conversations
• Qualitative validating process was very effective but time
consuming
• The Cases protect the rights and privacy of citizens