Citizen disengagement from politics is one of the main issues in modern democracy. Technologies can be used to tap into new internal motivations for people to take part and make sense of political debate. We present a case study of citizens engaging with the replay of political election debates with a novel hypervideo technology called Democratic Replay. Results of the study show that Democratic Replay increases people’s appetite for a new type of engagement with televised elections debates which is based on the realisation of key dimensions of deliberative democracy, such as: reflecting and focusing on different aspects of the political debate, reconstructing the arguments that politicians are making, and assessing facts and evidence. The study also shows that visual analytics narratives and hypervideo navigation improve sensemaking in that they trigger questioning and changing of personal assumptions that people hold before watching the debate. This is a very encouraging result, which addresses the ongoing concern about the real value of new media in the context of political debate and democratic deliberation: specifically, the scepticism toward their capability to support people’s critical thinking rather than promote polarisation of pre-existing groups and opinions. Our research into new sensemaking technologies and hypervideo shows that new media can crucially provide new ways for citizens to detect and make sense of political manipulations, check facts versus speculations, gain new insights, and confidently inform their political choices. Results of the demographic analysis also show that Democratic Replay appealed to different demographic sub-groups with different sensemaking behaviours. This means that democratic spaces and rights cannot be interpreted uniquely and should respond to people’s personal needs, interpretation and understanding of society. Therefore, technologies for democratic public deliberation need to be designed with a variety of users in mind, and they need to be customised to the needs of different demographic groups if they aim to reach all citizens.

1. Critical Thinking as a Skill for Democracy
A Case of Citizen Engagement with UK Televised Election Debates
Anna De Liddo
anna.delidd@open.ac.uk

3. Anna De Liddo
Research Fellow
Lucia Lupi
PhD Student Urban Informatics
Michelle Bachler
Senior Project Officer
Alberto Ardito
Web Developer
Retno Lasarti
PhD Student Explainable AI
Lucas Anastasiou
PhD Student

4. Why Critical Thinking Matters
“Everyone thinks; it is our nature to do so.
But much of our thinking, left to itself, is biased, distorted, partial, uninformed or
down-right prejudiced.
Yet the quality of our life and that of what we produce, make, or build depends precisely
on the quality of our thought. Shoddy thinking is costly, both in money and in quality of
life.
Elder, L., 1997
Critical thinking: The key to emotional intelligence. Journal of developmental education, 21(1), p.40

5. Defining Critical Thinking
"Critical thinking... means making reasoned judgments" (p. 8)
Beyer, B. K. (1995)
Critical thinking. Bloomington, IN: Phi Delta Kappa Educational Foundation.
…varies according to the motivation underlying it,
…is never universal in a individual,
…it is a skill and dispositions which need to be exercised in a life-long endeavor.
Modified from: The Foundation of Critical Thinking
https://www.criticalthinking.org/pages/defining-critical-thinking/766

6. Defining Critical Thinking
“To engage in high quality reasoning, one must have not only the cognitive ability to
do so, but the drive to do so as well.
…the affective dimension, comprised of feelings and volition, is a necessary
condition and component of high quality reasoning and problem solving.
In short, the truly intelligent person is not a disembodied intellect functioning in an
emotional wasteland, but a deeply committed mindful person, full of passion and
high values, engaged in effective reasoning, sound judgment, and wise conduct.”
Elder, L., 1997
Critical thinking: The key to emotional intelligence. Journal of developmental education, 21(1), p.40
.

7. CRITICAL THINKING CYCLE
Reconstruct
and Examine
patterns of
Personal
Believes
Recognise
problems and
attempt to
find workable
solutions
Gather and
Marshal
pertinent
information
and evidence
Recognise
unstated
Assumptions
and Values
Appraise
Evidence and
Evaluate
arguments
Test
Conclusion
and
Generalization
Based on Glaser’s seminal study on critical thinking and education - Edward M. Glaser, An
Experiment in the Development of Critical Thinking, Teacher’s College, Columbia University, 1941)
Affective Dimension

8. Critical Thinking, Emotional Intelligence and the Capability to Change
• “Despite the fact that thoughts, feelings, and desires are equally important functions of
the mind, it is cognition, or thinking, which is the key” [and can lead to “change”] “to the
other two”…
• We are able to change idea or feeling only by thinking differently about that which
we are experiencing, when we think that there is something we can do to change the
situation. Such a change in thinking leads to a change in motivation, and, ultimately, a
change in action.”
Adapted from Elder, L., 1997
Critical thinking: The key to emotional intelligence. Journal of developmental education, 21(1), p.40
“I see it feelingly”
Shakespeare, King Lear, act 4, scene 6

9. Our Open Questions
• Is critical thinking a recognized value and skill in modern society?
• What does it means to think critically in a hyper-connected and data-intensive
society?
• Are there any technologies to facilitate the emergence or practice of critical
thinking in public deliberation?
We look at public deliberation during political elections campaign, and in
particular at televised election debates as still the most prominent mean of
disseminating political information to a wide variety of citizens during an election
campaign.

10. We miss the right social context to practice critical thinking
• Critical thinking is hard to practice in a world of filter bubbles and echo chambers in
which we are used to benefit from the ”similarity” and support we receive from peers.
• …and the skill of ‘opinion shifting’ is seen as a sing of confusion and weakness rather
than the highest expression of knowledge and wisdom.
Socrates Quotes:
“I am the wisest man alive, for I know one thing, and that is that I know nothing.”

11. Information overload and balkanization makes it difficult for citizens to
extract useful and reliable insights to make better-informed decisions on
how to vote.
Information complexity and uncertainty (fake vs legitimate news)
contribute to citizens’ finding the political process obscure, confusing, and
difficult to relate to.
We miss the right social context to practice critical thinking

12. Our Hypothesis
• we hypothesize that people’s confusion and disconnection with politics is due mostly
to a lack of understanding of the complexity of the political debate, rather than a
lack of technologies for public debate, and as such it can be bridged by developing
individual socio-technical spaces for critical thinking, reflection and sensemaking.
• to test this hypothesis, we designed, implemented, and tested Democratic Replay, a
hypervideo and visual analytics technology to enhance citizens’ sensemaking of
televised election debates.

13. HomePage: 7 Analytical lenses on the Debate
A 3 minutes demo video
of Democratic Replay is
available at:
https://youtu.be/lc9htNN
NQdo

14. Hypermedia Repository combining Heterogeneous Data Sources,
Complex Analytics, and Interactive Visualizations

22. A/B TESTING
Control (BBC News interface)
57 people
Treatment (Democratic Replay interface)
56 people

23. EMPIRICAL EVALUATION DURING THE 2015 UK GENERAL
ELECTION CAMPAIGN
• 113 people - two groups: control (BBC News interface) and treatment (Democratic
Replay interface)
• groups balanced in term of age gender and political decidedness, with a nearly equal split
per demographic category
Analysis by demographic sub-group has been carried out:
• Gender: 52 participants - women (46%), 60 men (53%), and 1 preferred not to say.
• Age range18 to 65 - 45 people (40%) between 18 and 30, 32 (28%) between 31 and 40,
and 36 (32%) aged 41 or more.
• decided (60%) and undecided (40%) on how to vote
Conditions for selection:
(a) they knew how to use the Internet, and
(b) they answer affirmatively the question: “If there were to be a televised leaders’
debate at the time of the next general election, would you choose to watch it?”.

24. Critical Thinking and Sensemaking features
Sensemaking Features Otherwise Defined Adapted From
Reflection Retrospect Weick 1995
Insights Gaining insight
Focus Finding connections
Argumentation Structuring
Explanation
Assess Facts and Evidence
Distinguishing
Assess Assumptions
Change Assumptions
Reducing confusion, uncertainty, and
ambiguity
Gap-finding and gap-bridging.
Alsufiani et al. 2017

25. Results of Democratic Replay’s comparison with BBC replay
Democratic Replay enables 7 critical thinking and sensemaking capabilities:
1. experiencing surprise: “unexpected insights on the debaters and on what they said,”
2. “reflecting on the debate in a deeper way”
3. “evaluating facts and evidence
4. “focusing on different aspects of the debate”
5. “reconstructing the arguments that the speakers made.”
6. “assessing personal assumption” and
7. “changing some initial assumptions had before the debate.”

26. Gender Reactions to the Tool
Sensemaking Feature U p z r
Reflection 205.00 0.0152 2.4275 0.3366
Insight 214.50 0.0224 2.2830 0.3166
Focus 242.00 0.0804 1.7485 0.2425
Argumentation 211.00 0.0210 2.3072 0.3199
Explaination 237.50 0.0669 1.8321 0.2541
Evaluate Facts & Evidences 177.50 0.0029 2.9763 0.4127
Distinguish 246.00 0.1052 1.6201 0.2247
Assess Assumptions 220.50 0.0320 2.1440 0.2973
Change Assumptions 220.00 0.0344 2.1152 0.2933
Sensemaking Feature U p z r
Reflection 353.50 0.1620 1.3984 0.1805
Insight 306.00 0.0313 2.1532 0.2780
Focus 370.00 0.2433 1.1669 0.1506
Argumentation 364.50 0.2151 1.2396 0.1600
Explaination 395.50 0.4500 0.7554 0.0975
Evaluate Facts & Evidences 322.00 0.0596 1.8837 0.2432
Distinguish 425.50 0.8015 0.2514 0.0325
Assess Assumptions 383.00 0.3587 0.9179 0.1185
Change Assumptions 358.00 0.1857 1.3235 0.1709
• Women show a larger variety of sensemaking behaviours and bigger effect sizes,
they used the tool evaluate the debate and assess and change their personal
assumptions.
• Men focused on the capability of the tool to provide unexpected insights.

27. Age Range Reactions to the Tool
• Younger people, have effectively used Democratic Replay to ‘reconstruct the
arguments that the speakers made’ and to ‘assess their personal assumptions’.
• Adults between the ages of 30 and 40 showed the biggest reaction to Democratic
Replay, both in terms of number and sizes of the effects. They most prominently
used the tool to better assess facts and evidence.
• Older Adults (>40) are less likely to be affected by new technologies in the way
they shape their opinion and change their assumptions.
Sensemaking Feature U p z r
Reflection 179.00 0.1240 1.5381 0.2293
Insight 180.50 0.1238 1.5388 0.2294
Focus 202.00 0.3215 0.9913 0.1478
Argumentation 130.00 0.0053 2.7900 0.4159
Explaination 207.50 0.3844 0.8697 0.1297
Evaluate Facts & Evidences 160.00 0.0438 2.0157 0.3005
Distinguish 202.50 0.3342 0.9657 0.1440
Assess Assumptions 147.50 0.0204 2.3188 0.3457
Change Assumptions 176.00 0.1072 1.6108 0.2401
Sensemaking Feature U p z r
Reflection 62.00 0.0243 2.2525 0.3982
Insight 69.00 0.0453 2.0016 0.3538
Focus 84.00 0.1453 1.4563 0.2574
Argumentation 93.00 0.3506 0.9334 0.1650
Explaination 81.00 0.1351 1.4942 0.2641
Evaluate Facts & Evidences 50.50 0.0070 2.6990 0.4771
Distinguish 87.50 0.2465 1.1588 0.2048
Assess Assumptions 74.00 0.0847 1.7239 0.3048
Change Assumptions 58.00 0.0160 2.4086 0.4258
Sensemaking Feature U p z r
Reflection 143.00 0.5407 0.6118 0.1020
Insight 111.00 0.0962 1.6633 0.2772
Focus 119.50 0.1508 1.4367 0.2394
Argumentation 155.50 0.8531 0.1851 0.0309
Explaination 152.50 0.7684 0.2945 0.0491
Evaluate Facts & Evidences 119.00 0.1558 1.4193 0.2366
Distinguish 157.00 0.8932 0.1342 0.0224
Assess Assumptions 172.50 0.7240 0.3531 0.0588
Change Assumptions 172.00 0.7367 0.3362 0.0560
• >40 yrs• 31-40 yrs• 18-30 yrs

28. Design for Democracy on Demand
Results of the demographic analysis show that:
• Democratic Replay appealed to different demographic sub-groups with different
sensemaking behaviors.
Nowadays citizens require ‘democracy on demand’ (Coleman et al. 2015) and democratic
spaces and rights cannot be interpreted uniquely and should respond to people’s personal
needs, interpretation and understanding of society.
• Technologies for democratic public deliberation need to be designed with a variety of
needs in mind, of different demographic groups, if they aim to reach all citizens.
Coleman, S., Blumler, J., Moss, G. and Homer, M.S., 2015. The 2015 Televised Election Debates; Democracy on Demand?.

29. Future Research:
A Socio-Technical Challenge

30. “the best remedy for propaganda and misinformation... is
helping people engage in critical thinking and evidence-based
reasoning.” – Ricky Sethi
Citation credit - Juho Kim
https://theconversation.com/how-citizen-investigators-can-collaborate-on-crowdsourced-fact-checking-76890
CRITICAL THINKING AS A CRITICAL DEMOCRATIC SKILL

31. CRITICAL THINKING AS A CRITICAL DEMOCRATIC SKILL
“to live successfully in a democracy, people must be able to think
critically in order to make sound decisions about personal and civic
affairs.
If students learn to think critically, then they can use good thinking
as the guide by which they live their lives.” - Beyer, B. K. (1995)
Beyer, B. K. (1995) Critical thinking. Bloomington, IN: Phi Delta Kappa Educational Foundation.

32. CRITICAL THINKERS AS DEMOCRATIC CITIZENS
Attempt to:
• reason at the highest level of quality and in a fair- minded way
• Live rationally, reasonably and empathically
• Avoid thinking simplistically
• Appropriately consider the rights and needs of relevant others
Are aware:
• of the inherently flawed nature of human thinking when left unchecked
• That anyone at times fall prey to mistakes in reasoning, human irrationalities, biases,
distortions, uncritically accepted social rules and taboos, self-interest, vetted interests
Adapted from Linda Elder’s account on Critical thinking (2007)
Assistant Director of the Center for Critical Thinking, Sonoma State University, Rohnert Park, CA 94928.

33. “…recognize the complexities in developing as thinkers, and commit
themselves to life-long practice toward self-improvement.
They embody the Socratic principle: The unexamined life is not worth
living , because they realize that many unexamined lives together
result in an uncritical, unjust, dangerous world”
Linda Elder (2007)
Assistant Director of the Center for Critical Thinking, Sonoma State University, Rohnert
Park, CA 94928.
CRITICAL THINKERS AS DEMOCRATIC CITIZENS

34. Design for Sensemaking and Critical Thinking
• If we want to support people’s capability to question assumptions and think critically, we
need to design spaces for personal reflection and sensemaking.
New Human Interfaces and Interactions for “Deliberation Within” –
(Goodin and Niemeyer, 2016, p. 629)
Sensemaking support comes at a cost: aggregated data and advanced analytics required expert
human annotation and data integration between (often) proprietary platforms.
• sensemaking processes need human–machine support
• tools needed to bridge political debate across community platforms
Extended Intelligence (Collective Intelligence + AI) and (Open) Data Science Approaches
Goodin, R. E., and Niemeyer, S. J. (2016). When Does Deliberation Begin? Internal Reflection versus Public Discussion in Deliberative Democracy:. Political Studies,
51(4), 627–649. http://doi.org/10.1111/j.0032-3217.2003.00450.x

35. 2 RUNNING PROJECT
• ISOOKO EU project we are now exploring the use of Democratic Reflection
(democraticreflection.org) as a tool to support peace building and peace education in
developing countries.
• Democratic Reflections is a new crowdsourcing platform to harness real-time
audience feedback of live or video recorded events, such as televised election debates
and video testimonials.
• We are working with Aegis Trust (UK’s NGO), to engage local communities Rwanda
in actively listen and reflect on a testimonial video, called “Ubumuntu”, which tells a
range of stories that relate to the Genocide against the Tutsi in Rwanda.
• Our am is to assess to what extent Democratic Reflection helped users to relate to the
stories of Genoside in a personal way, promoted change in personal assumptions,
and developed empathic thinking and behaviour change.

36. JUST STARTING
• STRUCTURED DECENTRALISED DELIBERATION SYSTEMS for collective
decision making
• ONR - Office of Naval Research Global (US grant) - individual fellowship 3 years
• Will develop a new online discussion platform for digital journalism and public
deliberation around media articles, which is equally accessible, but more structured,
decentralised, and higher data quality than common social media.

37. Collective Intelligence For the Common Good Community - ci4cg.org
Several international workshops
and 2 Special issues

38. THANK YOU FOR LISTENING!
• Please fell free to contact me at anna.deliddo@open.ac.uk - @Anna_De_Liddo
• To know more about our work please visit the research group website at:
idea.kmi.open.ac.uk