Democratic Reflection and other contested collective intelligence tools aim to harness technology to enable people to build consensus even when they disagree. These tools use techniques like crowdsourcing and natural language processing to analyze online conversations, identify points of agreement, and generate visualizations to help people reflect on different perspectives. Trials of these tools showed they can improve critical thinking, challenge assumptions, and potentially bridge political and social divides. The tools are being used to facilitate collaboration and evidence-based discussions among groups addressing complex issues like public policy, education, and building peace in places affected by conflict.

1. Harnessing the power of coming together,
even when we disagree
Contested Collective Intelligence Research and Tools
Dr. Anna De Liddo
anna.delidd@open.ac.uk
Learning at Scale Special Interest Group - Invited Talk, Oct 12th, 2021

2. Anna De Liddo
Professor of
Human Computer Interaction
Dr. Lucia Lupi
Urban Informatics
Alberto Ardito
Senior Web Developer
Retno Lasarti
PhD Student Explainable AI
Lucas Anastasiou
PhD Student Argument Mining
Riccardo Pala
Web Developer
Michelle Bachler
Senior Project Officer

5. Collective Intelligence
• defined as the capability to collectively solve complex problems
• CI research seek to develop the conceptual foundations and
sociotechnical infrastructures to improve the ability of small to
large groups to act more intelligently that any person or machine
would do in isolation.

6. BUT.. PEOPLE
OFTEN
DISAGREE!!!

7. Contested Collective Intelligence (De Liddo 2012)
When tackling complex and contested problems
In contexts such as
collaborative learning,
public policy or business
strategy there will almost
always be contention over
the right answers.
There may
not be one
worldview,
or clear
option
Evidence can be
ambiguous or of dubious
reliability requiring the
construction of
plausible, possibly
competing narratives;
Growth in intelligence
results from learning,
which is socially
constructed through
different forms of
discourse, such as
dialogue and debate.

8. Collective Intelligence Spectrum
Model of Collective Intelligence (CI):
from sensing the environment, to interpreting it, to generating good
options, to taking decisions and coordinating action...
Collec&ve(
Ac&on(
Collec&ve(
Decision(
Collec&ve(
Idea&on(
Collec&ve(
Sensemaking(
Collec&ve(
Sensing((
(

9. The discussion spaces that we see on the
Web today are flawed
• are rudimental in the way
they structure data
• lack of content quality and
variety
• scarcely support evidence-
based reasoning
• lack features to enhance
personal understanding and
situational awareness

10. Public Deliberation on the internet is dispersed,
hard to make sense of and often occurs in
echo-chambers
• dispersed across different Web- sites and Social Media environments. This
makes it difficult for people to make sense of the state and progress of a
public debate.
• Poor summarization and poor visualizations issues have been widely
recognized as key detrimental barriers to the quality of online public
deliberation
• An open challenge to effective online deliberation is therefore connecting
communities that are "isolated" in different platforms island and allowing
cross fertilisation of ideas between community platforms.

11. Setting the Problem:
Social/Organisational Dimension
• … and produce well known negative socio-technical effects, such as
• ‘echo chambers” and the activation of biased information dynamics (Ditto
& Lopez, 1992; Taber & Lodge, 2006)
• Homophily and Lack of content variety (Huckfeldt & Sprague, 1995, Mutz
& Martin, 2001).
• polarisation, division and conflict (Sunstein, 2018; Golbeck et al., 2017;
Matias et al., 2015, Binder et al., 2009)
• degrade the quality, balance and safety (Golbeck, 2017; Guntuku, 2017) of
online discourse, up to undermining social tolerance (Mutz 2002)

12. Promising Solutions for Building Consensus
A new type of technology-enabled collective intelligence to help people make sense of and
co-create innovative solutions to complicated challenges.
Contested Collective Intelligence (CCI) tools to harness the power of technology to enable
people worldwide to build consensus – even when, on the face of it, they disagree.
Using advanced human and computational methods, such as crowdsourcing and Natural
Language Processing, to contribute and 'mine' online conversations (with people's consent),
these easy to use tools identify both stated and unstated points of agreement to help
summarise complex debates.
They then generate visualisations of these points of understanding, which we can show
people to help them reflect on each other's ideas and make better informed collective
decisions.
Mining online conversations also produces a wealth of data we can analyse further to
understand the fundamental nature of sensemaking, the social and cognitive process through
which humans make sense of their collective experience.

15. Contested
Collective
Intelligence
(De Liddo
2012)

16. Impact 1. Bridging divides and healing divisions
Society's division and disagreement are no more evident in the political arena or
post war conflicts.
Working with colleagues from The Open University's Knowledge Media Institute
and The University of Leeds to build a new collective intelligence platform to help
people think critically about political and societal issues and challenge their prior
assumptions.
These are crucial skills people need to bridge divides and reduce social conflict.

17. New Modes of Engagement with Televised
Political Debate through Audience Feedback
democraticreflection.org

18. A New Method to Harness Audience Reactions
• Instant
• Nuanced meaning
• Discourse-based: Provided in
form of discourse elements
• Voluntary and non-intrusive
• Enabling analytics and
visualisations
‘Soft’ Feedback:

19. Democratic Reflection’s Interactive Experience

20. Previous successful testing in Political Communication Contexts
Democratic Reflection was
tested with over 2000 UK citizen
in the last 2015, 2017 and 2019
political election debates, and
was reported to improve self-
reflection and learning, “change
the way people felt about
political leaders”, and showed to
“change personal assumptions”
that people had before using the
tool.

21. Emerging insights from the last 2015, 2017, 2019 UK Elections
• Democratic Reflection facilitated:
• High engagement and willingness to use the tool in the future
• No usability issues (also from people with very low digital literacy)
• Improved sensemaking
• “unexpected insights on the debaters and on what they said,”
• To “reflect on the debate in a deeper way”
• “assessing personal assumption” and
• “changing some initial assumptions had before the debate.”

22. Trials with broadcaster ITV during the 2015, 2017 and 2019 UK general election
• “I came into this group more or less decided on what I was going to vote for, but
the statistics completely changed my mind, so that helped”
• “To someone who's really undecided on who to vote for, it is quite interesting to
see what my reactions were […] I was a bit like oh, right. Okay, so obviously, I
agree with what they're saying. Maybe I need to go back and look at their
manifesto and read it all through again.”
2017 trial participants

23. Impact 2: Nudging
Behavior Change at
Scale to Build Peace
democraticreflection.org
Concept Video:
https://www.youtube.com/watch?time_contin
ue=4&v=xj0gB07yMuU

24. Setting the Problem
In 1994 Rwanda experienced large scale genocide.
Since 2001 Aegis Trust worked with the Government of Rwanda to establish the
Kigali Genocide Memorial (KGM www.kgm/rw ) in commemoration of the 1994
genocide against the Tutsi in Rwanda, as a place for remembrance and learning.
Video archival materials are part of the collective memory of the country, and are
used extensively by NGOs and piece building organisations as digital storytelling tools
to promote empathy and to build peace.
Challenges in using Digital Storytelling for peace building:
• to measure the impact of digital storytelling videos, and the changes they affect in
the viewers perceptions of the genocide.
• extent to which the viewing experience improves people’s critical thinking, active
listening, and other capabilities for peace.

25. User Study in Rwanda with Aegis Trust
In 2019 with the support of Kings Collage
London, the ISOOKO EU Project and Aegis
Trust
Aegis Trust: a British based NGO working
for the prevention of genocide and mass
atrocities worldwide, has been present in
Rwanda since 2001.

26. Testing DR with Video Testimonies from Rescuers
Working with Aegis Trust and Kings Collage Social Scientists colleagues to identify category of observations and
multimedia material: ‘Ubumuntu’ means ‘humanity’–Referring to the quality of being humane; greatness of heart,
benevolence, generosity and kindness.
https://www.youtube.com/watch?v=tMqFVF8lYEE&t=813s

27. Understanding (awareness and commitment to act)
Very low “S/he made a big commitment” = Very Low Understanding
Low “I can do good as he/she did” = Low Understanding
Medium “This shows how doing good is always possible” =Medium Understanding
High “I can be one of them” = High Understanding
Very high “I too can practice peace by helping others” = Very High Understanding
Personal connection (empathy)
Very low “It looks like s/he was doing a good thing” – Very Low Personal Connection
Low “That must have been very difficult” – Low Personal Connection
Medium “I know the feeling s/he is describing” – Medium Personal Connection
High “Listening to her/him makes me emotional” – High Personal Connection
Very high “What s/he is saying makes me proud” – Very High Personal Connection

28. Testing DR with Video Testimonies from Rescuers
Translations, recruitment and technical testing.
Access the TEST DRIVE at: https://democraticreflection.org/event/video/-L_O_AsF7Fykq2iXWdK2/-L_O_AsF7Fykq2iXWdK2

29. Feedback from Aegis Trust
• “What I liked is, the tool wasn't about commenting on the video as such, it was touching
on how people feel, and how people think in the place of the storyteller..."
• “The tool has that ability of giving you a chance to express yourself, to feel connected,
and connect with what you are watching, but also evaluate…It gave people a time for
self-reflection.”
• “When we engage people, when we call people for peace education training, we want
the change to start with individuals. When the change starts with individuals, then it
escalates to become a change for the group, and you can only achieve that if individually
people are connected, and people are really reflecting on their role in society. That is
something Democratic Reflection was really promoting. People had to respond not from
their mind, I would say, but from their heart.”
2019 trial interview with Head of Digital Resources and Communications at Aegis Trust

31. Impact 3. Evidence Hub - Crowdsourcing community capabilities
• For professional communities to pool resources,
expertise and crowdsource solutions to problems.
• Since 2014, worling with the UK's professional body
for health visitors, the Institute of Health Visiting
(iHV), to create and manage The Evidence Hub.
• The online platform allows iHV members to share,
reflect on and develop their professional practice
based on the latest scientific evidence and their
collective experience in the field.
• The network has created opportunities to collate
evidence and present it to public health
policymakers.

32. A Knowledge Tree to connect evidence to findings
Evidence
In Favor
Evidence
Against

33. Social Analytics

34. Evidence Hub Website: http://evidence-hub.net/

35. Users Engagement
• 14 Hubs communities
• Main Sectors: Healthcare and Education but also, sustainable development
and academic research networks
• Contributors from 3-10 small community groups to larger 100-900 users
community for bigger Hubs - from grassroots local initiatives to higher
national and international policy.
• Form hundreds to thousands user generated content.

36. NETLAB
Evidence Hub
¡ A tool for SDG communities of practice to gather
and debate evidence for ideas and solutions to
specific community issues.
¡ By aggregating and connecting single users’
contributions the Evidence Hub provides a collective
picture of what is the SDG challenges, debates,
projects and organizations in the Milan Metropolitan
Area
¡ This collective picture can be explored in form of a
simple textual interface (as list of organizations,
claims, evidences, issues, resources) or in a more visual
way as a knowledge tree, or a network map

37. Impact 4. LiteMap: Collaborative Web Annotation and Knowledge Mapping
for collaborative online learning
Since its first launch in 2015, has been used
• By over 2000 users
• in 10 different countries,
• Over 100 community groups
• 560 Maps to confirm an emerging public and
education impact.
• a Brazilian community of 1300 teachers carry
out collaborative work and coordinate online
course activities with, LiteMap improve
collaborative online learning and collective
inquiries.
http://litemap.net/

39. Crowdsourcing community capabilities in Brazil
• Brazilian open research community, COLEARN, facilitated by my Open
University colleague and open learning expert Dr Ale Okada, has also used
our cloud-based LiteMap tool to build collaboration between school
teachers and university researchers, remotely and at scale, in a way rarely
seen in the massive and geographically diverse country.
• The network has employed the tool's debate visualisation, collaborative
knowledge mapping, and argumentation technology to support thousands
of teachers and researchers to develop coordinated online learning
opportunities for school pupils on diverse themes from genetically
modified foods to Zika virus response.

40. LiteMap Strengths
• LiteMap main strength is the capability to provide user generated visual mapping of a
debate combined with Web annotation, that is the assimilation of information from any
web source.
• Litemap provides an instrument for reflecting across diverse forums and web data.
• By combining reflection with visual mapping LiteMap enables reflections to be
connected, commented and linked in a meta discussion space (the 2D canvas map) as
further element of co-creation and analysis.
• The functionality of hyperlinking external knowledge to the argument map was
considered highly valuable to see all the discussion about a given argument; and to
make arguments clearer.
• The capability of Web annotation to enable the collaborative harvesting of evidence
from external sources, is also an effective response to the lack of diversity and
shallowness of online conversations.
• Civic leaders, and communities alike, can use Web annotations to better ground
arguments, support counterarguments, while enabling provenance of ideas across
platforms and communities.
• Web annotation also naturally mitigates data duplication and plagiarism (since people
can link directly to the external source without the need to copy content).

41. LESSONS LEARNED FROM CONTESTED COLLECTIVE INTELLIGENCE
APPLICATIONS
• If we want to support people’s capability to question assumptions and think
critically, we need to design spaces for personal reflection and sensemaking.
Information overload and balkanization makes it difficult for people
to extract useful and reliable insights to make better-informed
decisions, to change biased assumptions or behaviours.
Information complexity and uncertainty (fake vs legitimate news)
contribute to citizens’ alienating from a healthy engagement in public
life.
We miss the right social context and tools to practice critical thinking

42. “the best remedy for propaganda and misinformation... is
helping people engage in critical thinking and evidence-
based reasoning.” – Ricky Sethi
Citation credit - Juho Kim
https://theconversation.com/how-citizen-investigators-can-collaborate-on-crowdsourced-fact-checking-76890
CRITICAL THINKING AS A CRITICAL DEMOCRATIC SKILL

43. Using technology to unite us, not divide us
• Powerful analytical tool are often used as persuasive tools but the same tools
can be used for improving civic engagement and learning
• Users profiling is more and more used by big corporations to target people but it
can be also used by government to provide better services and to design
effective civic learning experiences.
It is easy to scapegoat technology for society's ills, but we should never forget its
extraordinary potential to unite us.
How to we design for this second class of applications? How we create new tools
to do address society’s hardest challenges will define us.
Read the full impact article at: https://www.open.ac.uk/research/impact/using-technology-unite-us-not-divide-us

44. Thank you for listening!
Please fell free to contact me at anna.deliddo@open.ac.uk
to know more about our work please visit the research group website at:
idea.kmi.open.ac.uk