Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Anna De Liddo
Knowledge Media Institute, The Open University, UK
Simon Buckingham Shum
University of Technology Sydney, Au...
Wh
Model of Collective Intelligence (CI):
from sensing the environment, to interpreting it, to generating good
options, to ta...
Model of Collective Intelligence (CI):
from sensing the environment, to interpreting it, to generating good
options, to ta...
When tackling complex and contested problems:
v there may not be one worldview, or clear option
v evidence can be ambigu...
In the design space of CI systems
u  where there is insufficient data to confidently compute an
answer,
u  when there is...
(Social,	
  Visual	
  and	
  Argumentation-­‐based	
  CI)
Collective	
  Intelligence	
  Online	
  Deliberation	
  
Human	
...
¡  Poor Debate: No tools to identify were
ideas contrast, where people disagree
and why... popularity vs critical thinking
Flat listing of posts and no insight into the logical
structure of ideas and arguments the,	
  
such	
  as	
  the	
  coher...
These tools are increasingly used to support online debate and facilitate citizens’
engagement in policy and decision-maki...
 
	
  
	
  
LINK	
  to	
  QUORA:	
  	
  
	
  
http://www.quora.com/Physics/Do-­‐
wormholes-­‐always-­‐have-­‐black-­‐holes...
¡  Poor Debate: No tools to identify were ideas
contrast, where people disagree and why
¡  Poor idea evaluation: No mech...
That make the structure and status of a dialogue or
debate visible
Coming from research on Argumentation and CSAV,
these t...
!
!
•  referred to as “deliberative aggregators” (van Gelder 2012b)
•  produce a collective viewpoint or judgment on complex s...
Collaborative
Knowledge
Production
Collaborative Web
Annotation and
Knowledge mapping
Social Network Analysis
and Visualiz...
http://litemap.open.ac.uk
Flat listing of posts and no insight into the logical
structure of ideas and arguments ,	
  such	
  as	
  
the	
  coherenc...
Add rhetorical markers to distinguish post type
and role in the conversation	
  
agrees	
  with	
  
agrees	
  with	
  
disagrees	
  with	
  
Explore and Mark-up connections between
people and ideas, and ...
http://litemap.open.ac.uk
demo...
¡  used in a real user community to map the main issues, ideas
and arguments raised in a conversation hosted on the Utopi...
¡  The study lasted 6 weeks
¡  3 argument maps were created with LiteMap
and then embedded into the Utopia website.
¡  ...
¡  A newsletter was used to announce each new map to
online community.
¡  Over the period of the testing, an advert was ...
¡  Over the testing period, over 800 people navigated to the
argument maps.
¡  Most traffic was generated after the news...
¡  Understanding the argument mapping tasks, specifically
the challenges when collaboratively building a map in the
same ...
¡  Users reckon that a bottom-up approach to create an argument map
(from harvesting the arguments and then moving to ide...
“It was quite hard for us to agree what each clip could
mean, it is because the discussion in the Utopia website is
comple...
¡  the nature of the problem to map affected
the complexity of the argument mapping
process in a way that a collective
ma...
¡  If the issues are open ended and ill-defined
collaborative mapping can be trickier and even
the definition of issues a...
¡  advancement of personal knowledge of the topic,
¡  Acknowledge the power of structuring ideas for personal
reflection...
“The think I am not sure is how much this mapping task fits to the
Utopia’s online community because the discussion on Uto...
¡  Improving negotiation mechanisms
¡  Enabling different deliberation processes
for different issue types (open-ended v...
Thanks!
anna.deliddo@open.ac.uk
Twitter: Anna_De_Liddo
http://kmi.open.ac.uk/people/member/anna-de-liddo
Catalyst: http://...
¡  De Liddo, A., & Buckingham Shum, S. (2014). New Ways of Deliberating Online: An Empirical Comparison of
Network and Th...
De Liddo & Buckingham Shum ipp2014
De Liddo & Buckingham Shum ipp2014
De Liddo & Buckingham Shum ipp2014
De Liddo & Buckingham Shum ipp2014
De Liddo & Buckingham Shum ipp2014
De Liddo & Buckingham Shum ipp2014
De Liddo & Buckingham Shum ipp2014
De Liddo & Buckingham Shum ipp2014
De Liddo & Buckingham Shum ipp2014
De Liddo & Buckingham Shum ipp2014
Upcoming SlideShare
Loading in …5
×

De Liddo & Buckingham Shum ipp2014

842 views

Published on

Presentation to Internet Policy & Politics (IPP2014) Conference, Oxford, UK, Sept 2014

Published in: Technology
  • Be the first to comment

De Liddo & Buckingham Shum ipp2014

  1. 1. Anna De Liddo Knowledge Media Institute, The Open University, UK Simon Buckingham Shum University of Technology Sydney, Australia
  2. 2. Wh
  3. 3. Model of Collective Intelligence (CI): from sensing the environment, to interpreting it, to generating good options, to taking decisions and coordinating action... Collec&ve( Ac&on( Collec&ve( Decision( Collec&ve( Idea&on( Collec&ve( Sensemaking( Collec&ve( Sensing(( (
  4. 4. Model of Collective Intelligence (CI): from sensing the environment, to interpreting it, to generating good options, to taking decisions and coordinating action... Collec&ve( Ac&on( Collec&ve( Decision( Collec&ve( Idea&on( Collec&ve( Sensemaking( Collec&ve( Sensing(( (
  5. 5. When tackling complex and contested problems: v there may not be one worldview, or clear option v evidence can be ambiguous or of dubious reliability requiring the construction of plausible, possibly competing narratives; v growth in intelligence results from learning, which is socially constructed through different forms of discourse, such as dialogue and debate. Contested Collective Intelligence (De Liddo 2012)
  6. 6. In the design space of CI systems u  where there is insufficient data to confidently compute an answer, u  when there is ambiguity about the trustworthiness of environmental signals, u  and uncertainty about the impact of actions, u  And there is not one view that fits all …then a more powerful scaffolding for thinking and discourse is required, in order to support the emergence of CI around complex socio political dilemmas. Contested Collective Intelligence (De Liddo 2012)
  7. 7. (Social,  Visual  and  Argumentation-­‐based  CI) Collective  Intelligence  Online  Deliberation   Human  Dynamics  of  Engagements   Analytics, & Visualization Crowdsourcing ideas, arguments and facts Structured Discourse and Argumentation Democratic entitlements New class of Online Deliberation tools Citizen Voice Social Innovation Computational Services & Dialogic Agents
  8. 8. ¡  Poor Debate: No tools to identify were ideas contrast, where people disagree and why... popularity vs critical thinking
  9. 9. Flat listing of posts and no insight into the logical structure of ideas and arguments the,   such  as  the  coherence  or  evidential  basis  of  an  argument.    
  10. 10. These tools are increasingly used to support online debate and facilitate citizens’ engagement in policy and decision-making. These are fundamentally chronological views which offer: ¡  No support for idea refinement and improvement       LINK  to  PETITION:       http://www.change.org/en-­‐GB/petitions/ stand-­‐against-­‐russia-­‐s-­‐brutal-­‐crackdown-­‐ on-­‐gay-­‐rights-­‐urge-­‐winter-­‐ olympics-­‐2014-­‐sponsors-­‐to-­‐condemn-­‐ anti-­‐gay-­‐laws    
  11. 11.       LINK  to  QUORA:       http://www.quora.com/Physics/Do-­‐ wormholes-­‐always-­‐have-­‐black-­‐holes-­‐at-­‐ the-­‐beginning#answers  
  12. 12. ¡  Poor Debate: No tools to identify were ideas contrast, where people disagree and why ¡  Poor idea evaluation: No mechanisms to identify, contribute and discuss the evidence for an idea ¡  Poor Summarization and Visualization ¡  Shallow contributions and Cognitive clutters ¡  Platform Island & Balkanization This hampers both: ¡  quality of users’ participation and ¡  The quality of proposed ideas ¡  effective assessment of the state of the debate.
  13. 13. That make the structure and status of a dialogue or debate visible Coming from research on Argumentation and CSAV, these tools make visually explicit users’ lines of reasoning and (dis)agreements. ¡  Deliberatorium ¡  Debategraph ¡  Cohere ¡  CoPe_it! ¡  Problem&Proposals ¡  YourView ¡  The Evidence Hub
  14. 14. !
  15. 15. !
  16. 16. •  referred to as “deliberative aggregators” (van Gelder 2012b) •  produce a collective viewpoint or judgment on complex societal issues by crowdsourcing discourse •  inherent purpose to support large communities to tackle complex issues of public concern
  17. 17. Collaborative Knowledge Production Collaborative Web Annotation and Knowledge mapping Social Network Analysis and Visualization Structured Online Discussion and Argumentation Advanced Analytics for: Attention mediation & Deliberation diagnostic
  18. 18. http://litemap.open.ac.uk
  19. 19. Flat listing of posts and no insight into the logical structure of ideas and arguments ,  such  as   the  coherence  or  evidential  basis  of  an  argument.    
  20. 20. Add rhetorical markers to distinguish post type and role in the conversation  
  21. 21. agrees  with   agrees  with   disagrees  with   Explore and Mark-up connections between people and ideas, and give them a semantic  
  22. 22. http://litemap.open.ac.uk demo...
  23. 23. ¡  used in a real user community to map the main issues, ideas and arguments raised in a conversation hosted on the Utopia platform. ¡  Utopia is a large German online community on sustainable consumption and lifestyles, which uses a content management system (built on Synphony) with a space for online discussion to host online debates within the virtual community. ¡  Three community managers created summary argument maps to visualise different topics of sustainable living based on discussions and articles on the Utopia platform ¡  The resulting argument maps were then embedded in the Utopia website to display them to the wider online community.
  24. 24. ¡  The study lasted 6 weeks ¡  3 argument maps were created with LiteMap and then embedded into the Utopia website. ¡  Three mappers collaborated to the creation of each argument map. ¡  The mappers did not receive any training neither with the tool nor with the argumentation mapping process, and they were not familiar with the IBIS argumentation model.
  25. 25. ¡  A newsletter was used to announce each new map to online community. ¡  Over the period of the testing, an advert was placed in the sidebar of the Utopia homepage which pointed to the testing Website to maximize visibility and promote partecipation to the online discussion. ¡  The Webpage in which the maps were embedded contained an explanation on how to use the map, information on the user study, and a link to a survey for user to provide feedback on the usability and usefulness of the argument map to improve the sensemaking and summarization of the online debate.
  26. 26. ¡  Over the testing period, over 800 people navigated to the argument maps. ¡  Most traffic was generated after the newsletters were sent, after each one, around 130 visitors came to see the argument maps on the same day. ¡  57 users survey. Twice as many women as men participated in the survey.   ¡  Whereas the quantitative analysis of the surveys results is still under development, in the following we report initial insights from the qualitative analysis of three interviews with the argument mappers.
  27. 27. ¡  Understanding the argument mapping tasks, specifically the challenges when collaboratively building a map in the same virtual space (example questions: How did you build an understanding of the task at hand?, how would you summarize the main process you went through?) ¡  Understanding of the use of the tool (LiteMap) to carry on the summarization and arguments structuring task (example questions: How did you use the tool? How did you coordinate your activities? How did you reach agreement on the node labelling, connections and map layout?) ¡  Reflecting on how the tool could be improved by implementing new features which solve some of the encountered limitation.
  28. 28. ¡  Users reckon that a bottom-up approach to create an argument map (from harvesting the arguments and then moving to ideas and finally defining the issues) was not an effective process. “For the first topic we came from the lower end of all these clips and we basically had 30 clips in front of us and we had to cluster them. I.e. we had argument on the quality of organic food, arguments on the different work pays and condition in organic food shops, then arguments on strategies to attract people to buy organic etc... …many arguments each representing a different entry point to the conversation and all interlinked so it was hard for us to cluster them, make it into a logic and then fit them into the argument structure. If we just had the top down question to answer, then of course we could have just stick to adding the argument here or there and this would have been easy. But we had to find a good formulation to capture all the ideas that we wanted to put in there.”
  29. 29. “It was quite hard for us to agree what each clip could mean, it is because the discussion in the Utopia website is completely unstructured…you have a topic, and idea and all the argument packed as list of comments and there is no hierarchy between them. We had to create as mapper this hierarchy and then see how the higher elements related with the lowers and vice-versa. That was not very easy in practice, especially because is a very subjective task to do. Each of us would have done it in a different way and also the comment from some of the viewer was that they would have done it differently than us. That made it hard for any topic to map, and even harder for the ill-structured topics that were less defined.”
  30. 30. ¡  the nature of the problem to map affected the complexity of the argument mapping process in a way that a collective mapping procedure that may work for one issue can fail for a different one. ¡  Background knowledge and nature of the topic are notably key component to affect the effectiveness of collaborative mapping, as well as training with the tool and knowledge of the argumentation model.
  31. 31. ¡  If the issues are open ended and ill-defined collaborative mapping can be trickier and even the definition of issues as such would require a long negotiation effort between mappers. ¡  For these cases a bottom up approach to summarization seem to be even more counterproductive because it forces users to label and connect clips before the group has clarified the higher level structure of the map (the meta issues to map).
  32. 32. ¡  advancement of personal knowledge of the topic, ¡  Acknowledge the power of structuring ideas for personal reflection, ¡  and future use and topic exploration. “It was absolutely useful, especially for topics as sustainable food in which we had no common knowledge on the many aspects of the topic and in which there are no well established metrics and definitions to assess it, it is very good to see how you can cluster the issues around this topic. Also the aspects of organic food are now laid down and I can read them and then dig only on the arguments of the issues and ideas I am interested in.” “I think it is very useful to have the idea visualised and structured and see how they come together.”
  33. 33. “The think I am not sure is how much this mapping task fits to the Utopia’s online community because the discussion on Utopia are quite shallow, just one two opinions and no depth even to the level of a newspaper article. So doing argument maps based on people contributions is not only very subjective but also not very useful. I would find it more interesting if the arguments mapped were based on a more scientific level. In terms of the output form our point of view (the community managers) if we had more validated information behind the map the map would have more value as well.” ¡  online debates, even debates about complex socio technical issues, consist of often very shallow conversations. “An important added value that can be played by the community manager is the mapping of more validated information to support participants’ learning and reflection on new aspects of a given debate. Argument maps seems to be better suited to bring new evidence into the conversation rather than for mapping and summarising the existing online debate.”
  34. 34. ¡  Improving negotiation mechanisms ¡  Enabling different deliberation processes for different issue types (open-ended vs more focused problems) ¡  Building argument map to bring evidence to the debate and improve deliberation quality rather than summarization
  35. 35. Thanks! anna.deliddo@open.ac.uk Twitter: Anna_De_Liddo http://kmi.open.ac.uk/people/member/anna-de-liddo Catalyst: http://catalyst-fp7.eu/ EDV – Election Debate Visualizations: http://edv-project.net/ Follow us on Twitter @CATALYST_FP7 Watch the Demos on Youtube CATALYST FP7
  36. 36. ¡  De Liddo, A., & Buckingham Shum, S. (2014). New Ways of Deliberating Online: An Empirical Comparison of Network and Threaded Interfaces for Online Discussion. In E. Tambouris, A. Macintosh, & F. Bannister, Lecture Notes in Computer Science (Vol. 8654, pp. 90–101). Springer. ¡  De Liddo, A. (2014). “Enhancing Discussion Forum with Combined Argument and Social Netwrok Analytics. In Okada, A., Buckingham Shum, S. and Sherborne, T., (Eds) Knowledge Cartography. Springer. Second Edition. In press. ¡  De Liddo, A. and Buckingham Shum, S. (2013) The Evidence Hub: Harnessing the Collective Intelligence of Communities to Build Evidence-Based Knowledge, Workshop: Large Scale Ideation and Deliberation at 6th International Conference on Communities and Technologies, Munich, Germany ¡  De Liddo, A., Sándor, Á. and Buckingham Shum, S. (2012) Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study, Computer Supported Cooperative Work (CSCW) Journal : Volume 21, Issue 4 (2012), Page 417-448 ¡  Buckingham Shum, Simon (2008). Cohere: Towards Web 2.0 Argumentation. In: Proc. COMMA'08: 2nd International Conference on Computational Models of Argument, 28-30 May 2008, Toulouse, France. Available at:http://oro.open.ac.uk/10421/ ¡  De Liddo, Anna and Buckingham Shum, Simon (2010). Cohere: A prototype for contested collective intelligence. In: ACM Computer Supported Cooperative Work (CSCW 2010) - Workshop: Collective Intelligence In Organizations - Toward a Research Agenda, February 6-10, 2010, Savannah, Georgia, USA. Available at: http://oro.open.ac.uk/19554/ ¡  Buckingham Shum, Simon and De Liddo, Anna (2010). Collective intelligence for OER sustainability. In: OpenED2010: Seventh Annual Open Education Conference, 2-4 Nov 2010, Barcelona, Spain. Available at: http://oro.open.ac.uk/23352/ ¡  Buckingham Shum, Simon (2007). Hypermedia Discourse: Contesting networks of ideas and arguments. In: Priss, U.; Polovina, S. and Hill, R. eds. Conceptual Structures: Knowledge Architectures for Smart Applications. Berlin: Springer, pp. 29–44.

×