Connecting Levels and Methods of Analysis in Networked Learning Communities


Published on

Published in: Education, Technology
  • Be the first to comment

Connecting Levels and Methods of Analysis in Networked Learning Communities

  1. 1. Workshop onConnecting Levels and Methods of Analysis in Networked Communities Learning Analytics and Knowledge Conference 2012 Vancouver Version edited for SlideShare Dan Suthers, Ulrich Hoppe Maarten de Laat, Simon Buckingham-Shum
  2. 2. Motivations Multiple levels of learning agency in social settings  Individual - social setting as stimulus  Small Group - “maintaining a joint conception of a problem”; “group cognition”  Community - “knowledge building”  Networked - “networked individualism” Learners (could) participate in multiple simultaneous forms of learning in contextual and constitutive relationships Analytic Challenges  Ontology mismatches between data record and desired level of analysis  Summary representations needed to see emergent patterns but fail to capture how learning is actually accomplished  Distributed nature of (interaction in) socio-technical networks  And many others …
  3. 3. Questions: Networked LearningHow does learning take place through the interplay between individual and collective agency? How does local (individual and small group) activity aggregate to  Create resources that are then available network-wide for others individual and small group learning?  Drive advances in community knowledge building? (How) does the connectivity afforded by ICTs facilitate learners’ participation at multiple levels? What theoretical perspectives are relevant to bridging levels of analysis?  Do the different levels of analysis need different theories, and how can they be articulated?  Are there theories and methods that bridge the levels of analysis?
  4. 4. Questions: MethodHow can a plurality of methods help us make sense of levels of learning and their interplay? How can aggregate levels of analysis inform where to “dive in” for local analysis, for example to  make sense of results at the aggregate level?  find local sources of innovation? How can local levels of analysis  generate hypotheses to be tested at the network level?  identify events to be traced to other times and places? What practical techniques such as different types of triangulation or visualization can help to connect different levels and approaches of analysis? What are the prospects of technical integration of analysis tools through open architecture, adequate representations and user interface metaphors?
  5. 5. Questions: Multilevel Analysis inContext How can analyses be made available to participants of all types to support their awareness and reflection on learning processes? Dealing with the complexity of living practices in which people learn … and the potential added value of learning analytics to  raise awareness,  help reflect on (social) learning behavior and  connect learners in networks and communities where value is being created How should progress on these issues and appropriate applications be promoted in the context of the emerging Learning Analytics community?
  6. 6. Condensed History & Future ofWorkshopsOrigins in workshops by Koschmann, Suthers, StahlProductive Multivocality Series ICLS 2008: "A Common Framework for CSCL Interaction Analysis” CSCL 2009: "Common Objects for Productive Multivocality in Analysis” ARV 2009: "Pinpointing Pivotal Moments in Collaboration” ICLS 2010: “Productive Multivocality in the Analysis of Collaborative Learning” ARV 2011: ibidLevels of Analysis Series CSCL 2011: “Connecting Levels of Learning in Networked Communities” LAK 2012: “Connecting Levels and Methods of Analysis in Networked Learning Communities” ICLS 2012 (July): “Analyzing Collaborative Learning at Multiple Levels”
  7. 7. Today’s Workshop Activities Framing presentations Extended examples of concepts and tools applied to two data corpora Briefer presentations on other approaches Small group and full group discussion
  8. 8. Schedule09:00-10:30: Introductory Session 10m: Introduction to the workshop - Dan Suthers 20m: Introductions to each other 30m: Conceptual Framing Presentations - Organizers 20m: Visualizing Informal Networked Learning Activities (Issues and Tool Proposals) - Bieke Schreurs, Chris Teplovs & Maarten de Laat10:30-11:00: Coffee/Tea11:10-12:00: Part II: Issues & Tools 45m: Conceptual and Computational Tools (with examples from Wikiversity) - Ulrich Hoppe 05m: Form small groups to discuss topics after the break. (Suggested topics: Your examples of analytic questions; Key theoretical issues; Needs for computational support)12:00-13:00: Lunch (on site)13:00-14:30: Part III: Thinking about Levels 20m: Small Group Discussion: What are the key theoretical issues and needs for computational support? 25m: Small Groups report 15m: Multimodality in Levels of Analysis - Sharon Oviatt 20m: Multi-level Microanalysis - Alyssa Wise 15m: Full group open discussion14:30-15:00 Coffee/Tea15:00-17:00 Part IV: Outlook 45m: An Analytic Hierarchy (with examples from Tapped In) - Dan Suthers 45m: Discussion
  9. 9. IntroductionsOrganizers Dan Suthers, University of Hawai‘i Maarten de Laat, Open Universiteit Nederland Simon Buckingham-Shum, Open University UK Ulrich Hoppe, University of Duisburg-EssenOther Presenters Alyssa Wise, Simon Fraser University Bieke Schreurs, Open Universiteit Nederland Chris Teplovs, University of Windsor Sharon Oviatt, Incaa Designs
  10. 10. … and you Liaqat Ali  Shanta Rohse Bill Anderson  Toshiyuki Takeda Michael Atkisson  Ravi Vatrapu George Bradford  Sue Whale Al Byers  John Whitmer Darren Cambridge  Phil Winne Cathleen Galas  Robert Yerex Rabbi Zidnii Ilman  Gary Williams Murray Logan  Vladimir Stoyak Piotr Mitros  Caitlin Martin
  11. 11. Framing Comments Dan Suthers (with comments from others)
  12. 12. Learning in Socio-Technical NetworksHow do social settings foster learning?Agency EpistemologiesWho or what is the agent What is the process of that learns? learning? Individual  Acquisition Small groups  Intersubjective Networks (communities, meaning-making cultures, societies)  ParticipatoryThe correspondence is not strict, and analysis can be applied at local or network levels Based on  Suthers (ijCSCL 2006)
  13. 13. Levels of Agency and Epistemologies Individual Epistemologies Learning as acquisition of information, knowledge or skills  Local: contribution theory, given/new contract, explanation, conceptual change, role practice, etc.  Network: weak ties, diffusion theories (contagion theory, diffusion of innovations) Intersubjective epistemologies Learning as intersubjective meaning-making  Local: argumentation, co-construction, group cognition  Network: Knowledge building, communities of scientists Participatory epistemologies (most bridge both levels) Learning as changes in social participation and identity  Local: apprenticeship, mentoring ...  Network: apprenticeship as LPP, CoP
  14. 14. Let’s not get stuck at one level!Claim: individuals participate in the foregoing forms of learning simultaneously This leads to a fundamental question: How does learning take place through the interplay between individual and collective agency in socio-technical networks? Requires coordinated multi-level analysis Requires coordinated multi-level theorizing (I address analysis and theory after some examples)
  15. 15. Examples of Connecting LevelsQuestions Ive got a huge amount of data. Where to start?  Where are productive interactions that I should look at?  Who is playing important roles? I found some interesting patterns of participation and knowledge building in a network: how did these come about?  What are participants actually doing?  Can I account for network-level phenomena in terms of how persons follow STN affordances? I analyzed an interesting and productive session. Does it have any significance beyond the session?  Are ideas taken up by others who express them elsewhere?  Do personal encounters led to new participation elsewhere? Can I account for individual learning in terms of their participation in social phenomena?
  16. 16. Analytic Challenges Logs may record activity in the wrong ontology for analysis (e.g., media-level events rather than interaction or ties) Activity is distributed across time, sites, media  Traces of activity may be fragmented across multiple logs, breaking up participants’ singular experience  Thus, distributed activity may be analytically cloaked People draw on the resources of the setting in diverse ways  Referencing and modifying available media  Echoing notational and semantic elements  Sensitive to temporal and spatial setting The phenomenological situation is extended  The “situation” for participants need not be limited to our selection of a transcript and may be non-local in time and space
  17. 17. Comments on TheoriesTheories that make contributions, but are limited in scope: Socio-constructivism  All knowledge is constructed by the individual (possibly in social interaction)  Psychological; limited at social or more aggregate levels of analysis Distributed Cognition  Transformation of representations in an STN effects a “cognitive” computation  “Cognition” at multiple network levels?  Better for functional explanations than generativity Communities of Practice  Engagement, alignment, imagination; local/global duality  Limited to specific kinds of communities (shared domain, interacting to sustain a shared practice)
  18. 18. More Comments on TheoriesPromising: CHAT (Activity Theory) QuickTimeª and a decompressor are needed to see this picture.  Vygotsky, Leontiv, ... Cole & Engestrom (1993)  Intended to encompass all human activity  Explicit consideration of how artifacts bridge levels Actor-Network Theory Latour (2005):  Localizing the Global  Redistributing the Local (Action is Overtaken, Connecting Sites)  Artifacts (“actants”) again key for constructing the network  Critiqued for devaluing human agency  Methodological strategy that prioritizes the data
  19. 19. Incoherence due toIncommensurability?Methodological determinism: “Methods rest on philosophical presuppositions. These remain embedded in them, even if they are not taught or discussed or attended to explicitly.” (Yanow & Schwartz-Shea, 2006) In what ways (if any) do we risk incoherence by mixing methods from different theoretical traditions?  There are potentially intractable differences between objectives and between explanatory theories  Are these forced on us by methods?  How? What “carries” the presupposition?  Do researchers have agency in overcoming this?
  20. 20. Example: Social Network AnalysisMarin & Wellman (2010): claim that social network analysis is not just a method; it is a perspective “Causation / explanatory power is located in relations (social structure), not attributes” Intrinsic to the analysis method?  One could use network analysis under other theoretical commitments  Example: homophily of attributes due to selection, not diffusion We have some agency in applying the method But the network representation implies commitments  Actors or social entities can be discretely identified  Binary relationships are relevant and primary
  21. 21. Example: The Percentage or Ratio Schegloff (1993): “Reflections on Quantification in the Study of Conversation”  comments on a study of sociability that used “laughter per minute” and “backchannels per minute” as measures Numerator Laughs Opportunities taken Denominator Minute Opportunities available  Denominator: Laughter is responsive, relevant at some points and not others  Numerator and denominator are mutually constitutive Ratio commits one to quantify events relative to some set of potential events, but this can be refined One analytic tradition can inform (fill in the blind spots) of another
  22. 22. Unpacking Methodological Determinism Beyond “guilt by association”  Often methods are taught within a setting with a viewpoint  But a user of a method does not necessarily adopt the entire tradition that the method comes from Methods consist of notations and practices  Notations (inscriptions) and available operations carry largely inescapable biases  Suthers (2008) Knowledge Cartography  The associated practices by which inscriptions become representations (Medina et al, 2009) also carry biases  Biases of practices can be confronted and modified in reflective practice Reflective use of methods requires stepping out of the viewpoint provided by the method  A goal of the Productive Multivocality project  Suthers et (16) al., (CSCL 2011) and book in progress
  23. 23. Strategies for Productive Multivocality Shared data repositories, standards, metadata Dialogue about same data  Agree on what data is worth considering? Have shared analytic objective (e.g., pivotal moments)  Interpretable by each tradition (objective as boundary object)  Probably vague at first (projective stimulus) Pair up diverse methods  Different individuals representing those methods! Push methods outside of their comfort zone  Potential problems in appropriateness of data Align data and analytic representations Iteration is required.  Eliminate inconsequential differences in the first pass  Repeat to focus on more essential differences and convergences (conceptual, epistemological)
  24. 24. Simon Says …We know that learning is a multifaceted, extraordinarily complex phenomenon, certainly happening in the individuals mind, but shaped by interaction with the others and the environment. So it makes sense to build as rich a picture as possible of what is going on at many levels in a learning ecosystem, prior to, during and after critical incidents.The intriguing promise of learning analytics goes beyond detecting patterns more efficiently than has been possible, as more digital data becomes available (= simply doing what weve always done, but faster/better).More radically, could the learning sciences be entering a new era (following genomics and other big data fields), in which there is more data than we know what to do with, and far greater complexity in the relationships that may be embedded in it?This requires the deployment of exploratory techniques seeking robust patterns for which there may be no motivating hypothesis, or explanatory theory. New patterns between different levels in the ecosystem might then become the catalysts for revising theory to find plausible explanations.This is not Chris Andersons death of theory, but analytics as scientifically valid probes that report back to us as analysts, from the deep space that big data opens up.
  25. 25. Discussion Topics Examples of ways in which you have connected or would like to connect levels of analysis? What analytic methods or tools are needed at different levels (e.g., content analysis, sequential/process analysis, SNA) and how do we coordinate them? What approaches would you take to theoretical articulation or integration? Dan Suthers