1. Workshop on
Connecting Levels and Methods
of Analysis in Networked
Communities
Learning Analytics and Knowledge Conference 2012
Vancouver
Version edited for SlideShare
Dan Suthers, Ulrich Hoppe
Maarten de Laat, Simon Buckingham-Shum

2. Motivations
 Multiple levels of learning agency in social settings
 Individual - social setting as stimulus
 Small Group - “maintaining a joint conception of a problem”;
“group cognition”
 Community - “knowledge building”
 Networked - “networked individualism”
 Learners (could) participate in multiple simultaneous
forms of learning in contextual and constitutive
relationships
 Analytic Challenges
 Ontology mismatches between data record and desired level
of analysis
 Summary representations needed to see emergent patterns
but fail to capture how learning is actually accomplished
 Distributed nature of (interaction in) socio-technical networks
 And many others …

3. Questions: Networked Learning
How does learning take place through the interplay
between individual and collective agency?
 How does local (individual and small group) activity
aggregate to
 Create resources that are then available network-wide for
others' individual and small group learning?
 Drive advances in community knowledge building?
 (How) does the connectivity afforded by ICTs facilitate
learners’ participation at multiple levels?
 What theoretical perspectives are relevant to bridging
levels of analysis?
 Do the different levels of analysis need different theories,
and how can they be articulated?
 Are there theories and methods that bridge the levels of
analysis?

4. Questions: Method
How can a plurality of methods help us make sense of
levels of learning and their interplay?
 How can aggregate levels of analysis inform where to
“dive in” for local analysis, for example to
 make sense of results at the aggregate level?
 find local sources of innovation?
 How can local levels of analysis
 generate hypotheses to be tested at the network level?
 identify events to be traced to other times and places?
 What practical techniques such as different types of
triangulation or visualization can help to connect
different levels and approaches of analysis?
 What are the prospects of technical integration of
analysis tools through open architecture, adequate
representations and user interface metaphors?

5. Questions: Multilevel Analysis in
Context
 How can analyses be made available to participants
of all types to support their awareness and reflection
on learning processes?
Dealing with the complexity of living practices in
which people learn …
and the potential added value of learning
analytics to
 raise awareness,
 help reflect on (social) learning behavior and
 connect learners in networks and communities where
value is being created
 How should progress on these issues and appropriate
applications be promoted in the context of the
emerging Learning Analytics community?

6. Condensed History & Future of
Workshops
Origins in workshops by Koschmann, Suthers, Stahl
Productive Multivocality Series
ICLS 2008: "A Common Framework for CSCL Interaction
Analysis”
CSCL 2009: "Common Objects for Productive Multivocality in
Analysis”
ARV 2009: "Pinpointing Pivotal Moments in Collaboration”
ICLS 2010: “Productive Multivocality in the Analysis of
Collaborative Learning”
ARV 2011: ibid
Levels of Analysis Series
CSCL 2011: “Connecting Levels of Learning in Networked
Communities”
LAK 2012: “Connecting Levels and Methods of Analysis in
Networked Learning Communities”
ICLS 2012 (July): “Analyzing Collaborative Learning at Multiple
Levels”

7. Today’s Workshop Activities
 Framing presentations
 Extended examples of concepts and
tools applied to two data corpora
 Briefer presentations on other
approaches
 Small group and full group discussion

8. Schedule
09:00-10:30: Introductory Session
10m: Introduction to the workshop - Dan Suthers
20m: Introductions to each other
30m: Conceptual Framing Presentations - Organizers
20m: Visualizing Informal Networked Learning Activities (Issues and Tool Proposals) - Bieke
Schreurs, Chris Teplovs & Maarten de Laat
10:30-11:00: Coffee/Tea
11:10-12:00: Part II: Issues & Tools
45m: Conceptual and Computational Tools (with examples from Wikiversity) - Ulrich Hoppe
05m: Form small groups to discuss topics after the break. (Suggested topics: Your
examples of analytic questions; Key theoretical issues; Needs for computational support)
12:00-13:00: Lunch (on site)
13:00-14:30: Part III: Thinking about Levels
20m: Small Group Discussion: What are the key theoretical issues and needs for
computational support?
25m: Small Groups report
15m: Multimodality in Levels of Analysis - Sharon Oviatt
20m: Multi-level Microanalysis - Alyssa Wise
15m: Full group open discussion
14:30-15:00 Coffee/Tea
15:00-17:00 Part IV: Outlook
45m: An Analytic Hierarchy (with examples from Tapped In) - Dan Suthers
45m: Discussion

9. Introductions
Organizers
 Dan Suthers, University of Hawai‘i
 Maarten de Laat, Open Universiteit Nederland
 Simon Buckingham-Shum, Open University UK
 Ulrich Hoppe, University of Duisburg-Essen
Other Presenters
 Alyssa Wise, Simon Fraser University
 Bieke Schreurs, Open Universiteit Nederland
 Chris Teplovs, University of Windsor
 Sharon Oviatt, Incaa Designs

10. … and you
 Liaqat Ali  Shanta Rohse
 Bill Anderson  Toshiyuki Takeda
 Michael Atkisson  Ravi Vatrapu
 George Bradford  Sue Whale
 Al Byers  John Whitmer
 Darren Cambridge  Phil Winne
 Cathleen Galas  Robert Yerex
 Rabbi Zidnii Ilman  Gary Williams
 Murray Logan  Vladimir Stoyak
 Piotr Mitros  Caitlin Martin

11. Framing Comments
Dan Suthers
(with comments
from others)

12. Learning in Socio-Technical Networks
How do social settings foster learning?
Agency Epistemologies
Who or what is the agent What is the process of
that learns? learning?
 Individual  Acquisition
 Small groups  Intersubjective
 Networks (communities, meaning-making
cultures, societies)  Participatory
The correspondence is not strict, and analysis
can be applied at local or network levels
Based on  Suthers (ijCSCL 2006)

13. Levels of Agency and Epistemologies
 Individual Epistemologies
Learning as acquisition of information, knowledge or skills
 Local: contribution theory, given/new contract, explanation,
conceptual change, role practice, etc.
 Network: weak ties, diffusion theories (contagion theory,
diffusion of innovations)
 Intersubjective epistemologies
Learning as intersubjective meaning-making
 Local: argumentation, co-construction, group cognition
 Network: Knowledge building, communities of scientists
 Participatory epistemologies (most bridge both levels)
Learning as changes in social participation and identity
 Local: apprenticeship, mentoring ...
 Network: apprenticeship as LPP, CoP

14. Let’s not get stuck at one level!
Claim: individuals participate in the foregoing
forms of learning simultaneously
 This leads to a fundamental question:
How does learning take place through the
interplay between individual and collective
agency in socio-technical networks?
 Requires coordinated multi-level analysis
 Requires coordinated multi-level theorizing
(I address analysis and theory after some examples)

15. Examples of Connecting Levels
Questions
 I've got a huge amount of data. Where to start?
 Where are productive interactions that I should look at?
 Who is playing important roles?
 I found some interesting patterns of participation and
knowledge building in a network: how did these come
about?
 What are participants actually doing?
 Can I account for network-level phenomena in terms of how
persons follow STN 'affordances'?
 I analyzed an interesting and productive session. Does
it have any significance beyond the session?
 Are ideas taken up by others who express them elsewhere?
 Do personal encounters led to new participation elsewhere?
 Can I account for individual learning in terms of their
participation in social phenomena?

16. Analytic Challenges
 Logs may record activity in the wrong ontology for
analysis (e.g., media-level events rather than
interaction or ties)
 Activity is distributed across time, sites, media
 Traces of activity may be fragmented across multiple logs,
breaking up participants’ singular experience
 Thus, distributed activity may be analytically cloaked
 People draw on the resources of the setting in
diverse ways
 Referencing and modifying available media
 Echoing notational and semantic elements
 Sensitive to temporal and spatial setting
 The phenomenological situation is extended
 The “situation” for participants need not be limited to our
selection of a transcript and may be non-local in time and
space

17. Comments on Theories
Theories that make contributions, but are limited in
scope:
 Socio-constructivism
 All knowledge is constructed by the individual (possibly in
social interaction)
 Psychological; limited at social or more aggregate levels of
analysis
 Distributed Cognition
 Transformation of representations in an STN effects a
“cognitive” computation
 “Cognition” at multiple network levels?
 Better for functional explanations than generativity
 Communities of Practice
 Engagement, alignment, imagination; local/global duality
 Limited to specific kinds of communities (shared domain,
interacting to sustain a shared practice)

18. More Comments on Theories
Promising:
 CHAT (Activity Theory)
QuickTimeª and a
decompressor
are needed to see this picture.
 Vygotsky, Leontiv,
... Cole & Engestrom (1993)
 Intended to encompass all human activity
 Explicit consideration of how artifacts bridge levels
 Actor-Network Theory Latour (2005):
 Localizing the Global
 Redistributing the Local
(Action is Overtaken,
Connecting Sites)
 Artifacts (“actants”) again key for constructing the network
 Critiqued for devaluing human agency
 Methodological strategy that prioritizes the data

19. Incoherence due to
Incommensurability?
Methodological determinism:
“Methods rest on philosophical presuppositions.
These remain embedded in them, even if they are
not taught or discussed or attended to explicitly.”
(Yanow & Schwartz-Shea, 2006)
 In what ways (if any) do we risk incoherence
by mixing methods from different theoretical
traditions?
 There are potentially intractable differences
between objectives and between explanatory
theories
 Are these forced on us by methods?
 How? What “carries” the presupposition?
 Do researchers have agency in overcoming this?

20. Example: Social Network Analysis
Marin & Wellman (2010): claim that social network
analysis is not just a method; it is a perspective
 “Causation / explanatory power is located in relations
(social structure), not attributes”
 Intrinsic to the analysis method?
 One could use network analysis
under other theoretical commitments
 Example: homophily of attributes due
to selection, not diffusion
 We have some agency in applying the method
 But the network representation implies commitments
 Actors or social entities can be discretely identified
 Binary relationships are relevant and primary

21. Example: The Percentage or Ratio
 Schegloff (1993): “Reflections on Quantification in the
Study of Conversation”
 comments on a study of sociability that used “laughter per
minute” and “backchannels per minute” as measures
Numerator Laughs Opportunities taken
Denominator Minute Opportunities available
 Denominator: Laughter is responsive, relevant at some points
and not others
 Numerator and denominator are mutually constitutive
 Ratio commits one to quantify events relative to
some set of potential events, but this can be refined
 One analytic tradition can inform (fill in the blind
spots) of another

22. Unpacking Methodological Determinism
 Beyond “guilt by association”
 Often methods are taught within a setting with a viewpoint
 But a user of a method does not necessarily adopt the
entire tradition that the method comes from
 Methods consist of notations and practices
 Notations (inscriptions) and available operations carry largely
inescapable biases  Suthers (2008) Knowledge Cartography
 The associated practices by which inscriptions become
representations (Medina et al, 2009) also carry biases
 Biases of practices can be confronted and modified in
reflective practice
 Reflective use of methods requires stepping out of
the viewpoint provided by the method
 A goal of the Productive Multivocality project
 Suthers et (16) al., (CSCL 2011) and book in progress

23. Strategies for Productive Multivocality
 Shared data repositories, standards, metadata
 Dialogue about same data
 Agree on what data is worth considering?
 Have shared analytic objective (e.g., pivotal moments)
 Interpretable by each tradition (objective as boundary object)
 Probably vague at first (projective stimulus)
 Pair up diverse methods
 Different individuals representing those methods!
 Push methods outside of their comfort zone
 Potential problems in appropriateness of data
 Align data and analytic representations
 Iteration is required.
 Eliminate inconsequential differences in the first pass
 Repeat to focus on more essential differences and
convergences (conceptual, epistemological)

24. Simon Says …
We know that learning is a multifaceted, extraordinarily complex
phenomenon, certainly happening in the individual's mind, but shaped by
interaction with the others and the environment. So it makes sense to
build as rich a picture as possible of what is going on at many levels in a
learning ecosystem, prior to, during and after critical incidents.
The intriguing promise of learning analytics goes beyond detecting patterns
more efficiently than has been possible, as more digital data becomes
available (= simply doing what we've always done, but faster/better).
More radically, could the learning sciences be entering a new era (following
genomics and other big data fields), in which there is more data than we
know what to do with, and far greater complexity in the relationships that
may be embedded in it?
This requires the deployment of exploratory techniques seeking robust
patterns for which there may be no motivating hypothesis, or explanatory
theory. New patterns between different levels in the ecosystem might
then become the catalysts for revising theory to find plausible
explanations.
This is not Chris Anderson's 'death of theory', but analytics as scientifically
valid probes that 'report back' to us as analysts, from the deep space that
big data opens up.

25. Discussion Topics
 Examples of ways in which you have
connected or would like to connect levels of
analysis?
 What analytic methods or tools are needed at
different levels (e.g., content analysis,
sequential/process analysis, SNA) and how do
we coordinate them?
 What approaches would you take to
theoretical articulation or integration?
Dan Suthers
Suthers@hawaii.edu
lilt.ics.hawaii.edu