Suthers & Rosen, Learning Analytics and Knowledge 2011

1,453 views

Published on

Presentation of Suthers, D. D., & Rosen, D. (2011). A unified framework for multi-level analysis of distributed learning Proceedings of the First International Conference on Learning Analytics & Knowledge, Banff, Alberta, February 27-March 1, 2011.
Abstract: Learning and knowledge creation is often distributed across multiple media and sites in networked environments. Traces of such activity may be fragmented across multiple logs and may not match analytic needs. As a result, the coherence of distributed interaction and emergent phenomena are analytically cloaked. Understanding distributed learning and knowledge creation requires multi-level analysis of the situated accomplishments of individuals and small groups and of how this local activity gives rise to larger phenomena in a network. We have developed an abstract transcript representation that provides a unified analytic artifact of distributed activity, and an analytic hierarchy that supports multiple levels of analysis. Log files are abstracted to directed graphs that record observed relationships (contingencies) between events, which may be interpreted as evidence of interaction and other influences between actors. Contingency graphs are further abstracted to twomode directed graphs that record how associations between actors are mediated by digital artifacts and summarize sequential patterns of interaction. Transitive closure of these associograms yields sociograms, to which existing network analytic techniques may be applied, yielding aggregate results that can then be interpreted by reference to the other levels of analysis. We discuss how the analytic hierarchy bridges between levels of analysis and theory.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,453
On SlideShare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
24
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Suthers & Rosen, Learning Analytics and Knowledge 2011

  1. 1. A Unified Framework for Multi-Level Analysis of Distributed Learning Dan Suthers Department of Information and Computer Sciences and Communication and Information Sciences Program Devan Rosen Department of Speech University of Hawaii Funded by NSF VOSS
  2. 2. Learning in Social SettingsMultiple theories of how learning takes place in social settings ▪ From social as stimulus to social entity as learning agent ▪ From "networked individualism" to "maintaining a joint conception of a problem" ▪ From "diffusion of innovations" to "knowledge building"All involve uptake: when an actor takes (a trace of) anotheractors activity as being relevant in some way for his or hercurrent activity ▪ See Suthers (ijCSCL 2006) for discussion of learning epistemologies; and Suthers et al. (ijCSCL 2010) for uptakeUptake is evidenced by how individual actions are observablycontingent on the actions of others in their socio-technicalnetwork contexts
  3. 3. Analytic ChallengesFundamental question: how learning takes place through theinterplay between individual and collective agency ▪ Situated accomplishments of individuals and small groups ▪ How these local accomplishments give rise to larger phenomena in networksRequires coordinated multi-level analysisActivity can be distributed across multiple media and sites ▪ Traces of activity may be fragmented across multiple logs ▪ Logs may record activity in the wrong ontology for analysis (e.g., media-level events rather than interaction or ties)Distributed activity may be analytically cloaked ✓ Abstract transcript representation that collects relevant events into a single analytic artifact ✓ Analytic hierarchy that supports multiple levels of analysis
  4. 4. Sociogram (Tie Model) P2 P1 P3Associograms Uptake Graph (Interaction Model)(Mediation Model) m2 w1 w2 P2 P1 m3 m4 w3 P3 w4 m1Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3Events w4(Event Model) Threading m4 aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..." Logs aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..." aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..." ... xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." ... xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..." xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..." xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."
  5. 5. aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..." aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..." aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..." ... Logs(Process xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." Trace) ... xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..." xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..." xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."
  6. 6. Entity-Relation Types (Domain Model) Discussions Containment Actors Write, messages Read Threading aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..." aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..." aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..." ... Logs(Process xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." Trace) ... xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..." xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..." xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."
  7. 7. Entity-Relations (Domain Model) w1: P2 writes m1 @27/08/2007 16:03:38 Discussion 1 Containment m1 P2 WritesEvents(Event Model) aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..." aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..." aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..." ... Logs(Process xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." Trace) ... xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..." xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..." xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."
  8. 8. Entity-Relations (Domain Model) w1: P2 writes m1 @27/08/2007 16:03:38 Discussion 1 r1: P1 reads w1 @27/08/2007 22:09:47 Containment P1 m1 Reads P2Events(Event Model) aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..." aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..." aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..." ... Logs(Process xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." Trace) ... xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..." xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..." xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."
  9. 9. Entity-Relations (Domain Model) w1 Discussion 1 r1 w2: P1 writes m2 @27/08/2007 22:18:22 Containment P1 m1 P2 m2 WritesEvents(Event Model) Threading aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..." aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..." aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..." ... Logs(Process xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." Trace) ... xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..." xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..." xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."
  10. 10. Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3Events w4(Event Model) Threading m4 aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..." aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..." aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..." ... Logs(Process xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." Trace) ... xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..." xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..." xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."
  11. 11. Event Model as Abstract Transcript ▪ Log files no longer needed (except as we discover new information needs) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 Events w4 (Event Model) Threading m4▪ Straightforward extension to include events from other media▪ Sequence of events serves as unified transcript of distributed events
  12. 12. Contingency Graphs ▪ But what about interaction? Entity-Relations (Domain Model) w1 ? ? Discussion 1 r1 w2 ? ? r2 w3 ? Containment P1 r1 m1 ? r2 P2 ? m2 r3 ? m3 P3 Events w4 (Event Model) Threading m4 ▪ Identify empirical relationships between events that collectively evidence uptake ▪ Called contingencies after Garfinkels "contingently achieved accomplishments": how actors draw on the evolving context
  13. 13. Contingencies: Media Dependency ▪ To reply to a message, it must first be written Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 Events w4 (Event Model) Threading m4
  14. 14. Contingencies: Media Dependency ▪ To read a message, it must first be written Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 Events w4 (Event Model) Threading m4
  15. 15. Contingencies: Temporal Proximity & Same Actor ▪ Continuity of agents activity Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 Events w4 (Event Model) Threading m4
  16. 16. Contingencies: Lexical or Semantic Overlap▪ For example, reuse of noun phrases Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 Events w4 (Event Model) Threading m4
  17. 17. Contingency Graph as Contextualized Action Model▪ Analytically relevant manifest relationships between actors actions and other events have been recorded ▪ Next: raise analytic level of description to latent relationships and higher order structures Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 w4 Threading m4
  18. 18. Uptake Graph (Interaction Model) ▪ Interpret collections or subgraphs of contingencies as corroborating evidence for uptake ▪ Supports sequential analysis of interaction w1 Uptake Graph w2 (Interaction Model) w3 w4 Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 w4 Threading m4
  19. 19. Uptake Graph (Interaction Model) ▪ Finds uptake not manifest in threading structure ▪ P3 plays an integrative role in this discussion ▪ Other examples in Suthers et al. (ijCSCL 2010) w1 Uptake Graph w2 (Interaction Model) w3 w4 Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 w4 Threading m4
  20. 20. Associograms● Directed affiliation network of actors and artifacts● Mediation Model: How actors associations are mediated P2 m1 Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 w4 Threading m4
  21. 21. Associograms(Mediation Model) P2 P1 m1Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 w4 Threading m4
  22. 22. Associograms(Mediation Model) m2 P2 P1 m1Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 w4 Threading m4
  23. 23. Associograms(Mediation Model) m2 P2 P1 m3 m1Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 w4 Threading m4
  24. 24. Associograms(Mediation Model) m2 P2 P1 m3 m4 P3 m1Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3 w4 Threading m4
  25. 25. Finding Interaction Patterns: Round Trip m2 Associograms (Mediation Model) P2 m3 m4 P1 P3 m1 w1 r1 w2 r2 w3 r1 r2 r3Contingency Graph(Contextual Action Model) w4
  26. 26. Recovering Temporal Information m2 Associograms (Mediation Model) P2 m3 m4 P1 P3 m1 w1 r1 w2 r2 w3 r1 r2 r3Contingency Graph(Contextual Action Model) w4
  27. 27. Relationships ▪ Patterns of mediated associations reveal relationships Associograms Pairwise Associations (Relationship Model) (Mediation Model) m2 m2 P2 P1 P2 Dialogue P1 m3 m4 P3 m1 m1
  28. 28. Relationships ▪ Patterns of mediated associations reveal relationships Associograms Pairwise Associations (Relationship Model) (Mediation Model) m2 m3 m2 P2 P1 P2 P1 P2 Producer/ P3 m3 m4 Dialogue Consumer m1 m1 P3 m1
  29. 29. Relationships ▪ Patterns of mediated associations reveal relationships Associograms Pairwise Associations (Relationship Model) (Mediation Model) m2 m3 m2 P2 P1 P2 P1 P2 Producer/ P3 m3 m4 Dialogue Consumer m1 m1 P3 M1 M1 m1 P1 Shared P2 Shared P1 P2 Interest Authorship M2 M2
  30. 30. Multi-Media Associations ▪ Characterize pairwise relationships in terms of distribution across media ▪ Compare roles of various media in supporting associations (Suthers & Chu, Networked Learning 2010) w1 m2 P2 P1 m3 m4 P3 m1 w2
  31. 31. Ties ▪ Straightforward to collapse into sociogram by transitive closure or similar computations P2 P3 Associograms Pairwise Associations (Relationship Model) (Mediation Model) m2 m3 m2 P2 P1 P2 P1 P2 Producer/ P3 Dialogue m3 m4 Consumer m1 m1 P3 m1
  32. 32. Ties ▪ SNA methods can now be applied Sociogram (Tie Model) P2 P1 P3 Associograms Pairwise Associations (Relationship Model) (Mediation Model) m2 m3 m2 P2 P1 P2 P1 P2 Producer/ P3 Dialogue m3 m4 Consumer m1 m1 P3 m1
  33. 33. Prior ResearchContingency graphs used for ... ▪ Microanalysis of process through which learners achieved an insight ▪ Semi-automated analyses of graph manipulations to find pivotal moments
  34. 34. Current ResearchTapped In (SRI International) ▪ Network of educators: professional development and peer support (longest running educational online community) ▪ 8 years of data; focusing on 2 year peak ▪ 20K educators, 8K user-created spaces; 50 tenant organizations ▪ Chats, threaded discussions, wikis, resource sharing ...Current Focus ▪ Identifying where significant activity takes place and characterizing the nature of that activity (talk tomorrow afternoon) ▪ Nonlocal consequences of local activities, e.g., trace contingencies to find whether actors move ideas and other actors to new settings
  35. 35. AdvantagesAs a data representation ▪ Integration of distributed data: uncloak distributed interaction ▪ Common format for reuse of algorithmsAs an analytic framework ▪ Multi-Level Multi-Theoretical analysis possible ▪ Multiple ontologies allow for mapping between interaction, mediated affiliation and tie levels of analysisWorkshop: Connecting Levels of Learning in NetworkedCommunities ▪ July 5th @ CSCL in Hong Kong ▪ http://www.isls.org/cscl2011/ or http://engaged.hnlc.org/
  36. 36. Sociogram (Tie Model)Dan Suthers P2 P1suthers@hawaii.edu P3Associograms Uptake Graph (Interaction Model)(Mediation Model) m2 w1 w2 P2 P1 m3 m4 w3 P3 w4 m1Contingency Graph (Contextual Action Model) Entity-Relations (Domain Model) w1 Discussion 1 r1 w2 r2 w3 Containment P1 r1 m1 r2 P2 m2 r3 m3 P3Events w4(Event Model) Threading m4 aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..." Logs aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..." aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..." ... xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." ... xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..." xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..." xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."

×