Formation and Learning Analytics?

3,036 views

Published on

Formation by Design (FxD) Symposium
Georgetown University, Washington, June 30 – July 2, 2014
https://futures.georgetown.edu/formation

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,036
On SlideShare
0
From Embeds
0
Number of Embeds
126
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Formation and Learning Analytics?

  1. 1. Formation and Learning Analytics? Simon Buckingham Shum Knowledge Media Institute, The Open University Visiting Fellow, Grad. School of Education, University of Bristol From August: Connected Intelligence Centre, University of Technology Sydney http://simon.buckinghamshum.net http://twitter.com/sbskmi Formation by Design (FxD) Symposium Georgetown University, Washington, June 30 – July 2, 2014 https://futures.georgetown.edu/formation 1 Emergence   Learning  
  2. 2. It’s out of the labs and into products: every learning tool now has an “analytics dashboard” (a Google image search) 2
  3. 3. https://twitter.com/Wiswijzer2/status/414055472451575808 3
  4. 4. https://twitter.com/Wiswijzer2/status/414055472451575808 “Note:  check  the  huge   difference  between   knowing  and   measuring…”   4
  5. 5. 5 c.f.  “Data  is  the  new  oil” Scaleable   Precise   Quan?fiable   Reprocessable  
  6. 6. For Morozov, analytics is where technological solutionism hits education: 6 “This flight from thinking and the urge to replace human judgments with timeless truths produced by algorithms is the underlying driving force of solutionism.”
  7. 7. If something like this is the pedagogical architecture for (trans)formational learning… 7 Structuring Knowledge Performing Generating Learning Power Purposing
  8. 8. If something like this is the pedagogical architecture for (trans)formational learning… 8 Structuring Knowledge Performing Generating Learning Power Purposing …what roles can computational learning analytics play? The  worldview  clash   is  what  makes  this   fascina?ng!  
  9. 9. For discussion on how analytics shape educational worldviews 9http://people.kmi.open.ac.uk/sbs/2014/06/edmedia2014-keynote
  10. 10. thought  experiment:     could  we  imagine…   liminal  space   analy7cs?   10
  11. 11. liminal space = opportunity for transformation or withdrawal 11
  12. 12. Liminal space = opportunity for transformation, or withdrawal 12 So while we initially strive to make our students feel comfortable on campus, we then must help them balance their desire for security with the need to take risks and explore new ideas and possibilities. Rather than attempting to resolve the tension, a college should help students find their place on this precarious threshold, in the liminal space between the familiar and strange, the old and the new. Johansson & Felten, 2014
  13. 13. Liminal space = opportunity for transformation, or withdrawal 13 “It is when you have left the tried and true, but have not yet been able to replace it with anything else. [...] It is when you are between your old comfort zone and any possible new answer. If you are not trained in how to hold anxiety, how to live with ambiguity, how to entrust and wait, you will run... anything to flee this terrible cloud of unknowing.” Richard Rohr O.F.M. on the spirituality of liminal space
  14. 14. meaningful  behavioural  proxies?     some  clues…   14
  15. 15. interpersonal   engagement  and   social  capital   analy7cs   15
  16. 16. Social Network Analysis (SNAPP) 16 What’s going on in these discussion forums?
  17. 17. Social Network Analysis to gain insight into peer-peer and peer-mentor dynamics (SNAPP tool) http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation Bakharia A and Dawson S. (2011) SNAPP: a bird's-eye view of temporal participant interaction. Proceedings of the 1st International Conference on Learning Analytics and Knowledge. Banff, Alberta, Canada: ACM, 168-173.
  18. 18. Social Network Analysis (SNAPP) 18 http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation 2 learners connect otherwise separate clusters tutor only engaging with active students, ignoring disengaged ones on the edge
  19. 19. Visualizing and filtering online social ties: by topic and type of tie Schreurs B., Teplovs C., Ferguson R., De Laat M. and Buckingham Shum S. (2013). Visualizing Social Learning Ties by Type and Topic: Rationale and Concept Demonstrator. Proc. 3rd Int. Conf. on Learning Analytics & Knowledge. Leuven, BE: ACM, 33-37. Open Access Eprint: http://oro.open.ac.uk/36891
  20. 20. Visualizing and filtering online social ties: by topic and type of tie
  21. 21. Visualizing and filtering online social ties: by topic and type of tie
  22. 22. Visualizing and filtering online social ties: by topic and type of tie
  23. 23. Visualizing and filtering online social ties: by topic and type of tie
  24. 24. disposi7onal     learning  analy7cs   24
  25. 25. Dispositional Learning Analytics Workshop 25 http://learningemergence.net/events/lasi-dla-wkshp http://learningemergence.net/2014/03/01/ assessing-learning-dispositions-academic-mindsets
  26. 26. Observation informal and formal Self-Diagnostic informal and formal Behavioural Analytics Assessing Learning Dispositions/Mindsets Future sweetspot... multiple lenses to provoke self-reflection http://people.kmi.open.ac.uk/sbs/2014/02/assessing-learning-dispositions-academic-mindsets
  27. 27. Self-report through reflective blogging 9-10 yr old EnquiryBloggers • Bushfield School, Wolverton, UK EnquiryBlogger Wordpress Multisite plugins http://learningemergence.net/tools/enquiryblogger 27
  28. 28. Masters level EnquiryBloggers Graduate School of Education, University of Bristol EnquiryBlogger: blogging for Learning Power & Authentic Enquiry http://learningemergence.net/2012/06/20/enquiryblogger-for-learning-power-authentic-enquiry 28
  29. 29. 29 EnquiryBlogger teacher’s dashboard – direct navigation to learners’ blogs from the visual analytic
  30. 30. We are quantifying our lives
  31. 31. http://learningemergence.net/2014/03/01/assessing-learning-dispositions-academic-mindsets Dispositional profile from behavioural traces, to complement self-report? Shaofu Huang: Prototyping Learning Power Modelling in SocialLearn http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium Simon Knight: http://people.kmi.open.ac.uk/knight/2014/02/knowledge-in-search Social network patterns, teamwork effectiveness and initiation of relationships Questioning, arguing and search behaviours reveal intrinsic curiosity and epistemic commitments Tagging/sharing/ blogging/social patterns reveal how you see connections between ideas Behavioural and somatic traces associated with perseverance, grit, tenacity; overcoming panic/stress when stretched
  32. 32. narra7ve/   discourse   analy7cs   32 The  Storymaking  Project:  PhD  by  Joanna  Kwiat   hLp://projects.kmi.open.ac.uk/storymaking  
  33. 33. 1st International Workshop on Discourse-Centric Learning Analytics analytics that look beneath the surface, and quantify linguistic proxies for ‘deeper learning’ Beyond number / size / frequency of posts; ‘hottest thread’ http://www.glennsasscer.com/wordpress/wp-content/uploads/2011/10/iceberg.jpg http://solaresearch.org/events/lak/lak13/dcla13
  34. 34. Why are stories so important to formation? 34 “Fundamentally different to logical argument, the story positively demands those things that the other either denies or restricts: emotion, agency, character, perspective, and so on. To understand a story on some level is to become engaged in its telling and this will normally be achieved by allowing oneself to empathise or identify with the story’s characters or its narrator.”
  35. 35. Discourse analytics on webinar textchat Ferguson, R. and Buckingham Shum, S., Learning analytics to identify exploratory dialogue within synchronous text chat. In: 1st International Conference on Learning Analytics and Knowledge (Banff, Canada, 2011). ACM, pp. 99-103. Open Access Eprint http://oro.open.ac.uk/28955 Can we spot the quality learning conversations in a 2.5 hr webinar?
  36. 36. -60 -40 -20 0 20 40 60 80 9:28 9:32 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:13 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:48 11:52 11:54 12:00 12:03 12:04 12:05 Average Exploratory Discourse analytics on webinar textchat Sheffield, UK not as sunny as yesterday - still warm Greetings from Hong Kong Morning from Wiltshire, sunny here! See you! bye for now! bye, and thank you Bye all for now Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar… Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
  37. 37. -60 -40 -20 0 20 40 60 80 9:28 9:32 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:13 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:48 11:52 11:54 12:00 12:03 12:04 12:05 Average Exploratory Discourse analytics on webinar textchat …but if we zoom in on a peak… Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
  38. 38. Discourse analytics on webinar textchat -100 0 100 9:28 9:40 9:50 10:00 10:07 10:17 10:31 10:45 11:04 11:17 11:26 11:32 11:38 11:44 11:52 12:03 Averag Classified as “exploratory talk” (more substantive for learning) “non- exploratory” …language is used in a manner more akin to “Exploratory Talk” (Neil Mercer) Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
  39. 39. Rhetorical discourse analytics 41 OPEN QUESTION: “… little is known …” “… role … has been elusive” “Current data is insufficient …” CONTRASTING IDEAS: “… unorthodox view resolves …” “In contrast with previous hypotheses ...” “... inconsistent with past findings ...” SURPRISE: “We have recently observed ... surprisingly” “We have identified ... unusual” “The recent discovery ... suggests intriguing roles” http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
  40. 40. Rhetorical discourse analytics 42 Human analyst
  41. 41. Rhetorical discourse analytics 43 Human analyst Computational analyst http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
  42. 42. Rhetorical discourse analytics 44 Duygu Simsek’s PhD: http://people.kmi.open.ac.uk/simsek/research/ Glimpses of analytics capable of detecting higher order thinking. But humans will always read differently to machines Can we correlate this with “academic writing”, and can such analytics be used as formative feedback on drafts?
  43. 43. Rhetorical discourse analytics 45 Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: http://oro.open.ac.uk/37391 CONTRAST SUMMARY & CONTRIBUTION
  44. 44. Evidence Hub: semantic storytelling for students, practitioners and researchers Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net A wizard guides the user through the submission of a structured story: •  What’s the Issue? •  What claim are you making/ addressing? •  What kind of evidence supports/challenges this? •  Link it to papers/data •  Index it against the NIC’s core themes
  45. 45. Evidence Hub: Argument Maps Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net The wizard then generates a structured Knowledge Tree showing evidence-based claims (and disagreements)
  46. 46. http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning An alternative graphical network rendering can be generated to show the current Argument Map of supporting and challenging evidence around an Issue Evidence Hub: Argument Maps
  47. 47. Evidence Hub: professional development http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning Issue   Poten?al   Solu?on   Suppor?ng   Evidence   (prac??oner   story)  
  48. 48. epistemic  cogni7on   analy7cs   50
  49. 49. De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf. Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829 Argumentation analytics (in Debate Hub & Cohere apps) what epistemic contributions are learners making in the network? 51 Rebecca is playing the role of broker, connecting different peers’ contributions in meaningful ways We now have the basis for recommending that you engage with people NOT like you…
  50. 50. Generates novel visualizations to summarise community arguments on a given issue http://consider.it Structured Deliberation/Argument Maps
  51. 51. Shifts in epistemic commitments? (Simon Knight, KMi PhD research) What is it to ‘know’ when we search? http://sjgknight.com/finding-knowledge/2014/02/knowledge-in-search Danish exams permit Net: http://sjgknight.com/finding-knowledge/2013/07/danish-use-of-internet-in-exams-epistemology-pedagogy-assessment Epistemic networks for epistemic commitments: http://oro.open.ac.uk/39254 Learning sciences research into Epistemic Commitments/ Cognition combined with ‘researching search’ methodologies.
  52. 52. Shifts in epistemic commitments? (Simon Knight, KMi PhD research) Dimensions of Epistemic Belief Certainty The degree to which knowledge is conceived as stable or changing, ranging from absolute to tentative and evolving knowledge Simplicity The degree to which knowledge is conceived as compartmentalised or interrelated, ranging from knowledge as made up of discrete and simple facts to knowledge as complex and comprising interrelated concepts Source The relationship between knower and known, ranging from the belief that knowledge resides outside the self and is transmitted, to the belief that it is constructed by the self Justification What makes a sufficient knowledge claim, ranging from the belief in observation or authority as sources, to the belief in the use of rules of inquiry and evaluation of expertise Knight, Simon; Buckingham Shum, Simon and Littleton, Karen (2014). Epistemology, assessment, pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics (In press). http://oro.open.ac.uk/39226
  53. 53. Shifts in epistemic commitments? (Simon Knight, KMi PhD research) Behavioural Proxies of Epistemic Belief? Certainty Frequency and density of positive and negative polarity links (e.g. supports; builds on, is consistent with, versus challenges; refutes; is inconsistent with). Presence of stability markers – (e.g. current sources, bibliographic references, geographical spread). Frequency and density of meta-discourse markers in the text of nodes, which moderate assertions in scholarly ways (e.g. in contrast to; remains an unresolved issue) Simplicity Standard network analytics describing the topography of the structure Semantic analysis of node and link types describing the variety of types used by a learner, the balance in link polarity (positive/neutral/negative), and the balance of propositional (causes; refutes; solves) versus analogical thinking (is a metaphor of; is analogous to). Semantic analysis of tags on nodes and links Encapsulation of node clusters or sub-networks within a broader category expressing a higher order construct The use of different argumentation schemes, which reflect different sources of authority (argument from expertise; argument by analogy; argument from precedent) In collaborative maps, the social network view showing who is connected to whom, but what kinds of links Source Presence within the text of nodes of ‘I think’ or other restatements of fact Challenges to claims through negative link types and meta-discourse markers in node text Few additional nodes made other than those created as quotations. Justification Judgements of relevance, and supporting or explanatory notes (‘this evidences/ explains x’). Ties to method ‘ideas’. Use of well known Argumentation Schemes, especially evidence and arguments that grounded in the attribution of authority to people (‘experts’) or publications based on their status (‘valuing peer review’) Rhetorical constructions in node text which show a critical use of sources
  54. 54. analytics for wider educational outcomes Deakin Crick, R., Barr, S., Green, H. and Pedder, D. (2013). Evaluating the Wider Outcomes of Schools: Complex Systems Modelling. Centre for Systems Learning & Leadership, Graduate School of Education, University of Bristol, UK. http://learningemergence.net/library/reports/
  55. 55. Providing a more complete analytic on the health of the educational community 57 “Test grades are just ONE of a dozen indicators in our vision of a healthy educational community” • life-long/life-wide learners • citizenship • self-confidence • teamwork • emotional intelligence • “wellbeing”…
  56. 56. Hierarchical Process Modelling (Univ. Bristol Perimeta tool) 58 Seeing a learning community as a complex adaptive system requires the voices of learners, teachers, leaders, parents and ‘external’ stakeholders
  57. 57. The core mission and values of a network of school academies
  58. 58. 60 Hierarchical Process Modelling transforms qualitative and quantitative inputs into a multi-level visual analytic
  59. 59. ‘Italian Flag’ visual analytic 61 Degree of green / white / red reflects current (un)certainty over availability of evidence: supporting / unknown / challenging
  60. 60. analytics as mirrors to provoke (trans?)formative reflection
  61. 61. Your most recent mood comment: “Great, at last I have found all the resources that I have been looking for, thanks to Steve and Ellen. In your last discussion with your mentor, you decided to work on your resilience by taking on more learning challenges Your ELLI Spider shows that you have made a start on working on your resilience, and that you are also beginning to work on your creativity, which you identified as another area to work on. 1 2 3 45 Envisioning a social learning analytics dashboard Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, 29 Apr-2 May: ACM: New York, 23-33. DOI: http://dx.doi.org/10.1145/2330601.2330616 Eprint: http://oro.open.ac.uk/32910 63
  62. 62. liminal  space  analy7cs?   Use of computational techniques to reflect back to learners apparent shifts in • how they engage in a community of enquiry/practice • how they narrate their learning journeys • how they negotiate meaning • how their worldview seems to have shifted • how they respond cognitively (even somatically) when challenged.
  63. 63. 65
  64. 64. towards a networked improvement community (NIC) for FxD? hLp://learningemergence.net/2014/05/29/learning-­‐analy?cs-­‐plus-­‐nics-­‐for-­‐systemic-­‐edu-­‐improvement  
  65. 65. Defining features of a NIC ‘B’ + ‘C’ Level improvement ‘A’  —  what   prac??oners  do  in  a   given  ins?tu?on/ context   ‘B’  —  trying  to   improve  local   prac?ce   ‘C’  —  the  network   of  ‘B’  people   sharing  insight  and   coordina?ng   innova?on   Based  on  Doug  Engelbart’s  forma?ve  work:   hLp://www.dougengelbart.org/about/nics.html  
  66. 66. Defining features of an Educational-NIC a  sociotechnical  infrastructure  and  modus  operandi  for  coordinated  aLack  on   educa?onal  challenges    an  inten?onal  community    facilitated  by  a  coordina?ng  hub    using  agreed  representa?ons  of  the  system  and  improvement  strategies    in  order  to  orchestrate  ac?on    in  pursuit  of  measurable  goals    by  sharing  data,  insights  and  arguments    in  order  to  build  evidence-­‐based  claims   Bryk  et  al,  Carnegie  Founda?on  for  the  Advancement  of  Teaching:   hLp://www.carnegiefounda?on.org/sites/default/files/bryk-­‐gomez_building-­‐nics-­‐educa?on.pdf  
  67. 67. A NIC is in essence a “Collaboratory” “a collaboratory is more than an elaborate collection of ICT; it is a new networked organizational form that also includes social processes; collaboration techniques; formal and informal communication; and agreement on norms, principles, values, and rules” (Cogburn, 2003, p. 86). Cogburn, D. L. (2003). HCI in the so-called developing world: what’s in it for everyone, Interactions, 10(2), 80-87, New York: ACM Press. http://dx.doi.org/10.1145/637848.637866  
  68. 68. Building blocks of an Educational-NIC NIC Data Hub Collective Sensemaking Coordinating Maps (of the system, problem, and strategies) Coordinating Maps (of the system, problem, and strategies) Coordinating Maps (of the system, problem, and strategies) Human + Machine Analytics Educational ContextEducational Context
  69. 69. Building blocks of an Educational-NIC NIC Data Hub Collective Sensemaking Human + Machine Analytics Educational ContextEducational Context Coordinating Maps (of the system, problem, and strategies) Coordinating Maps (of the system, problem, and strategies) Coordinating Maps (of the system, problem, and strategies)
  70. 70. Shared Conceptual Maps: of system / problem / strategies http://www.carnegiefoundation.org/elibrary/getting-ideas-action- building-networked-improvement-communities-in-education http://www.carnegiealphalabs.org/persistence/
  71. 71. Shared Conceptual Maps: for knowledge navigation People / Organizations / Projects / Claims / Evidence Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
  72. 72. Shared Conceptual Maps: for knowledge navigation People / Organizations / Projects / Claims / Evidence Evidence Hub for Research by Children & Young People: http://rcyp.evidence-hub.net
  73. 73. Building blocks of an Educational-NIC NIC Data Hub Human + Machine Analytics Educational ContextEducational Context Coordinating Maps (of the system, problem, and strategies) Coordinating Maps (of the system, problem, and strategies) Coordinating Maps (of the system, problem, and strategies) Collective Sensemaking
  74. 74. Impact Map http://chaos.open.ac.uk/hypothesis-list/ Impact Maps were inspired by the Evidence Hub, and are organised around core hypotheses that the NIC is investigating.
  75. 75. Impact Map http://oermap.org/hypothesis/578/hypothesis-a-performance/
  76. 76. Impact Map http://oermap.org/hypothesis/578/hypothesis-a-performance/ Impact Maps use Sankey flow diagrams to show how findings from different contexts support or challenge each of the NIC’s core hypotheses
  77. 77. Impact Map http://oermap.org/hypothesis/578/hypothesis-a-performance/

×