Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Learning Analytics vs Cognitive Automation

1,347 views

Published on

Keynote, Learning Analytics Summer Institute, Ann Arbor, June 2017

Published in: Education
  • Be the first to comment

Learning Analytics vs Cognitive Automation

  1. 1. Learning Analytics vs Cognitive Automation: Rationale, Examples & Organisational Strategy Simon Buckingham Shum Connected Intelligence Centre • University of Technology Sydney utscic.edu.au Keynote, Learning Analytics Summer Institute, Ann Arbor, June 2017 @sbuckshum • Simon.BuckinghamShum.net
  2. 2. Acknowledgements to CIC colleagues whose work I’ll be sharing…
  3. 3. 3 UTS learning strategy, supported by analytics innovation + impact implications for learning analytics + some examples the machines are coming… cognitive automation
  4. 4. San Francisco, Fall Joint Computer Conference — Dec. 9th 1968
  5. 5. we need better tools to tackle “humanity’s complex, urgent problems”
  6. 6. 8 dougengelbart.org
  7. 7. US change index of work tasks 1960-2009 9 Frank Levy and Richard J. Murnane, Dancing with Robots: Human Skills for Computerized Work (Washington, DC: Third Way, 2013), Fig.3
  8. 8. Routine manual and cognitive work is being automated Only non-routine manual and cognitive work will be done by people (What counts as ‘non-routine’ will reduce as AI improves) 10
  9. 9. Jason Furman, Chairman of the Council of Economic Advisers https://artificialintelligencenow.com/schedule/conference/presentation/time-different-opportunities-and-challenges-artifi
  10. 10. learning 12 analytics? what are the implications for
  11. 11. utscic.e du.au As analytics aggregate lower level data & A.I. gradually automates routine cognitive work…
  12. 12. utscic.e du.au Humans must move to the higher ground… • Train data scientists to combine algorithmic intelligence with creative intelligence, and ethical mindsets • Deploy all the Educational & Data Science expertise we have to cultivate the higher order graduate qualities As analytics aggregate lower level data & A.I. gradually automates routine cognitive work…
  13. 13. utscic.e du.au Humans must move to the higher ground… • Train data scientists to combine algorithmic intelligence with creative intelligence, and ethical mindsets • Deploy all the Educational & Data Science expertise we have to cultivate the higher order graduate qualities As analytics aggregate lower level data & A.I. gradually automates routine cognitive work… Cultivate those qualities that are distinctively human and devise practical, authentic ways to evidence them
  14. 14. The endless cycle… 16 What we take to be “distinctively human” has always been in transition, but now at unprecedented pace
  15. 15. 17 At its best, education – esp. Higher Education – should provide the conditions for an integrated possibly even transformational experience
  16. 16. So we’re talking about… Providing authentic learning contexts, and assessments that build lifelong learning capabilities Focusing not solely on course completion, but on whether students have a sense of calling Cultivating curiosity, agency, resilience, an appetite for risk, a thirst for different perspectives Requiring evidence that academic knowledge is being integrated with personal experience, identity, increasingly professional disposition
  17. 17. Towards analytics for a holistic higher education Randy Bass Georgetown University Reinvent the University for the Whole Person http://reinvent.net/series/reinvent-the-university
  18. 18. Harnessing ‘disintegrative’ forces in higher education in the service of integrative learning for holistic devpt.? Randall Bass & Bret Eynon (2017). From Unbundling to Rebundling: Design Principles for Transforming Institutions in the New Digital Ecosystem. Change: The Magazine of Higher Learning, 49, (2), pp. 8-17. DOI: http://dx.doi.org/10.1080/00091383.2017.1286211 Based on: https://www.aacu.org/publications- research/publications/open-and-integrative- designing-liberal-education-new-digital An interest in narrative, e-portfolios + analytics that empower learners to reflect on their journey
  19. 19. disciplinary knowledge + transferable skills + open learning dispositions / innovation mindsets 21
  20. 20. “Knowledge of methods alone will not suffice: there must be the desire, the will, to employ them. This desire is an affair of personal disposition.” John Dewey Dewey, J. How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process. Heath and Co, Boston, 1933 Knowledge, Skills & Dispositions
  21. 21. “One of the key issues emerging from these findings was the learner’s orientation towards the unknown, uncertainty and ambiguity, and their tendency to either retreat from it or move into it. The former effectively precludes deep learning, and the latter is the beginning point for it.” Ruth Deakin Crick & Chris Goldspink Deakin Crick R. and Goldspink G. (2014) Learning Dispositions, Self-theories and Student Engagement, British Journal of Educational Studies, 62,1,1-17. DOI: http://dx.doi.org/10.1080/00071005.2014.904038 Knowledge, Skills & Dispositions
  22. 22. Institute for the Future http://www.iftf.org/futureworkskills
  23. 23. utscic.edu.au Institute for the Future http://www.iftf.org/futureworkskills
  24. 24. learning 26 analytics? what are the implications for
  25. 25. 27 learning in the wild
  26. 26. Analytics beyond the LMS: xAPI-based aggregation 28 ? ?
  27. 27. Connected Learning Analytics Toolkit K. Kitto, A. Bakharia, M. Lupton, D. Mallet, J. Banks, P. Bruza, A. Pardo, S. Buckingham Shum, S. Dawson, D. Gašević, G. Siemens, & G. Lynch, (2016). The connected learning analytics toolkit. Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (LAK '16). ACM, New York, NY, USA, 548-549. https://doi.org/10.1145/2883851.2883881 Student dashboard Educator dashboard
  28. 28. 30 learningembodied
  29. 29. Sensing co-located teamwork Martinez-Maldonado, R., Power, T., Hayes, C., Abdipranoto, A., Vo, T., Axisa, C., and Buckingham-Shum, S. (2017) Analytics Meet Patient Manikins: Challenges in an Authentic Small-Group Healthcare Simulation Classroom. International Conference on Learning Analytics and Knowledge, LAK 2017, 90-94.
  30. 30. Tracking nurses’ movement around the patient Martinez-Maldonado, R., Pechenizkiy, M., Power, T., Buckingham-Shum, S., Hayes, C. and Axisa, C. (2017) Modelling Embodied Mobility Teamwork Strategies in a Simulation-Based Healthcare Classroom. International Conference on User Modelling, Adaptation and Personalization, UMAP 2017,
  31. 31. Movement heatmaps –> sequence analysis Martinez-Maldonado, R., Pechenizkiy, M., Power, T., Buckingham-Shum, S., Hayes, C. and Axisa, C. (2017) Modelling Embodied Mobility Teamwork Strategies in a Simulation-Based Healthcare Classroom. International Conference on User Modelling, Adaptation and Personalization, UMAP 2017,
  32. 32. 34 analytics dispositional
  33. 33. Grade (x) vs Student Effort (y) from teacher observational assessments over the semester 35 Nagy, R. (2016). Tracking and visualizing student effort: Evolution of a practical analytics tool for staff and student engagement. Journal of Learning Analytics, 3 (2), 165–193. http://dx.doi.org/10.18608/jla.2016.32.8 Seminar: https://utscic.edu.au/events/niccic-redlands-school-8-june-2016
  34. 34. 36 Assessing learning dispositions: Crick Learning for Resilient Agency survey (CLARA) Deakin Crick, R., Huang, S., Ahmed Shafi, A. and Goldspink, C. (2015). Developing Resilient Agency in Learning: The Internal Structure of Learning Power. British Journal of Educational Studies: 62, (2), 121-160. http://dx.doi.org/10.1080/00071005.2015.1006574 https://utscic.edu.au/tools/clara • http://clara.learningemergence.com
  35. 35. 37 Structural Equation Model underpinning CLARA Deakin Crick, R., Huang, S., Ahmed Shafi, A. and Goldspink, C. (2015). Developing Resilient Agency in Learning: The Internal Structure of Learning Power. British Journal of Educational Studies: 62, (2), 121- 160. http://dx.doi.org/10.1080/00071005.2015.1006574 Knowing how constructs drive each other not only underpins the survey’s internal integrity, but could provide the empirical basis for the automated analysis of behavioural traces, as proxies for these constructs
  36. 36. Immediate visual analytic generated by CLARA Feedback to Stimulate Self-Directed Change A framework for reflection and coaching Deakin Crick, R., Huang, S., Ahmed Shafi, A. and Goldspink, C. (2015). Developing Resilient Agency in Learning: The Internal Structure of Learning Power. British Journal of Educational Studies: 62, (2), 121- 160. http://dx.doi.org/10.1080/00071005.2015.1006574 40 item survey
  37. 37. Scaling CLARA in UTS Barratt-See, G., Cheng, M., Deakin Crick, R. & Buckingham Shum, S. (2017). Assessing Resilient Agency with CLARA: Empirical Findings from Piloting a Visual Analytics Tool at UTS. Proceedings UniSTARS 2017: University Students, Transitions, Achievement, Retention & Success. (Adelaide, 1-4 July, 2017). https://utscic.edu.au/tools/clara Senior student mentors introduce 1st Years to CLARA through fictional personas, based on educator- specified archetypes
  38. 38. Scaling CLARA in UTS Barratt-See, G., Cheng, M., Deakin Crick, R. & Buckingham Shum, S. (2017). Assessing Resilient Agency with CLARA: Empirical Findings from Piloting a Visual Analytics Tool at UTS. Proceedings UniSTARS 2017: University Students, Transitions, Achievement, Retention & Success. (Adelaide, 1-4 July, 2017). https://utscic.edu.au/tools/clara n=876 n=957 n=548 n=602 § For the 921 students with post-profile data, there were significant positive changes on all 8 dimensions. § We can also derive through cluster analysis significantly different cohort profiles, inviting reflection and possibly intervention for those who might be judged at risk
  39. 39. Behavioural proxies for “Conscientiousness”? Shute, V. J. and M. Ventura (2013). Stealth Assessment: Measuring and supporting learning in video games. Cambridge, MA, MIT Press. Figure 5 from report to The John D. and Catherine T. MacArthur Foundation Reports on Digital Media and Learning http://myweb.fsu.edu/vshute/pdf/Stealth_Assessment.pdf
  40. 40. Behavioural proxies for “Conscientiousness”? Shute, V. J. and M. Ventura (2013). Stealth Assessment: Measuring and supporting learning in video games. Cambridge, MA, MIT Press. Figure 5 from report to The John D. and Catherine T. MacArthur Foundation Reports on Digital Media and Learning http://myweb.fsu.edu/vshute/pdf/Stealth_Assessment.pdf
  41. 41. Behavioural proxies for “Conscientiousness”? Shute, V. J. and M. Ventura (2013). Stealth Assessment: Measuring and supporting learning in video games. Cambridge, MA, MIT Press. Figure 5 from report to The John D. and Catherine T. MacArthur Foundation Reports on Digital Media and Learning http://myweb.fsu.edu/vshute/pdf/Stealth_Assessment.pdf
  42. 42. Behavioural proxies for “Conscientiousness”? Shute, V. J. and M. Ventura (2013). Stealth Assessment: Measuring and supporting learning in video games. Cambridge, MA, MIT Press. Figure 5 from report to The John D. and Catherine T. MacArthur Foundation Reports on Digital Media and Learning http://myweb.fsu.edu/vshute/pdf/Stealth_Assessment.pdf
  43. 43. reflective writing 45 as a window mind on the
  44. 44. Automated feedback on reflective writing § Reflection is critical to the integration of academic + experiential knowledge § This is where you disclose what you’re uncertain about, and how you’ve changed, in the first person § Scholarship clarifies the hallmarks of deeper reflective writing: Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C. and Knight, S. (2017). Reflective Writing Analytics for Actionable Feedback. Proc. LAK17: 7th Int. Conf. on Learning Analytics & Knowledge, March 13-17, 2017, Vancouver. (ACM Press). https://doi.org/10.1145/3027385.3027436 Best Paper Award
  45. 45. Automated feedback on reflective writing
  46. 46. For more examples see: Learning Analytics for 21st Century Competencies. (Eds.) Buckingham Shum S. & Deakin Crick, R. (2016). Journal of Learning Analytics (Special Section), 3, (2), pp. 6-212. http://dx.doi.org/10.18608/jla.2016.32.2
  47. 47. Caution! A Learning Analytic approach makes educational claims (implicitly or explicitly) 49 epistemology pedagogyassessment Knight, S., Buckingham Shum, S. and Littleton, K. (2014). Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space. Journal of Learning Analytics, 1, (2), pp.23-47. http://epress.lib.uts.edu.au/journals/index.php/JLA/article/download/3538/4156 Knight, S. and Buckingham Shum, S. (2017). Theory & Learning Analytics. Handbook of Learning Analytics. https://solaresearch.org/hla-17 the middle space of learning analytics In all of the examples shown, the analytics approach makes an impact when integrated coherently into the learning design, which is in turn driven by this triangle
  48. 48. 50 these kinds of analytics go far beyond current products: they require innovation
  49. 49. 51 But universities also seek impact
  50. 50. 52 notes on UTS organisational strategy
  51. 51. 2011 2011 Envisioning “the Data Intensive University” UTS-wide Forum to consider the profound implications of analytics “To make UTS a world leading university of technology, where staff and students understand data […] and can develop analytical tools” UTS-WIDE CONSULTATIONS & STRATEGIC ALIGNMENT
  52. 52. UTS-WIDE CONSULTATIONS & STRATEGIC ALIGNMENT 54 2011 2011 Envisioning “the Data Intensive University” DIU UTS-wide Forum 2012 Connected Intelligence Working Party UTS-wide consultations and interviews 2013 Connected Intelligence Strategy Privacy & Ethics Forum Plans for a Masters Degree 2014 Director Appointed Launch of the Centre UTS-wide engagement Collaboration Proposals Invited & Projects Initiated 2015 Masters Degree Launches Analytics Pilots in Faculties CIC Staffing Grows First Pilot Evaluation Data 2016 Growing number of UTS & Industry Partnerships Analytical Tools Maturing Academic Board
  53. 53. Faculties & Institutes Student Support Units CIC Business Units DEVELOP & DEPLOY ANALYTICS ACROSS UTS 2017 http://utscic.edu.au
  54. 54. UTS FACULTY OF TRANSDISCIPLINARY INNOVATION https://www.uts.edu.au/future-students/transdisciplinary-innovation
  55. 55. UTS INNOVATION & CREATIVE INTELLIGENCE STRATEGY BACHELORS IN CI & INNOVATION http://www.uts.edu.au/partners-and-community/initiatives/creative-intelligence • http://www.uts.edu.au/future-students/creative-intelligence-and-innovation
  56. 56. Architecting for Innovation + Impact?
  57. 57. Architecting for Innovation + Impact? VC DVC Operations IT BI Analytics LMS Analytics IT SERVICES MODEL: Analytics based in BI or LMS team
  58. 58. Architecting for Innovation + Impact? VC DVC Research Faculty School Centre Learning Analytics researchers FACULTY RESEARCH MODEL: Analytics based in a centre/school
  59. 59. Architecting for Innovation + Impact? VC DVC Education CIC HYBRID INNOVATION/SERVICES MODEL: Analytics based in a non-faculty centre reporting to DVC (Education), staffed by academics + professional admin team
  60. 60. Architecting for Innovation + Impact? VC DVC Research Faculty School Centre Academics DVC Education CIC DVC Operations IT BI Analytics LMS Analytics HYBRID INNOVATION/SERVICES MODEL: Analytics based in a non-faculty centre reporting to DVC (Education), staffed by academics + admin team
  61. 61. Architecting for Innovation + Impact? Board Room VC/DVCs/Deans/Directors Common Room Academic staff Server Room IT Division CIC Skillset: strong interpersonal skills + Education, Learning Design, Interface Design, Programming, Web Development, Text Analytics, Machine Learning, Statistics, Visualisation, Decision-Support, Sensemaking, Creativity & Risk, Participatory Design
  62. 62. ADVANTAGES THAT THIS ORG STRUCTURE BRINGS Operating within the DVC’s Education & Students Office enables close coupling with student services and teaching innovation Baseline funding provides invaluable stability for planning projects and staff Reporting directly to a DVC, and talking directly to other operational directors, gets stuff done Operating outside a faculty provides agility for decision- making, and helpful neutrality
  63. 63. 65 lesson 1 “LA+LD” learning analytics must be aligned with the learning design, or adds little value
  64. 64. 66 lesson 2 “IA not AI” a key role for analytics is less to judge, and more to provoke productive reflection
  65. 65. What we take to be “distinctively human” has always been in transition, but now at unprecedented pace The task of Learning Analytics may increasingly be to make persistent and visible what until now has remained ephemeral and invisible LA+LD designs this as formative feedback to provoke reflection, insight, creativity and deeper learning

×