• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Our Learning Analytics are Our Pedagogy
 

Our Learning Analytics are Our Pedagogy

on

  • 3,368 views

Keynote Address, Expanding Horizons 2012, Macquarie University ...

Keynote Address, Expanding Horizons 2012, Macquarie University

http://staff.mq.edu.au/teaching/workshops_programs/expanding_horizons

"Learning Analytics": unprecedented data sets and live data streams about learners, with computational power to help make sense of it all, and new breeds of staff who can talk predictive models, pedagogy and ethics. This means rather different things to different people: unprecedented opportunity to study, benchmark and improve educational practice, at scales from countries and institutions, to departments, individual teachers and learners. "Benchmarking" may trigger dystopic visions of dumbed down proxies for 'real teaching and learning', but an emu response is no good. For educational institutions, our calling is to raise the quality of debate, shape external and internal policy, and engage with the companies and open communities developing the future infrastructure. How we deploy these new tools rests critically on assessment regimes, what can be logged and measured with integrity, and what we think it means to deliver education that equips citizens for a complex, uncertain world.

Statistics

Views

Total Views
3,368
Views on SlideShare
2,163
Embed Views
1,205

Actions

Likes
7
Downloads
77
Comments
0

19 Embeds 1,205

http://learningemergence.net 773
http://people.kmi.open.ac.uk 211
http://www.scoop.it 172
http://galasresearch.blogspot.com 24
http://galasresearch.blogspot.it 6
http://tweetedtimes.com 5
http://translate.googleusercontent.com 2
http://galasresearch.blogspot.se 1
https://twitter.com 1
https://www.linkedin.com 1
http://galasresearch.blogspot.co.uk 1
http://galasresearch.blogspot.nl 1
http://galasresearch.blogspot.gr 1
http://galasresearch.blogspot.co.nz 1
http://galasresearch.blogspot.sg 1
https://si0.twimg.com 1
http://galasresearch.blogspot.tw 1
https://twimg0-a.akamaihd.net 1
http://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Our Learning Analytics are Our Pedagogy Our Learning Analytics are Our Pedagogy Presentation Transcript

    • Keynote Address, Expanding Horizons 2012, Macquarie Universityhttp://staff.mq.edu.au/teaching/workshops_programs/expanding_horizonsOur Learning Analyticsare Our Pedagogy Simon Buckingham Shum @sbskmi Knowledge Media Institute, The Open University UK simon.buckinghamshum.net 1
    • learning objective: walk out withbetter questionsthan you can ask right now 2
    • Urgent need: quality dialogue betweenanalytics stakeholders, to accelerateinvention innovation www.SoLAResearch.org Follow: @SoLAResearch 3
    • The global demand for learning Implications for assessment and feedback at John Daniel massive scale?http://www.col.org/resources/speeches/2012presentations/Pages/2012-02-01.aspx 4
    • is educationpoised to become a data-driven enterprise and science ? 5
    • Possibly 90% of the digital data we havetoday was generated in the last 2 years Volume outstrips old infrastructure Variety Internet of things, e-business transactions, environmental sensors, social media, audio, video, mobile… Velocity The speed of data access and analysis is exploding A quantitative shift on this scale is in fact a qualitative shift, requiring new ways of thinking about societal phenomena 6
    • edX: “this is big data, giving us the chanceto ask big questions about learning” Will the tomorrow’s educational researcher be as helpless without an analytics infrastructure, as a geneticist without genome databases, or a physicist without CERN? 7
    • Lifelogging: explosion of data captureand sharing about personal activities http://www.mirror-project.eu http://quantifiedself.com/guide 8
    • Educational Data Mining research community
    • Learning Analytics research community
    • different levels of analytic 11
    • ‘Learning Analytics’ and‘Academic Analytics’Long, P. and Siemens, G. (2011), Penetrating the fog: analytics in learning and education. Educause Review Online,46, 5, pp.31-40. http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education 12
    • Macro/Meso/Micro Learning Analytics Macro: region/state/national/international
    • Macro/Meso/Micro Learning Analytics Macro: region/state/national/international Meso: institution-wide
    • Macro/Meso/Micro Learning Analytics Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Will institutions be dazzled by the dashboards, or know what questions to ask at each level?
    • Macro/Meso/Micro Learning Analytics Macro: region/state/national/international
    • US states are getting the infrastructurein placedataqualitycampaign.org 17
    • Shared Learning Collaborativehttp://slcedu.org 18
    • National league tables for English schools 19
    • Macro/Meso/Micro Learning Analytics Meso: institution-wide
    • Analytics-savvy Leaders are the future?Parr-Rud, O. (2012). Drive Your Business with Predictive Analytics. SAS White Paperhttp://www.sas.com/reg/gen/corp/1800392 21
    • Business Intelligence companies see aneducation market opening up These are pedagogically agnostic: they seek to optimize operational efficiency whatever the sector These may make pedagogical assumptions: how will learning design and assessment regimes shape the analytics they offer?http://www.sas.com/industry/education/highered 22
    • Business Intelligence companies see aneducation market opening up …but do they know anything about the roles that language plays in learning and knowledge construction? 23
    • BI+HigherEd communities of practice 24
    • Business Intelligence ≠ Learning Analytics
    • Macro/Meso/Micro Learning Analytics Micro: individual user actions (and hence cohort)
    • Analytics in your VLE:Blackboard: feedback to studentshttp://www.blackboard.com/Platforms/Analytics/Overview.aspx 27
    • Purdue University Signals: real time traffic-lights for students based on predictive model Premise: academic success is defined as a function of aptitude (as measured by standardized test scores and similar information) and effort (as measured by participation within the online learning environment). Using factor analysis and logistic regression, a model was tested to predict student success based on: •  ACT or SAT score •  Overall grade-point average Predicted 66%-80% •  CMS usage composite of struggling •  CMS assessment composite students who •  CMS assignment composite needed help •  CMS calendar composite Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x 28
    • Desire2Learn visual analytics & predictive modelswhich can be interrogated on different dimensionshttp://www.desire2learn.com/products/analytics 29
    • Desire2Learn visual analytics & predictive modelswhich can be interrogated on different dimensionshttp://www.desire2learn.com/products/analytics 30
    • Socrato: train for SATshttp://www.socrato.com 31
    • Khan Academy: more data to teachers,finer-grained feedback to studentshttp://www.thegatesnotes.com/Topics/Education/Sal-Khan-Analytics-Khan-Academy 32
    • Adaptive platforms generate fine-grained analyticshttps://grockit.com/research 33
    • Adaptive platforms generate fine-grainedanalytics http://knewton.com
    • Adaptive platforms generate fine-grainedanalyticshttp://oli.cmu.edu
    • The VLE—BI convergence 36
    • Hard distinctions between Learning +Academic analytics may dissolve…as they get joined up, each level enriches the others Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Aggregation of user tracesenriches meso + macro analytics with finer-grained process data
    • Hard distinctions between Learning +Academic analytics may dissolve…as they get joined up, each level enriches the others Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Aggregation of user traces Breadth + depth from macroenriches meso + macro analytics + meso levels add power to with finer-grained process data micro analytics
    • …so everybody’s happy?dawn of a new data-driven enterprise + science? 39
    • wrong.a very healthy debate is brewing… 40
    • data (indeed technology) is not neutraldata does not wholly ‘speak for itself’ 41
    • Measurement tools are not neutral“accounting tools...do not simply aid the measurement of economic activity, they shape the reality they measure” Du Gay, P. and Pryke, M. (2002) Cultural Economy: Cultural Analysis and Commercial Life Sage, London. pp. 12-13
    • Beyond big data hubris1.  Automating Research Changes the Definition of Knowledge2.  Claims to Objectivity and Accuracy are Misleading3.  Bigger Data are Not Always Better Data4.  Not All Data Are Equivalent5.  Just Because it is Accessible Doesn’t Make it Ethical6.  Limited Access to Big Data Creates New Digital Divides boyd, d. and Crawford, K. (2001). Six Provocations for Big Data. Presented to: A Decade in Internet Time: Symposium on the Dynamics of the Internet and Society, Oxford Internet Institute, Sept. 21, 2011. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431
    • Analytics provide maps = systematicways of distorting reality“A marker of the health of the learning analytics field will be the quality of debate around what the technology renders visible and leaves invisible.” Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, 2012, Vancouver, BC). ACM: New York. Eprint: http://oro.open.ac.uk/32823
    • contextcontextcontext 45
    • Analytics in English schoolsRAISEonline platform: cohort visualization 46
    • Will our analytics reflect the progress that‘Joe’ has made on so many other fronts –but not his SATs? 47
    • Will our analytics reflect the progress that‘Joe’ has made on so many other fronts –but not his SATs? ?48
    • Video conferencing analyticsOU KMi’s FlashmeetingVideo conference spoken foreign language tutorials Mentor 1 Mentor 2 AV Chat AV ChatSession 2 3 49
    • Video conferencing analyticsOU KMi’s FlashmeetingVideo conference spoken foreign language tutorials Mentor 1 Mentor 2 AV Chat AV Chat 1Session 2 3 50
    • Video conferencing analyticsOU KMi’s FlashmeetingVideo conference spoken foreign language tutorials — which mentor would you want to have?... Mentor 1 Mentor 2 AV Chat AV Chat 1Session 2 3 51
    • Video conferencing analyticsOU KMi’s FlashmeetingVideo conference spoken foreign language tutorials — which mentor would you want to have?... Mentor 1 Mentor 2 AV Chat AV Chat 1Session Mentor 1 is doing the best job: at this introductory 2 level, students need intensive input and flounder if left 3 52
    • course completion is only one proxy for good learning and what’s easy tomeasure isn’t alwayswhat’s most important 53
    • The Wal-Martification of education?http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college 54http://lak12.wikispaces.com/Recordings
    • The Wal-Martification of education? “What counts as data, how do you get it, and what does it actually mean?” “The basic question is not what can we measure? The basic question is “data narrowness” what does a good “instrumental learning” education look like? “students with no curiosity” Big questions.http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college 55http://lak12.wikispaces.com/Recordings
    • Course completion as a proxy for learninghttp://annezelenka.com/2012/05/05/but-what-about-learning 56
    • Course completion as a proxy for learninghttp://annezelenka.com/2012/05/05/but-what-about-learning NOTE TO SELVES: We are the HigherEd market who make it worthwhile for major vendors to design analytics focused on 57 maximising course completion PS: HEIs may feel that they are trapped by external expectations and requirements. Systems thinking required…
    • let’s just pretend that learning analytics tookseriously the revolution going on outside the university front door…we would need to devise learning analytics for this?... 58
    • Learning analytics for this?“We are preparing students for jobs that do not exist yet, that will use technologies that have not been invented yet, in order to solve problems that are not even problems yet.” “Shift Happens” http://shifthappens.wikispaces.com 59
    • Learning analytics for this?“While employers continue to demand high academic standards, they also now want more. They want people who can adapt, see connections, innovate, communicate and work with others. This is true in many areas of work. The new knowledge-based economies in particular will increasingly depend on these abilities. Many businesses are paying for courses to promote creative abilities, to teach the skills and attitudes that are now essential for economic success…” All our Futures: Creativity, culture & education, May 1999 60
    • Learning analytics for this? “Knowledge of methods alone will not suffice: there must be the desire, the will, to employ them. This desire is an affair of personal disposition.” John Dewey, 1933Dewey, J. How We Think: A Restatement of the Relation of Reflective Thinking to theEducative Process. Heath and Co, Boston, 1933 61
    • Learning analytics for this?“The test of successful education is not the amount of knowledge that pupils take away from school, but their appetite to know and their capacity to learn.” Sir Richard Livingstone, 1941 62
    • Learning analytics for this? The Knowledge-Agency Window co-generation Expert-led enquiry Student-led enquiry Knowledge and use Teaching as Authenticity learning design Agency Identity Repetition, Pre-scribed Knowledge Abstraction Acquisition Expert-led teaching Student-led revision Teacher agency Student agency
    • Learning analytics for this? Creativity, Culture and Education (2009) Changing Young Lives 2012. Newcastle: CCE. http://www.creativitycultureeducation.org/ changing-young-lives-2012 64
    • Musicality ≠ Musical Reproduction In those early days the children were taught from the start to develop their own voice, whether literally singing, or through the instrument they played. They were not taught music, but musicality. Central to this tuition were the partimenti, many pages of detailed music notes which pose many questions, but leave the pupil to find the solutions. The music is not a literal transcript, which the musician reads and reproduces. set of rules and then The partimenti establish, at the start, a pose a set of conflicts for the musician to resolve, in their own way. 65http://bit.ly/U1vkNf
    • consider assessment for learning(not summative assessment for grading pupils, teachers, institutions or nations) 66
    • Assessment for Learninghttp://assessment-reform-group.org 67
    • Assessment for Learninghttp://assessment-reform-group.org 68
    • Assessment for Learninghttp://assessment-reform-group.org To what extent could automated feedback be designed and evaluated with emotional impact in mind? 69
    • Assessment for Learninghttp://assessment-reform-group.org Can analytics identify proxies for such advanced qualities? 70
    • Assessment for Learninghttp://assessment-reform-group.org Do analytics provide constructive next steps? 71
    • Assessment for Learninghttp://assessment-reform-group.org How do we provide analytics feedback that does not disempower and de- motivate struggling learners? 72
    • analytics for… dispositions discourse social networks See SoLAR Storm: Social Learning Analytics symposium 73http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium
    • Social Learning Analytics §  Analytics focused on social learning theories, practices and platforms, e.g. §  Discourse analytics: beyond quantitative summaries of online writing, to qualitative analysis §  Social network analytics: visualizing effective social ties for collective learning §  Dispositional analytics: measuring students’ readiness to engage in lifelong, lifewide learningFerguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, 29 Apr-2 May: ACM Press. Eprint: http://oro.open.ac.uk/32910Buckingham Shum, S. and Ferguson, R., Social Learning Analytics. Educational Technology & Society (Special Issue on Learning & Knowledge Analytics, Eds. G. Siemens & D. Gašević), 15, 3, (2012), 3-26. http://www.ifets.info Open Access Eprint: http://oro.open.ac.uk/34092
    • Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar Sheffield, UK not as sunny but if we zoom in on a peak… See you! as yesterday - still warm bye for now! Greetings from Hong Kong bye, and thank you Morning from Wiltshire, 80 sunny here! Bye all for now 60 40 20 0 9:28 9:32 10:13 11:48 12:00 12:05 12:04 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:52 11:54 12:03 -20 -40 Average Exploratory -60Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within SynchronousText Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955
    • Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Classified as “exploratory talk” (more substantive 100 for learning) 50 0 9:28 “non- 9:40 9:50 10:00 10:07 10:17 10:31 10:45 11:04 11:17 11:26 11:32 11:38 11:44 11:52 12:03 -50 exploratory” Averag -100Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within SynchronousText Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955
    • Discourse Network Analytics = Concept Network + Social Network AnalyticsDe Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1stInternational Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829
    • KMi’s Cohere: a web deliberation platform enabling semantic social network and discourse network analytics Rebecca is playing the role of broker, connecting 2 peers’ contributions in meaningful waysDe Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1stInternational Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829
    • Analytics for C21 learning skills Different social network patterns Questioning and in different challenging may contexts may load onto Critical load onto Curiosity Learning Relationships Repeated Sharing relevant attempts to pass resources from an online test other contexts may load onto may load onto Resilience Meaning MakingBuckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and LearningAnalytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, Vancouver). Eprint: http://oro.open.ac.uk/32823
    • LearningEmergence.net: embeddingdispositional analytics into practice + toolsEnquiryBlogger: Wordpress plugins for reflective learning journals 80
    • Discourse analysis (Xerox Incremental Parser)Detection of salient sentences in scholarly reports,based on the rhetorical signals authors use:BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:Recent studies indicate … ... new insights provide direct evidence ... … little is known …… the previously proposed … ... we suggest a new ... approach ... … role … has been elusive Current data is insufficient …… is universally accepted ... ... results define a novel role ...CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING:… unorthodox view resolves … studies ... have provided important The goal of this study ...paradoxes … advances Here, we show ...In contrast with previous Knowledge ... is crucial for ... Altogether, our results ... indicatehypotheses ... understanding... inconsistent with past findings ... valuable information ... from studiesGENERALIZING: SURPRISE:... emerging as a promising approach We have recently observed ... surprisinglyOur understanding ... has grownexponentially ... We have identified ... unusual... growing recognition of the The recent discovery ... suggests Ágnes Sándor & OLnet Project: http://olnet.org/node/512 intriguing rolesimportance ...De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-MachineAnnotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
    • Human and machine analysis of a text for keycontributions Document 1 19 sentences annotated 22 sentences annotated 11 sentences same as human annotation Document 2 71 sentences annotated 59 sentences annotated 42 sentences same as human annotationhttp://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotationDe Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-MachineAnnotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
    • Closing thoughts 83
    • “The basic question is not what can we measure? The basic question is what does a good education look like? Big questions. (Gardner Campbell) Do we value what we can measure, or measure what we really value? And just because this is tough to do, doesn’t mean we don’t do it. (Guy Claxton, BBC Radio 4 Education Debate, Nov. 2012) 84
    • Our analytics promote values, pedagogy and assessment regimes. Are we clear which master our analytics serve? Are we happy to be judged by them? simon.buckinghamshum.nethttp:// sbskmi http://twitter.com/ 85