Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Paul Prinsloo
University of South Africa (Unisa)
14prinsp
How do we know they learn?
The changing scope and role of
eviden...
Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/
I do not own the copyright of any of the im...
Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/
Overview of the presentation
• Thinking abo...
Fall 2017
Source credit: http://educationnext.org/should-professors-ban-laptops-classroom-computer-use-affects-
student-le...
Source credit: http://educationnext.org/should-professors-ban-laptops-classroom-computer-use-affects-
student-learning-stu...
Source credit: https://www.nytimes.com/2017/11/22/business/laptops-not-during-lecture-or-meeting.html
22 November 2017
“Me...
22 November 2017
“But a growing body of evidence shows that over all,
college students learn less when they use
computers ...
Haven’t we been here before?
Image credit: https://pixabay.com/en/shock-woman-portrait-hand-2757395/
11 September 2016
Source credit: https://www.chronicle.com/article/No-Banning-Laptops-Is-Not-the/237752
Source credit: http://activehistory.ca/2014/07/a-brief-history-of-the-laptop-ban/
From 2011 – Duke University – “If classr...
Image credit: https://pixabay.com/en/dinosaur-mirror-wing-mirror-behind-1564323/
The object in the rearview mirror may be
...
What may be haunting us, isn’t that they are not
learning, but that we are not solely in control of
their learning (as if ...
Image credit: https://pixabay.com/en/women-lay-monochrome-soledad-one-3118387/
Losing control is not the only reason why m...
Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2275202/
So, why do we need/want/have to know that
the...
Why do we need to know that they learn?
Because…
1. We can know
2. Metrics make the world go round
3. We know or the unbea...
Image credit: https://pixabay.com/en/turn-on-turn-off-industry-energy-2923046/
Reason 1: Because we can know
Source credit: https://www.amazon.com/Owned-Property-Privacy-Digital-
Serfdom/dp/1316612201/ref=sr_1_4?ie=UTF8&qid=1519453...
We have access to ever increasing volumes, velocity and
variety of student digital data, that allows us to expand not
only...
Reason 2:
Because
metrics make
the world go
round
Image credit: https://www.amazon.com/Metric-Power-David-
Beer/dp/1349717681/ref=sr_1_1?s=books&ie=UTF8&qid=1519507130&sr=1...
Image credit: https://pixabay.com/en/tape-measure-wrap-up-hands-finger-2406650/
“Measures define what is true and then are...
Image credit: https://pixabay.com/en/justitia-goddess-goddess-of-justice-2597016/
We can only know when
we measure and whe...
Image credit: https://memeguy.com/photo/12358/theres-always-that-one-guy-at-work
By measuring and allocating value, we can...
“Evidence-based education seems to favour a
technocratic [and quantitative] model in which it is
assumed that the only rel...
Source credit: http://onlinelibrary.wiley.com/doi/10.1111/j.1741-5446.2006.00241.x/full
Source credit: https://link.springer.com/article/10.1007/s11217-010-9191-x
Image credit: https://pixabay.com/en/pot-sticks-fire-smoke-ritual-2567719/
Our institutional rituals of verification sanct...
Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2209152/
Reason 3: Because we know…
… everything
Image credit: https://www.pexels.com/photo/big-data-business-cloud-communication-236906/
Where the institutional LMS resem...
Page credit: http://www.npr.org/sections/ed/2017/01/11/506361845/the-higher-ed-learning-revolution-tracking-each-
students...
• We feed our data fetish, believing that more data are always
better data
• We believe that big(ger) data gives us a tota...
Apophenia – “seeing patterns
where none actually exist,
simply because enormous
quantities of data can offer
connections t...
Citation: Pink, S., Ruckenstein, M., Willim, R., & Duque, M. (2018). Broken data: Conceptualising data in an
emerging worl...
Image credit: https://pixabay.com/en/camera-spy-pigeon-surveillance-712122/
Reason 4: Because we have the
tools…
Three sources of data
Directed
A digital form of
surveillance
wherein the
“gaze of the
technology is
focused on a
person o...
Source credit: http://timoelliott.com/blog/2013/02/gartnerbi-emea-2013-part-1-analytics-moves-to-the-core.html
Getting to ...
Image credit: https://pixabay.com/en/people-man-guy-camera-alone-photo-2587110/
Reason 5: Because our students are broken ...
Source credit: https://www.edsurge.com/amp/news/2017-11-13-what-can-higher-ed-learn-from-precision-medicine
“Focused hospi...
Image credit: https://commons.wikimedia.org/wiki/File:I_Dont_Know_What_to_Call_This..._(8455760157).jpg
Image credit: https://www.flickr.com/photos/highersights/6231641551
Because we want
to…
Reason 6
Image credit: https://pixabay.com/en/friends-trust-friendship-together-1015312/
We want to…
• Know what we (not our
studen...
• We know that we cannot cause learning to happen
and only create spaces and occasions where learning
can happen
• We know...
Image credit: https://pixabay.com/en/potter-hands-sound-tonkunst-2106654/
Reason 7: We want to understand where/how
we can...
Image credit: https://pixabay.com/en/wonder-woman-superman-superhero-552109/
Beware: They don’t need
superwoman or superman
Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/
Image credit: https://pixabay.com/en/person...
Image credit: https://pixabay.com/en/light-out-of-focus-focus-lamps-1068944/
What are the implications when we use their d...
Image credit: https://pixabay.com/en/camera-surveillance-camera-1778861/
Responsible and ethical learning analytics is fou...
Image credit: https://pixabay.com/en/listen-to-wall-listen-3014329/
Tell me again, why you want to measure
their learning?
Image credit: https://pixabay.com/en/camera-facade-wall-spy-film-1570429/
But do they learn? A
case study of broken
data, ...
Three weeks into a 12-week, fully
online course: Are they learning
(according to our structure, our
criteria and what we c...
Exhibit A
Exhibit B
Image credit: https://pixabay.com/en/person-listening-headphones-eyes-2157916/
What does this mean?
Exhibit C
Exhibit D
Student Z
At the end of a 12-
week course: Have
they learned
(according to our
structure, our
criteria and what
we can and...
Image credit: https://pixabay.com/en/buddha-statue-stone-statues-speak-546458/
If this data tell a story, whose story is i...
Prinsloo, P. (2016, November 7 ). Failing our students: not noticing the traces they leave behind. [Web log post]. Retriev...
I thought I cared.
But I did not notice her/him falling behind. I
had access to data abut her/his engagement.
And somehow ...
Imagecredit:https://pixabay.com/en/statue-camouflage-
hidden-discreet-2393168/
“… when a world is reduced to numbers, a me...
Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2275202/
Please tell me again, why do you want to know...
Paul Prinsloo (Prof)
Research Professor in Open Distance Learning (ODL)
College of Economic and Management Sciences, SP3-1...
How do we know they learn? The changing scope and role of evidence in student learning
How do we know they learn? The changing scope and role of evidence in student learning
Upcoming SlideShare
Loading in …5
×

How do we know they learn? The changing scope and role of evidence in student learning

395 views

Published on

Keynote at “Celebrating 15 years of Promoting Excellence in Teaching and Learning”, 25-27 February 2018, American University of Cairo, Egypt

Published in: Education
  • Be the first to comment

  • Be the first to like this

How do we know they learn? The changing scope and role of evidence in student learning

  1. 1. Paul Prinsloo University of South Africa (Unisa) 14prinsp How do we know they learn? The changing scope and role of evidence in student learning Image credit: https://pixabay.com/en/building-cctv-door-female-ladies-1839464/ Keynote at “Celebrating 15 years of Promoting Excellence in Teaching and Learning”, 25-27 February 2018, American University of Cairo
  2. 2. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/ I do not own the copyright of any of the images in this presentation. I therefore acknowledge the original copyright and licensing regime of every image used. This presentation (excluding the images) is licensed under a Creative Commons Attribution 4.0 International License.
  3. 3. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/ Overview of the presentation • Thinking about laptops in the classroom and dinosaurs • Dreaming of losing control • Considering why we need/want/have to know that students are learning • A case study of broken data, broken lenses and broken hopes • (In)conclusions
  4. 4. Fall 2017 Source credit: http://educationnext.org/should-professors-ban-laptops-classroom-computer-use-affects- student-learning-study/
  5. 5. Source credit: http://educationnext.org/should-professors-ban-laptops-classroom-computer-use-affects- student-learning-study/ “we do not claim that all computer use in the classroom is harmful. Exercises where computers or tablets are deliberately used may, in fact, improve student performance. Rather, our results relate to classes where using computers or tablets for note-taking is optional.” Fall 2017
  6. 6. Source credit: https://www.nytimes.com/2017/11/22/business/laptops-not-during-lecture-or-meeting.html 22 November 2017 “Measuring the effect of laptops on learning is tough. One problem is that students don’t all use laptops the same way.”
  7. 7. 22 November 2017 “But a growing body of evidence shows that over all, college students learn less when they use computers or tablets during lectures. They also tend to earn worse grades. The research is unequivocal: Laptops distract from learning, both for users and for those around them. It’s not much of a leap to expect that electronics also undermine learning in high school classrooms or that they hurt productivity in meetings in all kinds of workplaces.”
  8. 8. Haven’t we been here before? Image credit: https://pixabay.com/en/shock-woman-portrait-hand-2757395/
  9. 9. 11 September 2016 Source credit: https://www.chronicle.com/article/No-Banning-Laptops-Is-Not-the/237752
  10. 10. Source credit: http://activehistory.ca/2014/07/a-brief-history-of-the-laptop-ban/ From 2011 – Duke University – “If classroom time is primarily characterized by lectures, then laptops are probably a distraction, and if students bring them to class for note taking, it is quite likely that some will use them for activities unrelated to the class.” 31 July, 2014
  11. 11. Image credit: https://pixabay.com/en/dinosaur-mirror-wing-mirror-behind-1564323/ The object in the rearview mirror may be closer than it is… – and it is not technology…
  12. 12. What may be haunting us, isn’t that they are not learning, but that we are not solely in control of their learning (as if we ever were…) Image credit: http://geopoliticsrst.blogspot.com.eg/2014/12/isis-saudi-arabia-puppet-master-of-isis.html
  13. 13. Image credit: https://pixabay.com/en/women-lay-monochrome-soledad-one-3118387/ Losing control is not the only reason why many of us wonder: “But are they learning?”
  14. 14. Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2275202/ So, why do we need/want/have to know that they are learning?
  15. 15. Why do we need to know that they learn? Because… 1. We can know 2. Metrics make the world go round 3. We know or the unbearable lightness of arrogance 4. We have the tools 5. Our students are broken and need fixing 6. We want to understand, listen and act 7. We ant to understand where/how we can help them to do better
  16. 16. Image credit: https://pixabay.com/en/turn-on-turn-off-industry-energy-2923046/ Reason 1: Because we can know
  17. 17. Source credit: https://www.amazon.com/Owned-Property-Privacy-Digital- Serfdom/dp/1316612201/ref=sr_1_4?ie=UTF8&qid=1519453392&sr=8-4&keywords=owned As learning moves increasingly online, we (Silicon Valley, governments, data brokers, commercial entities, and higher education) have access to huge amounts of intersecting, and mutually constitutive data sets. And we own their data. And we are owned. The learning management systems we use have become a treasure trove of student data that we can process, scrape, clean, dissect, mine, massage to find patterns, and at times, make to fit our assumptions about whether they are learning or not
  18. 18. We have access to ever increasing volumes, velocity and variety of student digital data, that allows us to expand not only on the traditional scope of institutional research with regard to student data, but also to infer relations unthinkable ten years ago. We may therefore be tempted to rush to look for patterns without considering our own assumptions and epistemologies There is a real danger to mistake the noise for a signal, and not realise that the noise pollutes our data with false alarms and “setting back our ability to understand how the system really works” (Silver, 2012, p. 162)
  19. 19. Reason 2: Because metrics make the world go round
  20. 20. Image credit: https://www.amazon.com/Metric-Power-David- Beer/dp/1349717681/ref=sr_1_1?s=books&ie=UTF8&qid=1519507130&sr=1-1&keywords=metric+power “We are created and recreated by metrics; we live through them, with them, and within them. Metrics facilitate the making and the remaking of judgements about us, the judgements we make of ourselves and the consequences of those judgements as they are felt and experienced in our lives. We play with metrics and we are more often played by them” (Beer, 2016, p. 3)
  21. 21. Image credit: https://pixabay.com/en/tape-measure-wrap-up-hands-finger-2406650/ “Measures define what is true and then are used to verify the truth” (Beer, 2016, p. 28)
  22. 22. Image credit: https://pixabay.com/en/justitia-goddess-goddess-of-justice-2597016/ We can only know when we measure and when we know we have the power to to claim understanding, to define what is ‘true’, and based on this ’truth’, we assign positions, class, potential, and futures that often lock individuals into fulfilling our ‘truths’ about them
  23. 23. Image credit: https://memeguy.com/photo/12358/theres-always-that-one-guy-at-work By measuring and allocating value, we can find those who do not fit in our normal distribution curves, and isolate them and define them as abnormal, not normal
  24. 24. “Evidence-based education seems to favour a technocratic [and quantitative] model in which it is assumed that the only relevant research questions are about the effectiveness of educational means and techniques, forgetting, among other things, that what counts as “effective” crucially depends on judgments about what is educationally desirable” (Biesta, 2007, p. 5)
  25. 25. Source credit: http://onlinelibrary.wiley.com/doi/10.1111/j.1741-5446.2006.00241.x/full
  26. 26. Source credit: https://link.springer.com/article/10.1007/s11217-010-9191-x
  27. 27. Image credit: https://pixabay.com/en/pot-sticks-fire-smoke-ritual-2567719/ Our institutional rituals of verification sanction those of us who measure frequently, those of us who are measured frequently, and for all of us who submit our reports to those who need (our) data to exist
  28. 28. Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2209152/ Reason 3: Because we know… … everything
  29. 29. Image credit: https://www.pexels.com/photo/big-data-business-cloud-communication-236906/ Where the institutional LMS resembles not only a walled garden… …but increasingly Foucault’s Panopticon where our tracking systems and recommender engines watch over their every move
  30. 30. Page credit: http://www.npr.org/sections/ed/2017/01/11/506361845/the-higher-ed-learning-revolution-tracking-each- students-every-move 2017
  31. 31. • We feed our data fetish, believing that more data are always better data • We believe that big(ger) data gives us a total picture where n=all and that there is nothing we cannot know • We believe that data are neutral and objective and not as subjective as human decisions based on experience, intuition and wisdom • We create data proxies for the details of our students’ lives for which we do not have data, or that we cannot quantify • We think finding patterns is enough and we never care to ask ‘why’? And because we think we (can) know everything…
  32. 32. Apophenia – “seeing patterns where none actually exist, simply because enormous quantities of data can offer connections that radiate in all directions” (boyd & Crawford, 2012, p. 668) Our epistemological arrogance leads us to see Jesus in a piece of burned toast Seeing Jesus in toast: Irreverent ideas on some of the claims pertaining to learning analytics (Prinsloo, 2016) – https://opendistanceteachingandlearning.wo rdpress.com/2015/12/07/seeing-jesus-in- toast-irreverent-ideas-on-some-of-the-claims- pertaining-to-learning-analytics/ @Jesus_H_Toast
  33. 33. Citation: Pink, S., Ruckenstein, M., Willim, R., & Duque, M. (2018). Broken data: Conceptualising data in an emerging world. Big Data & Society, 5(1), 2053951717753228.
  34. 34. Image credit: https://pixabay.com/en/camera-spy-pigeon-surveillance-712122/ Reason 4: Because we have the tools…
  35. 35. Three sources of data Directed A digital form of surveillance wherein the “gaze of the technology is focused on a person or place by a human operator” Automated Generated as “an inherent, automatic function of the device or system and include traces …” Volunteered “gifted by users and include interactions across social media and the crowdsourcing of data wherein users generate data” (emphasis added) Kitchen, R. (2013). Big data and human geography: opportunities, challenges and risks. Dialogues in Human Geography, 3, 262-267. SOI: 10.1177/2043820613513388
  36. 36. Source credit: http://timoelliott.com/blog/2013/02/gartnerbi-emea-2013-part-1-analytics-moves-to-the-core.html Getting to prescribe on the basis of ‘students like you’…
  37. 37. Image credit: https://pixabay.com/en/people-man-guy-camera-alone-photo-2587110/ Reason 5: Because our students are broken and need fixing
  38. 38. Source credit: https://www.edsurge.com/amp/news/2017-11-13-what-can-higher-ed-learn-from-precision-medicine “Focused hospitals are built around a very specific value-adding process activity: They take incomplete or broken parts and then transform them into more complete outputs of higher value while charging a fee for the outcome.” Higher education should be like hospitals using precision medicine…
  39. 39. Image credit: https://commons.wikimedia.org/wiki/File:I_Dont_Know_What_to_Call_This..._(8455760157).jpg
  40. 40. Image credit: https://www.flickr.com/photos/highersights/6231641551 Because we want to… Reason 6
  41. 41. Image credit: https://pixabay.com/en/friends-trust-friendship-together-1015312/ We want to… • Know what we (not our students) can do different, better or stop doing • Know how we can and when we should step back, step in, not do anything, intervene, nudge and care
  42. 42. • We know that we cannot cause learning to happen and only create spaces and occasions where learning can happen • We know that there is a lot of learning that we plan for that does not happen. Some of the learning we plan for may happen (often despite our failures and missed opportunities • And we also know that there is also a lot of learning that we did not plan for, that actually happens Image credit: https://pixabay.com/en/guy-man-male-people-child-boy-2609607/
  43. 43. Image credit: https://pixabay.com/en/potter-hands-sound-tonkunst-2106654/ Reason 7: We want to understand where/how we can help them to do better
  44. 44. Image credit: https://pixabay.com/en/wonder-woman-superman-superhero-552109/ Beware: They don’t need superwoman or superman
  45. 45. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/ Image credit: https://pixabay.com/en/person-male-man-portrait-shadow-828630/ “If you have come to help us, you can go home. If you have come to accompany us, please come. We can talk” Glesne, C. (2016). Research as solidarity. In T. Kukutai and J. Taylor. (Eds), Indigenous data sovereignty. Toward an agenda (pp. 169-178). Canberra, Australia: Australian National University Press. Retrieved from https://press.anu.edu.au/publications/series/centre-aboriginal-economic- policy-research-caepr/indigenous-data-sovereignty
  46. 46. Image credit: https://pixabay.com/en/light-out-of-focus-focus-lamps-1068944/ What are the implications when we use their data points to describe, diagnose, predict and prescribe their learning journeys without ever asking them what these data points mean to them, and what data would matter to them to allow them to make more informed decisions, to complete their journeys and not ours?
  47. 47. Image credit: https://pixabay.com/en/camera-surveillance-camera-1778861/ Responsible and ethical learning analytics is found in the nexus between their stories and ours. We cannot afford to ignore the fact that it is their data, their aspirations, their learning journeys and that our data collection, analysis and use may not tell the whole story.
  48. 48. Image credit: https://pixabay.com/en/listen-to-wall-listen-3014329/ Tell me again, why you want to measure their learning?
  49. 49. Image credit: https://pixabay.com/en/camera-facade-wall-spy-film-1570429/ But do they learn? A case study of broken data, broken lenses and broken hope
  50. 50. Three weeks into a 12-week, fully online course: Are they learning (according to our structure, our criteria and what we can and want to measure)?
  51. 51. Exhibit A
  52. 52. Exhibit B
  53. 53. Image credit: https://pixabay.com/en/person-listening-headphones-eyes-2157916/ What does this mean?
  54. 54. Exhibit C
  55. 55. Exhibit D
  56. 56. Student Z At the end of a 12- week course: Have they learned (according to our structure, our criteria and what we can and want to measure)? Exhibit E
  57. 57. Image credit: https://pixabay.com/en/buddha-statue-stone-statues-speak-546458/ If this data tell a story, whose story is it? Is this the whole story? Who is listening? What did me miss?
  58. 58. Prinsloo, P. (2016, November 7 ). Failing our students: not noticing the traces they leave behind. [Web log post]. Retrieved from https://opendistanceteachingandlearning.wordpress.com/2016/11/07/failing-our-students-not-noticing-the-traces-they-leave- behind/
  59. 59. I thought I cared. But I did not notice her/him falling behind. I had access to data abut her/his engagement. And somehow did not notice Prinsloo, P. (2016, November 7 ). Failing our students: not noticing the traces they leave behind. [Web log post]. Retrieved from https://opendistanceteachingandlearning.wordpress.com/2016/11/07/failing-our-students-not-noticing-the-traces-they-leave- behind/
  60. 60. Imagecredit:https://pixabay.com/en/statue-camouflage- hidden-discreet-2393168/ “… when a world is reduced to numbers, a measure, to what is calculable and laid before us; when humans are summed, aggregated and accounted for; then much remains forgotten, unsaid, concealed” (Elden, 2006, in Beer, 2016, pp. 59-60)
  61. 61. Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2275202/ Please tell me again, why do you want to know that they are learning? And how will you know?
  62. 62. Paul Prinsloo (Prof) Research Professor in Open Distance Learning (ODL) College of Economic and Management Sciences, SP3-15, P O Box 392, Unisa, 0003, Republic of South Africa T: +27 (0) 12 433 4719 (office) T: +27 (0) 82 3954 113 (mobile) prinsp@unisa.ac.za Skype: paul.prinsloo59 Personal blog: http://opendistanceteachingandlearning.wordpress.com Twitter profile: @14prinsp Thank you

×