SlideShare a Scribd company logo
1 of 55
Download to read offline
Simon Buckingham Shum
Professor of Learning Informatics
Director, Connected Intelligence Centre
Learning Informatics as Inspiration
for Learning Analytics
cic.uts.edu.au • Simon.BuckinghamShum.net • @sbuckshum
eMadrid 2021 • emadridnet.org
overview
learning analytics (+ human-centred design)
informatics
learning informatics
4 informatics principles for human-centred LA
what this looks like in action
3
Learning Analytics
4
What is Learning Analytics?
Learning
student engagement
teaching practice
curriculum design
instructional design
pedagogy
assessment
epistemology
…
Analytics
data
statistics
classification
machine learning
text processing
visualisation
predictive models
…
this is not a straightforward dialogue!
5
A key circle is missing…
Human
Factors
stakeholder involvement
participatory design cycles
user interface design
privacy and ethics
end-user evaluation
organisational strategy
staff training
6
Learning Analytics: A Human-Centred Design Discipline
Learning Analytics
Human
Factors
7
Learning Analytics: A Human-Centred Design Discipline
Learning Analytics
Human
Factors
https://www.emadridnet.org/index.php/en/28-eventos-y-seminarios/1214-human-centred-design-and-learning-analytics-where-are-we-at-and-where-should-we-go-next
http://simon.buckinghamshum.net/2019/05/human-centred-analyticsai-in-education
Human-Centred Analytics/AI in Education
9
informatics
We need an overarching lens for designing
effective, usable, ethical learning technology
10
Kristen Nygaard
August 27, 1926 – August 10, 2002
Internationally acknowledged as the co-inventor
with Ole-Johan Dahl of object-oriented programming
and the programming language Simula.
He also was a pioneer of participatory design and the
“Scandinavian school of systems development”.
http://kristennygaard.no
11
Nygaard, K. (1986): Program Development as a Social Activity, Invited Lecture, Proceedings of the IFIP 10th World Computer Congress,
INFORMATION PROCESSING 86, Dublin. Elsevier Science Publishers, pp. 189-198. Available at: https://ojs.ruc.dk/index.php/pdc/article/view/97/89
12
“Informatics is the science that has as its
domain information processes and related
phenomena in artifacts, society and nature”.
Nygaard, K. (1986): Program Development as a Social Activity, Invited Lecture, Proceedings of the IFIP 10th World Computer Congress,
INFORMATION PROCESSING 86, Dublin. Elsevier Science Publishers, pp. 189-198. Available at: https://ojs.ruc.dk/index.php/pdc/article/view/97/89
13
No management hand-waving about valuing
“user participation”
Let’s get specific: who are we talking about?
Nygaard, K. (1986): Program Development as a Social Activity, Invited Lecture, Proceedings of the IFIP 10th World Computer Congress,
INFORMATION PROCESSING 86, Dublin. Elsevier Science Publishers, pp. 189-198. Available at: https://ojs.ruc.dk/index.php/pdc/article/view/97/89
14
How will such users get to “participate”?
Nygaard, K. (1986): Program Development as a Social Activity, Invited Lecture, Proceedings of the IFIP 10th World Computer Congress,
INFORMATION PROCESSING 86, Dublin. Elsevier Science Publishers, pp. 189-198. Available at: https://ojs.ruc.dk/index.php/pdc/article/view/97/89
“Informatics is a distinct scientific discipline,
characterised by its own concepts, methods,
body of knowledge and open issues.
It covers the foundations of computational
structures, processes, artefacts and
systems; and their software designs, their
applications, and their impact on society”
https://www.informatics-europe.org
This laid the foundations for Informatics today…
“Informatics concerns itself with the study of living,
working and building in a digital world. Wherever
technology touches people, it must be designed with
ultimate care. This requires mastery of technological
knowhow and a deep appreciation of the social,
cultural and organizational forces at work.”
https://www.informatics.uci.edu/explore/chairs-welcome
This laid the foundations for Informatics today…
17
learning
informatics
Note that the terms Education(al) Informatics have been proposed, and some definitions of this are similar, albeit
limited to educational institutions — I prefer a broader focus on lifelong/lifewide learning.
Levy, P., Ford, N., Foster, J., Madden, A., Miller, D., Nunes, M. B., et al. (2003). Educational informatics: An emerging research agenda. Journal
of Information Science, 29(4), 298-210. [reprint]
Collins, J.W., and Weiner, S.A..(2010). Proposal for the creation of a subdiscipline: Education informatics. Teachers College Record 112, no. 10:
2523–2536. [reprint]
18
learning informatics
How does this lens translate into deploying Analytics/AI for learning?
Design for educator and learner agency
Design socio-technical systems
Design for imperfect computational models
System Integrity > algorithmic accountability
19
Design socio-technical
systems
Learning Informatics Principle 1
Framework @UTS for educators to co-design Analytics/AI
à augment teaching practice
PhD by Antonette Shibani: http://simon.buckinghamshum.net/2019/11/congratulations-dr-antonette-shibani
Shibani, A., Knight, S. and Buckingham Shum, S. (2019). Contextualizable Learning Analytics Design: A Generic Model, and Writing Analytics Evaluations. Proc. 9th International
Conference on Learning Analytics & Knowledge (LAK19). ACM Press, NY, pp. 210-219. DOI: https://doi.org/10.1145/3303772.3303785. Eprint: https://tinyurl.com/lak19clad
Student
Task
Design
Feedback
& User
Interface
Features
in the
Data
Educators
Analytics/AI
designers
Assessment
Acknowledgement:
https://www.freevector.com/jigsaw-puzzle#
Educator
/Student
Resources
Learning
Design
Evaluation
Analytics
Capability
Bundling analytics with educator resources
Integrating Writing Activities with Writing Analytics
Higher Education Text Analytics
consortium:
http://heta.io/resources
UTS Orientation Portal:
https://uts.edu.au/acawriter
Automated feedback systems should enhance
educators’ and students’ feedback literacy
DAFFI 2020:
Designing Automated Feedback for Impact
8-9 Sept. 2020 • bit.ly/daffi2020
https://cic.uts.edu.au/ensuring-automated-feedback-is-pedagogically-sound-daffi2020
Raising data/learning analytics literacy
(staff and students) UTS Open: https://open.uts.edu.au
Disseminating innovations in Learning Analytics to the UTS community
Briefings • Hands-on Training • Teaching into degree programs
CIC Events archive: https://cic.uts.edu.au/events
Raising data/learning analytics literacy
(staff and students)
25
Design for educator
and learner agency
PhDs by Carlos Alvarez-Prieto, Vanessa Echeverria, Antonette Shibani
http://simon.buckinghamshum.net/2020/09/congratulations-dr-carlos-prieto-alvarez
http://simon.buckinghamshum.net/2020/05/congratulations-dr-vanessa-echeverria
http://simon.buckinghamshum.net/2019/11/congratulations-dr-antonette-shibani
Learning Informatics Principle 2
Multimodal student data from simulations
Who did what, when in a nursing simulation? Team Timeline for evidence-based debriefings
Patient’s state changes
Nurses 1-3
Uses a device
Administers medication
Personalised feedback on high performance teamwork
Echeverria, V., Martinez-Maldonado, R. and Buckingham Shum, S. (2019). Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data. In Proceedings of ACM Conference on Human
Factors in Computing (CHI’19). ACM: NY. Paper 39, pp. 1-16. https://doi.org/10.1145/3290605.3300269 Open Access Eprint: http://bit.ly/chi19utscic
Co-design techniques to engage students and
staff in designing multimodal learning analytics
Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. (2018).
Mapping Learner-Data Journeys: Evolution of a Visual Co-design Tool. Proceedings
of the 30th Australian Conference on Computer-Human Interaction (OzCHI’18),
Melbourne, Australia, Dec. 2018, ACM, New York, NY, USA, pp. 205–214.
DOI: https://doi.org/10.1145/3292147.3292168
29
Toolkit available http://ladeck.utscic.edu.au
LA-DECK: card-based co-design tool for LA
Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. (2020). LA-DECK: A Card-Based Learning Analytics Co-Design Tool. Proc.10th
International Conference on Learning Analytics and Knowledge, Frankfurt, March 2020, ACM. 10 pages. DOI: https://doi.org/10.1145/3375462.3375476
30
LA-DECK: card-based co-design tool for LA
Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. (2020). LA-DECK: A Card-Based Learning Analytics Co-Design Tool. Proc.10th
International Conference on Learning Analytics and Knowledge, Frankfurt, March 2020, ACM. 10 pages. DOI: https://doi.org/10.1145/3375462.3375476
Automated
formative
feedback on
reflective writing
Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan,
P., Sutton, N., Wight, R., Lucas, C., Sándor, Á.,
Kitto, K., Liu, M., Mogarkar, R. & Buckingham
Shum, S. (2020). AcaWriter: A learning analytics
tool for formative feedback on academic writing.
Journal of Writing Research, 12, (1), 141-186.
https://doi.org/10.17239/jowr-2020.12.01.06
Participatory prototyping with educators
to build trust in the NLP
http://heta.io/how-can-writing-analytics-researchers-rapidly-codesign-feedback-with-educators
Learning Analytics researchers work with
academics (3 hour workshop)
Goal: calibrate the parser detecting affect in
reflective writing, working through sample texts
Rapid prototyping with a Python notebook, then
integrated into full application for further testing
Design for imperfect
computational models
Kirsty Kitto, Simon Buckingham Shum, and Andrew Gibson. (2018). Embracing Imperfection in Learning Analytics. In Proceedings LAK18: International Conference on
Learning Analytics and Knowledge, March 5–9, 2018, Sydney, NSW, Australia, pp.451-460. (ACM, New York, NY, USA). https://doi.org/10.1145/3170358.3170413
Learning Informatics Principle 3
(Ergonomics principle: wise allocation of function between human and machine)
The Navajo rug
“In a Navajo rug there is always an imperfection
woven into the corner. And interestingly
enough, it’s where “the Spirit moves in and out
of the rug.” The pattern is perfect and then
there’s one part of it that clearly looks like a
mistake . . .
Perfection is not the elimination of imperfection.
[…] Perfection, rather, is the ability to
incorporate imperfection!
[…] You either incorporate imperfection,
or you fall into denial.”
http://exhibitions.kelsey.lsa.umich.edu/less-than-perfect/navajo.php
Richard Rohr (2011). Breathing Under Water: Spirituality and the Twelve Steps.
Cincinnati, OH: Franciscan Media
See also: https://medium.com/bedolabs/success-through-imperfection-c3ef21cb32ed
We must equip graduates with the
distinctive qualities that will keep
them in jobs that won’t be
automated…
…but if we want to use LA in such
teaching and learning,
tracking/assessing such
competencies will be imperfect
Machines will never fully model our higher order capabilities
– but that can be an educational feature not a bug
Reflection and metacognition
are among the highest order outcomes that we aspire to
cultivate in learners — the desire and skills to observe
one’s thoughts, emotions and actions, and glean insights
Learning Analytics in such
contexts will in principle have a
high degree of
imperfection
Embodied, skilled performance
an important part of the learning experience is physically
embodied (e.g. inspecting a forest; a social services risk
assessment)
Wicked problems (Horst Rittel)
problems with no correct answer, and no stopping rules
— even the definition of the problem is contested
Socially complex performance
scenarios where the outcome is emergent in nature, a function of
many drivers that result in unpredictable and/or unique outcomes
(e.g. a social worker client interview; conflict resolution)
Learning Analytics in such
contexts will in principle have a
high degree of
imperfection
Machines will never fully model our higher order capabilities
– but that can be an educational feature not a bug
Can we design for this?
Imperfect Learning Analytics à cognitive dissonance
38
Kirsty Kitto, Simon Buckingham Shum, and Andrew Gibson. (2018). Embracing Imperfection in Learning Analytics. In Proceedings LAK18: International Conference on
Learning Analytics and Knowledge, March 5–9, 2018, Sydney, NSW, Australia, pp.451-460. (ACM, New York, NY, USA). https://doi.org/10.1145/3170358.3170413
“…as D’Mello and Graesser [15] demonstrate, it is when the
student experiences dissonance because the
analytics fail to match their expectations that
they are likely to reflect on why they think the
machine is wrong. We believe that this form of critical
questioning is more likely to happen if the student has been given
an underlying reason to be a little distrustful of the classifier.”
39
Gavriel Salomon, David Perkins and Tamar Globerson (1991).
Partners in cognition: extending human intelligence with
intelligent technologies. Educational Researcher, 20, 3, pp.il2–9.
Learners’ engagement with intelligent
technology should not be mindless.
On the contrary we should design for…
“nonautomatic, effortful and
thus metacognitively guided
processes”
“Embracing Imperfection in Learning Analytics”
40
1. Robust learning design
ensures that the activity
involving automated feedback
is meaningful whether or not
the technology always works
2. Explicit encouragement
— in student briefings, and in
the user interface — to push
back if they disagree with
the feedback
41
System Integrity
> “algorithmic accountability”
Learning Informatics Principle 4
Remind me, why should I trust this system?
Growing public literacy around algorithmic bias and
the need for accountability is to be welcomed
43
…but is there anything distinctive about algorithms for teaching
and learning that shapes how we frame “accountability”?
Stakeholders and key transitions
in designing a Learning Analytics system
Ethical Principles
Educational/Learning
Sciences researcher
Learning Theory
Algorithm
Learning Analytics
Researcher
Educator
Learner
Learning Outcomes
Educational Insights
Programmer
Software, Hardware
User Interface
Data
Educational/Learning
Sciences researcher
Programmer
Software, Hardware
Educator
Learner
Ethical Principles
Algorithm
Learning Outcomes
Learning Theory
Learning Analytics
Researcher
User Interface
Educational Insights
Accountability in terms of: Computer Science
Data
Educational/Learning
Sciences researcher
Programmer
Software, Hardware
Educator
Learner
Ethical Principles
Algorithm
Learning Outcomes
Learning Theory
Learning Analytics
Researcher
User Interface
Educational Insights
Accountability in terms of: Data Science
Data Training
Data
Educational/Learning
Sciences researcher
Programmer
Software, Hardware
Educator
Learner
Ethical Principles
Algorithm
Learning Outcomes
Learning Theory
Learning Analytics
Researcher
User Interface
Educational Insights
Accountability in terms of: User-Centred Design
Data
Design Process
Educational/Learning
Sciences researcher
Programmer
Software, Hardware
Educator
Learner
Algorithm
Learning Outcomes
Learning Theory
Learning Analytics
Researcher
User Interface
Educational Insights
Accountability in terms of: Deliberative Democracy
Data
Ethical Principles
Accountability in terms of: Deliberative Democracy?
“A Deliberative Mini-Public has two core
features:
• It is deliberative: participants reach their
conclusions or recommendations after
receiving information and engaging in a careful
and open discussion about the issue or issues
before them.
• It is a mini-public: its members
constitute, so far as possible, a representative
subset of the wider population.” (Farrell et al, 2019)
• Escobar, O. and Elstub, S. (2017), Forms of Mini-Publics: An introduction to deliberative innovations in democratic practice. The newDemocracy Foundation.
• Farrell, D., Curato, N., Dryzek, J.S., Geißel, B., Grönlund, K., Marien, S., Niemeyer, S., Pilet, J.-B., Renwick, A., Rose, J., and Setälä, M. (2019), Deliberative Mini-Publics: Core Design Features.
Centre for Deliberative Democracy and Global Governance working paper 2019/5. Canberra, Australia: Centre for Deliberative Democracy and Global Governance.
Ethical LA
Principles
Educational/Learning
Sciences researcher
Programmer
Software, Hardware
Educator
Learner
Ethical Principles
Algorithm
Learning Outcomes
Learning Theory
Learning Analytics
Researcher
User Interface
Educational Insights
Accountability in terms of:
Learning Sciences & Educational Technology
Data
Remind me, why should I trust this system?
So we move from asking...
Remind me, why should I trust this system?
...to asking...
More detailed version of that analysis…
53
LASI tutorial: http://simon.buckinghamshum.net/2019/11/black-box-learning-analytics
54
ICLS 2018 keynote: h9p://simon.buckinghamshum.net/2018/06/icls2018-keynote
Elsewhere I have argued that this constitutes
a transition in education’s knowledge infrastructure
and discuss different theory à analytics mappings
55
learning informatics
How does this lens translate into deploying Analytics/AI for learning?
Design for educator and learner agency
Design socio-technical systems
Design for imperfect computational models
System Integrity > algorithmic accountability

More Related Content

What's hot

Computational Participation: Towards a National Education Policy in Uruguay ...
Computational Participation: Towards a National EducationPolicy in Uruguay  ...Computational Participation: Towards a National EducationPolicy in Uruguay  ...
Computational Participation: Towards a National Education Policy in Uruguay ...
@cristobalcobo
 
Challenges Facing Distance Education in the 21st Century: Implications for Se...
Challenges Facing Distance Education in the 21st Century:Implications for Se...Challenges Facing Distance Education in the 21st Century:Implications for Se...
Challenges Facing Distance Education in the 21st Century: Implications for Se...
EDEN Digital Learning Europe
 
Lightning talks: the student digital experience
Lightning talks: the student digital experienceLightning talks: the student digital experience
Lightning talks: the student digital experience
Jisc
 
Pro Technology Group2 Final
Pro Technology Group2 FinalPro Technology Group2 Final
Pro Technology Group2 Final
srosedahl
 
Group2-CEP 810 Ppt Presentation
Group2-CEP 810 Ppt PresentationGroup2-CEP 810 Ppt Presentation
Group2-CEP 810 Ppt Presentation
guest6423c6d
 

What's hot (20)

Learning in the Digital University
Learning in the Digital UniversityLearning in the Digital University
Learning in the Digital University
 
Computational Participation: Towards a National Education Policy in Uruguay ...
Computational Participation: Towards a National EducationPolicy in Uruguay  ...Computational Participation: Towards a National EducationPolicy in Uruguay  ...
Computational Participation: Towards a National Education Policy in Uruguay ...
 
Framing digital capabilities for staff in UK higher education (HE): meeting
Framing digital capabilities for staff in UK higher education (HE): meetingFraming digital capabilities for staff in UK higher education (HE): meeting
Framing digital capabilities for staff in UK higher education (HE): meeting
 
Challenges Facing Distance Education in the 21st Century: Implications for Se...
Challenges Facing Distance Education in the 21st Century:Implications for Se...Challenges Facing Distance Education in the 21st Century:Implications for Se...
Challenges Facing Distance Education in the 21st Century: Implications for Se...
 
Working in partnership to support student employability
Working in partnership to support student employabilityWorking in partnership to support student employability
Working in partnership to support student employability
 
Lightning talks: the student digital experience
Lightning talks: the student digital experienceLightning talks: the student digital experience
Lightning talks: the student digital experience
 
WA Curriculum Outline: Technologies
WA Curriculum Outline: TechnologiesWA Curriculum Outline: Technologies
WA Curriculum Outline: Technologies
 
Jisc Change Agents' Network Webinar 13 May 2015
Jisc Change Agents' Network Webinar 13 May 2015Jisc Change Agents' Network Webinar 13 May 2015
Jisc Change Agents' Network Webinar 13 May 2015
 
Digital Diagnostic: identifying staff digital capabilities at Staffordshire U...
Digital Diagnostic: identifying staff digital capabilities at Staffordshire U...Digital Diagnostic: identifying staff digital capabilities at Staffordshire U...
Digital Diagnostic: identifying staff digital capabilities at Staffordshire U...
 
How you can enhance your efficiency and effectiveness for teaching and learni...
How you can enhance your efficiency and effectiveness for teaching and learni...How you can enhance your efficiency and effectiveness for teaching and learni...
How you can enhance your efficiency and effectiveness for teaching and learni...
 
Learning technology is everywhere…or is it?
Learning technology is everywhere…or is it?Learning technology is everywhere…or is it?
Learning technology is everywhere…or is it?
 
Kathy Trinder, Glasgow Caledonian University
Kathy Trinder, Glasgow Caledonian UniversityKathy Trinder, Glasgow Caledonian University
Kathy Trinder, Glasgow Caledonian University
 
Towards portable learning analytics dashboards - Andrii Vozniuk, Sten Govaert...
Towards portable learning analytics dashboards - Andrii Vozniuk, Sten Govaert...Towards portable learning analytics dashboards - Andrii Vozniuk, Sten Govaert...
Towards portable learning analytics dashboards - Andrii Vozniuk, Sten Govaert...
 
Smart Educational Decision Support Systems for School Complexity Leadership: ...
Smart Educational Decision Support Systems for School Complexity Leadership: ...Smart Educational Decision Support Systems for School Complexity Leadership: ...
Smart Educational Decision Support Systems for School Complexity Leadership: ...
 
Wellbeing and responsibility: a new ethics for digital educators
Wellbeing and responsibility: a new ethics for digital educatorsWellbeing and responsibility: a new ethics for digital educators
Wellbeing and responsibility: a new ethics for digital educators
 
Understanding Learning Networks
Understanding Learning NetworksUnderstanding Learning Networks
Understanding Learning Networks
 
Building digital capability and digital experience insights community event
Building digital capability and digital experience insights community eventBuilding digital capability and digital experience insights community event
Building digital capability and digital experience insights community event
 
Digital Learning Projection - Learning state estimation from multimodal learn...
Digital Learning Projection - Learning state estimation from multimodal learn...Digital Learning Projection - Learning state estimation from multimodal learn...
Digital Learning Projection - Learning state estimation from multimodal learn...
 
Pro Technology Group2 Final
Pro Technology Group2 FinalPro Technology Group2 Final
Pro Technology Group2 Final
 
Group2-CEP 810 Ppt Presentation
Group2-CEP 810 Ppt PresentationGroup2-CEP 810 Ppt Presentation
Group2-CEP 810 Ppt Presentation
 

Similar to 2021_07_01 «Learning Informatics as Inspiration for Learning Analytics».

Learning Informatics: AI • Analytics • Accountability • Agency
Learning Informatics: AI • Analytics • Accountability • AgencyLearning Informatics: AI • Analytics • Accountability • Agency
Learning Informatics: AI • Analytics • Accountability • Agency
Simon Buckingham Shum
 
Learning design and data analytics: from teacher communities to CSCL scripts
Learning design and data analytics: from teacher communities to CSCL scriptsLearning design and data analytics: from teacher communities to CSCL scripts
Learning design and data analytics: from teacher communities to CSCL scripts
davinia.hl
 
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...
Simon Buckingham Shum
 
Supporting teachers with community, design and learning analytics, Davinia He...
Supporting teachers with community, design and learning analytics, Davinia He...Supporting teachers with community, design and learning analytics, Davinia He...
Supporting teachers with community, design and learning analytics, Davinia He...
davinia.hl
 
HI5030 Business Systems Analysis And Design.docx
HI5030 Business Systems Analysis And Design.docxHI5030 Business Systems Analysis And Design.docx
HI5030 Business Systems Analysis And Design.docx
write4
 

Similar to 2021_07_01 «Learning Informatics as Inspiration for Learning Analytics». (20)

Learning Informatics: AI • Analytics • Accountability • Agency
Learning Informatics: AI • Analytics • Accountability • AgencyLearning Informatics: AI • Analytics • Accountability • Agency
Learning Informatics: AI • Analytics • Accountability • Agency
 
Learning design and data analytics: from teacher communities to CSCL scripts
Learning design and data analytics: from teacher communities to CSCL scriptsLearning design and data analytics: from teacher communities to CSCL scripts
Learning design and data analytics: from teacher communities to CSCL scripts
 
KidsLoop-March2022-Ethically-driven analytics supporting educators.pptx
KidsLoop-March2022-Ethically-driven analytics supporting educators.pptxKidsLoop-March2022-Ethically-driven analytics supporting educators.pptx
KidsLoop-March2022-Ethically-driven analytics supporting educators.pptx
 
Who are you and makes you special?
Who are you and makes you special?Who are you and makes you special?
Who are you and makes you special?
 
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...
The Connected Intelligence Centre: Human-Centered Analytics for UTS Data Chal...
 
10_05_2019 Seminario eMadrid sobre «Tecnologías de la educación dentro y fuer...
10_05_2019 Seminario eMadrid sobre «Tecnologías de la educación dentro y fuer...10_05_2019 Seminario eMadrid sobre «Tecnologías de la educación dentro y fuer...
10_05_2019 Seminario eMadrid sobre «Tecnologías de la educación dentro y fuer...
 
Supporting teachers with community, design and learning analytics, Davinia He...
Supporting teachers with community, design and learning analytics, Davinia He...Supporting teachers with community, design and learning analytics, Davinia He...
Supporting teachers with community, design and learning analytics, Davinia He...
 
Artificial Intelligence in E-learning (AI-Ed): Current and future applications
Artificial Intelligence in E-learning (AI-Ed): Current and future applicationsArtificial Intelligence in E-learning (AI-Ed): Current and future applications
Artificial Intelligence in E-learning (AI-Ed): Current and future applications
 
Abreu portfolio
Abreu portfolioAbreu portfolio
Abreu portfolio
 
Plagiarism and AI tools: an example of linking information- and digital liter...
Plagiarism and AI tools: an example of linking information- and digital liter...Plagiarism and AI tools: an example of linking information- and digital liter...
Plagiarism and AI tools: an example of linking information- and digital liter...
 
Digital Literacies, Open Educational Resources (OER) & Open Badges
Digital Literacies, Open Educational Resources (OER) & Open BadgesDigital Literacies, Open Educational Resources (OER) & Open Badges
Digital Literacies, Open Educational Resources (OER) & Open Badges
 
Learning Analytics - A New Discipline and Bits of Semantics
Learning Analytics - A New Discipline and Bits of SemanticsLearning Analytics - A New Discipline and Bits of Semantics
Learning Analytics - A New Discipline and Bits of Semantics
 
2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja
2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja
2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja
 
HI5030 Business Systems Analysis And Design.docx
HI5030 Business Systems Analysis And Design.docxHI5030 Business Systems Analysis And Design.docx
HI5030 Business Systems Analysis And Design.docx
 
edmedia2014-learning-analytics-keynote
edmedia2014-learning-analytics-keynoteedmedia2014-learning-analytics-keynote
edmedia2014-learning-analytics-keynote
 
Taking on a New Leadership Challenge: Student-Focused Learning in Artificial ...
Taking on a New Leadership Challenge: Student-Focused Learning in Artificial ...Taking on a New Leadership Challenge: Student-Focused Learning in Artificial ...
Taking on a New Leadership Challenge: Student-Focused Learning in Artificial ...
 
Understanding everyday users’ perception of socio-technical issues through s...
Understanding everyday users’ perception of  socio-technical issues through s...Understanding everyday users’ perception of  socio-technical issues through s...
Understanding everyday users’ perception of socio-technical issues through s...
 
Short CfP #DISC2016
Short CfP #DISC2016Short CfP #DISC2016
Short CfP #DISC2016
 
Final call for #DISC2016
Final call for #DISC2016Final call for #DISC2016
Final call for #DISC2016
 
UTS CIC2 Briefing, 17 June 2016
UTS CIC2 Briefing, 17 June 2016UTS CIC2 Briefing, 17 June 2016
UTS CIC2 Briefing, 17 June 2016
 

More from eMadrid network

More from eMadrid network (20)

Recognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo Tovar
Recognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo TovarRecognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo Tovar
Recognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo Tovar
 
A study about the impact of rewards on student's engagement with the flipped ...
A study about the impact of rewards on student's engagement with the flipped ...A study about the impact of rewards on student's engagement with the flipped ...
A study about the impact of rewards on student's engagement with the flipped ...
 
Assessment and recognition in technical massive open on-line courses with and...
Assessment and recognition in technical massive open on-line courses with and...Assessment and recognition in technical massive open on-line courses with and...
Assessment and recognition in technical massive open on-line courses with and...
 
Recognition of learning: Status, experiences and challenges - Carlos Delgado ...
Recognition of learning: Status, experiences and challenges - Carlos Delgado ...Recognition of learning: Status, experiences and challenges - Carlos Delgado ...
Recognition of learning: Status, experiences and challenges - Carlos Delgado ...
 
Bootstrapping serious games to assess learning through analytics - Baltasar F...
Bootstrapping serious games to assess learning through analytics - Baltasar F...Bootstrapping serious games to assess learning through analytics - Baltasar F...
Bootstrapping serious games to assess learning through analytics - Baltasar F...
 
Meta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
Meta-review of recognition of learning in LMS and MOOCs - Ruth CobosMeta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
Meta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
 
Best paper Award - Miguel Castro
Best paper Award - Miguel CastroBest paper Award - Miguel Castro
Best paper Award - Miguel Castro
 
eMadrid Gaming4Coding - Possibilities of game learning analytics for coding l...
eMadrid Gaming4Coding - Possibilities of game learning analytics for coding l...eMadrid Gaming4Coding - Possibilities of game learning analytics for coding l...
eMadrid Gaming4Coding - Possibilities of game learning analytics for coding l...
 
Seminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdf
Seminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdfSeminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdf
Seminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdf
 
eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...
eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...
eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...
 
Open_principles_and_co-creation_for_digital_competences_for_students.pdf
Open_principles_and_co-creation_for_digital_competences_for_students.pdfOpen_principles_and_co-creation_for_digital_competences_for_students.pdf
Open_principles_and_co-creation_for_digital_competences_for_students.pdf
 
Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...
Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...
Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...
 
eMadrid_KatjaAssaf_DigiCred.pdf
eMadrid_KatjaAssaf_DigiCred.pdfeMadrid_KatjaAssaf_DigiCred.pdf
eMadrid_KatjaAssaf_DigiCred.pdf
 
Presentazione E-Madrid - 12-01-2023 Ruth Kerr.pdf
Presentazione E-Madrid - 12-01-2023 Ruth Kerr.pdfPresentazione E-Madrid - 12-01-2023 Ruth Kerr.pdf
Presentazione E-Madrid - 12-01-2023 Ruth Kerr.pdf
 
EDC-eMadrid_20230113 Ildikó Mázár.pdf
EDC-eMadrid_20230113 Ildikó Mázár.pdfEDC-eMadrid_20230113 Ildikó Mázár.pdf
EDC-eMadrid_20230113 Ildikó Mázár.pdf
 
2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...
2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...
2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...
 
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
 
2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...
2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...
2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...
 
2022_11_11 «AI and ML methods for Multimodal Learning Analytics»
2022_11_11 «AI and ML methods for Multimodal Learning Analytics»2022_11_11 «AI and ML methods for Multimodal Learning Analytics»
2022_11_11 «AI and ML methods for Multimodal Learning Analytics»
 
2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
2022_11_11 «The promise and challenges of Multimodal Learning Analytics»2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
 

Recently uploaded

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 

Recently uploaded (20)

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
 

2021_07_01 «Learning Informatics as Inspiration for Learning Analytics».

  • 1. Simon Buckingham Shum Professor of Learning Informatics Director, Connected Intelligence Centre Learning Informatics as Inspiration for Learning Analytics cic.uts.edu.au • Simon.BuckinghamShum.net • @sbuckshum eMadrid 2021 • emadridnet.org
  • 2. overview learning analytics (+ human-centred design) informatics learning informatics 4 informatics principles for human-centred LA what this looks like in action
  • 4. 4 What is Learning Analytics? Learning student engagement teaching practice curriculum design instructional design pedagogy assessment epistemology … Analytics data statistics classification machine learning text processing visualisation predictive models … this is not a straightforward dialogue!
  • 5. 5 A key circle is missing… Human Factors stakeholder involvement participatory design cycles user interface design privacy and ethics end-user evaluation organisational strategy staff training
  • 6. 6 Learning Analytics: A Human-Centred Design Discipline Learning Analytics Human Factors
  • 7. 7 Learning Analytics: A Human-Centred Design Discipline Learning Analytics Human Factors https://www.emadridnet.org/index.php/en/28-eventos-y-seminarios/1214-human-centred-design-and-learning-analytics-where-are-we-at-and-where-should-we-go-next
  • 9. 9 informatics We need an overarching lens for designing effective, usable, ethical learning technology
  • 10. 10 Kristen Nygaard August 27, 1926 – August 10, 2002 Internationally acknowledged as the co-inventor with Ole-Johan Dahl of object-oriented programming and the programming language Simula. He also was a pioneer of participatory design and the “Scandinavian school of systems development”. http://kristennygaard.no
  • 11. 11 Nygaard, K. (1986): Program Development as a Social Activity, Invited Lecture, Proceedings of the IFIP 10th World Computer Congress, INFORMATION PROCESSING 86, Dublin. Elsevier Science Publishers, pp. 189-198. Available at: https://ojs.ruc.dk/index.php/pdc/article/view/97/89
  • 12. 12 “Informatics is the science that has as its domain information processes and related phenomena in artifacts, society and nature”. Nygaard, K. (1986): Program Development as a Social Activity, Invited Lecture, Proceedings of the IFIP 10th World Computer Congress, INFORMATION PROCESSING 86, Dublin. Elsevier Science Publishers, pp. 189-198. Available at: https://ojs.ruc.dk/index.php/pdc/article/view/97/89
  • 13. 13 No management hand-waving about valuing “user participation” Let’s get specific: who are we talking about? Nygaard, K. (1986): Program Development as a Social Activity, Invited Lecture, Proceedings of the IFIP 10th World Computer Congress, INFORMATION PROCESSING 86, Dublin. Elsevier Science Publishers, pp. 189-198. Available at: https://ojs.ruc.dk/index.php/pdc/article/view/97/89
  • 14. 14 How will such users get to “participate”? Nygaard, K. (1986): Program Development as a Social Activity, Invited Lecture, Proceedings of the IFIP 10th World Computer Congress, INFORMATION PROCESSING 86, Dublin. Elsevier Science Publishers, pp. 189-198. Available at: https://ojs.ruc.dk/index.php/pdc/article/view/97/89
  • 15. “Informatics is a distinct scientific discipline, characterised by its own concepts, methods, body of knowledge and open issues. It covers the foundations of computational structures, processes, artefacts and systems; and their software designs, their applications, and their impact on society” https://www.informatics-europe.org This laid the foundations for Informatics today…
  • 16. “Informatics concerns itself with the study of living, working and building in a digital world. Wherever technology touches people, it must be designed with ultimate care. This requires mastery of technological knowhow and a deep appreciation of the social, cultural and organizational forces at work.” https://www.informatics.uci.edu/explore/chairs-welcome This laid the foundations for Informatics today…
  • 17. 17 learning informatics Note that the terms Education(al) Informatics have been proposed, and some definitions of this are similar, albeit limited to educational institutions — I prefer a broader focus on lifelong/lifewide learning. Levy, P., Ford, N., Foster, J., Madden, A., Miller, D., Nunes, M. B., et al. (2003). Educational informatics: An emerging research agenda. Journal of Information Science, 29(4), 298-210. [reprint] Collins, J.W., and Weiner, S.A..(2010). Proposal for the creation of a subdiscipline: Education informatics. Teachers College Record 112, no. 10: 2523–2536. [reprint]
  • 18. 18 learning informatics How does this lens translate into deploying Analytics/AI for learning? Design for educator and learner agency Design socio-technical systems Design for imperfect computational models System Integrity > algorithmic accountability
  • 20. Framework @UTS for educators to co-design Analytics/AI à augment teaching practice PhD by Antonette Shibani: http://simon.buckinghamshum.net/2019/11/congratulations-dr-antonette-shibani Shibani, A., Knight, S. and Buckingham Shum, S. (2019). Contextualizable Learning Analytics Design: A Generic Model, and Writing Analytics Evaluations. Proc. 9th International Conference on Learning Analytics & Knowledge (LAK19). ACM Press, NY, pp. 210-219. DOI: https://doi.org/10.1145/3303772.3303785. Eprint: https://tinyurl.com/lak19clad Student Task Design Feedback & User Interface Features in the Data Educators Analytics/AI designers Assessment
  • 21. Acknowledgement: https://www.freevector.com/jigsaw-puzzle# Educator /Student Resources Learning Design Evaluation Analytics Capability Bundling analytics with educator resources Integrating Writing Activities with Writing Analytics Higher Education Text Analytics consortium: http://heta.io/resources UTS Orientation Portal: https://uts.edu.au/acawriter
  • 22. Automated feedback systems should enhance educators’ and students’ feedback literacy DAFFI 2020: Designing Automated Feedback for Impact 8-9 Sept. 2020 • bit.ly/daffi2020 https://cic.uts.edu.au/ensuring-automated-feedback-is-pedagogically-sound-daffi2020
  • 23. Raising data/learning analytics literacy (staff and students) UTS Open: https://open.uts.edu.au
  • 24. Disseminating innovations in Learning Analytics to the UTS community Briefings • Hands-on Training • Teaching into degree programs CIC Events archive: https://cic.uts.edu.au/events Raising data/learning analytics literacy (staff and students)
  • 25. 25 Design for educator and learner agency PhDs by Carlos Alvarez-Prieto, Vanessa Echeverria, Antonette Shibani http://simon.buckinghamshum.net/2020/09/congratulations-dr-carlos-prieto-alvarez http://simon.buckinghamshum.net/2020/05/congratulations-dr-vanessa-echeverria http://simon.buckinghamshum.net/2019/11/congratulations-dr-antonette-shibani Learning Informatics Principle 2
  • 26. Multimodal student data from simulations
  • 27. Who did what, when in a nursing simulation? Team Timeline for evidence-based debriefings Patient’s state changes Nurses 1-3 Uses a device Administers medication Personalised feedback on high performance teamwork Echeverria, V., Martinez-Maldonado, R. and Buckingham Shum, S. (2019). Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data. In Proceedings of ACM Conference on Human Factors in Computing (CHI’19). ACM: NY. Paper 39, pp. 1-16. https://doi.org/10.1145/3290605.3300269 Open Access Eprint: http://bit.ly/chi19utscic
  • 28. Co-design techniques to engage students and staff in designing multimodal learning analytics Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. (2018). Mapping Learner-Data Journeys: Evolution of a Visual Co-design Tool. Proceedings of the 30th Australian Conference on Computer-Human Interaction (OzCHI’18), Melbourne, Australia, Dec. 2018, ACM, New York, NY, USA, pp. 205–214. DOI: https://doi.org/10.1145/3292147.3292168
  • 29. 29 Toolkit available http://ladeck.utscic.edu.au LA-DECK: card-based co-design tool for LA Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. (2020). LA-DECK: A Card-Based Learning Analytics Co-Design Tool. Proc.10th International Conference on Learning Analytics and Knowledge, Frankfurt, March 2020, ACM. 10 pages. DOI: https://doi.org/10.1145/3375462.3375476
  • 30. 30 LA-DECK: card-based co-design tool for LA Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. (2020). LA-DECK: A Card-Based Learning Analytics Co-Design Tool. Proc.10th International Conference on Learning Analytics and Knowledge, Frankfurt, March 2020, ACM. 10 pages. DOI: https://doi.org/10.1145/3375462.3375476
  • 31. Automated formative feedback on reflective writing Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sándor, Á., Kitto, K., Liu, M., Mogarkar, R. & Buckingham Shum, S. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12, (1), 141-186. https://doi.org/10.17239/jowr-2020.12.01.06
  • 32. Participatory prototyping with educators to build trust in the NLP http://heta.io/how-can-writing-analytics-researchers-rapidly-codesign-feedback-with-educators Learning Analytics researchers work with academics (3 hour workshop) Goal: calibrate the parser detecting affect in reflective writing, working through sample texts Rapid prototyping with a Python notebook, then integrated into full application for further testing
  • 33. Design for imperfect computational models Kirsty Kitto, Simon Buckingham Shum, and Andrew Gibson. (2018). Embracing Imperfection in Learning Analytics. In Proceedings LAK18: International Conference on Learning Analytics and Knowledge, March 5–9, 2018, Sydney, NSW, Australia, pp.451-460. (ACM, New York, NY, USA). https://doi.org/10.1145/3170358.3170413 Learning Informatics Principle 3 (Ergonomics principle: wise allocation of function between human and machine)
  • 34. The Navajo rug “In a Navajo rug there is always an imperfection woven into the corner. And interestingly enough, it’s where “the Spirit moves in and out of the rug.” The pattern is perfect and then there’s one part of it that clearly looks like a mistake . . . Perfection is not the elimination of imperfection. […] Perfection, rather, is the ability to incorporate imperfection! […] You either incorporate imperfection, or you fall into denial.” http://exhibitions.kelsey.lsa.umich.edu/less-than-perfect/navajo.php Richard Rohr (2011). Breathing Under Water: Spirituality and the Twelve Steps. Cincinnati, OH: Franciscan Media See also: https://medium.com/bedolabs/success-through-imperfection-c3ef21cb32ed
  • 35. We must equip graduates with the distinctive qualities that will keep them in jobs that won’t be automated… …but if we want to use LA in such teaching and learning, tracking/assessing such competencies will be imperfect
  • 36. Machines will never fully model our higher order capabilities – but that can be an educational feature not a bug Reflection and metacognition are among the highest order outcomes that we aspire to cultivate in learners — the desire and skills to observe one’s thoughts, emotions and actions, and glean insights Learning Analytics in such contexts will in principle have a high degree of imperfection
  • 37. Embodied, skilled performance an important part of the learning experience is physically embodied (e.g. inspecting a forest; a social services risk assessment) Wicked problems (Horst Rittel) problems with no correct answer, and no stopping rules — even the definition of the problem is contested Socially complex performance scenarios where the outcome is emergent in nature, a function of many drivers that result in unpredictable and/or unique outcomes (e.g. a social worker client interview; conflict resolution) Learning Analytics in such contexts will in principle have a high degree of imperfection Machines will never fully model our higher order capabilities – but that can be an educational feature not a bug Can we design for this?
  • 38. Imperfect Learning Analytics à cognitive dissonance 38 Kirsty Kitto, Simon Buckingham Shum, and Andrew Gibson. (2018). Embracing Imperfection in Learning Analytics. In Proceedings LAK18: International Conference on Learning Analytics and Knowledge, March 5–9, 2018, Sydney, NSW, Australia, pp.451-460. (ACM, New York, NY, USA). https://doi.org/10.1145/3170358.3170413 “…as D’Mello and Graesser [15] demonstrate, it is when the student experiences dissonance because the analytics fail to match their expectations that they are likely to reflect on why they think the machine is wrong. We believe that this form of critical questioning is more likely to happen if the student has been given an underlying reason to be a little distrustful of the classifier.”
  • 39. 39 Gavriel Salomon, David Perkins and Tamar Globerson (1991). Partners in cognition: extending human intelligence with intelligent technologies. Educational Researcher, 20, 3, pp.il2–9. Learners’ engagement with intelligent technology should not be mindless. On the contrary we should design for… “nonautomatic, effortful and thus metacognitively guided processes”
  • 40. “Embracing Imperfection in Learning Analytics” 40 1. Robust learning design ensures that the activity involving automated feedback is meaningful whether or not the technology always works 2. Explicit encouragement — in student briefings, and in the user interface — to push back if they disagree with the feedback
  • 41. 41 System Integrity > “algorithmic accountability” Learning Informatics Principle 4
  • 42. Remind me, why should I trust this system?
  • 43. Growing public literacy around algorithmic bias and the need for accountability is to be welcomed 43 …but is there anything distinctive about algorithms for teaching and learning that shapes how we frame “accountability”?
  • 44. Stakeholders and key transitions in designing a Learning Analytics system Ethical Principles Educational/Learning Sciences researcher Learning Theory Algorithm Learning Analytics Researcher Educator Learner Learning Outcomes Educational Insights Programmer Software, Hardware User Interface Data
  • 45. Educational/Learning Sciences researcher Programmer Software, Hardware Educator Learner Ethical Principles Algorithm Learning Outcomes Learning Theory Learning Analytics Researcher User Interface Educational Insights Accountability in terms of: Computer Science Data
  • 46. Educational/Learning Sciences researcher Programmer Software, Hardware Educator Learner Ethical Principles Algorithm Learning Outcomes Learning Theory Learning Analytics Researcher User Interface Educational Insights Accountability in terms of: Data Science Data Training Data
  • 47. Educational/Learning Sciences researcher Programmer Software, Hardware Educator Learner Ethical Principles Algorithm Learning Outcomes Learning Theory Learning Analytics Researcher User Interface Educational Insights Accountability in terms of: User-Centred Design Data Design Process
  • 48. Educational/Learning Sciences researcher Programmer Software, Hardware Educator Learner Algorithm Learning Outcomes Learning Theory Learning Analytics Researcher User Interface Educational Insights Accountability in terms of: Deliberative Democracy Data Ethical Principles
  • 49. Accountability in terms of: Deliberative Democracy? “A Deliberative Mini-Public has two core features: • It is deliberative: participants reach their conclusions or recommendations after receiving information and engaging in a careful and open discussion about the issue or issues before them. • It is a mini-public: its members constitute, so far as possible, a representative subset of the wider population.” (Farrell et al, 2019) • Escobar, O. and Elstub, S. (2017), Forms of Mini-Publics: An introduction to deliberative innovations in democratic practice. The newDemocracy Foundation. • Farrell, D., Curato, N., Dryzek, J.S., Geißel, B., Grönlund, K., Marien, S., Niemeyer, S., Pilet, J.-B., Renwick, A., Rose, J., and Setälä, M. (2019), Deliberative Mini-Publics: Core Design Features. Centre for Deliberative Democracy and Global Governance working paper 2019/5. Canberra, Australia: Centre for Deliberative Democracy and Global Governance. Ethical LA Principles
  • 50. Educational/Learning Sciences researcher Programmer Software, Hardware Educator Learner Ethical Principles Algorithm Learning Outcomes Learning Theory Learning Analytics Researcher User Interface Educational Insights Accountability in terms of: Learning Sciences & Educational Technology Data
  • 51. Remind me, why should I trust this system? So we move from asking...
  • 52. Remind me, why should I trust this system? ...to asking...
  • 53. More detailed version of that analysis… 53 LASI tutorial: http://simon.buckinghamshum.net/2019/11/black-box-learning-analytics
  • 54. 54 ICLS 2018 keynote: h9p://simon.buckinghamshum.net/2018/06/icls2018-keynote Elsewhere I have argued that this constitutes a transition in education’s knowledge infrastructure and discuss different theory à analytics mappings
  • 55. 55 learning informatics How does this lens translate into deploying Analytics/AI for learning? Design for educator and learner agency Design socio-technical systems Design for imperfect computational models System Integrity > algorithmic accountability