Skip to main content
https://twitter.com/Wiswijzer2/status/414055472451575808
1
Simon Buckingham Shum
Connected Intelligence Centre • University of Technology Sydney
@sbuckshum • http://utscic.edu.au • http://Simon.BuckinghamShum.net
Teaching, Assessment and Learning Analytics:
Time to Question Assumptions
University	of	Melbourne	•	Assessment	Research	Centre	&	Centre	for	the	Study	of	Higher	Education,	13th Sept.	2017
https://twitter.com/Wiswijzer2/status/414055472451575808
3
“Note:	check	the	huge	
difference	between	
knowing	and	
measuring…”
Scaleable
Precise
Quantifiable
Reprocessable
5
I was speaking at this UK ESRC symposium
http://codeactsineducation.wordpress.com
6
I asked Siri to find the conference website…
“Find code acts in
education”
7
“Find code acts in
education”
I asked Siri to find the conference website…
8
https://www.youtube.com/watch?v=Dlr4O1aEJvI&sns=em
Learning Analytics: a form of computational social science
9
Computing/
Data Sciences
Education &
Learning
Sciences
Human-Computer
Interaction
What do we mean by “Learning Analytics”?
“the measurement, collection,
analysis and reporting of data
about learners and their
contexts, for purposes of
understanding and optimizing
learning and the environments in
which it occurs”
Society for Learning Analytics Research
10
Er, isn’t this what educational
researchers have always done?
What do we mean by “Learning Analytics”?
“the process of developing
actionable insights through
problem definition and the
application of statistical models
and analysis against existing
and/or simulated future data”
UK Joint Information Systems Committee
11
For details and more definitions see http://www.laceproject.eu/faqs/learning-analytics
Er, so Learning Analytics =
statistical power tools for
educational researchers/
learning scientists, or institutional
analysts?
What do we mean by “Learning Analytics”?
“The potential of learning analytics is arguably far more
significant than as an enabler of data-intensive educational
research, exciting as this is. The new possibility is that
educators and learners — the stakeholders who constitute the
learning system studied for so long by researchers — are for
the first time able to see their own processes and progress
rendered in ways that until now were the preserve of
researchers outside the system.” (p.17)
12
Knight S. & Buckingham Shum, S. (2017). Theory and Learning Analytics. Handbook of Learning Analytics (Chapter 1).
Society for Learning Analytics Research. https://solaresearch.org/hla-17
Assumptions…
Suspicions…
Concerns…
Examples…
as provocations to conversation
13
Learning Analytics are here, but can promote different
educational futures: let’s invent good ones
14
epistemology
pedagogyassessment
Knight, S., Buckingham Shum, S. and Littleton, K. (2014). Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space.
Journal of Learning Analytics, 1, (2), pp.23-47. http://epress.lib.uts.edu.au/journals/index.php/JLA/article/download/3538/4156
learning
analytics
What kinds of learner activity do the
analytics value by tracking?
(so what is not valued?)
Do the analytics value the same things
as the assessment regime? (if not,
why will educators or learners care?)
Do learners see the analytics? What
does this say about the pedagogy? (is it
desirable that they change their
process mid-course in response to
their own and others’ feedback?)
Concerns around Learning Analytics
Schools and universities are outsourcing
their core business — e.g. resource
selection, instructional design, feedback
and grading — to black box algorithms
15
A student warning system, somewhere near you…
Student: “Being told by the LMS every time I login that I’m at
grave risk of failing is stressful. I already informed Student
Services about my disability and recent bereavement, and I’m
working with my tutor to catch up…”
A student warning system, somewhere near you…
University: “Don’t worry it’s nothing
personal. It’s just the algorithm.”
Student: “Being told by the LMS every time I login that I’m at
grave risk of failing is stressful. I already informed Student
Services about my disability and recent bereavement, and I’m
working with my tutor to catch up…”
A social network visualisation, somewhere near you…
Student: “Being picked out like this as some
sort of loner makes me feel uncomfortable…
I chat with peers all the time in the cafes.”
A social network visualisation, somewhere near you…
Student: “Being picked out like this as some
sort of loner makes me feel uncomfortable…
I chat with peers all the time in the cafes.”
University: “Don’t worry it’s nothing
personal. It’s just the algorithm.”
Algorithms are generating huge interest in the
media, policy, social justice, and academia
http://governingalgorithms.org
http://datasociety.net
Algorithmic accountability in learning?
21
http://simon.buckinghamshum.net/2016/03/algorithmic-accountability-for-learning-analytics
Concerns around Learning Analytics
LMS data often has little to do with learning —
why all the attention on those dashboards when
our most skilled educators are typically using
other platforms creatively and to great effect?
22
We can now aggregate activity from diverse platforms (with student consent)
23
?
?
Connected Learning Analytics Toolkit
K. Kitto, A. Bakharia, M. Lupton, D. Mallet, J. Banks, P. Bruza, A. Pardo, S. Buckingham Shum, S. Dawson, D. Gašević, G. Siemens, & G. Lynch, (2016). The connected
learning analytics toolkit. Proc. 6th Int. Conf. Learning Analytics & Knowledge (LAK '16). ACM, New York, NY, USA, 548-549. https://doi.org/10.1145/2883851.2883881
Student dashboard Educator dashboard
25
CLA Toolkit: Groupwork dashboard
We could use this to
grade summatively, or
as formative feedback to
provoke reflection
Kitto et al (2017). Designing for student-facing learning analytics. Australasian Journal of Educational Technology, 2017, 33(5)
Concerns around Learning Analytics
There’s far more to learning than
having students tethered to screens
26
A field exercise…
27
Posture analysis of fieldwork students
28Masaya Okada and Masahiro Tada. 2014. Formative assessment method of real-world learning by integrating heterogeneous elements of behavior, knowledge,
and the environment. Proc. 4th Int. Conf. on Learning Analytics and Knowledge. ACM, New York, NY, USA, 1-10. DOI= http://dx.doi.org/10.1145/2567574.2567579
Sensing co-located teamwork
Martinez-Maldonado, R., Power, T., Hayes, C., Abdipranoto, A., Vo, T., Axisa, C., and Buckingham Shum, S. (2017) Analytics Meet Patient Manikins: Challenges in an
Authentic Small-Group Healthcare Simulation Classroom. International Conference on Learning Analytics and Knowledge, LAK 2017, 90-94.
Tracking nurses’ movement around the patient
Martinez-Maldonado, R., Pechenizkiy, M., Power, T., Buckingham Shum, S., Hayes, C. and Axisa, C. (2017) Modelling Embodied Mobility Teamwork Strategies in a
Simulation-Based Healthcare Classroom. International Conference on User Modelling, Adaptation and Personalization, UMAP 2017,
Movement heatmaps –> sequence analysis
Martinez-Maldonado, R., Pechenizkiy, M., Power, T., Buckingham Shum, S., Hayes, C. and Axisa, C. (2017) Modelling Embodied Mobility Teamwork Strategies in a
Simulation-Based Healthcare Classroom. International Conference on User Modelling, Adaptation and Personalization, UMAP 2017,
Concerns around Learning Analytics
The vision of students being continually
prodded or reassured by AI agents is
antithetical to cultivating student qualities like
self-regulation, agency, curiosity, resilience…
32
33
Assessing learning dispositions:
Crick Learning for Resilient Agency survey (CLARA)
Deakin Crick, R., Huang, S., Ahmed Shafi, A. and Goldspink, C. (2015). Developing Resilient Agency in Learning: The Internal
Structure of Learning Power. British Journal of Educational Studies: 62, (2), 121-160. http://dx.doi.org/10.1080/00071005.2015.1006574
https://utscic.edu.au/tools/clara • http://clara.learningemergence.com
34
Structural Equation
Model underpinning
CLARA
Deakin Crick, R., Huang, S., Ahmed Shafi, A. and Goldspink, C. (2015). Developing Resilient Agency in Learning: The Internal
Structure of Learning Power. British Journal of Educational Studies: 62, (2), 121-160. http://dx.doi.org/10.1080/00071005.2015.1006574
Immediate visual analytic generated by CLARA
Feedback to Stimulate Self-Directed Change
A framework for reflection and coaching
40 item survey
Deakin Crick, R., Huang, S., Ahmed Shafi, A. and Goldspink, C. (2015). Developing Resilient Agency in Learning: The Internal
Structure of Learning Power. British Journal of Educational Studies: 62, (2), 121-160. http://dx.doi.org/10.1080/00071005.2015.1006574
Scaling CLARA in UTS
n=876 n=957
n=548 n=602
§ Approx. 3000 student profiles
§ For the 921 students with both pre-
and post-subject profiles, there
were significant positive changes
on all 8 dimensions.
§ We can also derive through cluster
analysis significantly different
cohort profiles: 4 examples
§ Next step: explore the relationships
of these self-reported profiles to
student outcomes
Student Effort (x) vs Grade (y)
from teacher observational assessments
over the semester: basis for a conversation
37
Nagy, R. (2016). Tracking and visualizing student effort: Evolution of a practical analytics tool for staff and
student engagement. Journal of Learning Analytics, 3 (2), 165–193. http://dx.doi.org/10.18608/jla.2016.32.8
UTS:CIC seminar: https://utscic.edu.au/events/niccic-redlands-school-8-june-2016
https://vimeo.com/168306314
38
From clicks to constructs in
MOOCs
Defining a C21 capability of
Crowd-Sourced
Learning
(Part of a
larger map)
Milligan, S. and Griffin, P. (2016).
Understanding learning and learning
design in MOOCs: A measurement-based
interpretation. Journal of Learning
Analytics, 3(2), 88– 115.
http://dx.doi.org/10.18608/jla.2016.32.5
39
From clicks to constructs in
MOOCs
Defining a C21 capability of
Crowd-Sourced
Learning
(Part of a
larger map)
Milligan, S. and Griffin, P. (2016).
Understanding learning and learning
design in MOOCs: A measurement-based
interpretation. Journal of Learning
Analytics, 3(2), 88– 115.
http://dx.doi.org/10.18608/jla.2016.32.5
40
From clicks to constructs in
MOOCs
Defining a C21 capability of
Crowd-Sourced
Learning
(Part of a
larger map)
Milligan, S. and Griffin, P. (2016).
Understanding learning and learning
design in MOOCs: A measurement-based
interpretation. Journal of Learning
Analytics, 3(2), 88– 115.
http://dx.doi.org/10.18608/jla.2016.32.5
41
From clicks to constructs in
MOOCs
Defining a C21 capability of
Crowd-Sourced
Learning
(Part of a
larger map)
Milligan, S. and Griffin, P. (2016).
Understanding learning and learning
design in MOOCs: A measurement-based
interpretation. Journal of Learning
Analytics, 3(2), 88– 115.
http://dx.doi.org/10.18608/jla.2016.32.5
Learning Analytics for 21st Century Competencies. (Eds.)
Buckingham Shum S. & Deakin Crick, R. (2016). Journal of
Learning Analytics (Special Section), 3(2), pp. 6-212.
http://dx.doi.org/10.18608/jla.2016.32.2
More examples
https://utscic.edu.au/lasi-asia-keynote2016
Concerns around Learning Analytics
Higher education – at its best –
teaches students to craft arguments, and
reflect deeply on how they’re developing as
learners and professionals
(Analytics seem to have nothing to say about such qualities.)
43
ANALYTICAL/CRITICAL	rhetorical	moves	in	academic	texts
BACKGROUND	KNOWLEDGE
Recent	studies	indicate	…
…	the	previously	proposed	…
…	is	universally	accepted	...	
NOVELTY
...	new	insights	provide	direct	evidence	...
...	we	suggest	a	new	...	approach	...
...	results	define	a	novel	role	...
OPEN	QUESTION/MISSING	INFORMATION
…	little	is	known	…
…	role	…	has	been	elusive
Current	data	is	insufficient	…
TREND
...	emerging	as	a	promising	approach	
Our	understanding	...	has	grown	
exponentially	...
...	growing	recognition	of	the	importance	...
CONTRASTING	IDEAS
…	unorthodox	view	resolves	…	
paradoxes	…
In	contrast	with	previous	
hypotheses	...
...	inconsistent	with	past	findings	
...
SIGNIFICANCE
studies	...	have	provided	important	
advances
Knowledge	...	is	crucial	for	...	understanding
valuable	information	...	from	studies
SURPRISE
We	have	recently	observed	...	surprisingly
We	have	identified	...	unusual
The	recent	discovery	...	suggests	intriguing	roles
SIGNALLING	AUTHOR	INTENT
The	goal	of	this	study	...
Here,	we	show	...
Altogether,	our	results	...	indicate
44
Academic Writing Analytics: feedback on
analytical/argumentative or reflective writing
Info	https://utscic.edu.au/tools/awa
46
Highlighted sentences are colour-
coded according to their broad type
Sentences with Function Keys have
more precise functions (e.g. Novelty)
CIC’s automated feedback tool: analytical writing
Info	https://utscic.edu.au/tools/awa
AWA:	ANALYTICAL	ACADEMIC	WRITING	(UTS	CIVIL	LAW)
47Knight, S., Buckingham Shum, S., Ryan, P., Sándor, Á., & Wang, X. (Forthcoming). Academic Writing Analytics for Civil Law: Participatory Design
through Academic and Student Engagement. International Journal of Artificial Intelligence in Education
AWA:	ANALYTICAL	ACADEMIC	WRITING
48
Roll over sentences with Fkeys for a
popup reminding you of their meaning
Knight, S., Buckingham Shum, S., Ryan, P., Sándor, Á., & Wang, X. (Forthcoming). Academic Writing Analytics for Civil Law: Participatory Design
through Academic and Student Engagement. International Journal of Artificial Intelligence in Education
New UI to promote re-drafting in response to feedback
Shibani, A., Knight, S., Buckingham Shum, S., & Ryan, P. (2017, Forthcoming). Design and Implementation of a Pedagogic Intervention Using Writing Analytics. In
W. Chen et al. (Eds.). Proceedings of the 25th International Conference on Computers in Education, New Zealand.
50
UTS	CIVIL	LAW	STUDENT	VIEWS
“takes	the	emotion	out	of	having	your	work	scrutinized”	respondent	12;	“it	was	
not	embarrassing	in	the	way	that	it	can	be	when	a	tutor	or	marker	gives	
feedback”	student	7	reflection	notes
“I	definitely	found	it	useful.	It	also	made	me	realise that	I	tend	to	use	bold,	
certain	language	in	making	my	point	towards	the	end	of	each	paragraph	rather	
than	up	front	at	the	beginning	(when	introducing	my	point).”	Respondent	5
“I	realise now	what	descriptive	writing	is	- the	software	had	quite	a	bit	to	say	
about	my	lack	of	justification	- also	true	- pressed	for	time	and	difficult	
circumstances	have	caused	this	for	me	in	this	instance	- good	to	see	it	sampled.”	
Respondent	9
Knight, S., Buckingham Shum, S., Ryan, P., Sándor, Á., & Wang, X. (Forthcoming). Academic Writing Analytics for Civil Law: Participatory Design
through Academic and Student Engagement. International Journal of Artificial Intelligence in Education
51
UTS	CIVIL	LAW	STUDENT	VIEWS
“it	is	possible	to	make	a	clearly	stated	point	in	an	academic	way	without	using	one	of	the	
markers	…saying	that	if	a	paper	does	not	use	specified	'signposts'	suggests	that	the	writing	
is	not	clear	and	‘academic’	(see	’tips’	on	the	results	page),	constricts	writing	style.	I	think	it	
is	possible	to	be	clear	about	your	position	without	explicitly	saying	'my	position	is…’.”	
"respondent	11
“found	that	the	tool	was	limited	in	its	ability	to	pick	up	on	summary	sentences.	It	was	able	
to	detect	phrases	such	as	‘ultimately,	this	essay	will	conclude,’	or	‘in	conclusion,’	but	the	
use	of	adverbs	such	as	‘thus,’	and	‘evidently,’	in	conclusive	statements	failed	to	be	
recognized.”…
“Another	limitation	is	that	certain	sentences,	which	were	recounts	or	mere	descriptions	
were	deemed	important,	whilst	more	substantive	parts	of	the	essay	containing	arguments	
and	original	voice	failed	to	be	detected.”	
Knight, S., Buckingham Shum, S., Ryan, P., Sándor, Á., & Wang, X. (Forthcoming). Academic Writing Analytics for Civil Law: Participatory Design
through Academic and Student Engagement. International Journal of Artificial Intelligence in Education
CIC’s automated feedback tool: reflective writing
An	early	paragraph	which	is	simply	setting	the	scene:
Info https://utscic.edu.au/tools/awa
CIC’s automated feedback tool: reflective writing
A	concluding	paragraph	moving	into	professional	reflection:
Info https://utscic.edu.au/tools/awa
https://twitter.com/Wiswijzer2/status/414055472451575808
54
Thank you!
Any assumptions
questioned/
answered?
Discussion…