Evidence Hub activity presentation at the LACE SoLAR Flare event, #LACEflare, at The Open University, Milton Keynes, UK, on Friday 24 October 2014.

1. Evidence Hub
activity
LACE SoLAR Flare
24 October 2014
The Open University, Milton Keynes, UK
Co-Chairs:
Doug Clow, Rebecca Ferguson, Simon Cross

2. “Big data is like teenage sex: everyone
talks about it, nobody really knows how
to do it, everyone thinks everyone else
is doing it, so everyone claims they are
doing it…”
– Dan Ariely, Facebook, 6 Jan 2013

3. “Big data is like teenage sex: everyone
talks about it, nobody really knows how
to do it, everyone thinks everyone else
is doing it, so everyone claims they are
doing it…”
– Dan Ariely, Facebook, 6 Jan 2013
… and the world of education seems
obsessed about it, but the little that
does go on is often done badly, and
leaves people disillusioned.

4. 4
Nobody seems to
know what they’re
talking about …
Let’s change that!
Juvenile boat-tailed grackles, Quiscalus major
Photo (CC)-BY-SA Andrea Westmoreland https://www.flickr.com/photos/andrea_pauline/4768171665

5. 5
What do we know,
collectively?
What do we not know?
[T]here are known knowns; there are things we
know we know. We also know there are known
unknowns; that is to say we know there are some
things we do not know. But there are also unknown
unknowns - the ones we don't know we don't know.
- Donald Rumsfeld
Photo public domain http://en.wikipedia.org/wiki/File:Donald_Rumsfeld_Tommy_Franks.jpg

6. 6
hypotheses
• claims
• research questions
• propositions
Evidence is for or against …
Photo (CC)-BY Brian Hillegas https://www.flickr.com/photos/seatbelt67/502255276

7. Hypothesis A: Learning
Learning is at the heart of learning analytics. Do we see real improvements in learning
outcomes for learners? We might be able to see patterns in learners’ data, but can we take
action based on those patterns that improves their learning? We might be able to
personalise learning based on learners’ data, but does that make any difference to how
much they learn?
This hypothesis is about improved learning outcomes: e.g. cognitive gains, improved
assessment marks, better scores on tests, attainment results.
Example positive evidence
A study showing measurable improvements in scores on a test among learners who
received study prompts from a learning analytics system compared to the usual teaching
approach.
Example negative evidence
A study showing no significant difference in assessment results before and after the
introduction of a learning analytics dashboard.
7
Learning analytics improves learning outcomes.

8. Hypothesis B: Teaching
Does learning analytics optimise the learning process? We would expect that to lead to
more efficient processes, allow resources to be better targeted, and save money and time.
We’d also expect performance metrics (other than attainment) to improve. Does learning
analytics lead to improvements in retention, completion and progression?
This hypothesis is about improvements to teaching and learning that are not direct
learning gains by the learner.
Example positive evidence
A study showing improved retention among student cohorts whose tutors were prompted
to contact at-risk learners identified by a predictive model compared to other cohorts.
Example negative evidence
A learning analytics project that increased costs but resulted in no improvements in
efficiency or performance.
8
Learning analytics improves learning support and teaching,
including retention, completion and progression.

9. Hypothesis C: Uptake
Is learning analytics a fad that will never really get off the ground at real scale? Will it ever
move beyond pilot projects and demos? If a system is deployed across a whole
organisation, do the teachers and learners actually use it?
This hypothesis looks at the level of usage of learning analytics, and is concerned with
institutional and policy perspectives.
Example positive evidence
A survey of usage of learning analytics dashboards across the university sector in one
country finds them in use in more than half of organisations.
Example negative evidence
A predictive modelling project is rolled out across a group of schools, but usage is low and
the project is discontinued.
9
Learning analytics are taken up and used widely,
including deployment at scale.

10. Hypothesis D: Ethics
Learning analytics raises many ethical issues, around privacy, transparency, surveillance,
data ownership and control, and data protection. Can these real concerns be addressed
effectively, or will they prove to be barriers?
This hypothesis is about the ‘should we’ questions, rather than the ‘can we’ ones
addressed by the other hypotheses.
Example positive evidence
An organisation develops an ethics policy for learning analytics that is warmly received by
learners and other stakeholders.
Example negative evidence
A large-scale project to gather analytics data across multiple schools is shut down because
of concerns about privacy.
10
Learning analytics is used in an ethical way.

11. LIVE DEMO

12. Your mission
• Engage with these questions
• Pool our expertise
• Inform, discuss, debate
(CC) Mike Licht on Flickr http://www.flickr.com/photos/notionscapital/4436135087/

13. Your task: build a SoLAR system
• You each have two hypotheses
• What evidence do you know in favour? Against?
– Wide definition – “this systematic review of RCTs in Science proved
it” to”I’ve heard of someone who is doing this and it seems to be Ok”
• Write on Post-Its:
– Positive = yellow, Negative = green
Photo (CC)-BY Image Editor https://www.flickr.com/photos/11304375@N07/2818891443

14. Go to your room
• Lists on the board outside
• Finish at 3.30 and come back here.
• I’ll circulate, take photos, and summarise.

15. www.laceproject.eu
@laceproject
“Evidence Hub Activity” by Doug Clow, Institute of Educational
Technology, The Open University, was presented at the LACE
SoLAR Flare, The Open University, Milton Keynes, UK, on 24
October 2014.
@dougclow
dougclow.org
doug.clow@open.ac.uk
This work was undertaken as part of the LACE Project, supported by the European Commission Seventh
Framework Programme, grant 619424.
These slides are provided under the Creative Commons Attribution Licence:
http://creativecommons.org/licenses/by/4.0/. Some images used may have different licence terms.
15