This presentation is to introduce a discussion session at the 2015 EUNIS Congress workshop session of the E-Learning Task Force. The LACE Project is very briefly introduced, followed by an explanation of the presenter's view of learning analytics and a critique of some common themes. Assessment Analytics is presented as an antithesis to these themes and an assessment lifecycle model (used in the Jisc Electronic Management of Assessment Programme) is used to outline some ways in which assessment analytics can be realised, as stimulus for discussion.
2. Agenda for this Session
Brief outline of LACE Project
Perspectives on Learning Analytics
A critical eye
Some ideas about Assessment Analytics
Group discussion and feedback
2
4. LACE Goals and objectives
4
• Objective 1 – Promote
knowledge creation and
exchange
• Objective 2 – Increase the
evidence base
• Objective 3 – Contribute to
the definition of future
directions
• Objective 4 – Build
consensus on inter-
operability and data
sharing
8. 8
Dear EUNIS,
The data here is lovely and
we’ve had such an adventure with
the analytics. We have a new data
warehouse and a cool dashboard and
can now predict which students will
fail witk 80% accuracy.
Wish you were here,
love Big D
EUNIS,
Abertay University.
Dundee,
Scotland
9. What do I Mean “Learning Analytics”?
Analytics is the process of developing actionable insights through
problem definition and the application of statistical models and
analysis against existing and/or simulated future data.
Adam Cooper, Cetis Analytics Series
Learning analytics is the measurement, collection, analysis and
reporting of data about learners and their contexts, for purposes of
understanding and optimizing learning and the environments in
which it occurs.
Society for Learning Analytics Research
I put primarily management matters out of scope. e.g. efficiency
9
10. The Three Themes Are:
• Data Warehouses and Big Data
• Dashboards and Fancy Visualisations
• Predictive Analytics for Retention and “Intervention” Systems
A Critical View!
10
14. I Think These Are Marginalised:
• What we can get from Small Data
• What can be achieved with “intermediate technology”
• Simple visualisations in a context
• Understanding without prediction
• Relevance to all learners
• A strong foundation in [pedagogic] practice
Assessment Analytics can be all of these!
14
15. What do I Mean “Assessment Analytics”?
• Almost a subset of Learning Analytics
Analytics is the process of developing actionable insights through problem
definition and the application of statistical models and analysis against existing
and/or simulated future data.
Learning analytics is the measurement, collection, analysis and reporting of data
about learners and their contexts, for purposes of understanding and optimizing
learning and the environments in which it occurs.
• In which the data derives from assessment
or
• The “environment” is assessment
PLUS
• Extending the forms of assessment (including summative-only)
15
16. Time to Un-pack!
Data about...
• Learner activity
• Learner performance
• Assessment quality
• Teacher activity
• Demographic and situational
facts
To guide...
• Learner
• Curriculum design
– “content”/instruction
– Structure/sequencing
– Assessment
– Pedagogic style
• Reflective practice
• ...
16
Analytics used by...
• Learner
• Teacher
23. Classical Test Theory
Mostly useful for objective testing.
Well-established and relatively simple.
Can quantify:
• Test reliability*
• Inter-rater consistency
• Item difficulty
• Discriminating power
• Learner misconception
And help to identify:
• Errors and
badly-designed items
23
* - validity is something different (is the measuring what we want to know)
25. Discussion Questions
• Where do you think data analytics could be used to enhance the
cycle for conventional assessment?
• From the perspective of teachers and learners, which are the
most attractive points in the lifecycle to adopt assessment
analytics?
• For which points would fine-grained eAssessment activity data
help?
• What new(er) forms of assessment can we introduce with
analytics, and what would the lifecycle look like for them?
Discuss in groups. Choose one or more of these questions, or pose
your own. Feedback.
25
27. Some Resources I Recommend
Watch Cath Ellis!
https://vimeo.com/85331242
Cetis Analytics Series
http://publications.cetis.org.uk/c/analytics
“Understanding Assessment Results” – Questionmark Analytics, Austin Fossey (2014 Users
Conference)
https://www.questionmark.com/us/conference/Documents/2014%20Handouts/2014_bp_results.pdf
27
28. “Assessment Analytics” by Adam Cooper, Cetis, was presented at
the EUNIS E-Learning Task Force Workshop on Electronic
Management of Assessment and Assessment Analytics held at
Abertay University in Dundee on June 9th 2015.
adam@cetis.org.uk
This work was undertaken as part of the LACE Project, supported by the European Commission Seventh
Framework Programme, grant 619424.
These slides are provided under the Creative Commons Attribution Licence:
http://creativecommons.org/licenses/by/4.0/. Some images used may have different licence terms.
www.laceproject.eu
@laceproject
28
Come and talk with
me
29. Credits
Data warehouse overview – public domain – Hhultgren
via wikimedia commons http://commons.wikimedia.org/wiki/File:Data_warehouse_overview.JPG
Big Data: Moving Parts – CC BY Dion Hinchcliffe
https://www.flickr.com/photos/dionh/7550578346
Learn Course At-a-Glance – Blackboard Analytics for Learn
http://www.blackboard.com/Platforms/Analytics/Products/Blackboard-Analytics-for-Learn.aspx
IBM Coremetircs Explore
http://www-03.ibm.com/software/products/en/digital-analytics
Arrivals – CC BY T. Gregorius
via wikimedia commons http://commons.wikimedia.org/wiki/File:Big_data_cartoon_t_gregorius.jpg
Special Intervention Unit – CC BY Hexogen
Via wikimedia commons http://commons.wikimedia.org/wiki/File:Special_Intervention_Unit-Kosovo.jpg
Small is Beautiful – (c) Blond and Briggs
Via wikimedia commons https://en.wikipedia.org/wiki/Small_Is_Beautiful#/media/File:SmallIsBeautiful1973.jpg
DEEDS screenshot – from DEEDS site
http://www.esng.dibe.unige.it/deeds/
Discrimination/Difficulty Chart from “Understanding Assessment Results”
29
Editor's Notes
Objective 1 – Promote knowledge creation and exchange
LACE’s response is a range of activities designed to actively integrate communities that are
conducting research, early practitioner adopters and those who are building first-generation
commercial or open-source software. This integration is used to stimulate creativity and
accelerate the identification of viable and effective solutions to real problems, and hence to
drive both current research and technology transfer.
• Objective 2 – Increase the evidence base
LACE’s response is to create and curate a knowledge base of evidence in the LACE
Evidence Hub. This captures evidence for the effectiveness of various tools and
techniques. It also takes into account the range of contexts in which LA and EDM are
applied, and the different objectives motivating their use. The Evidence Hub is designed to
accelerate beneficial uptake of LA and EDM as well as to indicate where research,
experimentation or piloting are necessary to test unsubstantiated claims.
• Objective 3 – Contribute to the definition of future directions
LACE aims to combine participatory workshops with structured methods to explore future
directions. Plausible futures for LA and EDM are captured from the imagination of people
we engage with and communicated through the medium of scenarios to bring them to life in
a believable setting for a wider audience. Studies assess the feasibility and desirability of
achieving future states, and the difference of perception among stakeholders, thus
informing research and policy agendas. -> ETHICS & Privacy
• Objective 4 – Build consensus on interoperability and data sharing
LACE’s response is to establish the current state of LA and EDM interoperability and to
clearly identify gaps. This includes the feasibility of increased data sharing for operational
and research purposes in addition to the information models necessary to make use of
exchanged data.
All four objectives have been