Assessment Analytics - EUNIS 2015 E-Learning Task Force Workshop
•Download as PPTX, PDF•
3 likes•1,313 views
This presentation is to introduce a discussion session at the 2015 EUNIS Congress workshop session of the E-Learning Task Force. The LACE Project is very briefly introduced, followed by an explanation of the presenter's view of learning analytics and a critique of some common themes. Assessment Analytics is presented as an antithesis to these themes and an assessment lifecycle model (used in the Jisc Electronic Management of Assessment Programme) is used to outline some ways in which assessment analytics can be realised, as stimulus for discussion.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Assessment Analytics - EUNIS 2015 E-Learning Task Force Workshop
Similar to Assessment Analytics - EUNIS 2015 E-Learning Task Force Workshop (20)
More from LACE Project
More from LACE Project (20)
Recently uploaded
Recently uploaded (20)
Assessment Analytics - EUNIS 2015 E-Learning Task Force Workshop
- 1. Assessment Analytics Adam Cooper, Cetis EUNIS E-Learning Task Force Workshop, Abertay University, Dundee, June 9th 2015 #laceproject
- 2. Agenda for this Session Brief outline of LACE Project Perspectives on Learning Analytics A critical eye Some ideas about Assessment Analytics Group discussion and feedback 2
- 3. Quick Overview of LACE 3
- 4. LACE Goals and objectives 4 • Objective 1 – Promote knowledge creation and exchange • Objective 2 – Increase the evidence base • Objective 3 – Contribute to the definition of future directions • Objective 4 – Build consensus on inter- operability and data sharing
- 5. Learning Analytics Community Exchange Project 5 Suppliers Entrepreneurs Teachers Learners Researchers Policy-makers Library Academic managers Institutional managers IT staff Student services
- 6. Who we are (47 Ass. Partners) 6 LACE Network LACE Consortium
- 7. A Critical View of Dominant Themes (in Learning Analytics) 7
- 8. 8 Dear EUNIS, The data here is lovely and we’ve had such an adventure with the analytics. We have a new data warehouse and a cool dashboard and can now predict which students will fail witk 80% accuracy. Wish you were here, love Big D EUNIS, Abertay University. Dundee, Scotland
- 9. What do I Mean “Learning Analytics”? Analytics is the process of developing actionable insights through problem definition and the application of statistical models and analysis against existing and/or simulated future data. Adam Cooper, Cetis Analytics Series Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs. Society for Learning Analytics Research I put primarily management matters out of scope. e.g. efficiency 9
- 10. The Three Themes Are: • Data Warehouses and Big Data • Dashboards and Fancy Visualisations • Predictive Analytics for Retention and “Intervention” Systems A Critical View! 10
- 11. Data Warehouses and Big Data OK, we need to manage data, but… 11
- 12. 12 Dashboards and Fancy Visualisations
- 13. Predictive Analytics, Retention & “Intervention” Technically sophisticated… but sometimes opaque 13
- 14. I Think These Are Marginalised: • What we can get from Small Data • What can be achieved with “intermediate technology” • Simple visualisations in a context • Understanding without prediction • Relevance to all learners • A strong foundation in [pedagogic] practice Assessment Analytics can be all of these! 14
- 15. What do I Mean “Assessment Analytics”? • Almost a subset of Learning Analytics Analytics is the process of developing actionable insights through problem definition and the application of statistical models and analysis against existing and/or simulated future data. Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs. • In which the data derives from assessment or • The “environment” is assessment PLUS • Extending the forms of assessment (including summative-only) 15
- 16. Time to Un-pack! Data about... • Learner activity • Learner performance • Assessment quality • Teacher activity • Demographic and situational facts To guide... • Learner • Curriculum design – “content”/instruction – Structure/sequencing – Assessment – Pedagogic style • Reflective practice • ... 16 Analytics used by... • Learner • Teacher
- 17. Broad Areas for Assessment Analytics (Some Ideas) 17
- 20. 20 From “Case Study: Acting on Assessment Analytics” by Sheila MacNeill and Dr Cath Ellis (Cetis Analytics Series Vol. 2, No. 2)
- 23. Classical Test Theory Mostly useful for objective testing. Well-established and relatively simple. Can quantify: • Test reliability* • Inter-rater consistency • Item difficulty • Discriminating power • Learner misconception And help to identify: • Errors and badly-designed items 23 * - validity is something different (is the measuring what we want to know)
- 24. More Authentic Assessment 24 Mehrnoosh Vahdat
- 25. Discussion Questions • Where do you think data analytics could be used to enhance the cycle for conventional assessment? • From the perspective of teachers and learners, which are the most attractive points in the lifecycle to adopt assessment analytics? • For which points would fine-grained eAssessment activity data help? • What new(er) forms of assessment can we introduce with analytics, and what would the lifecycle look like for them? Discuss in groups. Choose one or more of these questions, or pose your own. Feedback. 25
- 27. Some Resources I Recommend Watch Cath Ellis! https://vimeo.com/85331242 Cetis Analytics Series http://publications.cetis.org.uk/c/analytics “Understanding Assessment Results” – Questionmark Analytics, Austin Fossey (2014 Users Conference) https://www.questionmark.com/us/conference/Documents/2014%20Handouts/2014_bp_results.pdf 27
- 28. “Assessment Analytics” by Adam Cooper, Cetis, was presented at the EUNIS E-Learning Task Force Workshop on Electronic Management of Assessment and Assessment Analytics held at Abertay University in Dundee on June 9th 2015. adam@cetis.org.uk This work was undertaken as part of the LACE Project, supported by the European Commission Seventh Framework Programme, grant 619424. These slides are provided under the Creative Commons Attribution Licence: http://creativecommons.org/licenses/by/4.0/. Some images used may have different licence terms. www.laceproject.eu @laceproject 28 Come and talk with me
- 29. Credits Data warehouse overview – public domain – Hhultgren via wikimedia commons http://commons.wikimedia.org/wiki/File:Data_warehouse_overview.JPG Big Data: Moving Parts – CC BY Dion Hinchcliffe https://www.flickr.com/photos/dionh/7550578346 Learn Course At-a-Glance – Blackboard Analytics for Learn http://www.blackboard.com/Platforms/Analytics/Products/Blackboard-Analytics-for-Learn.aspx IBM Coremetircs Explore http://www-03.ibm.com/software/products/en/digital-analytics Arrivals – CC BY T. Gregorius via wikimedia commons http://commons.wikimedia.org/wiki/File:Big_data_cartoon_t_gregorius.jpg Special Intervention Unit – CC BY Hexogen Via wikimedia commons http://commons.wikimedia.org/wiki/File:Special_Intervention_Unit-Kosovo.jpg Small is Beautiful – (c) Blond and Briggs Via wikimedia commons https://en.wikipedia.org/wiki/Small_Is_Beautiful#/media/File:SmallIsBeautiful1973.jpg DEEDS screenshot – from DEEDS site http://www.esng.dibe.unige.it/deeds/ Discrimination/Difficulty Chart from “Understanding Assessment Results” 29
Editor's Notes
- Objective 1 – Promote knowledge creation and exchange LACE’s response is a range of activities designed to actively integrate communities that are conducting research, early practitioner adopters and those who are building first-generation commercial or open-source software. This integration is used to stimulate creativity and accelerate the identification of viable and effective solutions to real problems, and hence to drive both current research and technology transfer. • Objective 2 – Increase the evidence base LACE’s response is to create and curate a knowledge base of evidence in the LACE Evidence Hub. This captures evidence for the effectiveness of various tools and techniques. It also takes into account the range of contexts in which LA and EDM are applied, and the different objectives motivating their use. The Evidence Hub is designed to accelerate beneficial uptake of LA and EDM as well as to indicate where research, experimentation or piloting are necessary to test unsubstantiated claims. • Objective 3 – Contribute to the definition of future directions LACE aims to combine participatory workshops with structured methods to explore future directions. Plausible futures for LA and EDM are captured from the imagination of people we engage with and communicated through the medium of scenarios to bring them to life in a believable setting for a wider audience. Studies assess the feasibility and desirability of achieving future states, and the difference of perception among stakeholders, thus informing research and policy agendas. -> ETHICS & Privacy • Objective 4 – Build consensus on interoperability and data sharing LACE’s response is to establish the current state of LA and EDM interoperability and to clearly identify gaps. This includes the feasibility of increased data sharing for operational and research purposes in addition to the information models necessary to make use of exchanged data. All four objectives have been
- Add numbers for key indicators to slides