Measuring the impact of learning in Moodle
•Download as PPTX, PDF•
2 likes•948 views
The document discusses measuring the impact of learning in Moodle. It outlines the value of measuring learning impact, including being able to prove what works, see where to target learning, create more effective learning programs, and prove ROI/effectiveness. Common objections to measuring impact like it being not scientific, too hard, or too expensive are addressed. The document provides practical tips for getting started with measuring impact, including determining where to start, getting buy-in, designing an experiment, collecting and analyzing data, communicating findings, and using findings to improve learning.
More Related Content
What's hot
What's hot (16)
Similar to Measuring the impact of learning in Moodle
Similar to Measuring the impact of learning in Moodle (20)
More from LEO
More from LEO (12)
Recently uploaded
Recently uploaded (20)
Measuring the impact of learning in Moodle
- 1. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Measuring the impact of learning in Moodle Peter Dobinson Solutions Architect
- 2. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Points we will cover today What is the value of measuring learning impact? Common objections How to get started Practical tips
- 3. FOLLOW US @leolearning #mootieuk17 #moodle-xapiFOLLOW US @leolearning #mootieuk17 #moodle-xapi
- 4. FOLLOW US @leolearning #mootieuk17 #moodle-xapi The value You can prove what works See where to target learning More effective learning programmes It allows you to prove your ROI/ effectiveness Learning personalisation
- 5. FOLLOW US @leolearning #mootieuk17 #moodle-xapiFOLLOW US @leolearning #mootieuk17 #moodle-xapi
- 6. FOLLOW US @leolearning #mootieuk17 #moodle-xapi It’s not scientific 1
- 7. FOLLOW US @leolearning #mootieuk17 #moodle-xapi It’s too hard 2
- 8. FOLLOW US @leolearning #mootieuk17 #moodle-xapi It’s too expensive 3
- 9. FOLLOW US @leolearning #mootieuk17 #moodle-xapiFOLLOW US @leolearning #mootieuk17 #moodle-xapi
- 10. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Work out where you want to start 1
- 11. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Get buy in from all levels of the business 2
- 12. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Design your experiment 3
- 13. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Get the data 4
- 14. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Measure the impact 5
- 15. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Communicate your findings 6
- 16. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Use your findings to improve what you do 7
- 17. FOLLOW US @leolearning #mootieuk17 #moodle-xapi DON’T PANIC!
- 18. FOLLOW US @leolearning #mootieuk17 #moodle-xapi
- 19. FOLLOW US @leolearning #mootieuk17 #moodle-xapi
- 20. FOLLOW US @leolearning #mootieuk17 #moodle-xapi The value Better budget justification For L&D Better outcomes for organisations Better outcomes for learners
- 21. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Any questions?
- 22. FOLLOW US @leolearning #mootieuk17 #moodle-xapi Thank you
Editor's Notes
- Why measure the effectiveness of your learning? I never guess. It is a capital mistake to theorise before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts. -Sir Arthur Conan Doyle, Author of Sherlock Holmes stories “In God we trust. All others must bring data.” – W. Edwards Deming, statistician, professor, author, lecturer, and consultant. Because if your not you are at best bumbling along, surviving by guessing what works, grasping tightly onto the small tidbits of information you get from your users. At worse your letting yourself be lead by your built in bias or opinions of what works, and what dosent forcing your users to do the learning you like. Either way your not striving for the best outcomes for your users and organisation.
- Promise to L&D – more budget Promise to organisations – better training Promise to learners – personalised and tailored learning
- The three main objections to Learning Analytics and my response to them
- It’s not scientific Many academics would argue that any kind of learning analytics fall short of the strict requirements of the scientific method, therefore any hypothesis not disproved by your analysis of the data are moot. My response: It’s better than doing nothing… which is what most people are doing. Think about what level of evidence will be good enough for you. As the Kirkpatricks have pointed out, a chain of evidence that would not stand up to scientific rigour may be enough to convict someone in a court of law. If a level of evidence is good enough for a court then it’s probably good enough to convince your board to invest in L&D. Good enough develops into great. Lastly as pointed out my Mike Rustici one of the leaders in this space: How many other business decisions get made with scientifically rigorous and known information? It's not many. Business is a series of judgement calls made with imperfect information See http://leolearning.com/2017/03/learning-analytics/ for more information.
- It’s too hard. All I can say is that getting the right people and tools involved makes it easy. Making a car is hard, its why we buy one off the shelf. Someone else has designed the car, bought/created the tools to do it, hired and managed the workforce needed and created the supply chain to deliver it to the dealer so you can just walk in and buy it. Get help, hire or buy in the expertise to get this going. Eventually L&D will consider analytics a skill that everyone needs like marketing do, but until then seek out the expertise out there to help.
- It’s too expensive. This again is a fair point, however later on I will point out some tools that can be used to start to build your analytics ecosystem quite a few of which are free/low cost.
- How to get started in learning analytics
- Do you want to do a small pilot using a small piece of learning. Maybe consider using an A/B approach with two distinct cohorts and approaches as well as a control, and measuring the outcomes Or do you want to start gathering as much data as you can and trying to find correlations, the big data approach
- Get buy in from all levels of the business It's important to get backing not only as high up as possible, but also with your end users Explain your not tracking, your measuring in order to improve the efficacy of your organisation and its learning, it’s not trying to show people how badly they are doing!
- Design your experiment Work out what you want to compare It's no good having learning data in isolation If you're using the traditional method work out what the goal of the learning is, and think of 3-4 measurable impacts you want it to have. With this method these need to be qualitative bits of data, IE sales figures, complains, our produced output. If you want to use a big data approach to start to identify trends then you need to get as much data as possible! We will often in this situation pass things like LMS log data, attempts status, time, question data etc. Then you can slowly start adding in your other data sources and teasing out trends.
- Get the data. Buy in’s are needed from the owners of the data. Overcome by ensuring that this need for measuring comes from as high up as possible. Dont worry too much about technically how your going to get the data. Most of the time integrating it can be quite easy, even if you use something like excel or a csv file. At this stage you can also work out how you can start to generate more learning related data than you already have. Ideally you want to start to track as much data as you can as you may find you want to utilise it in the future.
- Actually measure the impact Have you sold more courses? Built more do-dads? Had less complaints? Had more passes? Had a measurable impact on sales? Can you correlate your learning to a measurable change? “Correlation doesn't equal causation, but it's a good place to start!”
- Communicate your findings Consider how you're going make sure your organisation understands the results Consider when you want to present your findings. When you first begin to use analytics you will have few insights often as your data set grows you will be able to provide a more real time snapshot Who needs access to the all data or a dashboard? Will they be able to understand it? Will they draw the wrong conclusions from it? You are always better off presenting a report back to your senior leaders. But line managers may well want these dashboards but tailored to them. Quite often a simple report, or a description of your outcome (ie.. our best performers do these courses, or our people with the highest NPS score completed customer service training with this trainer) may be the best way to present data upwards.
- Use the data to design a better learning experience/product It’s okay to have an opinion, as long as you can back it up with data Example of AB testing in marketing Practise continual improvement. Use the data to design better learning interventions. Use a big data to find the gaps in your learning provision and then use it again to design the your response. This will be the hardest problem we have to solve in learning, but is the holy grail of what measuring learning impact, Marketing have been doing this for over 10 years now, so L&D can hopefully quickly get there!
- Top practical tips 1) How are you going to get at the data? xAPI and the LRS have given us a wonderful tool allowing us to get at learning data in all its forms. However you don't necessarily need it. Data in Excel, CSV or SQL format can be just as useful in many situations. However, you will only get so far with this approach. Most IT blockers can be easily overcome with a conversation with the right people. IE LEO/Rustici/Watershed !! 2) Unless you have a good understanding of data and analytics don’t do it on your own, get help from people who do. Often organisations will already have data teams, there are companies out there like Watershed who do this all the time. People have been trying to do this for years and failing, we now have the skills and the toolset in the industry to do this They know how to combine data sources, and often have the technology to do it automatically. They speak the right language, and can help you with any issues with IT They understand how to compare data in order to quickly and easily work out correlations They often know how to present the data back in a form that people can understand 3) You don’t need great analysis tools but it helps. You could do it in Excel (really basic statistical analysis, for a traditional experimental approach) but you will spend hours doing this, and won't be able to do big data analysis with it. For big data approaches you can use tools Envivo, Kibana, Rapidminer or R but consider the cost of your time compared to using something off the shelf like Watershed, especially if that tools to automatically help you ingest and understand data. I’d suggest only using those kind of tools if you really know what you're doing Consider as well that a tool may well allow you to use live data in the future, so you can easily present back learning dashboards to users, or even start to build learning personalisation through recommendation engines etc. Consulting and setup support will help you overcome any issues A good data analysis tool, should also be able to present the information back to those who need it
- Some of the great tools you can use to start to build a learning ecosystem that generates you more in depth learning data that you can analyse.
- Promise to L&D – more budget Promise to organisations – better training Promise to learners – personalised and tailored learning