How to Evaulate Learning Technologies
•Download as PPTX, PDF•
0 likes•186 views
This is the slide deck from my Training Magazine Network webinar about CPR, Context, Pilot, Report. CPR is an evaluation framework for learning technologies.
Recommended
More Related Content
Similar to How to Evaulate Learning Technologies
Similar to How to Evaulate Learning Technologies (20)
More from Cara North
More from Cara North (6)
Recently uploaded
Recently uploaded (20)
How to Evaulate Learning Technologies
- 1. @caranorth11 How To Evaluate Your Learning Technology Cara North
- 2. SpongeBob SquarePants is copyright of Viacom International Inc, all rights reserved
- 3. @caranorth11
- 4. @caranorth11 Share an example of learning technology that DID NOT work for your organization
- 6. CPR Creators • Cara North • Anna Leach • Natalie Gintert • Tim Nunn • Ana-Paula Correia (listed in order of authorship) North, C. A., Leach, A., Gintert, N., Nunn, T., Correia, A.-P. (2018). Evaluation of the Duolingo English Test: Implications for K-12 English Language Learners. In M. Simonson (Ed.), 40th Annual Proceedings of Association for Educational Communications and Technology: Volume 2 – Practice of Educational Communications and Technology (204-207). Bloomington, IN: Association for Educational Communications and Technology (AECT).
- 8. “identification, clarification, and application of defensible criteria to determine an evaluation object's value (worth or merit) in relation to those criteria.” Fitzpatrick, Sanders & Worthen (2011). Program Evaluation: Chapter 1 (Evaluation’s basic purpose, uses, and conceptual distinctions) What is Evaluation?
- 9. Savings
- 10. The Poster Child for Technology Evaluation
- 12. Satisfaction
- 13. CPR Context P R
- 15. @caranorth11 What do you think your organization focuses on when selecting a new learning technology?
- 18. Evaluation Matrix 18 Evaluation Questions Instruments [Evidence] Student writing sample Previous Duolingo data Duolingo English Test results Student pre-test survey Teacher pre-test survey Student post- test survey Does the Duolingo English Test categorize learners current level appropriately? X X X X X X Does the Duolingo English Test provide the instructor areas of opportunity (ie-understanding of language strength/weaknesses) for the learner to improve their English? X X X Does the Duolingo English Test delivery method (online assessment) impact the testing environment (online environment in an office setting)? X X X X How do the Duolingo English Test results compare to the teacher’s evaluations? X X X
- 19. CPR C Pilot R
- 20. Pilot Considerations Representation Evaluation criteria Environment
- 21. @caranorth11 How does your organization test learning technologies for evaluation?
- 22. Observation is your friendObservation is your friend
- 23. Don’t give them lip service, give them a K.I.S.S.
- 24. CPR C P Report
- 25. What Does your Organization Value?
- 27. Contact Information Podcast: Instructional Redesign http://www.instructionalredesign.com Twitter: @caranorth11 Skype username: caranorth11 LinkedIn: https://www.linkedin.com/in/caranorth11/ Website: Cara’s Learning Experience http://www.caranorth.com Training Learning and Development Community http://www.tldc.us Email: north.129@osu.edu Connect with me
Editor's Notes
- You go into work wanting to share this enthusiasm with your organization. You know this learning tool is the answer to ALL of the problems with your learning population and you even think it will make coffee for everyone. Alright, how can you demonstrate that this tool is the right fit? The angel on your shoulder appears and tells you that to get buy in you should do an evaluation of the tool, prove it’s worth. The devil on your other shoulder tells you how cumbersome it will be and has you think about how much time and energy it will take to evaluate it. Fear not, good reigns supreme over evil and we are here to share an approach to learning tool evaluation that is heaven sent.
- For our evaluation course, we selected to evaluate the Duolingo English Exam. We were familiar with the app so we wanted to see how this new product worked and if it could have a big impact in the English as a second language market
- Each student who took the DET has a writing sample. This writing sample is part of an English immersion exercise in the student’s English writing course. The writing sample was critical to evaluating the Duolingo English exam for several reasons. First, it provided an example of the student’s writing in the context of an assignment. The assignment has a rubric attached to it and it allowed the evaluators to see where the students are in their English skills. Students who completed the DET are already using Duolingo’s app in the classroom to take a Duolingo English Course (DEC). The Duolingo app provides students with lessons bundled in a microlearning approach. Students have the option to go through previous lessons or try new lessons in order to earn experience points (XP). When students reach a particular threshold of XP, new lessons unlock and Duolingo provides a percentage that they are fluent in a language. The course the students are enrolled in recommends Duolingo participation. However, the teacher doesn’t require a certain level of completion but to work based on their current ability levels. The pre-test survey asks students to provide self-assessment data about their own English levels. The DET results are the main performance aspect of the evaluative data being collected. After the Duolingo rater watches the video of the student taking the test, a score is assigned based on six different levels: beginner, elementary, intermediate, advanced, proficient, and expert. The levels in the pre-test survey correspond to the levels categorized in the DET. The verbiage was similar and the name of each level is the same, however, this was first time that learners had seen these categories and verbiage since they did not complete the DET. Students were given an opportunity in an open response field to explain their reasoning for choosing the category. This type of self-assessment data asked learners to categorize their ability and was not a performance test. This was critical because criterion measures typically have a performance aspect, it should not be expected that ability evaluations have a one-to-one correspondence with criterion measures (Mabe & West,1982, p. 282). In the case of this evaluation as a whole, the performance test is the Duolingo English test.