Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

"Its on the exam" - Affecting student engagement through crowd-sourced assessment


Published on

Costello, E., Brown, M., Brunton, J., & Delaney, L. (2015). It's on the exam! Affecting student engagement through crowd-sourced assessment. Paper at Shaping the Future of Learning Together, Association for Learning Technology Annual Conference (ALT-C), Manchester, 10th September.

Published in: Education
  • Be the first to comment

"Its on the exam" - Affecting student engagement through crowd-sourced assessment

  1. 1. “It’s on the exam!”: Affecting Student Engagement Through Crowd-Sourced Assessment Eamon Costello @eam0 Mark Brown James Brunton Lorraine Delaney Open Education, National Institute for Digital Learning, Dublin City University Alt Annual Conference 2015, September 10th, Manchester
  2. 2. Backstory
  3. 3. Learning online
  4. 4. Assessment & Feedback
  5. 5. Some Key Principles in Learning Programing Feedback Reflection Practice Kahneman, D. (2011). Thinking, fast and slow. Macmillan. Chicago Wing, J. M. (2006). Computational thinking. Communications of the ACM,49(3), 33-35. Chicago
  6. 6. How can we increase engagement? and create a virtuous practice circle that includes reflection and feedback
  7. 7. Designing Tasks for Peer Assessment • Create Multiple Choice Questions on PHP Web programming topics according to question creation criteria • Answer questions • Provide feedback • Rate your peers • Required: number of questions to – create, – answer, – and comment-on/rate
  8. 8. How to ask • Purchase, H., Hamer, J., Denny, P. and Luxton-Reilly, A. (2010) The quality of a PeerWise MCQ repository. IN: Proceedings of the Twelfth Australasian Conference on Computing Education-Volume 103. Australian Computer Society, Inc. • Tarrant, M., Knierim, A., Hayes, S. K. and Ware, J. (2006) The frequency of item writing flaws in multiple- choice questions used in high stakes nursing assessments. Nurse Educ.Today. vol. 26, no. 8, pp 662-671. Many educators are not good question authors (Tarrant et. al., 2006) But well guided students may be (Purchase et. al., 2010) Student-friendly guide to best practice in authoring MCQ questions: – The question is clearly stated; – The question is error free; – The distractors (incorrect answers) are feasible; – The accompanying explanation is good; – the specified answer is correct.
  9. 9. What frequency is indicated by each of the following words? • associated with • can • characteristically • rarely • frequently • may • never • always
  10. 10. What frequency is indicated by each of the following words? (n = 70) • associated with • can • characteristically • rarely • frequently • may • never – 6 said 99% of time, 98% of time, 3 said 80% of time! • always – 4 said 1%, 1 said 5%!
  11. 11. Results
  12. 12. Requirement: Create 5 questions • 56% of the 27 students created more than 5 questions. Engagement 0 5 10 1 5 9 13 17 21 25 Questions Created per Student Required
  13. 13. Requirement: Answer 7 questions • 85% of the 27 students answered more than 7 questions. Engagement 7 0 50 100 1 5 9 13 17 21 25 Questions Answered per Student Required
  14. 14. Engagement Denny, P. (2013). The effect of virtual achievements on student engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 763- 772). ACM. Chicago Large-scale (n > 1000) randomized, controlled experiment (Denny, 2013) found a significant positive effect: • on the quantity of student contributions • the period of time over which students engaged
  15. 15. Student Feedback
  16. 16. Student Feedback “Peerwise depends on the knowledge of the person asking the questions. As a secondary teacher a lot of the people asking questions didn't balance the style of the question with the type of information required. People made questions unclear by trying to make them too hard. Also what could be done is if the people have two write questions in the future. Do so as they have to write 2 easy, 2 harder and one in depth question. This will allow them to develop their skills. It was an interesting experience and a nice break from coding. ”
  17. 17. Crowd-Sourced Assessment
  18. 18. Lessons Learned • Peerwise very easy to use for students and well received • Promotes engagement • Requires time thought to implement • Tip: Start simply and iterate
  19. 19. Questions? More about Peerwise: Me: @eam0