Rapid Reaction and Response Project


Published on

Presentation by Dr Ann Ooms , Kingston University and St Georges, University of London, at the "Improving Assessment and Feedback Practices in a Technology-Enhanced Teaching and Learning Environment: Theory and Practice" Event, 19th May 2010 at Kingston University. Part of the "Higher Education Academy : Evidence Based Practice Seminar Series 2010"

The presentation provides and overview of the findings from the HEA Pathfinder Project 'Rapid Reaction and Response' concerning the use of mobile classroom technologies to enhance feedback.

Published in: Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Rapid Reaction and Response Project

  1. 1. Rapid Reaction and Response - R³ The in-class use of mobile technologies to support formative assessment and feedback Dr. Ann Ooms Senior Lecturer Faculty of Health and Social Care Sciences Kingston University and St George’s University of London
  2. 2. National Student Satisfaction Survey Question 2007 NSS UK 2008 NSS UK 2007 NSS KU 2008 NSS KU 1 - 4 The teaching on my course 82 83 79 82 5 – 9 Assessment and feedback 62 64 59 64 10 – 12 Academic support 71 73 67 72 13 – 15 Organisation & management 71 73 68 73 16 – 18 Learning resources 80 81 73 77 19 - 21 Personal development 77 78 75 77 22 Overall satisfaction 81 82 78 81
  3. 3. Rationale <ul><li>In-class use of mobile technologies </li></ul>Students  identify misunderstandings and weaknesses  focus their learning activities and efforts  diminish misunderstandings Educators  identify students’ misconceptions, challenges  information regarding teaching practices  adapt teaching practices Immediate feedback
  4. 4. Technologies to support rapid feedback <ul><li>Interactive Blackboard Tools </li></ul><ul><ul><li>Blog </li></ul></ul><ul><ul><li>Wiki </li></ul></ul><ul><li>Electronic Voting Systems </li></ul><ul><li>Tablet PC’s </li></ul><ul><li>Interactive tablets </li></ul><ul><li>Mobile phones (EduTXT Texting Service) </li></ul>
  5. 5. Electronic Voting Systems
  6. 6. Tablet PC
  7. 7. Interactive Tablets
  8. 8. EduTXT Texting Service
  9. 9. Rapid Reaction and Response Project <ul><li>HEA Funded Research Project : £42,600 </li></ul><ul><li>Aim: To examine the effective use of in-class mobile technologies to enhance formative assessment and feedback </li></ul>
  10. 10. Research Questions <ul><li>Technology </li></ul><ul><ul><li>Under which conditions can each of the technologies be effectively and efficiently used for formative assessment and feedback? </li></ul></ul>
  11. 11. Research Questions <ul><li>Learning </li></ul><ul><ul><li>What is the impact of the in-class use of mobile technologies for formative assessment and feedback on: </li></ul></ul><ul><ul><ul><li>Students’ attitudes toward the module? </li></ul></ul></ul><ul><ul><ul><li>On students’ conceptual understanding? </li></ul></ul></ul><ul><ul><ul><li>On students’ test results? </li></ul></ul></ul>
  12. 12. Research Questions <ul><li>Teaching </li></ul><ul><ul><li>What is the impact of the R 3 project on: </li></ul></ul><ul><ul><ul><li>Teaching practices (and will they be sustained?) </li></ul></ul></ul><ul><ul><ul><li>Assessment practices (and will they be sustained?) </li></ul></ul></ul><ul><ul><ul><li>What indicators are there of institutional commitment? </li></ul></ul></ul>
  13. 13. Research Methodology <ul><li>Mixed methods methodology </li></ul><ul><ul><li>Student data </li></ul></ul><ul><ul><ul><li>Questionnaires </li></ul></ul></ul><ul><ul><ul><li>Attendance records </li></ul></ul></ul><ul><ul><li>Participant data </li></ul></ul><ul><ul><ul><li>Questionnaires </li></ul></ul></ul><ul><ul><ul><li>Interviews </li></ul></ul></ul><ul><ul><ul><li>Reflective journals </li></ul></ul></ul>
  14. 14. Lecturers Faculty Lecturers FADA 2 FASS 1 Business and Law 2 CISM 2 Engineering 2 HSCS 2 Science 2 13
  15. 15. Support Provided to Lecturers <ul><li>Workshop on assessment and item-writing (Bob Rotherham) </li></ul><ul><li>Workshops on use of the technologies </li></ul><ul><li>Support from two mentors </li></ul><ul><li>Peer support meetings (with mentors) </li></ul><ul><li>Support from e-assessment developer </li></ul><ul><li>Group meetings (reflection and additional workshops) </li></ul>
  16. 16. Data from Lecturers
  17. 17. Use of Technology Technology Number of lecturers Electronic Voting System 9 Tablet PC 5 BlackBoard Content System 2 SchoolPad Interactive Pads 1 Text Messaging 1 i-Pods 2
  18. 18. Effectiveness of technologies <ul><li>Rank order of effectiveness of technologies per interaction and feedback pattern </li></ul>Technology L-S S-L S-S Electronic Voting System 1 1 3 Tablet PC 3 1 1 BlackBoard Content System 2 2 2 SchoolPad Interactive Pads Text Messaging 2
  19. 19. How satisfied are your students in terms of formative assessment activities you provided? Dissatisfied Somewhat dissatisfied Somewhat satisfied Satisfied Pre 3 15 3 Post 0 10 9
  20. 20. How satisfied are your students in terms of feedback they received? Dissatisfied Somewhat dissatisfied Somewhat satisfied Satisfied Pre 1 11 6 3 Post 6 13
  21. 21. How satisfied are you in terms of your ability to provide feedback to your students? Dissatisfied Somewhat dissatisfied Somewhat satisfied Satisfied Pre 9 11 2 Post 1 12 6
  22. 22. Impact of project No impact Minor impact Impact Major impact Mean Students’ enjoyment of classroom sessions 0 6 12 2 3.67 Students’ attendance 7 5 6 1 2.22 Students’ engagement 1 2 14 3 4.00 Students’ conceptual understanding 3 5 12 0 3.00 Students’ test results 6 9 4 0 1.78
  23. 23. Impact of project No impact Minor impact Impact Major impact Mean Overall teaching experience 0 3 15 2 4.00 Information you gained about your students’ misunderstandings 0 2 12 5 4.22 Information you gained about your teaching practices 0 1 16 2 4.22 Informed your reflection on your teaching practices 1 1 15 1 4.00
  24. 24. Positive impact of the use of mobile technologies on the interaction and feedback patterns No impact Minor impact Impact Major impact Mean M1 L – S 1 1 7 1 1.80 S – L 9 1 2.10 S – S 1 3 6 1.50 M2 L – S 1 2 3 4 2.00 S – L 7 3 2.30 S – S 2 6 1 1.89
  25. 25. To what extent do you feel the use of mobile technologies enhanced your teaching
  26. 26. Learning curve <ul><li>“ I could have used it more. I think maybe in the future when I am more confident with it, probably.” </li></ul><ul><li>“ I did the right thing by not trying to do too much, and gauging what’s comfortable for me and the students.” </li></ul><ul><li>“ I only used it at the beginning and end of the lecture. I did not use it continuously throughout and that probably might be my next aim.” </li></ul><ul><li>“ I need to think deeper away from the technology and what I am doing.” </li></ul>
  27. 27. Benefits <ul><li>Students were more engaged </li></ul><ul><ul><li>“ I was able to engage students who would not normally participate” </li></ul></ul><ul><ul><li>“ Students responded quicker than they normally do when I ask them a question” </li></ul></ul><ul><ul><li>“ There was a real ‘buzz’ in the room, with students engaging with each other.” </li></ul></ul><ul><ul><li>“ I liked the way they worked as teams and they did too!” </li></ul></ul><ul><ul><li>“ Students were generally excited at using the pads.” </li></ul></ul>
  28. 28. Benefits <ul><li>Gained insights on students’ understanding </li></ul><ul><ul><li>“ It made the students more aware of what they didn’t know, rather than what they did know.” </li></ul></ul><ul><ul><li>“ I think the real time feedback and the anonymity help to highlight some of the weaknesses or the strengths.” </li></ul></ul><ul><ul><li>“ When I see a third of them don’t actually answer the question correctly, then I think either ‘Well they are not good enough students’ or ‘I need to explain it in a different way’, or a mixture of both.” </li></ul></ul><ul><ul><li>“ Reassured me that my verbal explanations were being understood.” </li></ul></ul>
  29. 29. Benefits <ul><li>Lecturers gained insights on their teaching </li></ul><ul><ul><li>“ It made me more aware to interact with the students in the class” </li></ul></ul><ul><ul><li>“ I am understanding more about what the students understand so I can act on that” </li></ul></ul><ul><ul><li>“ I was disappointed by their responses, they were lower that I had anticipated” </li></ul></ul><ul><ul><li>“ The technology allowed me to identify areas of concept development that needed to emphasised and to improve my own input for future lectures.” </li></ul></ul>
  30. 30. Challenges <ul><li>Access to the technologies </li></ul><ul><ul><li>“ Collecting and returning the equipment from other campuses is a bit time consuming.” </li></ul></ul><ul><li>Unreliability of the technologies </li></ul><ul><ul><li>“ Students did get impatient with the failure of the wireless network.” </li></ul></ul><ul><li>Time it takes to set it up in class </li></ul><ul><ul><li>“ Allow time to test the technology before the students arrive.” </li></ul></ul><ul><ul><li>“ Allow 10 minutes to set up.” </li></ul></ul><ul><ul><li>“ Have the technology pre-installed in some classrooms to avoid long set up time.” </li></ul></ul>
  31. 31. Challenges <ul><li>Time it takes in class </li></ul><ul><ul><li>“ I ended up about a lecture behind so that is about 10%.” </li></ul></ul><ul><ul><li>“ Allow 5 minutes minimum per voting slide.” </li></ul></ul><ul><li>Time to redesign lecture </li></ul><ul><ul><li>“ To use it effectively you need to integrate it with your lecture, and not just as an add-on and in order to do that I need time to redesign the lecture.” </li></ul></ul>
  32. 32. Data from Students
  33. 33. Impact of the in-classroom use of technologies n Disagree Somewhat disagree Somewhat agree Agree Made it more enjoyable to attend the classroom sessions 455 3.7 7.0 40.7 48.6 Made the classroom sessions more interactive 454 3.1 6.4 32.6 57.9 Had a positive impact on my motivation to study for this module 452 6.0 19.2 43.1 31.6
  34. 34. Impact of the in classroom use of technologies n Disagree Somewhat disagree Somewhat agree Agree Was useful for feedback on my understanding 452 3.8 17.1 40.9 38.2 Provided information about my misunderstanding of certain components 455 7.0 20.0 44.2 28.8 Had a positive impact on my understanding of the material taught 451 3.8 14.9 47.7 33.7 Assisted me in focusing my study efforts 450 7.3 23.1 38.4 31.1
  35. 35. Descriptive Statistics Question Mean More interactive 2.45 More enjoyable 2.34 Feedback on understanding 2.14 Impact on motivation 2.00 Impact on understanding 2.11 Info about misunderstandings 1.95 Focusing study efforts 1.93
  36. 36. Overall, the use of the new technologies was a positive experience for me as a student n = 456 Percent Disagree 3.5 Somewhat disagree 9.2 Somewhat agree 36.8 Agree 50.4
  37. 37. I would advise my lecturer to keep using new technologies in his/her modules n = 456 Percent Disagree 2.6 Somewhat disagree 5.7 Somewhat agree 31.8 Agree 59.9
  38. 38. I would like other lecturers to used new technologies in their modules n = 452 Percent Disagree 2.9 Somewhat disagree 8.6 Somewhat agree 33.0 Agree 55.5
  39. 39. Open-ended Questions: Positive Responses <ul><li>“ The clickers made things more “fun” and made us involved. Keep them!” </li></ul><ul><li>“ They were fun to draw on.” </li></ul><ul><li>“ Innovative and enjoyable to use.” </li></ul><ul><li>“ For less motivated students it gave them an opportunity to get involved instead of just sitting there.” </li></ul><ul><li>“ Areas of weakness and strength are highlighted.” </li></ul><ul><li>“ It makes the lecture very engaging and interesting.” </li></ul><ul><li>“ It would be nicer if they did it in every module.” </li></ul><ul><li>“ Using the tablets for group work ensured that you had to do the work because there was a chance that it was going to be shown on screen for everyone to see. That was good.” </li></ul>
  40. 40. Open-ended Questions: Suggestions for improving <ul><li>“ Use them more regularly” </li></ul><ul><li>“ Have the technology pre-installed” </li></ul><ul><li>“ More clickers so that everyone can have one” </li></ul><ul><li>“ Use in seminars” </li></ul><ul><li>“ Useful for revision lectures” </li></ul><ul><li>“ Make sure they all have batteries” </li></ul><ul><li>“ More questions and more time explaining the answers” </li></ul><ul><li>“ Increase the reliability of the technologies” </li></ul>
  41. 41. Sustainability - Dissemination <ul><li>Seven lecturers became mentors </li></ul><ul><li>One mentor became master-mentor </li></ul><ul><li>Conference presentations </li></ul><ul><li>Publications: in progress </li></ul><ul><ul><li>Two-day writing retreat in April 2009 </li></ul></ul><ul><ul><ul><li>7 participants </li></ul></ul></ul><ul><ul><ul><li>1 mentor </li></ul></ul></ul><ul><ul><ul><li>Management team </li></ul></ul></ul><ul><ul><ul><li>Dr Martin Oliver (Institute of Education) </li></ul></ul></ul><ul><ul><ul><li>Dr Jane Seale (University of Southampton) </li></ul></ul></ul>
  42. 42. Contact Information <ul><li>Ann Ooms: [email_address] </li></ul><ul><li>R 3 Web site: </li></ul><ul><ul><li>http://staffnet.kingston.ac.uk/~ku36708/RRR </li></ul></ul>