Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014
•Download as PPSX, PDF•
1 like•639 views
This year's Association of Test Publishers (ATP) Innovations in Testing Conference focused more than ever on test security, and the Caveon team was there. We share with you not only the concepts which we presented, but also new things we learned at the conference. Caveon leaders John Fremer and Steve Addicott summarize the test security ideas and strategies from ATP and the things you can do to protect your high-stakes tests.
Recommended
Recommended
More Related Content
Similar to Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014
Similar to Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014 (20)
More from Caveon Test Security
More from Caveon Test Security (20)
Recently uploaded
Recently uploaded (20)
Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014
- 1. Caveon Webinar Series: Ten Test Security Lessons from ATP March 19, 2014 John Fremer, Ph.D. President Caveon Consulting Services Steve Addicott Vice President Caveon Test Security
- 2. Agenda for Today • Steve – Emphasis on Test Security – Highlights from some TERRIFIC Security Sessions – Online proctoring – Next Step in Exam Security • John – Exam Security Processes and Procedures – “Telling it to the Judge” – User Agreements – ATP Security Committee Briefing – Closing Keynote • Q&A
- 4. Test Security is Front Burner 4www.caveon.com
- 5. Test Security is Front Burner 5www.caveon.com
- 6. Ten Years of Test Security • Star-studded cast – Schoenig, Wollack, Cizek (and Addicott ) • Key trends explored • Important “Lessons” 6www.caveon.com
- 7. Schoenig—Test Security is a Process ACT Process: • Deter, Detect, & Decide • Supports the 3-legged stool of 7www.caveon.com
- 8. Wollack—Data Forensics Goes Mainstream • An amazing paradigm shift Past: DF detects cheating = public distrust in testing Future: DF enhances validity = increased public trust • Industry Standards for DF • Proprietary Dissemination • CSDPTF COTS • DF moving closer to test instance 8www.caveon.com
- 9. Cizek—Test Security is about Validity (not cheating) What we have said: “a statistical procedure... can never prove that, for example, teachers in a school are cheating” “such statistical criteria are never ‘proof’ of cheating” 9www.caveon.com
- 10. Cizek—Test Security is about Validity (not cheating What we can say: “There is strong reason to be concerned that these scores may not be valid. We recommend that additional information be gathered or that testing be replicated before interpreting these scores to mean...” 10www.caveon.com
- 11. “Exam Inoculation and Other Crazy Ideas to Stop Cheaters from Passing Exams” Burns, Van de Velde, and Maynes • Pirates persist • Test Design is powerful! • Most measures are reactive, so… • Inoculate exams against pirates – Trojan Horse Items – Embedded Verification Tests – Item cloning – DOMC 11www.caveon.com
- 12. Online Proctoring Is Here to Stay • ATP SC orchestrated demo-rama • Security concerns abound • Are they justified? • More research required • Test Security Standards for Online Proctoring now available – http://bit.ly/1ifSv0F – Caveon.com Resources White Papers 12www.caveon.com
- 13. Next Step in Exam Security’s Evolution Institute for Exam Security (I4ES) • An independent, non-profit association by/for Test Security professionals • www.cespcert.org 13www.caveon.com
- 14. Fremer Areas to be Covered • Multidisciplinary Approaches Workshop • “Telling it to the Judge” • Critical Importance of User Agreements • ATP Security Committee Breakfast • Closing Keynote 14www.caveon.com
- 15. Multidisciplinary Approaches Workshop • Program Manager, Investigator, Attorney, Test Security Guy • Four Group Exercises • Each Dealing with Real Challenges in Testing Programs • Common Elements across Discussions www.caveon.com 15
- 16. One Exercise – The Context • Limited staff • Limited staff expertise • Limited resources: software, money, time Problem • Efficiently allocate resources to threats based on – The risk they pose – The likelihood of finding a solution 16www.caveon.com
- 17. Activity #2 • Assess the Risks! 17www.caveon.com
- 18. The Scenario • CBT administration, 3,000 examinees. Following the administration, you analyze your data for statistical indicators of cheating. • 180 examinees were statistically flagged for similar response patterns. They did not pass at a higher rate than expected. • Of those 180, a group of 80 examinees had been flagged in previous administrations. They did not pass at a higher rate than expected. • What are your next steps in addressing this issue? 18www.caveon.com
- 19. Common Elements Across Reports • Notify Your Boss Promptly • Collect Relevant Information Based on Your Response Plan • Seek Appropriate Expertise – Legal – Psychometric – Messaging/Communications 19www.caveon.com
- 20. I Told It To The Judge, But S/He Didn’t Listen • Jennifer Semko – Baker & Mckenzie • Camille Thompson – ACT • Dennis Maynes - Caveon 20www.caveon.com
- 21. I Told It To The Judge (cont.) • Advice re testifying – Answer only what you are asked – Don’t be a smart alec – KISS Principle • Courts have been Supportive • Follow Your Policies and Procedures or Stay Home www.caveon.com 21
- 22. I Told It to The Judge - User Agreements • User Agreement is a Binding Contract • Defines Relationship • Memorializes Obligations • You will be Judged by How Well you Fulfilled Your Commitments • Must Behave Responsibly and Fairly 22www.caveon.com
- 24. ATP Security Committee • High Attendance at Breakfast • Great Deal of Productive Energy • Many Projects (Examples) – Removing Copyrighted Material from the Internet – Test Taker Rights and Responsibilities 24www.caveon.com
- 25. Closing Keynote • Hilary Mason • Big Data – Tremendously Useful • Gives Power to Many to Influence Events • It is the Effective Holders of Information who Can Wield Power • Some Problems Cut Across Fields – Unwillingness to Act on Data Alone 25www.caveon.com
- 26. Overall Impression • Sharp Increase in Awareness of Importance of Test Security to Test Program Management • Other Domains Catching up to IT in Recognition of Problems • Focus on User Agreements and Online Proctoring • Willingness to Try New Approaches 26www.caveon.com
- 27. Questions? Please type questions for our presenters in the GoToWebinar control panel on your screen. 27www.caveon.com
- 28. Thank you! John Fremer President Caveon Consulting Services 28www.caveon.com Steve Addicott Vice President Caveon Test Security Follow Caveon on twitter @caveon Check out our blog www.caveon.com/blog LinkedIn Group “Caveon Test Security”