Caveon Webinar Series:
Ten Test Security
Lessons from ATP
March 19, 2014
John Fremer, Ph.D.
President
Caveon Consulting Se...
Agenda for Today
• Steve
– Emphasis on Test Security
– Highlights from some TERRIFIC Security
Sessions
– Online proctoring...
Test Security is…..
3www.caveon.com
Test Security is Front Burner
4www.caveon.com
Test Security is Front Burner
5www.caveon.com
Ten Years of Test Security
• Star-studded cast
– Schoenig, Wollack, Cizek (and Addicott )
• Key trends explored
• Importa...
Schoenig—Test Security is a Process
ACT Process:
• Deter, Detect, & Decide
• Supports the 3-legged stool of
7www.caveon.com
Wollack—Data Forensics Goes Mainstream
• An amazing paradigm shift
Past: DF detects cheating = public distrust in testing
...
Cizek—Test Security is about Validity (not
cheating)
What we have said:
“a statistical procedure... can never prove that,
...
Cizek—Test Security is about Validity (not cheating
What we can say:
“There is strong reason to be concerned that these
sc...
“Exam Inoculation and Other Crazy Ideas to Stop
Cheaters from Passing Exams”
Burns, Van de Velde, and Maynes
• Pirates per...
Online Proctoring Is Here to Stay
• ATP SC orchestrated demo-rama
• Security concerns abound
• Are they justified?
• More ...
Next Step in Exam Security’s Evolution
Institute for Exam Security (I4ES)
• An independent, non-profit association by/for
...
Fremer Areas to be Covered
• Multidisciplinary Approaches
Workshop
• “Telling it to the Judge”
• Critical Importance of Us...
Multidisciplinary Approaches Workshop
• Program Manager, Investigator, Attorney,
Test Security Guy
• Four Group Exercises
...
One Exercise – The Context
• Limited staff
• Limited staff expertise
• Limited resources: software, money,
time
Problem
• ...
Activity #2
• Assess the Risks!
17www.caveon.com
The Scenario
• CBT administration, 3,000 examinees. Following
the administration, you analyze your data for
statistical in...
Common Elements Across Reports
• Notify Your Boss Promptly
• Collect Relevant Information Based
on Your Response Plan
• Se...
I Told It To The Judge, But S/He Didn’t
Listen
• Jennifer Semko – Baker &
Mckenzie
• Camille Thompson – ACT
• Dennis Mayne...
I Told It To The Judge (cont.)
• Advice re testifying
– Answer only what you are asked
– Don’t be a smart alec
– KISS Prin...
I Told It to The Judge - User
Agreements
• User Agreement is a Binding Contract
• Defines Relationship
• Memorializes Obli...
Poll #4
23www.caveon.com
ATP Security Committee
• High Attendance at Breakfast
• Great Deal of Productive Energy
• Many Projects (Examples)
– Remov...
Closing Keynote
• Hilary Mason
• Big Data – Tremendously Useful
• Gives Power to Many to Influence
Events
• It is the Effe...
Overall Impression
• Sharp Increase in Awareness of
Importance of Test Security to Test
Program Management
• Other Domains...
Questions?
Please type questions for our presenters in
the GoToWebinar control panel on your
screen.
27www.caveon.com
Thank you!
John Fremer
President
Caveon Consulting Services
28www.caveon.com
Steve Addicott
Vice President
Caveon Test Sec...
Upcoming SlideShare
Loading in …5
×

Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014

489 views
374 views

Published on

This year's Association of Test Publishers (ATP) Innovations in Testing Conference focused more than ever on test security, and the Caveon team was there. We share with you not only the concepts which we presented, but also new things we learned at the conference.

Caveon leaders John Fremer and Steve Addicott summarize the test security ideas and strategies from ATP and the things you can do to protect your high-stakes tests.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
489
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014

  1. 1. Caveon Webinar Series: Ten Test Security Lessons from ATP March 19, 2014 John Fremer, Ph.D. President Caveon Consulting Services Steve Addicott Vice President Caveon Test Security
  2. 2. Agenda for Today • Steve – Emphasis on Test Security – Highlights from some TERRIFIC Security Sessions – Online proctoring – Next Step in Exam Security • John – Exam Security Processes and Procedures – “Telling it to the Judge” – User Agreements – ATP Security Committee Briefing – Closing Keynote • Q&A
  3. 3. Test Security is….. 3www.caveon.com
  4. 4. Test Security is Front Burner 4www.caveon.com
  5. 5. Test Security is Front Burner 5www.caveon.com
  6. 6. Ten Years of Test Security • Star-studded cast – Schoenig, Wollack, Cizek (and Addicott ) • Key trends explored • Important “Lessons” 6www.caveon.com
  7. 7. Schoenig—Test Security is a Process ACT Process: • Deter, Detect, & Decide • Supports the 3-legged stool of 7www.caveon.com
  8. 8. Wollack—Data Forensics Goes Mainstream • An amazing paradigm shift Past: DF detects cheating = public distrust in testing Future: DF enhances validity = increased public trust • Industry Standards for DF • Proprietary  Dissemination • CSDPTF  COTS • DF moving closer to test instance 8www.caveon.com
  9. 9. Cizek—Test Security is about Validity (not cheating) What we have said: “a statistical procedure... can never prove that, for example, teachers in a school are cheating” “such statistical criteria are never ‘proof’ of cheating” 9www.caveon.com
  10. 10. Cizek—Test Security is about Validity (not cheating What we can say: “There is strong reason to be concerned that these scores may not be valid. We recommend that additional information be gathered or that testing be replicated before interpreting these scores to mean...” 10www.caveon.com
  11. 11. “Exam Inoculation and Other Crazy Ideas to Stop Cheaters from Passing Exams” Burns, Van de Velde, and Maynes • Pirates persist • Test Design is powerful! • Most measures are reactive, so… • Inoculate exams against pirates – Trojan Horse Items – Embedded Verification Tests – Item cloning – DOMC 11www.caveon.com
  12. 12. Online Proctoring Is Here to Stay • ATP SC orchestrated demo-rama • Security concerns abound • Are they justified? • More research required • Test Security Standards for Online Proctoring now available – http://bit.ly/1ifSv0F – Caveon.com Resources White Papers 12www.caveon.com
  13. 13. Next Step in Exam Security’s Evolution Institute for Exam Security (I4ES) • An independent, non-profit association by/for Test Security professionals • www.cespcert.org 13www.caveon.com
  14. 14. Fremer Areas to be Covered • Multidisciplinary Approaches Workshop • “Telling it to the Judge” • Critical Importance of User Agreements • ATP Security Committee Breakfast • Closing Keynote 14www.caveon.com
  15. 15. Multidisciplinary Approaches Workshop • Program Manager, Investigator, Attorney, Test Security Guy • Four Group Exercises • Each Dealing with Real Challenges in Testing Programs • Common Elements across Discussions www.caveon.com 15
  16. 16. One Exercise – The Context • Limited staff • Limited staff expertise • Limited resources: software, money, time Problem • Efficiently allocate resources to threats based on – The risk they pose – The likelihood of finding a solution 16www.caveon.com
  17. 17. Activity #2 • Assess the Risks! 17www.caveon.com
  18. 18. The Scenario • CBT administration, 3,000 examinees. Following the administration, you analyze your data for statistical indicators of cheating. • 180 examinees were statistically flagged for similar response patterns. They did not pass at a higher rate than expected. • Of those 180, a group of 80 examinees had been flagged in previous administrations. They did not pass at a higher rate than expected. • What are your next steps in addressing this issue? 18www.caveon.com
  19. 19. Common Elements Across Reports • Notify Your Boss Promptly • Collect Relevant Information Based on Your Response Plan • Seek Appropriate Expertise – Legal – Psychometric – Messaging/Communications 19www.caveon.com
  20. 20. I Told It To The Judge, But S/He Didn’t Listen • Jennifer Semko – Baker & Mckenzie • Camille Thompson – ACT • Dennis Maynes - Caveon 20www.caveon.com
  21. 21. I Told It To The Judge (cont.) • Advice re testifying – Answer only what you are asked – Don’t be a smart alec – KISS Principle • Courts have been Supportive • Follow Your Policies and Procedures or Stay Home www.caveon.com 21
  22. 22. I Told It to The Judge - User Agreements • User Agreement is a Binding Contract • Defines Relationship • Memorializes Obligations • You will be Judged by How Well you Fulfilled Your Commitments • Must Behave Responsibly and Fairly 22www.caveon.com
  23. 23. Poll #4 23www.caveon.com
  24. 24. ATP Security Committee • High Attendance at Breakfast • Great Deal of Productive Energy • Many Projects (Examples) – Removing Copyrighted Material from the Internet – Test Taker Rights and Responsibilities 24www.caveon.com
  25. 25. Closing Keynote • Hilary Mason • Big Data – Tremendously Useful • Gives Power to Many to Influence Events • It is the Effective Holders of Information who Can Wield Power • Some Problems Cut Across Fields – Unwillingness to Act on Data Alone 25www.caveon.com
  26. 26. Overall Impression • Sharp Increase in Awareness of Importance of Test Security to Test Program Management • Other Domains Catching up to IT in Recognition of Problems • Focus on User Agreements and Online Proctoring • Willingness to Try New Approaches 26www.caveon.com
  27. 27. Questions? Please type questions for our presenters in the GoToWebinar control panel on your screen. 27www.caveon.com
  28. 28. Thank you! John Fremer President Caveon Consulting Services 28www.caveon.com Steve Addicott Vice President Caveon Test Security Follow Caveon on twitter @caveon Check out our blog www.caveon.com/blog LinkedIn Group “Caveon Test Security”

×