Caveon webinar lessons learned at atp and nces


Published on

Late February presented two auspicious testing industry events: ATP's “Innovations in Testing” Conference, and the US Department of Education’s National Center for Education Statistics' “Testing Integrity Symposium.”

Join Caveon leaders John Fremer and Steve Addicott as they share the Top Ten Security Lessons Learned from the two events.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Caveon webinar lessons learned at atp and nces

  1. 1. Upcoming Caveon Events• NCME in Vancouver, April 12-16 – NATD Breakfast Plenary, Dr. John Fremer, April 13• CCSSO TILSA SCASS at National Harbor, MD• Conference for the Statistical Detection of Potential Test Fraud, University of Kansas, May 23-24• National Conference on Student Assessment in Minneapolis, June 26-29• International Testing Commission Conference in Amsterdam, July 3-5• National Collegiate Testing Association in Minneapolis, August 1-4
  2. 2. Caveon Online• Caveon Security Insights Blog –• Twitter – @Caveon• LinkedIn – Caveon Company Page – Caveon Test Security Group • Please contribute!• Facebook – Will you be our “friend?” – “Like” us!
  3. 3. Caveon Webinar Series“Lessons Learned at ATP and NCES” Steve Addicott Dr. John Fremer Vice President President Caveon Caveon Consulting Services March 28, 2012
  4. 4. Agenda• ATP 2012 Security Lessons Learned – Steve Addicott• NCES 2012 Security Lessons Learned – Dr. John Fremer• Q&A
  5. 5. About Caveon….• Our mission: To help protect the tests, programs, and reputations of our clients, some of the most important test programs and test publishers in the world – Certification & Licensure – State DOEs & School Districts – DOD – International• Entering our 9th year of operation• Provide practical, measureable security enhancements
  6. 6. Test Security Highlights from ATPTheme:• A dozen sessions focused on security – Too many at the same time!• Caveon sessions with our clients – Conducting investigations – Managing the media in cheating scandals – How to implement Data Forensics – Embedded verification tests• Security Committee breakfast – Standing room only, despite the 7am start time!
  7. 7. Key Industry Veterans Focusing onSecurity• Liberty Munson, • Ashok Sarathy, Microsoft GMAT• Chad Buckendahl, • Steven Barkley, Alpine Testing CFP• Mark Poole, Pearson • Daniel Eyob, VUE GMAT• Cathy Donath, • Nikki Eatchel, Donath Group Questar• Ray Nicosia, ETS
  8. 8. Conducting InvestigationsBen Mannes, Director of Test Security, ABIMLessons Learned• Was a security breach (due to a possible vulnerability in your policies or infrastructure) identified during the incident?• If so, what can be done to change policies, procedures, or technologies to fix it?• A cost/benefit analysis should be done to address the vulnerability identified by the incident.• Are there other ramifications to the organization from the incident?
  9. 9. Gimme Shelter: Managing the Media Stormof a Security Breach Jim Vasalek, Sr Director of Public Affairs, LSAC NON-Traditional Media Blogs, bulletin boards, chat rooms, file-sharing sites, and social media No professional code of ethics No editors No fact checkers No obligation to get your side of the story Can keep a story alive well after traditional media has lost interest, even well after it actually has died May be more influential with your key audiences than traditional media
  10. 10. Implementing Data ForensicsAimee Hobby Rhodes, Director of Exam Security, CFA InstituteCollect All the Evidence You Can!!• Use multiple statistical analyses – Similarity Analysis, Erasure Analysis, Prior score analysis (gains/losses), identical response analysis, etc• Suspected Cheating/Incident Reports• Seating Chart/Floor plans• Candidate Connections• Anything else you can find!
  11. 11. Implementing Data ForensicsJennifer Semko, Partner, Baker & McKenzieYour Candidate Agreement is foundational!• Are candidates on notice that – sharing items is a breach? – studying from recalled items is improper? – Data Forensics may be used?• Did you reserve the right to: – Invalidate scores? Suspend access to examination? Take other action?• What are the grounds for action? Is there a “catch all”?• Do you: – regularly review your agreement language? – have uniform security procedures and policies in place?
  12. 12. Embedded Verification TestLiz Burns, Senior Manager of Technical Certifications Programs, Juniper NetworksKey Benefits of New Security Tactic, EVT• Legally defensible “proof” for enforcement actions – Does not require “intent”• Enforces against unqualified candidate• Works on broad range of content theft• Fairly inexpensive to do• Provides intelligence on vulnerability of your exams• Can do “results hold” for proactive enforcement – Changes definition of “PASS” to reaching the cut score AND the fraud detection threshold
  13. 13. Social Media• Large emphasis placed on Social Media at the conference. – twitter hash tags, QR codes, or LinkedIn links• Many were tweeting and blogging about the conference during live sessions – Getting the word out instantaneously.• Get on the train….NOW! – be a part of the online conversation – how information is being shared.
  14. 14. Security Committee Breakfast• Standing room only…really – Early morning didn’t deter anyone – Despite myriad “celebrations” the evening before• Lots of important work – Security Survey – Enforcement – Lobbying – Standards• All volunteers – Pat yourself on the back!!
  16. 16. SECRETARY DUNCAN• “States, districts, schools, and testing companies should have sensible procedures in place to ensure tests accurately reflect student learning.”• Quoted in recent Atlanta Journal Constitution article – “Cheating our children: Suspicious school test scores across the nation.”
  17. 17. ETS – CARSWELL WHITEHEAD • All High Stakes Tests are Subject to Attempts to Gain an Unfair Advantage • Must Build in Processes to Detect Irregularities – Before – During – After
  18. 18. GREG CIZEK – UNC, CHAPEL HILLPREVENTION OF IRREGULARITIES IN ACADEMIC TESTING:• Test Providers / Contractors * Clear definition of cheating * Clear, educator-referenced materials * Web-based qualification utility, database? * Less corruptible formats * CBT/CAT delivery
  19. 19. DAVID FOSTER – KRYTERION SEVEN DETECTION PRINCIPLES1. Focus…Concentrate on high-risk threats first2. Adapt…Match detection methods to the threat3. Backup…Use layers of detection methods4. Predict…Watch out for new threats5. Filter…Rule out other explanations6. Evaluate…Use detection to evaluate security7. Plan…Set up to succeed
  20. 20. BRIAN JACOBS - UNIV OF MICHIGAN DETECTING IRREGULARITES• Benefits of statistical analysis – Relatively low cost – Covers the entire population – Measure of the “extent” of the irregularity – e.g., likelihood it would have occurred by chance – Systemic patterns as well as individual cases of concern• Limits of statistical analysis – Like preliminary screens for rare diseases, statistical analyses to detect cheating can have a high rate of false positives and false negatives • Critical to complement statistical analyses with other methods – Can never identify the individual responsible for the manipulation
  21. 21. JAMES LIEBMAN –COLUMBIA UNIVERSITY LAW SCHOOL • Keep an open line for reports of infractions • Teachers as allies • Duty to report immediately • Multiple locations (principal, monitors, local test office, state test office, “special investigations”) • Allow anonymity; offer confidentiality
  22. 22. SCOTT NORTON – LA DEPT OF EDAreas for Improvement at State Level• More state oversight is needed for district-led investigations.• Standardization across states may be needed for established procedures such as erasure analysis.• Better information is needed about other statistical analysis procedures for detecting suspect patterns of responses, unusual gains or losses, etc.
  23. 23. STEVE FERRARA – PEARSON• [Speaking primarily as a former Maryland SAD]• School Personnel Not Well Equipped to Handle Investigations• Don’t have Training• Teaching Draws on Nurturing and Supporting Skills
  24. 24. John Fremer – Caveon Test Security Ten Recommendations Moving Forward1. Acknowledge the seriousness of security issues2. Expect cheating and plan to be proactive3. Use multiple detection methods and forensic statistics4. Minimize testing windows5. Strengthen the chain of custody
  25. 25. Ten Recommendations Moving Forward…Cont.6. Increase the emphasis on security training7. Allocate adequate resources for test security8. Pilot techniques for detection of cheating9. Continue to learn from others10. Monitor new advances in anomaly detection and prevention (e.g. “Epidemiological Model”)
  26. 26. RESOURCES• CCSSO/ATP – Operational Best Practices• ATP Security Committee• Caveon Test Security – Blog – Webinars• (In development) NCME Guidelines• (In development) TILSA Guidebook for State Assessment Directors on Data Forensics
  27. 27. Continue The Conversation- Follow us on twitter @caveon for updates and events- Check out our blog… LinkedIn Group – Join our Caveon Test Security group to ask questions and join discussions about test security topics- Slides of this and past Webinars are available at
  28. 28. Thank you!Steve Addicott Dr. John FremerVice President PresidentCaveon, LLC Caveon Consulting