Late February presented two auspicious testing industry events: ATP's “Innovations in Testing” Conference, and the US Department of Education’s National Center for Education Statistics' “Testing Integrity Symposium.” Join Caveon leaders John Fremer and Steve Addicott as they share the Top Ten Security Lessons Learned from the two events.

1. Upcoming Caveon Events
• NCME in Vancouver, April 12-16
– NATD Breakfast Plenary, Dr. John Fremer, April 13
• CCSSO TILSA SCASS at National Harbor, MD
• Conference for the Statistical Detection of Potential
Test Fraud, University of Kansas, May 23-24
• National Conference on Student Assessment in
Minneapolis, June 26-29
• International Testing Commission Conference in
Amsterdam, July 3-5
• National Collegiate Testing Association in
Minneapolis, August 1-4
www.caveon.com

2. Caveon Online
• Caveon Security Insights Blog
– http://www.caveon.com/blog/
• Twitter
– @Caveon
• LinkedIn
– Caveon Company Page
– Caveon Test Security Group
• Please contribute!
• Facebook
– Will you be our “friend?”
– “Like” us!
www.caveon.com

3. Caveon Webinar Series
“Lessons Learned at ATP and NCES”
Steve Addicott Dr. John Fremer
Vice President President
Caveon Caveon Consulting Services
March 28, 2012

4. Agenda
• ATP 2012 Security Lessons Learned
– Steve Addicott
• NCES 2012 Security Lessons Learned
– Dr. John Fremer
• Q&A

5. About Caveon….
• Our mission: To help protect the tests, programs,
and reputations of our clients, some of the most
important test programs and test publishers in the
world
– Certification & Licensure
– State DOEs & School Districts
– DOD
– International
• Entering our 9th year of operation
• Provide practical, measureable security
enhancements
www.caveon.com

6. Test Security Highlights from ATP
Theme:
• A dozen sessions focused on security
– Too many at the same time!
• Caveon sessions with our clients
– Conducting investigations
– Managing the media in cheating scandals
– How to implement Data Forensics
– Embedded verification tests
• Security Committee breakfast
– Standing room only, despite the 7am start time!

7. Key Industry Veterans Focusing on
Security
• Liberty Munson, • Ashok Sarathy,
Microsoft GMAT
• Chad Buckendahl, • Steven Barkley,
Alpine Testing CFP
• Mark Poole, Pearson • Daniel Eyob,
VUE GMAT
• Cathy Donath, • Nikki Eatchel,
Donath Group Questar
• Ray Nicosia,
ETS

8. Conducting Investigations
Ben Mannes, Director of Test Security, ABIM
Lessons Learned
• Was a security breach (due to a possible vulnerability in your
policies or infrastructure) identified during the incident?
• If so, what can be done to change policies, procedures, or
technologies to fix it?
• A cost/benefit analysis should be done to address the
vulnerability identified by the incident.
• Are there other ramifications to the organization from the
incident?

9. Gimme Shelter: Managing the Media Storm
of a Security Breach
Jim Vasalek, Sr Director of Public Affairs, LSAC
NON-Traditional Media
Blogs, bulletin boards, chat rooms, file-sharing sites, and social
media
No professional code of ethics
No editors
No fact checkers
No obligation to get your side of the story
Can keep a story alive well after traditional media has lost
interest, even well after it actually has died
May be more influential with your key audiences than traditional
media

10. Implementing Data Forensics
Aimee Hobby Rhodes, Director of Exam Security,
CFA Institute
Collect All the Evidence You Can!!
• Use multiple statistical analyses
– Similarity Analysis, Erasure Analysis, Prior score analysis
(gains/losses), identical response analysis, etc
• Suspected Cheating/Incident Reports
• Seating Chart/Floor plans
• Candidate Connections
• Anything else you can find!

11. Implementing Data Forensics
Jennifer Semko, Partner, Baker & McKenzie
Your Candidate Agreement is foundational!
• Are candidates on notice that
– sharing items is a breach?
– studying from recalled items is improper?
– Data Forensics may be used?
• Did you reserve the right to:
– Invalidate scores? Suspend access to examination? Take
other action?
• What are the grounds for action? Is there a “catch all”?
• Do you:
– regularly review your agreement language?
– have uniform security procedures and policies in place?

12. Embedded Verification Test
Liz Burns, Senior Manager of Technical
Certifications Programs, Juniper Networks
Key Benefits of New Security Tactic, EVT
• Legally defensible “proof” for enforcement actions
– Does not require “intent”
• Enforces against unqualified candidate
• Works on broad range of content theft
• Fairly inexpensive to do
• Provides intelligence on vulnerability of your exams
• Can do “results hold” for proactive enforcement
– Changes definition of “PASS” to reaching the cut
score AND the fraud detection threshold

13. Social Media
• Large emphasis placed on Social Media at the
conference.
– twitter hash tags, QR codes, or LinkedIn links
• Many were tweeting and blogging about the
conference during live sessions
– Getting the word out instantaneously.
• Get on the train….NOW!
– be a part of the online conversation
– how information is being shared.

14. Security Committee Breakfast
• Standing room only…really
– Early morning didn’t deter anyone
– Despite myriad “celebrations” the evening before
• Lots of important work
– Security Survey
– Enforcement
– Lobbying
– Standards
• All volunteers
– Pat yourself on the back!!

15. JOHN FREMER – CAVEON TEST SECURITY
STATE ASSESSMENTS IN TRANSITION
The Perfect
Test Security
Storm

16. SECRETARY DUNCAN
• “States, districts, schools, and testing
companies should have sensible
procedures in place to ensure tests
accurately reflect student learning.”
• Quoted in recent Atlanta Journal
Constitution article
– “Cheating our children: Suspicious school
test scores across the nation.”

17. ETS – CARSWELL WHITEHEAD
• All High Stakes Tests are Subject to
Attempts to Gain an Unfair Advantage
• Must Build in Processes to Detect
Irregularities
– Before
– During
– After

18. GREG CIZEK – UNC, CHAPEL HILL
PREVENTION OF IRREGULARITIES IN
ACADEMIC TESTING:
• Test Providers / Contractors
* Clear definition of cheating
* Clear, educator-referenced materials
* Web-based qualification utility, database?
* Less corruptible formats
* CBT/CAT delivery

19. DAVID FOSTER – KRYTERION
SEVEN DETECTION PRINCIPLES
1. Focus…Concentrate on high-risk threats first
2. Adapt…Match detection methods to the threat
3. Backup…Use layers of detection methods
4. Predict…Watch out for new threats
5. Filter…Rule out other explanations
6. Evaluate…Use detection to evaluate security
7. Plan…Set up to succeed

20. BRIAN JACOBS - UNIV OF MICHIGAN
DETECTING IRREGULARITES
• Benefits of statistical analysis
– Relatively low cost
– Covers the entire population
– Measure of the “extent” of the irregularity – e.g.,
likelihood it would have occurred by chance
– Systemic patterns as well as individual cases of concern
• Limits of statistical analysis
– Like preliminary screens for rare diseases, statistical
analyses to detect cheating can have a high rate of false
positives and false negatives
• Critical to complement statistical analyses with other
methods
– Can never identify the individual responsible for the
manipulation

21. JAMES LIEBMAN –
COLUMBIA UNIVERSITY LAW SCHOOL
• Keep an open line for reports of
infractions
• Teachers as allies
• Duty to report immediately
• Multiple locations (principal, monitors,
local test office, state test office,
“special investigations”)
• Allow anonymity; offer confidentiality

22. SCOTT NORTON – LA DEPT OF ED
Areas for Improvement at State Level
• More state oversight is needed for district-led
investigations.
• Standardization across states may be needed
for established procedures such as erasure
analysis.
• Better information is needed about other
statistical analysis procedures for detecting
suspect patterns of responses, unusual gains
or losses, etc.

23. STEVE FERRARA – PEARSON
• [Speaking primarily as a former Maryland SAD]
• School Personnel Not Well Equipped to Handle
Investigations
• Don’t have Training
• Teaching Draws on Nurturing and Supporting
Skills

24. John Fremer – Caveon Test Security
Ten Recommendations Moving Forward
1. Acknowledge the seriousness of security issues
2. Expect cheating and plan to be proactive
3. Use multiple detection methods and forensic statistics
4. Minimize testing windows
5. Strengthen the chain of custody

25. Ten Recommendations Moving Forward…Cont.
6. Increase the emphasis on security training
7. Allocate adequate resources for test security
8. Pilot techniques for detection of cheating
9. Continue to learn from others
10. Monitor new advances in anomaly detection and
prevention (e.g. “Epidemiological Model”)

26. RESOURCES
• CCSSO/ATP – Operational Best Practices
• ATP Security Committee
• Caveon Test Security
– Blog
– Webinars
• (In development) NCME Guidelines
• (In development) TILSA Guidebook for State
Assessment Directors on Data Forensics

28. Continue The Conversation
- Follow us on twitter @caveon for updates and events
- Check out our blog…www.caveon.com/blog/
- LinkedIn Group – Join our Caveon Test Security group
to ask questions and join discussions about test
security topics
- Slides of this and past Webinars are available at
www.caveon.com/resources/webinars

29. Thank you!
Steve Addicott Dr. John Fremer
Vice President President
Caveon, LLC Caveon Consulting Services
steve.addicott@caveon.com john.fremer@caveon.com