The Endless Security Summer - Caveon Webinar 8-22-2012


Published on

Testing industry veterans John Fremer and Steve Addicott of Caveon are joined by Lou Woodruff, past president of the National College Testing Association to share their "lessons learned" from several of this summer's biggest testing conferences. For more information, please go to

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • lou
  • lou
  • lou
  • Polls:-Has there been a larger focus on security at your org. in last 5 years? before lou after Steve. -Did you attend any of these? (Select all that apply) right after agenda.Note: John’s last slide last before Q&A.Steve to add anima. And tweaks.Skyler to compile and add polls.At end of each speaker, ask questions if any come up.
  • The Endless Security Summer - Caveon Webinar 8-22-2012

    1. 1. Caveon Webinar Series“The Endless Security Summer: Testing Conference Lessons” August 22, 2012
    2. 2. “The Endless Security Summer”  Steve Addicott • Vice President • Caveon Test Security  John Fremer, Ph.D. • President • Caveon Consulting Services  Lou Woodruff, Ph.D. • Senior Security Consultant, Caveon • Former President, National College Testing Association2
    3. 3. Agenda • Conference on Statistical Detection of Potential Test Fraud, May 23-24 • National Conference on Student Assessment, June 26-29 • International Test Commission, July 3-5 • Dutch Exam Society, July 6 • National College Testing Association, August 1-4 • Closing Thoughts3
    4. 4. Poll Question #14
    5. 5. Conference on Statistical Detection of Potential Test Fraud • University of Kansas’ CETE; Lawrence, KS • First-ever Data Forensics Conference • Gathering of measurement experts & researchers – Certification – Education – Vendors – Test Publishers – Academics5
    6. 6. Conference on Statistical Detection of Potential Test Fraud • Most important takeaway THIS CONFERENCE EXISTS!! • Second most important take away Protecting our tests grows ever harder, and ever more important6
    7. 7. National Conference on Student Assessment (NCSA) • June 26-29 in Minneapolis, MN • Council of Chief State School Officers (CCSSO) • 2012 Theme: Teaching, Learning, and Assessment through the Next Decade7
    8. 8. Top Three NCSA Key Takeaways8
    9. 9. NCSA Key Takeaways #3—Bombshell! CCSSO is partnering with Center for Assessment to “re-focus” the conference – Reduced attendance? – Budgets? – Changing priorities?9
    10. 10. NCSA Key Takeaways #2—A number of dynamics in state assessments impacting security – Teacher Accountability • Legislatures passing laws • Using existing assessments to now grade teaching performance – Assessment Consortia – Technology • Classrooms • Administration • Learning10 • Assessments
    11. 11. #1 NCSA Takeaway • Despite all these changes, THERE WAS NOT ONE SESSION FOCUSED ON TEST SECURITY. NOT ONE.11
    12. 12. Topics That Could Have Covered… • Thoughtful piloting of security strategies • By 2014-2015, all state assessments will be CBT – Colossal logistical challenge – Too few PCs and test centers • Blending individual test programs – Dozens into two – Exams will be shared across states and time zones – Administration windows will last weeks – More technology, more sharing?12
    13. 13. International Test Commission • July 3 – 5, 2012 • Group representing testing organizations and national agencies world wide • Amsterdam, The Netherlands13
    14. 14. ITC – Major Takeaways• Growing recognition throughout the world of the importance of protecting IP and preventing cheating• Attention to standards – ATP, ITC, ANSI, Caveon• More attention to how to use data to detect misbehavior• “Trust but verify”14
    15. 15. CITO & Caveon Session • CITO – very major provider of testing services to Europe and many non-European countries • Collaborating with other Dutch testing agencies to promote security standards • Caveon – evidence of importance of security – Internationalization of testing – Challenges of technologically savvy cheats – Notorious US cases, especially Atlanta PS15
    16. 16. Faking and Test Security • Steve Addicott Contribution • Dealing with Media – Be prepared – Draw on all parts of your organization – communications, legal, membership, etc. – Have a spokesman – Tell the story from your perspective – Don’t expect the story to go away16
    17. 17. Dutch Exam Society (NVE) Security Summit • NVE has 400+ members from dozens of test programs in The Netherlands • Eight leaders gathered to: – Address Test Security concerns – Consider adoption of Security Standards for members • Caveon invited to play a supporting role17
    18. 18. Top NVE Security Summit Takeaways #3—Times are changing – Test Security was not, but is now, a front-burner issue #2—Security Risks are global #1—Rising tide lifts all boats – Organizations which are considered “competitive” can collaborate and create positive change18
    19. 19. Poll Question #219
    20. 20. National College Testing Association 11th Annual Conference Minneapolis 201220
    21. 21. College testing administrators and staff representing over 600 colleges More than 50 companies that do business in the testing field21
    22. 22. Consortium of College Testing Centers Listing of hundreds of college testing centers offering proctoring services for individuals needing to take a secure, proctored examination Access the listing for free on the NCTA website22
    23. 23. Six Sessions Devoted to Test Security: The Keynote Address The General Session Four Breakout Sessions23
    24. 24. Keynote Address Mark Poole Product Strategist and Security Expert Pearson VUE “The Future of Secure Testing in a Digital World”24
    25. 25. Highlighted five key trends that are shaping the next decade of testing: o a global cheating culture o high tech test fraud o testing anywhere securely o candidate authentication o data forensics25
    26. 26. Breakout Session: “Demystifying Data Forensics: An Overview of the Logic Underlying Cheating Detection Techniques” Jim Wollack, Ph.D. Director, Testing and Evaluation Services University of Wisconsin, Madison Presented the basic principles of applied data forensics in a way that everyone could understand. Reviewed some forensics approaches to identify examinees who copy answers, had prior access to test content, or hired surrogates to test for them26
    27. 27. Breakout Session: “Gimme Shelter: Weathering the Media Storm of a Security Breach” Steve Addicott Vice President, Client Services Caveon Test Security Ray Nicosia Executive Director of the Office of Testing Integrity Educational Testing Services (ETS) Faisel Alam Test Security Specialist Law School Admission Council (LSAC)27
    28. 28. This panel of security experts shared key aspects of media strategy for testing programs and techniques for positioning incidents to various audiences High profile security breaches were reviewed, highlighting details of what went well and what did not The use and effectiveness of various media tools was analyzed, and a checklist of media “Do’s and Donts” was presented28
    29. 29. Breakout Session: “Security Strategies for Every Stage of the Testing Process” DeDe Hedlund Jeff Place Creighton University Questionmark Looked at best practices for colleges to protect test content and minimize cheating Demonstrated the potential for using mobile test centers with examinees testing on tablets such as the Apple iPad29
    30. 30. Breakout Session: “Innovations in Online Proctoring” Rick Beaudry Denis Gravel CEO Director, Workforce Development B Virtual Inc. School of Applied Technology Presented a proctoring service that proctored test takers in any location via web-cam and microphone, with a proctor monitoring the session remotely30
    31. 31. General Session “Detection and Prevention of Test Fraud The State of the Art” John Fremer President Caveon Consulting Services Lou Woodruff Senior Security Consultant Caveon Consulting Services31
    32. 32. o Perspectives on the Perpetrators o Test Security Anatomy o Contemporary Test Security Responses o The Technology of Detection o Large Scale Security Investigations o Future Test Security Challenges for Higher Ed o Implications for College Testing Professionals32
    33. 33. Survey data on the prevalence of cheating among the nation’s youth was shared which showed an alarming level of cheating Cheating has grown from a strategy for increasing test scores to the manifestation of a growing social norm that accepts dishonesty in many areas of life The cheating/detection arms race will continue with advances in biometric, data forensic and test delivery techniques enhancing prevention and detection33
    34. 34. Result of Using Data Forensics in a State Assessment Program Reduction in Data Forensic Analyses of "Irregularities" school districts statewide 6.0% 5.0% Decisive actions taken on the forensic results 4.0% 3.0% Two years of follow-up test 2.0% administration monitoring 1.0% Found a 72% reduction in 0.0% test security irregularities Report 1 Report 2 Report 3 Report 4 Report 5 Report 6 Report 7 Report 8 Report 934
    35. 35. The Atlanta Public Schools The Perfect Test Security Storm35
    36. 36. Mandated assessments tied to federal funding The Perfect Schools and teachers Test Security evaluated on test scores Storm Administrators feeling unjustly treated Test security left in the hands of the teachers and the principals36
    37. 37. Atlanta Public Schools Test Security Case Governor brought in the GA Bureau of Investigation Sixty-one agents on the case with subpoena power District Superintendent, all Deputy Superintendents, and many Principals and Teachers were removed Who do you trust? Sometimes – no one!37
    38. 38. Take-Aways from the NCTA Conference College testing professionals are highly aware of and strongly interested in test security NCTA is a valuable platform for promoting security related initiatives and partnerships between colleges the testing industry Colleges are and will remain a rich resource for implementing cheating detection and prevention strategies38
    39. 39. Questions?39
    40. 40. Today’s Presenters Steve Addicott John Fremer Lou Woodruff LCWoodruff@aol.com40