Caveon Webinar Series: The Handbook of Test Security – Learn more about preventing and detecting cheating on high stakes tests

Uploaded on

Caveon Webinar Series presents it's March Webinar session: …

Caveon Webinar Series presents it's March Webinar session:

"The Handbook of Test Security – Learn more about preventing and detecting cheating on high stakes tests”

Handbook editors, Professor Jim Wollack, University of Wisconsin, and Dr. John Fremer, Caveon Consulting Services, provide a synopsis of the brand new “Handbook of Test Security,” and present key ideas covering the full spectrum of testing types and domains.

All of our sessions are offered at NO CHARGE, and last less than an hour.

Follow @caveon on twitter for discussion & updates about this and other Webinars!

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • Test are used as gatekeepers for so many professional goals: licensure/certification, advancement in school/graduation, admissions to higher education, job promotion.
  • Create a comprehensive resource so that testing entities can build upon one another’s expertise and aren’t stuck recreating their own test security wheels.
  • Richelle to help on this one… Make changes as Jim has mentioned
  • Examples of these before, during, and after the testWhere do we draw the line between proper and improper test prep
  • Rooms aren’t designed specifically for testing, students know the layout of everything, students often have unsupervised access to testing rooms in advanceTeachers aren’t professional proctors and generally prefer not to police their classrooms. Chapter provides practical, realistic, and straightforward approaches to prevent and deter cheating, as well as to reduce student’s motivation to cheat
  • Stand-alone writing prompts require students to produce a written response to a given brief topic or prompt. Text-based writing prompts reflect the reading and writing connection, in that students areasked to read about a topic from one or more sources, analyze it from a particular perspective, and then write a response
  • Protect data: test content, IP, examinee’s personal identification dataMore than just testing environment: layout of test materials environments, including dealing with access issues for testing and non-testing personnelHandling of test materials, not just final copy, but items and materials in different phases. Deals with issues related to access and monitoring, firewalls/encryption, working with outside non-testing contractors (printers/shippers),
  • Item preknowledge (Illegal test preparation or Internet-based discussion groups or exam wholesalers), test tampering, organized proxy testing, etc.Similarity and copying analyses, erasure analysis, gain score analysis, person-fit, response latency, What’s required for statistical evidence to be admissible in court? Advice on managing one’s false positive rate, particularly when using data forensics for data mining. Thoughts on using multiple indexes in combination
  • What has led to organizations winning or losing in court?Limits to how biometric data can be usedCarry out their duties and contractual obligations in “Good faith”
  • Should reference Bill Hatherill’s studies
  • Know that it really is happening. You have to stay vigilant. You have to look for cheating. You have to actively safeguard against it. Sometimes it’s right after, like item harvesting. Hacking, tampering, illegal coaching, item sharing/distributionDuring the exam: proxy testing, item capturing/transmission, copying, communication


  • 1. Upcoming Caveon Events• NCME Conference, April 26 – 30th, San Francisco Test Security I: Policy Issues: Technical/ Statistical/Methodological Issues. – Presenters include: Dr. John J. Fremer• USDLA Conference, April 28- May 1, St. Louis Will the real learner please stand up? – Presenters include Caveon’s Jamie Mulkey, Ed.D. and Patrick Martin
  • 2. Latest Publications• Handbook of Test Security – Pre-order your copy today. To be published March 27th. We’ll share a discount code before end of session.• TILSA Guidebook for State Assessment Directors on Data Forensics – just released
  • 3. Caveon Online• Caveon Security Insights Blog –• twitter – Follow @Caveon• LinkedIn – Caveon Company Page – “Caveon Test Security” Group • Please contribute!• Facebook – Will you be our “friend?” – “Like” us!
  • 4. Caveon Webinar Series: “The Handbook of Test Security”Learn more about preventing & detecting cheating on high-stakes tests Dr. James A. Wollack Dr. John Fremer Associate Professor President Educational Psychology Caveon Consulting Service Director, Testing & Evaluation Services University of Wisconsin-Madison March 21, 2013
  • 5. Agenda for today• James Wollack – Introduction to the Handbook of Test Security – Parts 1 & 2• John Fremer – Part 2 & 3• James Wollack – Contributing Authors – 10 Takeaways
  • 6. HANDBOOK OF TEST SECURITY• Editors - James Wollack & John Fremer• Slated for Publication – March 2013• Preventing, Detecting, and Investigating Cheating• Testing in Many Domains – Certification/Licensure – Clinical – Educational – Industrial/Organizational
  • 7. CURRENT STATE OF TESTING• Large-scale, Standardized Testing is an Integral Part of our Culture• Incentive to Cheat on Tests has never been Higher – Cheating is a very real problem – Extends across all walks of testing• Presence of Test Security Themes at Professional Conferences has Skyrocketed – Much of what is known about test security resides in testing companies and is proprietary
  • 8. PRIMARY AIMS OF THE HANDBOOK• Advance the science of test security• Draw on insights of psychometricians and policy-makers to provide a comprehensive resource that deals with all aspects of test security for all genres of testing. • Best practices for designing secure tests • Analysis of security vulnerabilities • Practical cheating prevention/detection strategies • Lessons learned in actual security violations and initiatives
  • 9. ORGANIZATION OF HANDBOOK• Part I: Designing Secure Delivery Systems – Security vulnerabilities unique to modality – Chapters focus on prevalence, major threats, prevention, detection• Part II: Important Issues in Test Security – Maintaining security before and after exam – Developing a security culture throughout all phases of organization – Reconnaissance effort following breach• Part III: Lessons Learned from Practice – Case studies from a variety of disciplines
  • 10. INTRODUCTION The Test Security Threat James A. Wollack & John J. Fremer– Cheating is pervasive and no testing program is immune– Extremely costly to programs and reputations of both testing industry and discipline– Very difficult to catch
  • 11. PART I: DESIGNING SECURE DELIVERY SYSTEMS Security in Large-Scale Paper and Pencil Testing John J. Fremer and Steve Ferrara – K-12 accountability and national admissions testing – Methods • Cheating by examinees and administrators • Exposure/disclosure of secure test materials • Inappropriate test preparation • Inappropriate test administration – Prevention & Detection – Recommendations for Testing Program Managers
  • 12. PART I: DESIGNING SECURE DELIVERY SYSTEMS Security Issues in Technology-Based Testing David Foster • Use of technology to build and deliver tests has created many new vulnerabilities • on-demand testing • Internet-based testing • electronic distribution of test files • storage of item banks, test files, results, examinee info • test question over-exposure • testing centers – Theft vs. Cheating – Prevention, Deterrence, and Detection strategies
  • 13. PART I: DESIGNING SECURE DELIVERY SYSTEMS Security Issues in Classroom Testing Louis Woodruff – Teacher-made tests, predominantly paper-based – Many of the types of cheating are the same as those on standardized tests, but particulars are often quite different • Teacher usually knows students, sometimes quite well • Students know each other, often very well • Much greater physical security vulnerabilities – Deterring cheating & mitigating motivations to cheat • Practical, realistic, straightforward approaches
  • 14. PART I: DESIGNING SECURE DELIVERY SYSTEMS Security Issues in Writing Assessment Suzanne Lane – Both large-scale and classroom writing assessments – Security vulnerabilities: • Prompts (and related text, if applicable) • Responses – Plagiarism • Detection systems • Prevention of plagiarism – Automated essay scoring and gaming
  • 15. PART II: IMPORTANT ISSUES IN TEST SECURITYPhysical Security at Test Centers and the Testing Company Anthony R. Scicchitano and Roger D. Meade – Controlling physical testing environment to protect data • layout of the testing environment • check-in/examinee authentication process • proctoring • handling of test materials • security audits
  • 16. PART II: IMPORTANT ISSUES IN TEST SECURITY Educator Cheating and the Statistical Detection of Group-Based Test Security Threats Dennis Maynes – Data Forensics with focus on detecting collusion or groups of individuals • largely non-technical description • credibility of various sources of data – Standards for statistical evidence of test security breaches
  • 17. PART II: IMPORTANT ISSUES IN TEST SECURITY Legal Matters in Test Security Jennifer Ancona Semko and Robert Hunt – Examined case law related to score cancelation and the basics of U.S. Copyright law – Before the exam: Candidate agreements and organizational security policies – During the exam: legal issues with biometric data & proctor intervention – After the exam: No need to “prove” cheating, taking action against copyright infringement – Emphasis on being prepared for a possible lawsuit before the incident.
  • 18. PART II: IMPORTANT ISSUES IN TEST SECURITY Security Planning, Training, and Monitoring Cyndy Thomas Fitzgerald and Jamie R. Mulkey – Need comprehensive written security plan – Develop related training materials – Provide and keep current training – Follow clear guidelines
  • 19. PART II: IMPORTANT ISSUES IN TEST SECURITY Conducting Investigations of Misconduct Deborah J. Harris and Rachel R. Watkins Schoenig – Importance of examinee/sponsor agreements – Decide on “triggers” for investigations – Provide “understandable” results – Scrupulously follow your rules
  • 20. PART II: IMPORTANT ISSUES IN TEST SECURITY Security Related Communications Susan M. Case and Beth E. Donahue – Need thoughtful plan of communication – Regular briefings and updates – Decision makers identified • Operational • Policy – Document security threats and their resolution
  • 21. PART III – CASE STUDIESCertification/Licensure Testing Case Studies Janet Duffy Carson– Healthy level of paranoia– Commitment to follow up on problems identified– Careful attention to training– Include dealing with “unusual events”– Register copyrights– Use outside resources when appropriate
  • 22. PART III – CASE STUDIES Prevention, Detection, and Management of Falsified Test Development Data (Clinical) Kevin M. Williams, Sara Rzepa, Gill Sitarenios, Hazel Wheldon• Very real danger of fraudulent data – Develop industry standards • Preventative methods • Warning signals – Compare data from different sites
  • 23. PART III – CASE STUDIES Educational Testing Neal Kingston– Pressures to cheat increasing– Make unethical behavior very clear– Use multiple statistical methods– Must act on data– Management of communications essential
  • 24. PART III – CASE STUDIESIndustrial/Organizational Testing Case Studies Dave Bartram and Eugene Burke – Body of knowledge growing – Use multiple testing and scoring mechanisms – Benefit to using honesty contracts
  • 25. PART III – CASE STUDIES Commentary on Case Studies William Hatherill- Offers useful observations that cut across all of the casestudies
  • 26. ADVICE FOR THE READER• Identify your most significant vulnerabilities• “Skim read” chapters relating to your program – Earmark chapters and sections for you and/or staff to read carefully – Cross-reference those sections with list of vulnerabilities – Update your list of vulnerabilities as you read• Develop a plan for addressing these vulnerabilities• Re-read sections as you deal with the issues addressed• Follow the advice of the leading security experts
  • 27. HANDBOOK CONTRIBUTORSDave Bartram, SHL Group Ltd. Suzanne Lane, University of PittsburghEugene Burke, SHL Group Ltd. Dennis Maynes, Caveon Test SecurityJanet Duffy Carson, Legal Consultant Roger D. Meade, Prometric, Inc.Susan M. Case, National Conference of Bar Examiners Jaime R. Mulkey, , Caveon Test SecurityBeth E. Donahue, Data Recognition Corporation Sara Rzepa , Multi-Health Systems, Inc.Steve Ferrara, Pearson Rachel R. Watkins Schoenig, ACT, Inc.Cyndy Thomas Fitzgerald, Fitzgerald & Associates Anthony R. Scicchitano , Prometric, Inc.David Foster, Kryterion, Inc. Jennifer Ancona Semko, Baker & McKenzie LLPJohn J. Fremer, Caveon Test Security Gill Sitarenios , Multi-Health Systems, Inc.Deborah J. Harris, ACT, Inc. Hazel Wheldon, Multi-Health Systems, Inc.William Hatherill, Fed. of State Boards of Phys. Therapy Kevin M. Williams , Multi-Health Systems, Inc.Robert Hunt, Western Governors University James A. Wollack, University of WisconsinNeal Kingston, University of Kansas Louis Woodruff, Caveon Test Security
  • 28. TOP 10 TAKEAWAYS FROM HANDBOOK 1. Don’t become desensitized to cheating 2. Know your vulnerabilities 3. Many tools are available to identify, reduce likelihood of and mitigate damage from security breaches 4. Most serious threats to security happen before or after the test 5. Legal standards exist for score cancelation policies and admissibility of statistical evidence
  • 29. TOP 10 TAKEAWAYS FROM HANDBOOK6. Security planning is critically important7. Technology is both a friend and enemy8. We can learn important lessons from the test compromise experiences of our colleagues9. It takes a village to protect the integrity of our tests10. Test fraud is dynamic…we must respond in kind
  • 30. RESOURCES• CCSSO/ATP – Operational Best Practices• ATP Security Committee• Caveon Test Security – Blog – Webinars• (Just released) NCME Guidelines• (Just released) TILSA Guidebook for State Assessment Directors on Data Forensics
  • 31. WHERE TO PURCHASE• Don’t forget to pre-order your copy of the Handbook of Test Security• – (Case Sensitive) – Save 20% - Enter discount code: HYJ82
  • 32. THANK YOU!Dr. James A. Wollack Dr. John FremerAssociate Professor PresidentEducational Psychology Caveon Consulting ServicesDirector, Testing & Evaluation Services john.fremer@caveon.comUniversity of - Follow Caveon on twitter @caveon - Check out our blog… - LinkedIn Group – “Caveon Test Security”