Your SlideShare is downloading. ×
Caveon Webinar - "Preventing & Detecting Cheating On High Stakes Tests"
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Caveon Webinar - "Preventing & Detecting Cheating On High Stakes Tests"


Published on

Published in: Education

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • Fix size of text to fit on slide.
  • Insert Poll Question #2 Here! For the last three years has your testing program conducted cheating detection analyses (data forensics)?Yes, for every administrationYes, but only occasionallyNo
  • Insert Poll Question #1 Here! Has there been media coverage of a cheating incident within your testing program in the last three years? Yes/No
  • Increase font size- skyler
  • Transcript

    • 1. Hashtag #HStakes Caveon Webinar Series “Preventing and Detecting Cheating on High Stakes Tests”Presented by:John Fremer, President, Caveon Consulting ServicesVickie Cartwright, Sr. Director, Accountability, Research, and Assessment, Orange County School District
    • 2. Presentation Overview Overall Summary External Assistance  Training  Security Audit  Data Forensics  Investigations Dealing with Media
    • 3. Overall Summary Provide Appropriate Training Evaluate the Adequacy of Your Security Procedures Perform Cheating Analyses Routinely Follow Up Thoroughly on All Suspicions Take Appropriate and Public Action
    • 4. Providing Appropriate Training Focus on Test Administration  Greatest vulnerability for all programs  Biggest “bang for the buck” Build on State and Vendor Materials Make Sure Everyone is Trained Have Building Administrators Participate
    • 5. Issues for which ExternalAssistance is Recommended External Test Security Audit Data Forensics Web Patrol (Monitoring Internet) Security Investigations
    • 6. Test Security Audit: The Importance of Knowing Validates your Security Practices Identifies Vulnerabilities Recommends and Prioritizes Actions to Correct Shows Your Seriousness about Security
    • 7. Test Security Audit: What is it? Systematic Review by Experienced Professionals Objective Third Party Evaluation Use Formal Standards Seek Practical and Specific Advice
    • 8. Need External Security Auditors Do Your Own Security Reviews Regularly? Have Such Reviews as an Assignment for a Key Staff Member Can’t do a Full Security Review Internally  It’s like evaluating your own children  Are they more beautiful or talented or smart?  Need objective and well-informed perspective
    • 9. Caveon Test Security Audit Topics1. Security Plan 8. Test Administration2. Roles & Responsibilities 9. Test Scores & Results3. Budget & Funding 10. Physical security4. Legal Precautions & 11. Information SecurityAgreements 12. Web & Media Monitoring5. Test & Item Design 13. Security Awareness &6. Test & Item Development & TrainingMaintenance 14. Security Incidence7. Test Publication Response Plan 15. Conducting Security Investigations [New]
    • 10. Questions?
    • 11. Psychometric Analyses -Detecting CheatingData Forensics Looking at Your Test Data Has a Test been Compromised? When and Where was it Compromised?“Traditional” Procedures No Longer Good Enough
    • 12. Data Forensics – School Districts Type of Data Available to Districts Usually Limited  Gains Analyses  Erasure Analyses in Paper and Pencil - by Inspection  Not practical except when there is a tip or something observed More Sophisticated Analyses Require More Data and are Very Desirable
    • 13. Data Forensics Benefits Use Data to make Informed Decisions Train from Known Threats Detect and Deter Cheaters Stay Ahead of Cheating Methods Evaluate Test Admin Security Procedures Evidence of Strong Action – Bringing in Independent Experts
    • 14. Other Data Forensics Analyses Similarity of Responses for Pairs and Groups of Students Aberrance – Failure to Have a Logical Pattern, e.g., Answering More Hard Items Correctly than Easy Ones Complete Erasure or Answer Changing Analyses – What % Wrong to Right
    • 15. Success Story - State Assessment Reduction in "Irregularities" 72% reduction – 2 years 6.0% Test admin monitoring 5.0% Conditional status 4.0% Districts follow through 3.0% “Students confess” 2.0% Coordinator training 1.0% 0.0% Report 1 Report 2 Report 3 Report 4 Report 5 Report 6 Report 7 Report 8 Report 9
    • 16. Web Patrol Are “Secure” Items on the Internet? Occurs in High Stakes Testing Long Testing Windows Tend to Lead to “Leakage” (Here comes the consortia) Involves Searching Tools, Expert Sleuthing, and Quick Follow-Up Very Common with Certification Programs
    • 17. Any Questions?
    • 18. Atlanta Public Schools Story – Persistent Accusations of Cheating in APS Schools Consistent Denials  Reports are wrong  Newspaper is prejudiced Data Revealed Unmistakable Evidence of Improprieties  Extreme cases very pronounced – one over 10 to 52nd power
    • 19. Media Lessons from High Profile Incidents  Expect Media Attention  Get Out Front with Your Own Story  Cultivate Local Media  Most do not understand test results  Looking for interesting story – a no problem administration is not a good story  Learn Who You Can Trust  Don’t Expect a Cheating Story to Go Away – You Cannot Control It
    • 20. FURTHER INFORMATION Slides & recording available at: Caveon Security Insights Blog  Twitter  @Caveon LinkedIn  Caveon Company Page  Caveon Test Security Group  Please contribute!