Caveon Webinar - "Preventing & Detecting Cheating On High Stakes Tests"


Published on

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Fix size of text to fit on slide.
  • Insert Poll Question #2 Here! For the last three years has your testing program conducted cheating detection analyses (data forensics)?Yes, for every administrationYes, but only occasionallyNo
  • Insert Poll Question #1 Here! Has there been media coverage of a cheating incident within your testing program in the last three years? Yes/No
  • Increase font size- skyler
  • Caveon Webinar - "Preventing & Detecting Cheating On High Stakes Tests"

    1. 1. Hashtag #HStakes Caveon Webinar Series “Preventing and Detecting Cheating on High Stakes Tests”Presented by:John Fremer, President, Caveon Consulting ServicesVickie Cartwright, Sr. Director, Accountability, Research, and Assessment, Orange County School District
    2. 2. Presentation Overview Overall Summary External Assistance  Training  Security Audit  Data Forensics  Investigations Dealing with Media
    3. 3. Overall Summary Provide Appropriate Training Evaluate the Adequacy of Your Security Procedures Perform Cheating Analyses Routinely Follow Up Thoroughly on All Suspicions Take Appropriate and Public Action
    4. 4. Providing Appropriate Training Focus on Test Administration  Greatest vulnerability for all programs  Biggest “bang for the buck” Build on State and Vendor Materials Make Sure Everyone is Trained Have Building Administrators Participate
    5. 5. Issues for which ExternalAssistance is Recommended External Test Security Audit Data Forensics Web Patrol (Monitoring Internet) Security Investigations
    6. 6. Test Security Audit: The Importance of Knowing Validates your Security Practices Identifies Vulnerabilities Recommends and Prioritizes Actions to Correct Shows Your Seriousness about Security
    7. 7. Test Security Audit: What is it? Systematic Review by Experienced Professionals Objective Third Party Evaluation Use Formal Standards Seek Practical and Specific Advice
    8. 8. Need External Security Auditors Do Your Own Security Reviews Regularly? Have Such Reviews as an Assignment for a Key Staff Member Can’t do a Full Security Review Internally  It’s like evaluating your own children  Are they more beautiful or talented or smart?  Need objective and well-informed perspective
    9. 9. Caveon Test Security Audit Topics1. Security Plan 8. Test Administration2. Roles & Responsibilities 9. Test Scores & Results3. Budget & Funding 10. Physical security4. Legal Precautions & 11. Information SecurityAgreements 12. Web & Media Monitoring5. Test & Item Design 13. Security Awareness &6. Test & Item Development & TrainingMaintenance 14. Security Incidence7. Test Publication Response Plan 15. Conducting Security Investigations [New]
    10. 10. Questions?
    11. 11. Psychometric Analyses -Detecting CheatingData Forensics Looking at Your Test Data Has a Test been Compromised? When and Where was it Compromised?“Traditional” Procedures No Longer Good Enough
    12. 12. Data Forensics – School Districts Type of Data Available to Districts Usually Limited  Gains Analyses  Erasure Analyses in Paper and Pencil - by Inspection  Not practical except when there is a tip or something observed More Sophisticated Analyses Require More Data and are Very Desirable
    13. 13. Data Forensics Benefits Use Data to make Informed Decisions Train from Known Threats Detect and Deter Cheaters Stay Ahead of Cheating Methods Evaluate Test Admin Security Procedures Evidence of Strong Action – Bringing in Independent Experts
    14. 14. Other Data Forensics Analyses Similarity of Responses for Pairs and Groups of Students Aberrance – Failure to Have a Logical Pattern, e.g., Answering More Hard Items Correctly than Easy Ones Complete Erasure or Answer Changing Analyses – What % Wrong to Right
    15. 15. Success Story - State Assessment Reduction in "Irregularities" 72% reduction – 2 years 6.0% Test admin monitoring 5.0% Conditional status 4.0% Districts follow through 3.0% “Students confess” 2.0% Coordinator training 1.0% 0.0% Report 1 Report 2 Report 3 Report 4 Report 5 Report 6 Report 7 Report 8 Report 9
    16. 16. Web Patrol Are “Secure” Items on the Internet? Occurs in High Stakes Testing Long Testing Windows Tend to Lead to “Leakage” (Here comes the consortia) Involves Searching Tools, Expert Sleuthing, and Quick Follow-Up Very Common with Certification Programs
    17. 17. Any Questions?
    18. 18. Atlanta Public Schools Story – Persistent Accusations of Cheating in APS Schools Consistent Denials  Reports are wrong  Newspaper is prejudiced Data Revealed Unmistakable Evidence of Improprieties  Extreme cases very pronounced – one over 10 to 52nd power
    19. 19. Media Lessons from High Profile Incidents  Expect Media Attention  Get Out Front with Your Own Story  Cultivate Local Media  Most do not understand test results  Looking for interesting story – a no problem administration is not a good story  Learn Who You Can Trust  Don’t Expect a Cheating Story to Go Away – You Cannot Control It
    20. 20. FURTHER INFORMATION Slides & recording available at: Caveon Security Insights Blog  Twitter  @Caveon LinkedIn  Caveon Company Page  Caveon Test Security Group  Please contribute!