Don't Be The Next Atlanta Public Schools - Caveon Webinar


Published on

This session was presented by Dr. John Fremer, Ph.D. on May 16th, 2012 to help other state departments of education and school districts learn how to avoid being the next big cheating scandal. Atlanta Public Schools was the subject of one of the biggest cheating scandals in history, and if one thing is certain, it's that history will repeat itself. Use these tips to learn how to avoid being the center of a cheating scandal.

Published in: Education, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Insert Poll Question #1 Here! Has there been media coverage of a cheating incident within your testing program in the last three years? Yes/No
  • Insert Poll Question #2 Here! For the last three years has your testing program conducted cheating detection analyses (data forensics)?Yes, for every administrationYes, but only occasionallyNo
  • Don't Be The Next Atlanta Public Schools - Caveon Webinar

    1. 1. Upcoming Caveon Events Conference for the Statistical Detection of Potential Test Fraud, University of Kansas, May 23-24 – Register Now! National Conference on Student Assessment in Minneapolis, June 26-29 International Testing Commission Conference in Amsterdam, July 3-5 Joint Statistical Meetings in San Diego, July 28 – August 2 National Collegiate Testing Association in Minneapolis, August 1-4 Visit for more info
    2. 2. Caveon Online Caveon Security Insights Blog  Twitter  @Caveon LinkedIn  Caveon Company Page  Caveon Test Security Group  Please contribute! Facebook  Will you be our “friend?”  “Like” us!
    3. 3. Background The following is an encore presentation of Dr. Fremer’s Plenary Session at April’s National Association of Test Directors Conference We are pleased to share his Certificate of Recognition for “Outstanding Contributions to Educational Assessment”
    4. 4. National Association of Test Directors Certificate of Recognition• 40 Years Improving Educ. Assess.• Revision of the SAT• Book “Measuring Up”• Writing about Computer Assessment• As NCME President - Advocated Inclusion for States, Districts, and Schools• Helping Promote Testing Integrity
    5. 5. Caveon Webinar Series“How to Avoid Becoming Another Atlanta Public Schools” John Fremer President, Caveon Consulting Services May 16, 2012
    6. 6. Presentation Overview  The Atlanta Story  Lessons  External Assistance
    7. 7. Caveon Role in APS Contract with Atlanta Educational Foundation  Not APS  Not state of Georgia Security Audit Limited Data Forensics
    8. 8. Atlanta Story – Short Form Persistent Accusations of Cheating in APS Schools Consistent Denials  Reports are wrong  Newspaper is prejudiced  Governor out to get Superintendent Governor’s Office of Student Achievement and Atlanta Journal Constitution Stayed on Case
    9. 9. What Did the Data Show? Unmistakable Evidence of Improprieties  Impossible to miss  Extreme cases very pronounced – one over 10 to 52nd power! Governor Lost All Patience  Georgia Bureau of Investigation  61 agents  Subpoena powers  “Deals” offered
    10. 10. Outcome for APS Superintendent Out – A National and Highly Regarded Educator All Deputy Superintendents Out Many Principals and Teachers Confess Those Pleading Fifth Amendment Declared Guilty Names of all Judged Guilty Published APS Situation Known Nationwide
    11. 11. Questions?
    12. 12. Testing Management Lessonsfrom Atlanta Don’t Make Political Leaders Mad at You! Perform Cheating Analyses Routinely Evaluate the Adequacy of Your Security Procedures Follow Up Thoroughly on All Suspicions Take Appropriate and Public Action
    13. 13. Media Lessons from Atlanta Expect Media Attention Get Out Front with Your Own Story Cultivate Local Media  Most do not understand test results  Looking for interesting story – a no problem administration is not a good story Learn Who You Can Trust Don’t Expect a Cheating Story to Go Away – You Cannot Control It
    14. 14. Any Questions?
    15. 15. Possible External Assistance  Test Security Audit  Data Forensics  Web Patrol (Monitoring Internet)  Security Investigations
    16. 16. Test Security Audit: The Importance of Knowing Validates your Security Practices Identifies Vulnerabilities Recommends and Prioritizes Actions to Correct Shows Your Seriousness about Security
    17. 17. Test Security Audit: What is it? Systematic Review by Experienced Professionals Objective Third Party Evaluation Over 250 Standards Immediate Feedback Practical and Specific
    18. 18. Test Security Audit Topics 8. Test Administration1. Security Plan 9. Test Scores & Results2. Roles & Responsibilities 10. Physical security3. Budget & Funding 11. Information Security4. Legal Precautions &Agreements 12. Web & Media Monitoring5. Test & Item Design 13. Security Awareness & Training6. Test & Item Development &Maintenance 14. Security Incidence Response Plan7. Test Publication 15. Conducting Security Investigations [New]
    19. 19. Psychometric Analyses - Detecting CheatingData Forensics Looking at Your Test Data Has a Test been Compromised? When and Where was itCompromised? “Traditional” Procedures NoLonger Good Enough
    20. 20. Data Forensics Benefits Use Data to make Informed Decisions Train from Known Threats Detect and Deter Cheaters Stay Ahead of Cheating Methods Evaluate Test Admin Security Procedures Evidence of Strong Action – Bringing in Independent Experts
    21. 21. Success Story - State Assessment Reduction in "Irregularities" 72% reduction – 2 years 6.0% Test admin monitoring 5.0% Conditional status 4.0% Districts follow through 3.0% “Students confess” 2.0% Coordinator training 1.0% 0.0% Report 1 Report 2 Report 3 Report 4 Report 5 Report 6 Report 7 Report 8 Report 9
    22. 22. Web Patrol Are “Secure” Items on the Internet? Occurs in High Stakes Testing Long Testing Windows Tend to Lead to “Leakage” Involves Searching Tools, Expert Sleuthing, and Quick Follow-Up Very Common with Certification Programs
    23. 23. Conducting Investigations Evaluating Cases Warranting Follow- Up Using all Available Data Interviewing Teachers, Test Coordinators, Principals First Response is Always – “There is no problem” Careers are at Stake – Need Very Solid Evidence
    24. 24. Continue The Conversation- Follow us on twitter @caveon for updates and events- Check out our blog… LinkedIn Group – Join our Caveon Test Security group to ask questions and join discussions about test security topics- Slides of this and past Webinars are available at
    25. 25. Please Join Us Next Time!“Tales from an Examnivore: Utilizing testdevelopment resources to support yourtesting program”Featuring Jamie MulkeyVice President, Test Development ServicesCaveonSave The Date - June 20th
    26. 26. Thank you!Dr. John Fremer, Ph.D.PresidentCaveon Consulting