This session was presented by Dr. John Fremer, Ph.D. on May 16th, 2012 to help other state departments of education and school districts learn how to avoid being the next big cheating scandal. Atlanta Public Schools was the subject of one of the biggest cheating scandals in history, and if one thing is certain, it's that history will repeat itself. Use these tips to learn how to avoid being the center of a cheating scandal.

1. Upcoming Caveon Events
 Conference for the Statistical Detection of Potential Test
Fraud, University of Kansas, May 23-24 – Register Now!
 National Conference on Student Assessment in
Minneapolis, June 26-29
 International Testing Commission Conference in
Amsterdam, July 3-5
 Joint Statistical Meetings in San Diego, July 28 – August 2
 National Collegiate Testing Association in Minneapolis,
August 1-4
Visit www.caveon.com/events for more info

2. Caveon Online
 Caveon Security Insights Blog
 http://www.caveon.com/blog/
 Twitter
 @Caveon
 LinkedIn
 Caveon Company Page
 Caveon Test Security Group
 Please contribute!
 Facebook
 Will you be our “friend?”
 “Like” us!

3. Background
 The following is an encore presentation
of Dr. Fremer’s Plenary Session at
April’s National Association of Test
Directors Conference
 We are pleased to share his Certificate
of Recognition for “Outstanding
Contributions to Educational
Assessment”

5. National Association of Test Directors
Certificate of Recognition
• 40 Years Improving Educ. Assess.
• Revision of the SAT
• Book “Measuring Up”
• Writing about Computer Assessment
• As NCME President - Advocated
Inclusion for States, Districts, and
Schools
• Helping Promote Testing Integrity

6. Caveon Webinar Series
“How to Avoid Becoming Another
Atlanta Public Schools”
John Fremer
President, Caveon Consulting Services
May 16, 2012

7. Presentation Overview
 The Atlanta Story
 Lessons
 External Assistance

8. Caveon Role in APS
 Contract with Atlanta Educational
Foundation
 Not APS
 Not state of Georgia
 Security Audit
 Limited Data Forensics

9. Atlanta Story – Short Form
 Persistent Accusations of Cheating in
APS Schools
 Consistent Denials
 Reports are wrong
 Newspaper is prejudiced
 Governor out to get Superintendent
 Governor’s Office of Student
Achievement and Atlanta Journal
Constitution Stayed on Case

10. What Did the Data Show?
 Unmistakable Evidence of
Improprieties
 Impossible to miss
 Extreme cases very pronounced – one
over 10 to 52nd power!
 Governor Lost All Patience
 Georgia Bureau of Investigation
 61 agents
 Subpoena powers
 “Deals” offered

11. Outcome for APS
 Superintendent Out – A National and
Highly Regarded Educator
 All Deputy Superintendents Out
 Many Principals and Teachers Confess
 Those Pleading Fifth Amendment
Declared Guilty
 Names of all Judged Guilty Published
 APS Situation Known Nationwide

12. Questions?

13. Testing Management Lessons
from Atlanta
 Don’t Make Political Leaders Mad at
You!
 Perform Cheating Analyses Routinely
 Evaluate the Adequacy of Your
Security Procedures
 Follow Up Thoroughly on All
Suspicions
 Take Appropriate and Public Action

14. Media Lessons from Atlanta
 Expect Media Attention
 Get Out Front with Your Own Story
 Cultivate Local Media
 Most do not understand test results
 Looking for interesting story – a no
problem administration is not a good
story
 Learn Who You Can Trust
 Don’t Expect a Cheating Story to Go
Away – You Cannot Control It

15. Any Questions?

16. Possible External Assistance
 Test Security Audit
 Data Forensics
 Web Patrol (Monitoring Internet)
 Security Investigations

17. Test Security Audit: The Importance
of Knowing
 Validates your Security Practices
 Identifies Vulnerabilities
 Recommends and Prioritizes Actions to
Correct
 Shows Your Seriousness about Security

18. Test Security Audit: What is it?
 Systematic Review by Experienced
Professionals
 Objective Third Party Evaluation
 Over 250 Standards
 Immediate Feedback
 Practical and Specific

19. Test Security Audit Topics
8. Test Administration
1. Security Plan
9. Test Scores & Results
2. Roles & Responsibilities
10. Physical security
3. Budget & Funding
11. Information Security
4. Legal Precautions &
Agreements 12. Web & Media Monitoring
5. Test & Item Design 13. Security Awareness &
Training
6. Test & Item Development &
Maintenance 14. Security Incidence
Response Plan
7. Test Publication
15. Conducting Security
Investigations [New]

20. Psychometric Analyses -
Detecting Cheating
Data Forensics
 Looking at Your Test Data
 Has a Test been Compromised?
 When and Where was it
Compromised?
 “Traditional” Procedures No
Longer Good Enough

21. Data Forensics Benefits
 Use Data to make Informed Decisions
 Train from Known Threats
 Detect and Deter Cheaters
 Stay Ahead of Cheating Methods
 Evaluate Test Admin Security
Procedures
 Evidence of Strong Action – Bringing in
Independent Experts

22. Success Story - State Assessment
Reduction in
"Irregularities"
 72% reduction – 2 years 6.0%
 Test admin monitoring 5.0%
 Conditional status 4.0%
 Districts follow through 3.0%
 “Students confess” 2.0%
 Coordinator training 1.0%
0.0%
Report 1
Report 2
Report 3
Report 4
Report 5
Report 6
Report 7
Report 8
Report 9

23. Web Patrol
 Are “Secure” Items on the Internet?
 Occurs in High Stakes Testing
 Long Testing Windows Tend to Lead to
“Leakage”
 Involves Searching Tools, Expert
Sleuthing, and Quick Follow-Up
 Very Common with Certification
Programs

24. Conducting Investigations
 Evaluating Cases Warranting Follow-
Up
 Using all Available Data
 Interviewing Teachers, Test
Coordinators, Principals
 First Response is Always – “There is no
problem”
 Careers are at Stake – Need Very Solid
Evidence

26. Continue The Conversation
- Follow us on twitter @caveon for updates and events
- Check out our blog…www.caveon.com/blog/
- LinkedIn Group – Join our Caveon Test Security group
to ask questions and join discussions about test
security topics
- Slides of this and past Webinars are available at
www.caveon.com/resources/webinars

27. Please Join Us Next Time!
“Tales from an Examnivore: Utilizing test
development resources to support your
testing program”
Featuring Jamie Mulkey
Vice President, Test Development Services
Caveon
Save The Date - June 20th

28. Thank you!
Dr. John Fremer, Ph.D.
President
Caveon Consulting Services
john.fremer@caveon.com