Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
UPCOMING CAVEON EVENTS
• EATP – European Association of
Test Publishers - September 25-27,
St. Julian’s, Malta.
• ICE - In...
CAVEON ONLINE
• Caveon Security Insights Blog
– http://www.caveon.com/blog/
• Twitter
– @Caveon
– Follow us!
• LinkedIn
– ...
"Preventing and Detecting Cheating: Lessons from
the CCSSO National Conference on Student
Assessment"
Steve Addicott Dr. J...
AGENDA FOR TODAY
• State Assessment Directors Meeting
• CCSSO TILSA Guidebook
• Greg Cizek Recommendations
• ―The Critical...
STATE ASSESSMENT DIRECTORS
• State Assessment Directors
Meeting
• ―Hallway‖ Conversations
• ―Plan B‖ Dominant Theme
– Virt...
CCSSO TILSA GUIDEBOOK SESSION
• Context
– Charlene Tucker – TILSA Perspective
– John Olson – Project Leader’s Overview
• S...
KENTUCKY TEST SECURITY –
FIRST STEPS
1. Commitment From Senior Leadership
2. Professional Audit
KENTUCKY: POLICIES & PRACTICE -
PREVENTION
• Security Plan
o Findings: The Department of Education does
not maintain a sin...
KENTUCKY: POLICIES & PRACTICE -
DETECTION
Data Forensics Analysis
o Action Tasks:
• Adding additional data forensics analy...
WEST VIRGINIA PERSPECTIVE –
AN IMPORTANT ADDITION
• A compilation of ―Best Practices‖
• 3 Key Areas
– Prevention
– Detecti...
WEST VIRGINIA - STATE AWARENESS
TIMELINE
• Test security has always been of interest to
SEAs
• Security breaches have thru...
WEST VIRGINIA - A STATE’S
POTENTIAL CONTEXT
• The 3 Ps of Assessment Integrity
– Policy
– Politics
– Practical Implication...
WEST VIRGINIA - WHERE ARE
WE NOW?
• Waiting for the next AJC article
• (Quality?) Information everywhere about test
securi...
OVERVIEW OF FREMER PRESENTATION
Recommendations (Sample Presented)
Prevention
Detection
Test Security Investigations
PREVENTION
• Communicate Zero Tolerance
1. Clear
2. Consistent
3. Frequently emphasized messages
• Develop a Security Plan...
DETECTION
• Use Multiple Data Forensics Methods
1. Unusual Gains
2. Similarity
3. Erasures (Answer Changing)
4. Person Fit...
TEST SECURITY INVESTIGATIONS
• Areas Needing Attention
– Level of evidence warranting an
investigation
– Roles for state, ...
TEST SECURITY INVESTIGATIONS (cont)
• Model Investigations Kit
– Clear definition of responsibilities
– Procedures for eva...
TEST SECURITY INVESTIGATIONS (cont.)
– Focus Primarily on Improving Security in
the Future
• Follow ―Innocent Until Proven...
GREG CIZEK RECOMMENDATIONS
A SAMPLE
• More Cheating than You Think!
– “Anyone who believes that his or her
program doesn’t...
SESSION: THE CRITICAL ROLE OF POLICY
IN MAINTAINING TEST SECURITY
Best thing —This session was on program!
• Presentations...
SESSION: THE CRITICAL ROLE OF POLICY
IN MAINTAINING TEST SECURITY
Mississippi Department of Education (MDE)
• James Mason ...
SESSION: THE CRITICAL ROLE OF POLICY
IN MAINTAINING TEST SECURITY
Florida Department of Education
• Formal Data Forensics ...
SESSION: THE CRITICAL ROLE OF POLICY
IN MAINTAINING TEST SECURITY
Maryland State Department of Education
• Dr. Tamara Lewi...
SESSION: THE CRITICAL ROLE OF POLICY
IN MAINTAINING TEST SECURITY
MSDE Mission ―POSSIBLE‖
• Policy on test security
• Oppo...
SESSION: CCSSO/ATP OPERATIONAL
BEST PRACTICES UPDATE
 Christopher Hanczrik – State of WA, Office of
Superintendent of Pub...
SESSION: CCSSO/ATP OPERATIONAL
BEST PRACTICES UPDATE
Chapter 8: Test Security
Define and design a comprehensive security p...
SCHOOL DISTRICT CHALLENGES
Non-session oriented takeaways
• School District leaders are in a crucible
– Tons of Exams
– Ev...
OTHER SIGNIFICANT EVENTS
International Test Commission (ITC)
• Publishing ―Best Practices‖
• Adapted from Caveon Security ...
RESOURCES
• CCSSO/ATP – Operational Best Practices
• ATP Security Committee
• Caveon Test Security
– Blog
– Webinars
• NCM...
Caveon’s 10 Year Anniversary Video,
Take a look and celebrate with us!
http://lnkd.in/XFBKR7
Thank you!
Steve Addicott Dr. John Fremer
Vice President President
Caveon Test Security Caveon Consulting Services
Steve.a...
Upcoming SlideShare
Loading in …5
×

Caveon Webinar Series: Preventing and Detecting Cheating, Lessons from CCSSO's National Conference on Student Assessment 6-2013

666 views

Published on

Discover security lessons learned from the program sessions and face-to-face meetings with state assessment and national organization staff at the June 2013 National Conference on Student Assessment in National Harbor, Maryland. Caveon Leaders John Fremer and Steve Addicott summarize the test security ideas and strategies that they drew from the conference. Among the points covered are the recommendations included in the new TILSA Test Security Guidebook which was released at the conference.

Published in: Education, Technology, Business
  • Be the first to comment

  • Be the first to like this

Caveon Webinar Series: Preventing and Detecting Cheating, Lessons from CCSSO's National Conference on Student Assessment 6-2013

  1. 1. UPCOMING CAVEON EVENTS • EATP – European Association of Test Publishers - September 25-27, St. Julian’s, Malta. • ICE - Institute for Credentialing Excellence - November 11-14, Amelia Island, FL • ATP 2014 - March 2-5, 2014, Scottsdale, AZ
  2. 2. CAVEON ONLINE • Caveon Security Insights Blog – http://www.caveon.com/blog/ • Twitter – @Caveon – Follow us! • LinkedIn – Caveon Company Page – Caveon Test Security Group • Please contribute! • Facebook – Will you be our ―friend?‖ – ―Like‖ us! www.caveon.com
  3. 3. "Preventing and Detecting Cheating: Lessons from the CCSSO National Conference on Student Assessment" Steve Addicott Dr. John Fremer Vice President President Caveon Test Security Caveon Consulting Services July 17, 2013 Caveon Webinar Series:
  4. 4. AGENDA FOR TODAY • State Assessment Directors Meeting • CCSSO TILSA Guidebook • Greg Cizek Recommendations • ―The Critical Role of Policy in Test Security‖ • CCSSO/ATP Operational Best Practices • Ongoing Challenges for School Districts • Other events – ITC Security Standards – 2nd Annual CSDPTF
  5. 5. STATE ASSESSMENT DIRECTORS • State Assessment Directors Meeting • ―Hallway‖ Conversations • ―Plan B‖ Dominant Theme – Virtually all Consortium States have one – One year extension of current pattern very common alternative – Some states have not done much fleshing out an alternative – Security and test difficulty could be major problems
  6. 6. CCSSO TILSA GUIDEBOOK SESSION • Context – Charlene Tucker – TILSA Perspective – John Olson – Project Leader’s Overview • State Perspectives – Kentucky – West Virginia • Key Recommendations in Guidebook • Wonderful Greg Cizek Discussant Comments
  7. 7. KENTUCKY TEST SECURITY – FIRST STEPS 1. Commitment From Senior Leadership 2. Professional Audit
  8. 8. KENTUCKY: POLICIES & PRACTICE - PREVENTION • Security Plan o Findings: The Department of Education does not maintain a single, comprehensive security plan/manual o Action Tasks: • Developing a comprehensive security plan/manual • Providing security plan/manual to all key staff and portions to outside contributors as appropriate
  9. 9. KENTUCKY: POLICIES & PRACTICE - DETECTION Data Forensics Analysis o Action Tasks: • Adding additional data forensics analysis • Erasure Analysis • Similarity Analysis • Score Growth and Losses
  10. 10. WEST VIRGINIA PERSPECTIVE – AN IMPORTANT ADDITION • A compilation of ―Best Practices‖ • 3 Key Areas – Prevention – Detection – Investigation • Key Implications: – Potential source to conduct a gap analysis – Preparation for any test security requirements
  11. 11. WEST VIRGINIA - STATE AWARENESS TIMELINE • Test security has always been of interest to SEAs • Security breaches have thrust assessment issues into the spotlight • SEAs, LEAs, and schools now have a dedicated interest in test security and assessment integrity
  12. 12. WEST VIRGINIA - A STATE’S POTENTIAL CONTEXT • The 3 Ps of Assessment Integrity – Policy – Politics – Practical Implications (i.e., reliability of results) • What is the driver? – Timing + Severity + Pervasiveness + Climate = – Crapshoot
  13. 13. WEST VIRGINIA - WHERE ARE WE NOW? • Waiting for the next AJC article • (Quality?) Information everywhere about test security practices, audits, and requirements • The guidebook providing a comprehensive look at the three major categories • Taking recommendations to create actionable information that practitioners can use to revise practice and policy
  14. 14. OVERVIEW OF FREMER PRESENTATION Recommendations (Sample Presented) Prevention Detection Test Security Investigations
  15. 15. PREVENTION • Communicate Zero Tolerance 1. Clear 2. Consistent 3. Frequently emphasized messages • Develop a Security Plan – Comprehensive – Up to date
  16. 16. DETECTION • Use Multiple Data Forensics Methods 1. Unusual Gains 2. Similarity 3. Erasures (Answer Changing) 4. Person Fit Analyses • Introduce Data Forensics Very Carefully – Think Through Planned Uses – Prepare LEAs (and others) for Data Forensics Use – Planful, Multi-step Public Information Program
  17. 17. TEST SECURITY INVESTIGATIONS • Areas Needing Attention – Level of evidence warranting an investigation – Roles for state, district, and other staff – Timelines – Model investigations kit
  18. 18. TEST SECURITY INVESTIGATIONS (cont) • Model Investigations Kit – Clear definition of responsibilities – Procedures for evaluating evidence – Planning and conducting interviews – Developing a report with recommendations
  19. 19. TEST SECURITY INVESTIGATIONS (cont.) – Focus Primarily on Improving Security in the Future • Follow ―Innocent Until Proven Guilty‖ Approach, but be Firm in your Questioning • Give Most Attention to the ―Worst of the Worst‖ • Consider Other Possible Explanations for Results
  20. 20. GREG CIZEK RECOMMENDATIONS A SAMPLE • More Cheating than You Think! – “Anyone who believes that his or her program doesn’t have a serious cheating problem has simply not investigated the concern deeply enough.” • Don’t do Cheating Statistical Analyses until you have a Plan for Using the Results • Focus on Score Validity. Look to Skilled Investigators for Help if Needed
  21. 21. SESSION: THE CRITICAL ROLE OF POLICY IN MAINTAINING TEST SECURITY Best thing —This session was on program! • Presentations from Mississippi, Florida, and Maryland DOEs • 3 different perspectives on security policies
  22. 22. SESSION: THE CRITICAL ROLE OF POLICY IN MAINTAINING TEST SECURITY Mississippi Department of Education (MDE) • James Mason & Walt Drane • Shared three incidents • Important lessons learned – No going back! – Have multiple enforcement options – Consistent use of data and policies – Play to win – Plan for the future
  23. 23. SESSION: THE CRITICAL ROLE OF POLICY IN MAINTAINING TEST SECURITY Florida Department of Education • Formal Data Forensics Program in Place – Student level and school level – Analyses drive invalidations (students) and investigations (schools) – Appeals process implemented – ―Measure to Manage‖ – Make resources available to all stakeholders
  24. 24. SESSION: THE CRITICAL ROLE OF POLICY IN MAINTAINING TEST SECURITY Maryland State Department of Education • Dr. Tamara Lewis, State Test Security Officer • State Test Administration & Security Committee (STASC)
  25. 25. SESSION: THE CRITICAL ROLE OF POLICY IN MAINTAINING TEST SECURITY MSDE Mission ―POSSIBLE‖ • Policy on test security • Opportunities for training • School security plans • Standards for detecting infractions • Investigation of suspected infractions • Being prompt in reporting infractions • Levying appropriate action/sanction • Examining areas for improvement
  26. 26. SESSION: CCSSO/ATP OPERATIONAL BEST PRACTICES UPDATE  Christopher Hanczrik – State of WA, Office of Superintendent of Public Instruction (Co-Chair)  Pat Roschewski – Data Recognition Corporation  Sarah Susbury – Virginia Department of Education  Lisa R. Ehrlich – Measured Progress (Co-Chair)
  27. 27. SESSION: CCSSO/ATP OPERATIONAL BEST PRACTICES UPDATE Chapter 8: Test Security Define and design a comprehensive security plan  Internal processes  Computer systems  Electronic transfers  Test administration BEST ADVICE  Develop the Plan  Implement it!!
  28. 28. SCHOOL DISTRICT CHALLENGES Non-session oriented takeaways • School District leaders are in a crucible – Tons of Exams – Ever greater stakes • Incumbent on us to provide the tools they need to measure and manage security risks • Provide the right resources to investigate
  29. 29. OTHER SIGNIFICANT EVENTS International Test Commission (ITC) • Publishing ―Best Practices‖ • Adapted from Caveon Security Audit Conference on Statistical Detection of Potential Test Fraud • October 17-19, 2013 • University of Wisconsin at Madison • Program: http://bit.ly/1b6jM42 • Registration: http://bit.ly/18b1fzM
  30. 30. RESOURCES • CCSSO/ATP – Operational Best Practices • ATP Security Committee • Caveon Test Security – Blog – Webinars • NCME Guidelines – http://ncme.org/default/assets/File/Committee%20Docs/Test%20Score% 20Integrity/Test%20Integrity- NCME%20Endorsed%20(2012%20FINAL).pdf • TILSA Guidebook for State Assessment Directors on Data Forensics – Buy the Guidebook at https://www.createspace.com/4302978 – Print for free at http://www.ccsso.org/Resources/Publications/TILSA_Test_Security _Guidebook.html – Read Greg Cizek’s observations – http://www.caveon.com/tilsa- test-security-guidebook-next-steps/
  31. 31. Caveon’s 10 Year Anniversary Video, Take a look and celebrate with us! http://lnkd.in/XFBKR7
  32. 32. Thank you! Steve Addicott Dr. John Fremer Vice President President Caveon Test Security Caveon Consulting Services Steve.addicott@caveon.com john.fremer@caveon.com - Follow Caveon on twitter @caveon and - (Fremer @TestSecurityGuy) - Check out our blog…www.caveon.com/blog/ - LinkedIn Group – Caveon Test Security

×