Caveon Webinar Series: Key Security Lessons Learned atATP’s “Innovations in Testing” 2013    Nikki Shepherd Eatchel       ...
Upcoming Caveon Events• National Council on Measurement in  Education Conference, San Francisco,  April 26-30• United Stat...
Caveon Online• Caveon Security Insights Blog   – http://www.caveon.com/blog/• twitter   – @Caveon   – Follow us!• LinkedIn...
Agenda for today• ATP Test Security Committee & Live Lab• Truth & Consequences, Conducting Effective Test  Security Invest...
But, before we get started…Caveon is proud to announce we are celebrating         10 years in business in 2013!
Security Committee Briefing                & Live LabGoals for the Year•   Security Survey•   Test Security Options•   Enf...
Survey ActivitiesActivities Completed• Three surveys designed by the ATPSC• Survey review and feedback provided by the  In...
Sneak Peak at the Data TrendsGeneral Information• Wider variety of respondents for 2012 survey• Increased focus on securit...
Sneak Peak at the Data Trends (Cont.)• Larger percentage of respondents with formal,  internal security plans• Significant...
Additional ATPSC Activities• Security Options Document• DMCA Takedown Effort• Candidate Rights and Responsibilities
Additional ATPSC Activities (Cont.)Security Options Document• Assessment Models• Delivery ChannelsCurrently available atww...
Additional ATPSC Activities (Cont.)DMCA Takedown Effort• 6 ISP’s were Identified hosting 17 Infringing  Websites   – One I...
Truth & Consequences:Conducting Effective Test Security Investigations    Benjamin Mannes, CPP, SSI, CHS-III    Director, ...
Truth & Consequences:Conducting Effective Test Security Investigations (Cont.) • The Proliferation of Risk • Core Elements...
Truth & Consequences:Conducting Effective Test Security Investigations (Cont.) Identification and Vetting • Strong web pre...
Truth & Consequences:Conducting Effective Test Security Investigations (Cont.)   Investigation   • Gather   • Preserve
Truth & Consequences:Conducting Effective Test Security Investigations (Cont.)  Field Interviews  • Purpose  • Participant...
Truth & Consequences:Conducting Effective Test Security Investigations (Cont.) Enforcement • Options • Organizational Goal...
HANDBOOK OF TEST SECURITY• Editors - James Wollack & John Fremer• Participating Authors – Deborah Harris, William  Hatheri...
HANDBOOK – PRIMARY AIMS• Insights of Psychometricians and Policy-Makers• Best Practices for Designing Secure Tests• Analys...
HANDBOOK – FIVE TAKEAWAYS• Security Vulnerabilities for All Genres of Testing• Critical Importance of Security Planning• P...
HANDBOOK – FIVE PREDICTIONS• In Many High Stakes Testing Programs  – Cheating detection statistical analyses will be perfo...
DATA FORENSICS,      OPENING THE BLACK BOX•   John Fremer•   Neal Kingston•   James Wollack•   Dennis Maynes
DATA FORENSICS,  OPENING THE BLACK BOX (cont.)• Features of “Mature” Forensics Approaches  –   Explicit standards  –   One...
DATA FORENSICS,   OPENING THE BLACK BOX (cont.)• Useful Statistics   – Similarity – Unusual number of same answers   – Abe...
THE GAME’S AFOOT:       SLEUTHS MATCH WITS• Kim Brunnert – Elsevier• Joy Matthews-Lopez and Paul Jones – National  Associa...
THE GAME’S AFOOT (cont.)• Overview of Problem• In 2010, a test preparation course   – Harvested items   – Imputed a key   ...
THE GAME’S AFOOT (cont.)• Live Data Set• Three Teams• Illustration of Problem   – 32 identical tests with a score of 95%  ...
THE GAME’S AFOOT (cont.)• Similarities of Team Approaches  –   Examination of test scores and histograms  –   Estimation o...
THE GAME’S AFOOT (cont.)• Summary of Project  – Optimal solution is not obvious  – Several models can and were used  – Pro...
TELL IT TO THE JUDGE    Winning With Data Forensics Evidence in Court•   Aimee Hobby Rhodes, JD•   James Wollack, Ph.D.•  ...
TELL IT TO THE JUDGE (cont.)•   Global Financial Certification Program•   Newly Using Data Forensics•   Strange Results Ob...
TELL IT TO THE JUDGE (cont.)1.   Use an Expert who is truly an Expert2.   Use a Trusted and Tried Methodology3.   Report F...
TELL IT TO THE JUDGE (cont.)7. Use Your Expert as a Teacher – Explaining what wasdone and Why the Evidence is Compelling8....
Balancing Security and AccessibilitySheryl LazarusMartha ThurlowNational Center on Educational Outcomes (NCEO)University o...
Balancing Test Security and Accessibility (Cont.)  Accessibility Issues  • Paper-Based  • Computer-Based  Students can uni...
Balancing Test Security and Accessibility (Cont.)  Typical Accommodation Options  •   Test Administrator Access  •   Visua...
Balancing Test Security and Accessibility (Cont.)Best Practices•   Test Security Guidelines Addressing Accommodations•   T...
Balancing Test Security and Accessibility (Cont.)ReferencesStandards for Educational and Psychological Testing(APA/AERA/NC...
RESOURCES• CCSSO/ATP – Operational Best Practices• ATP Security Committee• Caveon Test Security   – Blog   – Webinars• (Ju...
CAVEON’S 10 YEAR ANNIVERSARY VIDEO  Take a look and celebrate with us!        http://lnkd.in/XFBKR7
THANK YOU!    Nikki Shepherd Eatchel            Dr. John Fremer    Vice President                    President    Questar ...
Upcoming SlideShare
Loading in …5
×

Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013

431 views
303 views

Published on

The annual Association of Test Publishers Innovations in Testing Conference was held last week in Fort Lauderdale, Florida and Caveon was there! As expected, test security was front and center on the conference program, as more than 17 sessions included test security topics.

Dr. John Fremer, President of Caveon Consulting Services, joined by Nikki Eatchel, Vice President, Program Management, Questar Assessment, Inc and the 2011-2012 ATP Security Committee Chairperson, for an informational webinar that explores key takeaways and lessons learned on security from this year’s ATP conference.

If you missed the conference or some of the important sessions on test security, this is one way to stay updated on the latest and greatest industry security trends

If you have any questions or would like to watch the recording of this session, please contact Richelle Gruber at richelle.gruber@caveon.com

Thank you!

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
431
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • New item typesPerformance based itemsRemote proctoring technologies in high risk areasDevelopment a deeper item poolsData forensicsWeb patrolOnlineBiometricTechnology AdvancementsBroader use of Data Forensics
  • Which test security group are we talking about here, Caveon’s? Then it’s Caveon Test Security - Skyler
  • Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013

    1. 1. Caveon Webinar Series: Key Security Lessons Learned atATP’s “Innovations in Testing” 2013 Nikki Shepherd Eatchel Dr. John Fremer Vice President President Program Management Caveon Consulting Service Questar Assessment February 21, 2013
    2. 2. Upcoming Caveon Events• National Council on Measurement in Education Conference, San Francisco, April 26-30• United States Distance Education Association Conference, St. Louis, April 28 - May 1• National Conference on Student Assessment, National Harbor, June 19-21
    3. 3. Caveon Online• Caveon Security Insights Blog – http://www.caveon.com/blog/• twitter – @Caveon – Follow us!• LinkedIn – Caveon Company Page – Caveon Test Security Group • Please contribute!• Facebook – Will you be our “friend?” – “Like” us! www.caveon.com
    4. 4. Agenda for today• ATP Test Security Committee & Live Lab• Truth & Consequences, Conducting Effective Test Security Investigations• The Handbook of Test Security• Data Forensics, Opening the Black Box• The Game’s Afoot: Sleuths Match Wits• Tell it to the Judge, Winning with Data Forensics Evidence in Court• Balancing Test Security and Accessibility
    5. 5. But, before we get started…Caveon is proud to announce we are celebrating 10 years in business in 2013!
    6. 6. Security Committee Briefing & Live LabGoals for the Year• Security Survey• Test Security Options• Enforcement• Candidate Rights and Responsibilities• Newsletters• Live Lab
    7. 7. Survey ActivitiesActivities Completed• Three surveys designed by the ATPSC• Survey review and feedback provided by the Institute for Credentialing Excellence (ICE)• Survey Distributed, 117 respondents – 79 Certification/Licensure – 16 Education – 22 Vendor• Initial analysis complete
    8. 8. Sneak Peak at the Data TrendsGeneral Information• Wider variety of respondents for 2012 survey• Increased focus on security by all respondent organizations• Large percentage of “Don’t Know” responses
    9. 9. Sneak Peak at the Data Trends (Cont.)• Larger percentage of respondents with formal, internal security plans• Significant gap in formal, external security processes• Lack of standardization implementation• Few audits
    10. 10. Additional ATPSC Activities• Security Options Document• DMCA Takedown Effort• Candidate Rights and Responsibilities
    11. 11. Additional ATPSC Activities (Cont.)Security Options Document• Assessment Models• Delivery ChannelsCurrently available atwww.testpublishers.org
    12. 12. Additional ATPSC Activities (Cont.)DMCA Takedown Effort• 6 ISP’s were Identified hosting 17 Infringing Websites – One ISP – all websites removed – Approximately 30% of the websites no longer exist and affiliate websites were also removedCandidate Rights and Responsibilities• 8 organizations participated• Available May 2013
    13. 13. Truth & Consequences:Conducting Effective Test Security Investigations Benjamin Mannes, CPP, SSI, CHS-III Director, Test Security American Board of Internal Medicine Marc J. Weinstein, Esq. Partner Dillworth Paxson, LLP
    14. 14. Truth & Consequences:Conducting Effective Test Security Investigations (Cont.) • The Proliferation of Risk • Core Elements of Exam Integrity – Protect – Investigate – Enforce
    15. 15. Truth & Consequences:Conducting Effective Test Security Investigations (Cont.) Identification and Vetting • Strong web presence • Ongoing review and analysis • Vetting Analyze • Collusion • Sharing • Inappropriate Study Techniques • Violations • Proxy
    16. 16. Truth & Consequences:Conducting Effective Test Security Investigations (Cont.) Investigation • Gather • Preserve
    17. 17. Truth & Consequences:Conducting Effective Test Security Investigations (Cont.) Field Interviews • Purpose • Participants • Nonverbal communication
    18. 18. Truth & Consequences:Conducting Effective Test Security Investigations (Cont.) Enforcement • Options • Organizational Goals • Policy/Procedures, Laws/Regulations • Civil versus Criminal Join the conversation at: @ExamIntegrity on twitter LinkedIn Group: “Exam Integrity”
    19. 19. HANDBOOK OF TEST SECURITY• Editors - James Wollack & John Fremer• Participating Authors – Deborah Harris, William Hatherill, David Foster• Slated for Publication – March 2013• Preventing, Detecting, and Investigating Cheating• Testing in Many Domains – Certification/Licensure – Clinical – Educational – Industrial/Organizational
    20. 20. HANDBOOK – PRIMARY AIMS• Insights of Psychometricians and Policy-Makers• Best Practices for Designing Secure Tests• Analysis of Security Vulnerabilities• Practical Strategies for Cheating Prevention and Detection• Lessons Learned – Actual security violations – Security initiatives
    21. 21. HANDBOOK – FIVE TAKEAWAYS• Security Vulnerabilities for All Genres of Testing• Critical Importance of Security Planning• Practical and Proven Prevention & Detection Strategies• How Security Breaches Have Been Dealt With• Lessons We Have Learned From Past Instances of Testing Misbehavior
    22. 22. HANDBOOK – FIVE PREDICTIONS• In Many High Stakes Testing Programs – Cheating detection statistical analyses will be performed routinely – Computer-based testing will increasingly become the norm – Technology developments will be critically important to test security – Internationalization of testing programs will increase – “Test Security Manager” will become a recognized and valued position
    23. 23. DATA FORENSICS, OPENING THE BLACK BOX• John Fremer• Neal Kingston• James Wollack• Dennis Maynes
    24. 24. DATA FORENSICS, OPENING THE BLACK BOX (cont.)• Features of “Mature” Forensics Approaches – Explicit standards – One or more associations – Formal training programs – Regular conferences – One or more journals – Body of published literature – Regular positions in many agencies and companies
    25. 25. DATA FORENSICS, OPENING THE BLACK BOX (cont.)• Useful Statistics – Similarity – Unusual number of same answers – Aberrance – Answering hard questions correctly while missing easy ones – Gains (or drops) Atypical score changes over time – Erasures – Unusual patterns such as almost always wrong to right – Fast Responding – Much faster than others – superhuman? – Shared Information – Biometrics or personal information – Email or home addresses, photos, etc.
    26. 26. THE GAME’S AFOOT: SLEUTHS MATCH WITS• Kim Brunnert – Elsevier• Joy Matthews-Lopez and Paul Jones – National Association of Boards of Pharmacy• Lawrence Rudner – Graduate Management Admissions Council• Dennis Maynes – Caveon Test Security
    27. 27. THE GAME’S AFOOT (cont.)• Overview of Problem• In 2010, a test preparation course – Harvested items – Imputed a key – Distributed the content and key• Test Program Manager Reviewed Score Results• Many Scores at 95 %
    28. 28. THE GAME’S AFOOT (cont.)• Live Data Set• Three Teams• Illustration of Problem – 32 identical tests with a score of 95% – Probability of occurrence = 1 in 4 trillion• 100 Item Test• 387 Test Takers
    29. 29. THE GAME’S AFOOT (cont.)• Similarities of Team Approaches – Examination of test scores and histograms – Estimation of imputed key – Inspection of item performance – Filtering and splitting of test takers and items• Differences among Team Approaches – Types of classification rules and models – Degree of confidence in the number of test takers involved – Use of IRT models –may not apply when disclosed test content involved
    30. 30. THE GAME’S AFOOT (cont.)• Summary of Project – Optimal solution is not obvious – Several models can and were used – Problem is harder • Only some items and keys are compromised • Imputed answer keys are the actual keys – More research is needed • Detection of imputed answer keys • Bayes classifiers (need probability models given disclosure) – Harvesting and disclosure one of most damaging security breaches
    31. 31. TELL IT TO THE JUDGE Winning With Data Forensics Evidence in Court• Aimee Hobby Rhodes, JD• James Wollack, Ph.D.• Rachel Schoenig, JD• Jennifer Ancona Semko, JD• Steve Addicott
    32. 32. TELL IT TO THE JUDGE (cont.)• Global Financial Certification Program• Newly Using Data Forensics• Strange Results Observed• Test Program Manager Investigates• Ethics Board Votes to “De-certify”• Candidate Takes Case to Court
    33. 33. TELL IT TO THE JUDGE (cont.)1. Use an Expert who is truly an Expert2. Use a Trusted and Tried Methodology3. Report Fairly4. Set Up Procedures Properly5. Be Objective6. Be Reasonable in What You are Concluding
    34. 34. TELL IT TO THE JUDGE (cont.)7. Use Your Expert as a Teacher – Explaining what wasdone and Why the Evidence is Compelling8. Ensure that your Expert has all the Facts9. Take into Consideration all Reasonably AvailableEvidence10. Apply the Terms of your Candidate AgreementReasonably and in Good Faith
    35. 35. Balancing Security and AccessibilitySheryl LazarusMartha ThurlowNational Center on Educational Outcomes (NCEO)University of Minnesota
    36. 36. Balancing Test Security and Accessibility (Cont.) Accessibility Issues • Paper-Based • Computer-Based Students can unintentionally be denied meaningful access to examinations.
    37. 37. Balancing Test Security and Accessibility (Cont.) Typical Accommodation Options • Test Administrator Access • Visual Cues • Teacher Highlighting • Accessibility Software
    38. 38. Balancing Test Security and Accessibility (Cont.)Best Practices• Test Security Guidelines Addressing Accommodations• Test Security Agreements Addressing Accommodations• Consideration of Student IEP, 504, or LEP• Security Training• Protocols for Access to Testing Materials• Protocols for Distribution of Testing Materials
    39. 39. Balancing Test Security and Accessibility (Cont.)ReferencesStandards for Educational and Psychological Testing(APA/AERA/NCME, 1999)Testing and Data Integrity in the Assessment of StudentAchievement (NCME) – Draft documentOperational Best Practices for Statewide Large-ScaleAssessment Programs (ATP/CCSSO, 2010)
    40. 40. RESOURCES• CCSSO/ATP – Operational Best Practices• ATP Security Committee• Caveon Test Security – Blog – Webinars• (Just released) NCME Guidelines• (Just released) TILSA Guidebook for State Assessment Directors on Data Forensics
    41. 41. CAVEON’S 10 YEAR ANNIVERSARY VIDEO Take a look and celebrate with us! http://lnkd.in/XFBKR7
    42. 42. THANK YOU! Nikki Shepherd Eatchel Dr. John Fremer Vice President President Questar Assessment Caveon Consulting Services neatchel@questarai john.fremer@caveon.com- LinkedIn Group – Test Security- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog/- LinkedIn Group – Caveon Test Security

    ×