This webinar discusses on-site monitoring of standardized test administrations. On-site monitoring involves trained monitors visiting schools during tests to document test security practices, observe administrators and students, and ensure compliance with procedures. Monitoring supports test validity, provides quality assurance of the testing program, and complies with federal requirements. It was recommended that monitoring be both announced and unannounced. Observations are documented using checklists and any issues require follow-up. Monitoring data should be analyzed, compared across schools, and used to improve training, policies and future monitoring. The webinar outlined six steps for districts to establish an effective monitoring program.

1. www.caveon.com1
The What, Why and How of On-Site
Monitoring
March 1, 2017 12:00 p.m. ET
Caveon Webinar Series

2. Today’s Presenters
www.caveon.com2
Steve Addicott
Vice President , Sales
Marc Weinstein
Vice President, Caveon Investigative
Services

3. Today’s Questions
• What is on-site monitoring?
• Why implement on-site monitoring?
• How do we get started with on-site
monitoring?
• What do we do with the information from
monitoring?
www.caveon.com3

4. On-Site Monitoring
www.caveon.com4

5. What Is On-site Monitoring?
Trained monitors visit schools and testing rooms
during assessment administrations.
Monitors will:
Review and document school-level test security
compliance practices and procedures.
Observe test administrators, proctors and students.
Review all parts of the testing day.
Capture observations using pre-defined checklists for later
review.
www.caveon.com5

6. Why Have A Monitoring Program?
• Support the validity of assessment results.
• Provide quality assurance for your testing program.
– Gather information about implementation and efficacy of training
programs.
– Measure compliance with test administration processes and
protocols.
– Identify areas for improvement of process/policies.
• Comply with USED Peer Review Requirements.
www.caveon.com6

7. What Happens During On-site
Monitoring?
www.caveon.com7

8. Should We Announce Monitoring?
• Announced
– Pro: Administrators are more likely to ensure
compliance with policies and procedures.
– Con: School personnel on “best behavior,” so not a
true indicator or what happens during assessments
that are not monitored.
www.caveon.com8

9. Should We Announce Monitoring?
• Unannounced
– Pro: Provides a true indicator of what happens
during testing and encourages increased
compliance at all times
– Con: Added stress for test administrators
www.caveon.com9

10. During On-Site Monitoring
Look at all parts of the testing day
–Training logs and proctor agreements.
–School-level secure testing materials management.
–Proctor and student login procedures.
–Actual testing sessions in testing rooms.
–Final sign-off/ secure testing materials management.
www.caveon.com10

11. During On-Site Monitoring
Focus on process and people
– Test administration process.
– School staff/proctor roles and
behaviors.
– Student behaviors.
– Testing environment
(classroom, computer lab).
www.caveon.com11

12. During On-Site Monitoring
• Use objective checklist or form
– Yes/no (binary) checklist items to eliminate
discretion of monitors and allow for data analytics
across schools.
– Require explicit narrative descriptions of any
observed potential irregularity or deviation from
policies.
– Clear procedures for data capture and management.
– Archiving and evidence preservation.
www.caveon.com12

13. During On-Site Monitoring
• Consider electronic tools for incident
management
– Real-time data capture.
– Data analytics across schools.
– Workflow tools for incident management.
– Archiving and evidence preservation.
www.caveon.com13

14. What to do after On-Site
Monitoring
www.caveon.com14

15. Use of Monitoring Data and Results
• Do irregularities or potential violations
observed during monitoring require follow up
investigation?
• What level of monitoring data and results will
be shared with schools and staff?
• Should public reports of monitoring be
released?
www.caveon.com15

16. Use of Monitoring Data and Results
• Evaluate with other test security data sources.
–Incident or irregularity reports.
–Data Forensics or other test results analysis.
• Compare monitoring data points across
schools.
• Evaluate monitoring data within overall test
security context.
www.caveon.com16

17. Take Action
• Provide feedback to staff at observed sites.
• Conduct follow up investigations if warranted.
• Select sites for future monitoring.
• Improve test materials, instructions, or training.
www.caveon.com17

18. Getting Started
www.caveon.com18

19. Three questions to ask
• What are the goals of monitoring?
– Quality assurance.
– Ensuring compliance.
– Evaluating training effectiveness.
• What resources are available for monitoring?
• What will we do with the data from the
monitoring?
www.caveon.com19

20. Resource Document
www.caveon.com20

21. Six Steps
1. Set purpose and goals.
2. Determine budget/funding source.
3. Select sites.
4. Evaluate staffing options.
5. Develop audit forms.
6. Set periodic program reviews.
www.caveon.com21

22. Other Important Things to Consider
• Communication plans to stakeholders.
• Post test surveys of stakeholders about testing
experience.
• Staffing.
www.caveon.com22