SlideShare a Scribd company logo
1 of 37
Download to read offline
Upcoming Caveon Events
• NCME Conference, April 26 – 30th, San Francisco
Test Security I: Policy Issues: Technical/Statistical/Methodological Issues
– Presenters include: Dr. John Fremer, Caveon
• USDLA Conference, April 28- May 1, St. Louis
Will the real learner please stand up?
– Presenters include Caveon’s Jamie Mulkey, Ed.D. and Patrick Martin
• Caveon Webinar Series: Next session, May 2
What You Need to Know about High Stakes Cheating in Your Schools
– Presenters include former State Assessment Director, Dr. Mike Stetter,
and Dr. John Fremer, President, Caveon Consulting Services
Latest Publications
• Handbook of Test Security – Now available for
purchase! We’ll share a discount code before end
of session.
• TILSA Guidebook for State Assessment Directors
on Data Forensics – soon to be released
Caveon Online
• Caveon Security Insights Blog
– http://www.caveon.com/blog/
• twitter
– Follow @Caveon
• LinkedIn
– Caveon Company Page
– “Caveon Test Security” Group
• Please contribute!
• Facebook
– Will you be our “friend?”
– “Like” us!
www.caveon.com
“The Good and Bad of Online Proctoring”
Harry Layman
Executive Director
Digital Assessment Planning
The College Board
April 17, 2013
Caveon Webinar Series:
Dr. David Foster
CEO
Caveon Test Security
Agenda for today
• Recap of Structure and Content of recent paper “Online
Proctoring Systems Compared”
• Introduction and Definitions for Online Proctoring Discussion
• Test Security Threats Addressed by Online Proctoring
• Online Proctoring Features and Threat Impacts
• Seven Critical Online Proctoring Best Practices
• Odds and Ends
– Research Support
– Future Features
– Trends
• Lessons for Test Program Users
Online Proctoring Systems Compared:
Purpose of Paper
• Published by Dave Foster and Harry Layman
– http://bit.ly/proctoring
• Background of paper
• Recent paper provides a framework for describing different
offerings and several tables of feature comparisons
• Lots of detail, not entirely complete – will remain a work in
progress
• A good starting point as you begin to think about features /
benefits and your specific risk profile and threats of concern
Online Proctoring Systems Compared:
Structure and Content
• Overview of Online Proctoring – what it is, what it is not, and primary value
proposition
• An enumeration of 8 vendor products / services included in the scope of the
review
– Mix of capabilities and approaches; offerings not homogenous/equivalent
and so comparisons need to be more holistic
– We have not yet investigated all options equally, so the “comparisons”
are informative but not complete; look for updates over time
• Seven groups of features or characteristics of Online Proctoring offerings
organized into a series of comparison tables
• This presentation will “hit the highlights,” setting online proctoring in the
general context of test security threats, and provide a high-level discussion
of the attributes and features of Online Proctoring systems that can mitigate
those threats.
Online Proctoring Is / Is not:
• Online Proctoring refers specifically to using an internet-based
approach to remotely monitor individual test administrations, replacing
traditional approaches that have relied on “eyes in the room”
– Not CCTV, roving personnel, or using volunteers in user-selected
locations (libraries, etc.)
• Online Proctoring is NOT a “find-your-own-proctor” model as used for
decades in distance education exams
Online Proctoring Provides:
• Key value proposition in a nutshell:
– Independent, bias-free proctors
– Professional proctors (training, supervision, logging,
accountability, etc)
– Technology-assisted audio, visual and activity monitoring
support (e.g., log files, video, lock-down, keystroke monitoring,
test session control, real-time intervention, etc.)
General Proctoring Requirements
• Online Proctor During Entire Exam
• Continuous Internet
• Encryption for Data Transfer
• Schedule Availability
• Proctor Management
– Supervised
– Training
– Career Path
– Certification
• Program Customization
Security Threats / Scope
• Online Proctoring is primarily concerned with
– Threats during test administration; and hence
– Attempts to either capture test content, or
– Get inappropriate assistance in any way.
Security Threats / Scope
• Online Proctoring is only concerned with a subset of the
overall set of program vulnerabilities, threats and risks.
What are they?
• The subset are those threats that are detectable by the
online proctor (or onsite proctor) and over which he or
she has some control
Test Security Threats: A Finite Universe
• Only Six Categories of Cheating Threats
– Using pre-knowledge of test content
– Colluding with others
– Using cheating aids
– Using a proxy test taker
– Hacking into scoring system
– Copying answers from other test takers
• Only Six Categories of Stealing Threats
– Hacking into a system and stealing test files
– Capturing content by digital photography devices
– Capturing content by electronic recording of the screen
– Memorizing content
– Transcribing content verbally (on paper or recording device)
– Getting content from test program insider
Test Security Threats: A Finite Universe
• Only Six Categories of Cheating Threats
– Using pre-knowledge of test content
– Colluding with others
– Using cheating aids
– Using a proxy test taker
– Hacking into scoring system
– Copying answers from other test takers
• Only Six Categories of Stealing Threats
– Hacking into a system and stealing test files
– Capturing content by digital photography devices
– Capturing content by electronic recording of the screen
– Memorizing content
– Transcribing content verbally (on paper or recording device)
– Getting content from test program insider
Down to Six! Test Security Threats
Relevant for Online Proctors
• Cheating Threats
– C1. Colluding with others
– C2. Using cheating aids
– C3. Using a proxy test taker
– C4. Copying answers from other test takers
• Stealing Threats
– S1. Capturing content by digital photography devices
– S2. Transcribing questions verbally (on paper or recording
device)
C1. Colluding with Others
• How can a proctor effectively detect and deal with
collusion?
– Camera that can see a large portion of the room
– Camera that can pick up sounds and talking
– Ability to pause test session, and perhaps suspend it
– Ability to contact test taker, record/store
session and time-stamp incident
– Proctors should be prohibited
from viewing test taker screen
C2. Using Cheating Aids
• How can a proctor effectively detect and deal with a test
taker using a cheating aid?
– Camera that can see a large portion of the room, particularly
desk, computer, keyboard, test taker head/arms/hands
– Strong lockdown program to prevent access to online aids
– Ability to contact test taker, pause
test session, and perhaps suspend it
– Ability to record/store session and
time-stamp incident
C3. Using a Proxy Test Taker
• How can a proctor effectively detect and deal with
a proxy test taker?
– Strong initial authentication
– Continuous authentication methods
– Ability to contact test taker, pause test session, and
perhaps suspend it
– Ability to record/store session and time-stamp incident
C4. Copying Answers from Another Test
Taker
• How can a proctor effectively detect and deal with
a test taker copying from another?
– Camera that can see a large portion of the room,
including other test takers allowed in the room
– Ability to contact test taker, pause test session, and
perhaps suspend it
– Ability to record/store session and time-stamp incident
S1. Capture Test Content with Digital
Photography
• How can a proctor effectively detect and deal with
the digital photography of the test questions?
– Camera that can see a large portion of the room, along
with head/hands/torso of test taker
– Prevent proctor from viewing test screens
– Ability to contact test taker, pause test session, and
perhaps suspend it
– Ability to record/store session and time-stamp incident
S2. Transcribing Questions Verbally
• How can a proctor effectively detect and deal with
the transcription of test questions?
– Camera that can see a large portion of the room, along
with head/hands/torso of test taker
– Prevent proctors from viewing the test taker screen
– Ability to contact test taker, pause test session, and
perhaps suspend it
– Ability to record/store session and
time-stamp incident
Seven Critical Online Proctoring
Best Practices
1. Use wide-view camera with microphone
2. Require lockdown program
3. Prevent proctor view of test screens
4. Allow and require proctor to communicate with test taker
5. Allow proctor to control the test session
6. Use strong authentication methods
7. Record, store and time-stamp test session, along with
proctor logs
First: Use a Capable Webcam
• Is external to the computer; connected by 2-4 feet
of USB cable and set back
• Has large field of view (approx. 80° up to 360°)
• Allows proctor to see testing workstation and head,
torso, arms, hands of test taker
• Camera angle and placement
does not allow viewing the
workstation screen
• Has integrated microphone
Second: Use a Capable Lockdown Program
Operating System/Computer/Browser
• Prevent right-click • Prevent min/max windows
• Prevent printing • Prevent Copy/Paste
• Prevent function keys • Prevent running of applications
• Prevent important key combos • Prevent launch of applications
• Hide Taskbar and Desktop • Prevent communication tools
• Hide menus and icons • Prevent browser control/navigation
Third: Prevent Proctor View of Test
Workstation Screen
• What are the reasons a proctor should not view the
content of a test taker’s workstation screen?
– Security Reason #1: Viewing the screen may encourage
collusion between proctor and test taker
– Security Reason #2: Seeing the items of a high-stakes
exam on a proctor’s screen may encourage the theft of
the test content
– Privacy Reason: It’s not appropriate for a proctor to view
how a test taker answers test questions
Fourth: Allow Proctor to Communicate
with Test Taker
• Chat
• Canned Messages
• Technical Support call if necessary
Fifth: Allow Proctor to Control Test Session
• Control allows proctor to deal effectively with
detected threats
– Pause the test
• cover or remove the current question
• require acknowledgement and compliance from the test
taker
• provide a warning
– Suspend the test (can restart at later time)
– Cancel the test
• Proctor rules for these actions are subject to
customization by the testing program
Sixth: Use Strong Authentication Methods
• Strong
– Fingerprint readers
– Facial recognition
– Keystroke pattern recognition
– Voice recognition
• Avoid weak authentication methods
– Viewing of government-issued IDs through webcam
Seventh: Record and Store Sessions
• Record entire test session and proctor
logs/communications
• Time stamp incidents to make them easy to locate
• Provide easy-to-use review application for both video of
test session and proctor logs
• Store long-term or until no longer needed
Best Practices Effects on Security Incidents
• If these best practices are followed what benefits can
be expected?
– Better detection of threats or attempted breaches
– Professional, un-biased, and competent response by
proctors
– Deterrence: Fewer incidents going forward
– Stronger evidence basis for decision making and taking any
action, including legal action
Research Support
• Reliable data – on site or online – is hard to come by
• Limited data available for online proctoring; what is
there is encouraging
• More reporting by participating organizations will
increase the data pool
• Need studies also from academic researchers
It’s Only Going to Get Better:
Future Proctoring Features
• Improvements (e.g., better cameras; better proctor
training, etc.)
• New features: Automated proctoring technologies
– Inappropriate keystroke detection
– Audio level detection and display
– Detection of suspicious response patterns with Real-Time
Data Forensics
– Better facial, voice recognition software on the horizon
More Trends
• More providers, may see increasing stratification
between cost and quality / professionalism (as with
other security services)
• New technologies may allow better detection, or
lower costs – or both
– Augmenting human monitoring of online video with real-
time “artificial intelligence” could allow for greater proctor-
to-examinee ratios with even better security
Lessons for Program Users
• Do it right or be an easy target
• Strong security is always needed for tests where
cheating and theft occur, but not everyone has the
same set threats or concerns
• Different audiences may have different levels of
technical sophistication, risk, time sensitivity, etc.
• Online proctoring when done properly can be as
capable as any other well-run proctoring method at
reducing test security risks
HANDBOOK OF TEST SECURITY
• Editors - James Wollack & John Fremer
• Published March 2013
• Preventing, Detecting, and Investigating Cheating
• Testing in Many Domains
– Certification/Licensure
– Clinical
– Educational
– Industrial/Organizational
• Don’t forget to order your copy at www.routledge.com
– http://bit.ly/HandbookTS (Case Sensitive)
– Save 20% - Enter discount code: HYJ82
Questions?
Please join our “Caveon Test Security” LinkedIn
group and submit questions into the discussion
box or contact richelle.gruber@caveon.com
THANK YOU!
- Follow Caveon on twitter @caveon
- Check out our blog…www.caveon.com/blog
- LinkedIn Group – “Caveon Test Security”
Harry Layman
Executive Director
Digital Assessment Planning
The College Board
Dr. David Foster
CEO
Caveon Test Security
Again, find our Online Proctoring Systems Compared paper at http://bit.ly/proctoring

More Related Content

What's hot

Fingerprint technology in our lives
Fingerprint technology in our livesFingerprint technology in our lives
Fingerprint technology in our livescodykong
 
How to Get the Most Out of Security Tools
How to Get the Most Out of Security ToolsHow to Get the Most Out of Security Tools
How to Get the Most Out of Security ToolsSecurity Innovation
 
Fighting advanced malware using machine learning (English)
Fighting advanced malware using machine learning (English)Fighting advanced malware using machine learning (English)
Fighting advanced malware using machine learning (English)FFRI, Inc.
 
Vulnerability assessment and penetration testing
Vulnerability assessment and penetration testingVulnerability assessment and penetration testing
Vulnerability assessment and penetration testingAbu Sadat Mohammed Yasin
 
Ccsit cyberlympics 2017 workshop 3 - presentation
Ccsit cyberlympics 2017  workshop 3 - presentationCcsit cyberlympics 2017  workshop 3 - presentation
Ccsit cyberlympics 2017 workshop 3 - presentationNasir Bhutta
 
Ethical Hacking n VAPT presentation by Suvrat jain
Ethical Hacking n VAPT presentation by Suvrat jainEthical Hacking n VAPT presentation by Suvrat jain
Ethical Hacking n VAPT presentation by Suvrat jainSuvrat Jain
 
Professional Hacking in 2011
Professional Hacking in 2011Professional Hacking in 2011
Professional Hacking in 2011securityaegis
 
How to Break Software: Embedded Edition
How to Break Software: Embedded EditionHow to Break Software: Embedded Edition
How to Break Software: Embedded EditionTechWell
 
Vulnerability and Assessment Penetration Testing
Vulnerability and Assessment Penetration TestingVulnerability and Assessment Penetration Testing
Vulnerability and Assessment Penetration TestingYvonne Marambanyika
 
Remote Proctor: Software Secure SACSCOC Conference 2013
Remote Proctor: Software Secure SACSCOC Conference 2013Remote Proctor: Software Secure SACSCOC Conference 2013
Remote Proctor: Software Secure SACSCOC Conference 2013Software Secure, Inc.
 
Remote Proctor: Software Secure MSCHE 2013 Annual Conference Booth Presentation
Remote Proctor: Software Secure MSCHE 2013 Annual Conference Booth PresentationRemote Proctor: Software Secure MSCHE 2013 Annual Conference Booth Presentation
Remote Proctor: Software Secure MSCHE 2013 Annual Conference Booth PresentationSoftware Secure, Inc.
 
Remote Proctor: Software Secure Sloan-C 2013 Booth Presentation
Remote Proctor: Software Secure Sloan-C 2013 Booth PresentationRemote Proctor: Software Secure Sloan-C 2013 Booth Presentation
Remote Proctor: Software Secure Sloan-C 2013 Booth PresentationSoftware Secure, Inc.
 
Fuzzing101 uvm-reporting-and-mitigation-2011-02-10
Fuzzing101 uvm-reporting-and-mitigation-2011-02-10Fuzzing101 uvm-reporting-and-mitigation-2011-02-10
Fuzzing101 uvm-reporting-and-mitigation-2011-02-10Codenomicon
 
Watchful Eyes: A Solution Spotlight on Remote Proctoring Technology - ET4Onli...
Watchful Eyes: A Solution Spotlight on Remote Proctoring Technology - ET4Onli...Watchful Eyes: A Solution Spotlight on Remote Proctoring Technology - ET4Onli...
Watchful Eyes: A Solution Spotlight on Remote Proctoring Technology - ET4Onli...Software Secure, Inc.
 

What's hot (19)

Fingerprint technology in our lives
Fingerprint technology in our livesFingerprint technology in our lives
Fingerprint technology in our lives
 
How to Get the Most Out of Security Tools
How to Get the Most Out of Security ToolsHow to Get the Most Out of Security Tools
How to Get the Most Out of Security Tools
 
What is pentest
What is pentestWhat is pentest
What is pentest
 
Fighting advanced malware using machine learning (English)
Fighting advanced malware using machine learning (English)Fighting advanced malware using machine learning (English)
Fighting advanced malware using machine learning (English)
 
Vulnerability assessment and penetration testing
Vulnerability assessment and penetration testingVulnerability assessment and penetration testing
Vulnerability assessment and penetration testing
 
Ccsit cyberlympics 2017 workshop 3 - presentation
Ccsit cyberlympics 2017  workshop 3 - presentationCcsit cyberlympics 2017  workshop 3 - presentation
Ccsit cyberlympics 2017 workshop 3 - presentation
 
Ethical Hacking n VAPT presentation by Suvrat jain
Ethical Hacking n VAPT presentation by Suvrat jainEthical Hacking n VAPT presentation by Suvrat jain
Ethical Hacking n VAPT presentation by Suvrat jain
 
The VTC experience
The VTC experienceThe VTC experience
The VTC experience
 
Iscsp apt
Iscsp aptIscsp apt
Iscsp apt
 
Professional Hacking in 2011
Professional Hacking in 2011Professional Hacking in 2011
Professional Hacking in 2011
 
How to Break Software: Embedded Edition
How to Break Software: Embedded EditionHow to Break Software: Embedded Edition
How to Break Software: Embedded Edition
 
Vulnerability and Assessment Penetration Testing
Vulnerability and Assessment Penetration TestingVulnerability and Assessment Penetration Testing
Vulnerability and Assessment Penetration Testing
 
Remote Proctor: Software Secure SACSCOC Conference 2013
Remote Proctor: Software Secure SACSCOC Conference 2013Remote Proctor: Software Secure SACSCOC Conference 2013
Remote Proctor: Software Secure SACSCOC Conference 2013
 
Remote Proctor: Software Secure MSCHE 2013 Annual Conference Booth Presentation
Remote Proctor: Software Secure MSCHE 2013 Annual Conference Booth PresentationRemote Proctor: Software Secure MSCHE 2013 Annual Conference Booth Presentation
Remote Proctor: Software Secure MSCHE 2013 Annual Conference Booth Presentation
 
Remote Proctor: Software Secure Sloan-C 2013 Booth Presentation
Remote Proctor: Software Secure Sloan-C 2013 Booth PresentationRemote Proctor: Software Secure Sloan-C 2013 Booth Presentation
Remote Proctor: Software Secure Sloan-C 2013 Booth Presentation
 
Vapt pci dss methodology ppt v1.0
Vapt pci dss methodology ppt v1.0Vapt pci dss methodology ppt v1.0
Vapt pci dss methodology ppt v1.0
 
Fuzzing101 uvm-reporting-and-mitigation-2011-02-10
Fuzzing101 uvm-reporting-and-mitigation-2011-02-10Fuzzing101 uvm-reporting-and-mitigation-2011-02-10
Fuzzing101 uvm-reporting-and-mitigation-2011-02-10
 
Watchful Eyes: A Solution Spotlight on Remote Proctoring Technology - ET4Onli...
Watchful Eyes: A Solution Spotlight on Remote Proctoring Technology - ET4Onli...Watchful Eyes: A Solution Spotlight on Remote Proctoring Technology - ET4Onli...
Watchful Eyes: A Solution Spotlight on Remote Proctoring Technology - ET4Onli...
 
Intro to Network Vapt
Intro to Network VaptIntro to Network Vapt
Intro to Network Vapt
 

Similar to Caveon Webinar Series: The Good and Bad of Online Proctoring

Remote usability testing and remote user research for usability
Remote usability testing and remote user research for usabilityRemote usability testing and remote user research for usability
Remote usability testing and remote user research for usabilityUser Vision
 
2020 FRSecure CISSP Mentor Program - Class 10
2020 FRSecure CISSP Mentor Program - Class 102020 FRSecure CISSP Mentor Program - Class 10
2020 FRSecure CISSP Mentor Program - Class 10FRSecure
 
CohenNancyPresentation.ppt
CohenNancyPresentation.pptCohenNancyPresentation.ppt
CohenNancyPresentation.pptmypc72
 
Software Testing Life Cycle
Software Testing Life CycleSoftware Testing Life Cycle
Software Testing Life CycleUdayakumar Sree
 
Testing- Fundamentals of Testing-Mazenet solution
Testing- Fundamentals of Testing-Mazenet solutionTesting- Fundamentals of Testing-Mazenet solution
Testing- Fundamentals of Testing-Mazenet solutionMazenetsolution
 
Safety webinar with mark friend
Safety webinar with mark friendSafety webinar with mark friend
Safety webinar with mark friendERAUWebinars
 
Five Key Findings: Foolproof Computerized Examination
Five Key Findings: Foolproof Computerized ExaminationFive Key Findings: Foolproof Computerized Examination
Five Key Findings: Foolproof Computerized ExaminationPrometric
 
Bab 1 Fundamentals Of Testing
Bab 1 Fundamentals Of TestingBab 1 Fundamentals Of Testing
Bab 1 Fundamentals Of Testinglolayoriva
 
How to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingHow to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingTechWell
 
Caveon webinar lessons learned at atp and nces
Caveon webinar lessons learned at atp and ncesCaveon webinar lessons learned at atp and nces
Caveon webinar lessons learned at atp and ncesCaveon Test Security
 
Educause 2012 - Protecting Academic Integrity in Online Exam Environments
Educause 2012 - Protecting Academic Integrity in Online Exam Environments Educause 2012 - Protecting Academic Integrity in Online Exam Environments
Educause 2012 - Protecting Academic Integrity in Online Exam Environments Software Secure, Inc.
 
NETWORK PENETRATION TESTING
NETWORK PENETRATION TESTINGNETWORK PENETRATION TESTING
NETWORK PENETRATION TESTINGEr Vivek Rana
 
Stc 2015 regional-round-ppt-exlopratory mobile testing with risk analysis
Stc 2015 regional-round-ppt-exlopratory mobile testing with risk analysisStc 2015 regional-round-ppt-exlopratory mobile testing with risk analysis
Stc 2015 regional-round-ppt-exlopratory mobile testing with risk analysisArchana Krushnan
 
Michael Bolton - Two Futures of Software Testing
Michael Bolton - Two Futures of Software TestingMichael Bolton - Two Futures of Software Testing
Michael Bolton - Two Futures of Software TestingTEST Huddle
 
01. foundamentals of testing
01. foundamentals of testing01. foundamentals of testing
01. foundamentals of testingTricia Karina
 

Similar to Caveon Webinar Series: The Good and Bad of Online Proctoring (20)

Remote usability testing and remote user research for usability
Remote usability testing and remote user research for usabilityRemote usability testing and remote user research for usability
Remote usability testing and remote user research for usability
 
2020 FRSecure CISSP Mentor Program - Class 10
2020 FRSecure CISSP Mentor Program - Class 102020 FRSecure CISSP Mentor Program - Class 10
2020 FRSecure CISSP Mentor Program - Class 10
 
CohenNancyPresentation.ppt
CohenNancyPresentation.pptCohenNancyPresentation.ppt
CohenNancyPresentation.ppt
 
Fundamentals of Testing Section 1/6
Fundamentals of Testing   Section 1/6Fundamentals of Testing   Section 1/6
Fundamentals of Testing Section 1/6
 
Software Testing
Software TestingSoftware Testing
Software Testing
 
Software Testing Life Cycle
Software Testing Life CycleSoftware Testing Life Cycle
Software Testing Life Cycle
 
Testing- Fundamentals of Testing-Mazenet solution
Testing- Fundamentals of Testing-Mazenet solutionTesting- Fundamentals of Testing-Mazenet solution
Testing- Fundamentals of Testing-Mazenet solution
 
Safety webinar with mark friend
Safety webinar with mark friendSafety webinar with mark friend
Safety webinar with mark friend
 
Five Key Findings: Foolproof Computerized Examination
Five Key Findings: Foolproof Computerized ExaminationFive Key Findings: Foolproof Computerized Examination
Five Key Findings: Foolproof Computerized Examination
 
Bab 1 Fundamentals Of Testing
Bab 1 Fundamentals Of TestingBab 1 Fundamentals Of Testing
Bab 1 Fundamentals Of Testing
 
Module 6.pptx
Module 6.pptxModule 6.pptx
Module 6.pptx
 
ISTQB foundation level - day 2
ISTQB foundation level - day 2ISTQB foundation level - day 2
ISTQB foundation level - day 2
 
How to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingHow to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated Testing
 
Caveon webinar lessons learned at atp and nces
Caveon webinar lessons learned at atp and ncesCaveon webinar lessons learned at atp and nces
Caveon webinar lessons learned at atp and nces
 
Educause 2012 - Protecting Academic Integrity in Online Exam Environments
Educause 2012 - Protecting Academic Integrity in Online Exam Environments Educause 2012 - Protecting Academic Integrity in Online Exam Environments
Educause 2012 - Protecting Academic Integrity in Online Exam Environments
 
NETWORK PENETRATION TESTING
NETWORK PENETRATION TESTINGNETWORK PENETRATION TESTING
NETWORK PENETRATION TESTING
 
Stc 2015 regional-round-ppt-exlopratory mobile testing with risk analysis
Stc 2015 regional-round-ppt-exlopratory mobile testing with risk analysisStc 2015 regional-round-ppt-exlopratory mobile testing with risk analysis
Stc 2015 regional-round-ppt-exlopratory mobile testing with risk analysis
 
Michael Bolton - Two Futures of Software Testing
Michael Bolton - Two Futures of Software TestingMichael Bolton - Two Futures of Software Testing
Michael Bolton - Two Futures of Software Testing
 
01. foundamentals of testing
01. foundamentals of testing01. foundamentals of testing
01. foundamentals of testing
 
Experience Sharing on School Pentest Project (Updated)
Experience Sharing on School Pentest Project (Updated)Experience Sharing on School Pentest Project (Updated)
Experience Sharing on School Pentest Project (Updated)
 

More from Caveon Test Security

Unpublished study indicates high chance of fraud in thousands of tests of enem
Unpublished study indicates high chance of fraud in thousands of tests of enemUnpublished study indicates high chance of fraud in thousands of tests of enem
Unpublished study indicates high chance of fraud in thousands of tests of enemCaveon Test Security
 
Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon Test Security
 
Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon Test Security
 
Caveon Webinar Series - A Guide to Online Protection Strategies - March 28, ...
Caveon Webinar Series -  A Guide to Online Protection Strategies - March 28, ...Caveon Webinar Series -  A Guide to Online Protection Strategies - March 28, ...
Caveon Webinar Series - A Guide to Online Protection Strategies - March 28, ...Caveon Test Security
 
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...Caveon Test Security
 
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217Caveon Test Security
 
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...Caveon Test Security
 
Caveon Webinar Series - Four Steps to Effective Investigations in School Dis...
Caveon Webinar Series -  Four Steps to Effective Investigations in School Dis...Caveon Webinar Series -  Four Steps to Effective Investigations in School Dis...
Caveon Webinar Series - Four Steps to Effective Investigations in School Dis...Caveon Test Security
 
Caveon Webinar Series - On-site Monitoring in Districts 0317
Caveon Webinar Series - On-site Monitoring in Districts 0317Caveon Webinar Series - On-site Monitoring in Districts 0317
Caveon Webinar Series - On-site Monitoring in Districts 0317Caveon Test Security
 
CESP Study Session #1 October 2016
CESP Study Session #1 October 2016CESP Study Session #1 October 2016
CESP Study Session #1 October 2016Caveon Test Security
 
A Tale of Two Cities - School District Webinar #1 Jan 2017
A Tale of Two Cities - School District Webinar  #1 Jan 2017A Tale of Two Cities - School District Webinar  #1 Jan 2017
A Tale of Two Cities - School District Webinar #1 Jan 2017Caveon Test Security
 
Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Te...
Caveon Webinar Series  - Discrete Option Multiple Choice:  A Revolution in Te...Caveon Webinar Series  - Discrete Option Multiple Choice:  A Revolution in Te...
Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Te...Caveon Test Security
 
Caveon Webinar Series - Test Cheaters Say the Darnedest Things! - 072016
Caveon Webinar Series -  Test Cheaters Say the Darnedest Things! - 072016Caveon Webinar Series -  Test Cheaters Say the Darnedest Things! - 072016
Caveon Webinar Series - Test Cheaters Say the Darnedest Things! - 072016Caveon Test Security
 
Caveon Webinar Series - The Test Security Framework- Why Different Tests Nee...
Caveon Webinar Series -  The Test Security Framework- Why Different Tests Nee...Caveon Webinar Series -  The Test Security Framework- Why Different Tests Nee...
Caveon Webinar Series - The Test Security Framework- Why Different Tests Nee...Caveon Test Security
 
Caveon Webinar Series - Conducting Test Security Investigations in School Di...
Caveon Webinar Series -  Conducting Test Security Investigations in School Di...Caveon Webinar Series -  Conducting Test Security Investigations in School Di...
Caveon Webinar Series - Conducting Test Security Investigations in School Di...Caveon Test Security
 
Caveon Webinar Series - Creating Your Test Security Game Plan - March 2016
Caveon Webinar Series -  Creating Your Test Security Game Plan - March 2016Caveon Webinar Series -  Creating Your Test Security Game Plan - March 2016
Caveon Webinar Series - Creating Your Test Security Game Plan - March 2016Caveon Test Security
 
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...Caveon Test Security
 
Caveon Webinar Series - Will the Real Cloned Item Please Stand Up? final
Caveon Webinar Series -  Will the Real Cloned Item Please Stand Up? finalCaveon Webinar Series -  Will the Real Cloned Item Please Stand Up? final
Caveon Webinar Series - Will the Real Cloned Item Please Stand Up? finalCaveon Test Security
 
Caveon Webinar Series - Lessons Learned at the 2015 National Conference on S...
Caveon Webinar Series -  Lessons Learned at the 2015 National Conference on S...Caveon Webinar Series -  Lessons Learned at the 2015 National Conference on S...
Caveon Webinar Series - Lessons Learned at the 2015 National Conference on S...Caveon Test Security
 
Caveon Webinar Series - Learning and Teaching Best Practices in Test Security...
Caveon Webinar Series - Learning and Teaching Best Practices in Test Security...Caveon Webinar Series - Learning and Teaching Best Practices in Test Security...
Caveon Webinar Series - Learning and Teaching Best Practices in Test Security...Caveon Test Security
 

More from Caveon Test Security (20)

Unpublished study indicates high chance of fraud in thousands of tests of enem
Unpublished study indicates high chance of fraud in thousands of tests of enemUnpublished study indicates high chance of fraud in thousands of tests of enem
Unpublished study indicates high chance of fraud in thousands of tests of enem
 
Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...
 
Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...
 
Caveon Webinar Series - A Guide to Online Protection Strategies - March 28, ...
Caveon Webinar Series -  A Guide to Online Protection Strategies - March 28, ...Caveon Webinar Series -  A Guide to Online Protection Strategies - March 28, ...
Caveon Webinar Series - A Guide to Online Protection Strategies - March 28, ...
 
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...
 
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217
 
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...
 
Caveon Webinar Series - Four Steps to Effective Investigations in School Dis...
Caveon Webinar Series -  Four Steps to Effective Investigations in School Dis...Caveon Webinar Series -  Four Steps to Effective Investigations in School Dis...
Caveon Webinar Series - Four Steps to Effective Investigations in School Dis...
 
Caveon Webinar Series - On-site Monitoring in Districts 0317
Caveon Webinar Series - On-site Monitoring in Districts 0317Caveon Webinar Series - On-site Monitoring in Districts 0317
Caveon Webinar Series - On-site Monitoring in Districts 0317
 
CESP Study Session #1 October 2016
CESP Study Session #1 October 2016CESP Study Session #1 October 2016
CESP Study Session #1 October 2016
 
A Tale of Two Cities - School District Webinar #1 Jan 2017
A Tale of Two Cities - School District Webinar  #1 Jan 2017A Tale of Two Cities - School District Webinar  #1 Jan 2017
A Tale of Two Cities - School District Webinar #1 Jan 2017
 
Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Te...
Caveon Webinar Series  - Discrete Option Multiple Choice:  A Revolution in Te...Caveon Webinar Series  - Discrete Option Multiple Choice:  A Revolution in Te...
Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Te...
 
Caveon Webinar Series - Test Cheaters Say the Darnedest Things! - 072016
Caveon Webinar Series -  Test Cheaters Say the Darnedest Things! - 072016Caveon Webinar Series -  Test Cheaters Say the Darnedest Things! - 072016
Caveon Webinar Series - Test Cheaters Say the Darnedest Things! - 072016
 
Caveon Webinar Series - The Test Security Framework- Why Different Tests Nee...
Caveon Webinar Series -  The Test Security Framework- Why Different Tests Nee...Caveon Webinar Series -  The Test Security Framework- Why Different Tests Nee...
Caveon Webinar Series - The Test Security Framework- Why Different Tests Nee...
 
Caveon Webinar Series - Conducting Test Security Investigations in School Di...
Caveon Webinar Series -  Conducting Test Security Investigations in School Di...Caveon Webinar Series -  Conducting Test Security Investigations in School Di...
Caveon Webinar Series - Conducting Test Security Investigations in School Di...
 
Caveon Webinar Series - Creating Your Test Security Game Plan - March 2016
Caveon Webinar Series -  Creating Your Test Security Game Plan - March 2016Caveon Webinar Series -  Creating Your Test Security Game Plan - March 2016
Caveon Webinar Series - Creating Your Test Security Game Plan - March 2016
 
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...
 
Caveon Webinar Series - Will the Real Cloned Item Please Stand Up? final
Caveon Webinar Series -  Will the Real Cloned Item Please Stand Up? finalCaveon Webinar Series -  Will the Real Cloned Item Please Stand Up? final
Caveon Webinar Series - Will the Real Cloned Item Please Stand Up? final
 
Caveon Webinar Series - Lessons Learned at the 2015 National Conference on S...
Caveon Webinar Series -  Lessons Learned at the 2015 National Conference on S...Caveon Webinar Series -  Lessons Learned at the 2015 National Conference on S...
Caveon Webinar Series - Lessons Learned at the 2015 National Conference on S...
 
Caveon Webinar Series - Learning and Teaching Best Practices in Test Security...
Caveon Webinar Series - Learning and Teaching Best Practices in Test Security...Caveon Webinar Series - Learning and Teaching Best Practices in Test Security...
Caveon Webinar Series - Learning and Teaching Best Practices in Test Security...
 

Recently uploaded

CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...Nguyen Thanh Tu Collection
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxheathfieldcps1
 
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...M56BOOKSTORE PRODUCT/SERVICE
 
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdfP4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdfYu Kanazawa / Osaka University
 
How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17Celine George
 
Work Experience for psp3 portfolio sasha
Work Experience for psp3 portfolio sashaWork Experience for psp3 portfolio sasha
Work Experience for psp3 portfolio sashasashalaycock03
 
In - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptxIn - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptxAditiChauhan701637
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICESayali Powar
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapitolTechU
 
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptxSandy Millin
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxiammrhaywood
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesCeline George
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...CaraSkikne1
 
3.26.24 Race, the Draft, and the Vietnam War.pptx
3.26.24 Race, the Draft, and the Vietnam War.pptx3.26.24 Race, the Draft, and the Vietnam War.pptx
3.26.24 Race, the Draft, and the Vietnam War.pptxmary850239
 
EBUS5423 Data Analytics and Reporting Bl
EBUS5423 Data Analytics and Reporting BlEBUS5423 Data Analytics and Reporting Bl
EBUS5423 Data Analytics and Reporting BlDr. Bruce A. Johnson
 
Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....Riddhi Kevadiya
 
Over the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxOver the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxraviapr7
 
Department of Health Compounder Question ‍Solution 2022.pdf
Department of Health Compounder Question ‍Solution 2022.pdfDepartment of Health Compounder Question ‍Solution 2022.pdf
Department of Health Compounder Question ‍Solution 2022.pdfMohonDas
 

Recently uploaded (20)

CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
 
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdfP4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
 
How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17
 
Work Experience for psp3 portfolio sasha
Work Experience for psp3 portfolio sashaWork Experience for psp3 portfolio sasha
Work Experience for psp3 portfolio sasha
 
In - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptxIn - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptx
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICE
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptx
 
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 Sales
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...
 
3.26.24 Race, the Draft, and the Vietnam War.pptx
3.26.24 Race, the Draft, and the Vietnam War.pptx3.26.24 Race, the Draft, and the Vietnam War.pptx
3.26.24 Race, the Draft, and the Vietnam War.pptx
 
EBUS5423 Data Analytics and Reporting Bl
EBUS5423 Data Analytics and Reporting BlEBUS5423 Data Analytics and Reporting Bl
EBUS5423 Data Analytics and Reporting Bl
 
Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....
 
March 2024 Directors Meeting, Division of Student Affairs and Academic Support
March 2024 Directors Meeting, Division of Student Affairs and Academic SupportMarch 2024 Directors Meeting, Division of Student Affairs and Academic Support
March 2024 Directors Meeting, Division of Student Affairs and Academic Support
 
Finals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quizFinals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quiz
 
Over the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxOver the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptx
 
Department of Health Compounder Question ‍Solution 2022.pdf
Department of Health Compounder Question ‍Solution 2022.pdfDepartment of Health Compounder Question ‍Solution 2022.pdf
Department of Health Compounder Question ‍Solution 2022.pdf
 

Caveon Webinar Series: The Good and Bad of Online Proctoring

  • 1. Upcoming Caveon Events • NCME Conference, April 26 – 30th, San Francisco Test Security I: Policy Issues: Technical/Statistical/Methodological Issues – Presenters include: Dr. John Fremer, Caveon • USDLA Conference, April 28- May 1, St. Louis Will the real learner please stand up? – Presenters include Caveon’s Jamie Mulkey, Ed.D. and Patrick Martin • Caveon Webinar Series: Next session, May 2 What You Need to Know about High Stakes Cheating in Your Schools – Presenters include former State Assessment Director, Dr. Mike Stetter, and Dr. John Fremer, President, Caveon Consulting Services
  • 2. Latest Publications • Handbook of Test Security – Now available for purchase! We’ll share a discount code before end of session. • TILSA Guidebook for State Assessment Directors on Data Forensics – soon to be released
  • 3. Caveon Online • Caveon Security Insights Blog – http://www.caveon.com/blog/ • twitter – Follow @Caveon • LinkedIn – Caveon Company Page – “Caveon Test Security” Group • Please contribute! • Facebook – Will you be our “friend?” – “Like” us! www.caveon.com
  • 4. “The Good and Bad of Online Proctoring” Harry Layman Executive Director Digital Assessment Planning The College Board April 17, 2013 Caveon Webinar Series: Dr. David Foster CEO Caveon Test Security
  • 5. Agenda for today • Recap of Structure and Content of recent paper “Online Proctoring Systems Compared” • Introduction and Definitions for Online Proctoring Discussion • Test Security Threats Addressed by Online Proctoring • Online Proctoring Features and Threat Impacts • Seven Critical Online Proctoring Best Practices • Odds and Ends – Research Support – Future Features – Trends • Lessons for Test Program Users
  • 6. Online Proctoring Systems Compared: Purpose of Paper • Published by Dave Foster and Harry Layman – http://bit.ly/proctoring • Background of paper • Recent paper provides a framework for describing different offerings and several tables of feature comparisons • Lots of detail, not entirely complete – will remain a work in progress • A good starting point as you begin to think about features / benefits and your specific risk profile and threats of concern
  • 7. Online Proctoring Systems Compared: Structure and Content • Overview of Online Proctoring – what it is, what it is not, and primary value proposition • An enumeration of 8 vendor products / services included in the scope of the review – Mix of capabilities and approaches; offerings not homogenous/equivalent and so comparisons need to be more holistic – We have not yet investigated all options equally, so the “comparisons” are informative but not complete; look for updates over time • Seven groups of features or characteristics of Online Proctoring offerings organized into a series of comparison tables • This presentation will “hit the highlights,” setting online proctoring in the general context of test security threats, and provide a high-level discussion of the attributes and features of Online Proctoring systems that can mitigate those threats.
  • 8. Online Proctoring Is / Is not: • Online Proctoring refers specifically to using an internet-based approach to remotely monitor individual test administrations, replacing traditional approaches that have relied on “eyes in the room” – Not CCTV, roving personnel, or using volunteers in user-selected locations (libraries, etc.) • Online Proctoring is NOT a “find-your-own-proctor” model as used for decades in distance education exams
  • 9. Online Proctoring Provides: • Key value proposition in a nutshell: – Independent, bias-free proctors – Professional proctors (training, supervision, logging, accountability, etc) – Technology-assisted audio, visual and activity monitoring support (e.g., log files, video, lock-down, keystroke monitoring, test session control, real-time intervention, etc.)
  • 10. General Proctoring Requirements • Online Proctor During Entire Exam • Continuous Internet • Encryption for Data Transfer • Schedule Availability • Proctor Management – Supervised – Training – Career Path – Certification • Program Customization
  • 11. Security Threats / Scope • Online Proctoring is primarily concerned with – Threats during test administration; and hence – Attempts to either capture test content, or – Get inappropriate assistance in any way.
  • 12. Security Threats / Scope • Online Proctoring is only concerned with a subset of the overall set of program vulnerabilities, threats and risks. What are they? • The subset are those threats that are detectable by the online proctor (or onsite proctor) and over which he or she has some control
  • 13. Test Security Threats: A Finite Universe • Only Six Categories of Cheating Threats – Using pre-knowledge of test content – Colluding with others – Using cheating aids – Using a proxy test taker – Hacking into scoring system – Copying answers from other test takers • Only Six Categories of Stealing Threats – Hacking into a system and stealing test files – Capturing content by digital photography devices – Capturing content by electronic recording of the screen – Memorizing content – Transcribing content verbally (on paper or recording device) – Getting content from test program insider
  • 14. Test Security Threats: A Finite Universe • Only Six Categories of Cheating Threats – Using pre-knowledge of test content – Colluding with others – Using cheating aids – Using a proxy test taker – Hacking into scoring system – Copying answers from other test takers • Only Six Categories of Stealing Threats – Hacking into a system and stealing test files – Capturing content by digital photography devices – Capturing content by electronic recording of the screen – Memorizing content – Transcribing content verbally (on paper or recording device) – Getting content from test program insider
  • 15. Down to Six! Test Security Threats Relevant for Online Proctors • Cheating Threats – C1. Colluding with others – C2. Using cheating aids – C3. Using a proxy test taker – C4. Copying answers from other test takers • Stealing Threats – S1. Capturing content by digital photography devices – S2. Transcribing questions verbally (on paper or recording device)
  • 16. C1. Colluding with Others • How can a proctor effectively detect and deal with collusion? – Camera that can see a large portion of the room – Camera that can pick up sounds and talking – Ability to pause test session, and perhaps suspend it – Ability to contact test taker, record/store session and time-stamp incident – Proctors should be prohibited from viewing test taker screen
  • 17. C2. Using Cheating Aids • How can a proctor effectively detect and deal with a test taker using a cheating aid? – Camera that can see a large portion of the room, particularly desk, computer, keyboard, test taker head/arms/hands – Strong lockdown program to prevent access to online aids – Ability to contact test taker, pause test session, and perhaps suspend it – Ability to record/store session and time-stamp incident
  • 18. C3. Using a Proxy Test Taker • How can a proctor effectively detect and deal with a proxy test taker? – Strong initial authentication – Continuous authentication methods – Ability to contact test taker, pause test session, and perhaps suspend it – Ability to record/store session and time-stamp incident
  • 19. C4. Copying Answers from Another Test Taker • How can a proctor effectively detect and deal with a test taker copying from another? – Camera that can see a large portion of the room, including other test takers allowed in the room – Ability to contact test taker, pause test session, and perhaps suspend it – Ability to record/store session and time-stamp incident
  • 20. S1. Capture Test Content with Digital Photography • How can a proctor effectively detect and deal with the digital photography of the test questions? – Camera that can see a large portion of the room, along with head/hands/torso of test taker – Prevent proctor from viewing test screens – Ability to contact test taker, pause test session, and perhaps suspend it – Ability to record/store session and time-stamp incident
  • 21. S2. Transcribing Questions Verbally • How can a proctor effectively detect and deal with the transcription of test questions? – Camera that can see a large portion of the room, along with head/hands/torso of test taker – Prevent proctors from viewing the test taker screen – Ability to contact test taker, pause test session, and perhaps suspend it – Ability to record/store session and time-stamp incident
  • 22. Seven Critical Online Proctoring Best Practices 1. Use wide-view camera with microphone 2. Require lockdown program 3. Prevent proctor view of test screens 4. Allow and require proctor to communicate with test taker 5. Allow proctor to control the test session 6. Use strong authentication methods 7. Record, store and time-stamp test session, along with proctor logs
  • 23. First: Use a Capable Webcam • Is external to the computer; connected by 2-4 feet of USB cable and set back • Has large field of view (approx. 80° up to 360°) • Allows proctor to see testing workstation and head, torso, arms, hands of test taker • Camera angle and placement does not allow viewing the workstation screen • Has integrated microphone
  • 24. Second: Use a Capable Lockdown Program Operating System/Computer/Browser • Prevent right-click • Prevent min/max windows • Prevent printing • Prevent Copy/Paste • Prevent function keys • Prevent running of applications • Prevent important key combos • Prevent launch of applications • Hide Taskbar and Desktop • Prevent communication tools • Hide menus and icons • Prevent browser control/navigation
  • 25. Third: Prevent Proctor View of Test Workstation Screen • What are the reasons a proctor should not view the content of a test taker’s workstation screen? – Security Reason #1: Viewing the screen may encourage collusion between proctor and test taker – Security Reason #2: Seeing the items of a high-stakes exam on a proctor’s screen may encourage the theft of the test content – Privacy Reason: It’s not appropriate for a proctor to view how a test taker answers test questions
  • 26. Fourth: Allow Proctor to Communicate with Test Taker • Chat • Canned Messages • Technical Support call if necessary
  • 27. Fifth: Allow Proctor to Control Test Session • Control allows proctor to deal effectively with detected threats – Pause the test • cover or remove the current question • require acknowledgement and compliance from the test taker • provide a warning – Suspend the test (can restart at later time) – Cancel the test • Proctor rules for these actions are subject to customization by the testing program
  • 28. Sixth: Use Strong Authentication Methods • Strong – Fingerprint readers – Facial recognition – Keystroke pattern recognition – Voice recognition • Avoid weak authentication methods – Viewing of government-issued IDs through webcam
  • 29. Seventh: Record and Store Sessions • Record entire test session and proctor logs/communications • Time stamp incidents to make them easy to locate • Provide easy-to-use review application for both video of test session and proctor logs • Store long-term or until no longer needed
  • 30. Best Practices Effects on Security Incidents • If these best practices are followed what benefits can be expected? – Better detection of threats or attempted breaches – Professional, un-biased, and competent response by proctors – Deterrence: Fewer incidents going forward – Stronger evidence basis for decision making and taking any action, including legal action
  • 31. Research Support • Reliable data – on site or online – is hard to come by • Limited data available for online proctoring; what is there is encouraging • More reporting by participating organizations will increase the data pool • Need studies also from academic researchers
  • 32. It’s Only Going to Get Better: Future Proctoring Features • Improvements (e.g., better cameras; better proctor training, etc.) • New features: Automated proctoring technologies – Inappropriate keystroke detection – Audio level detection and display – Detection of suspicious response patterns with Real-Time Data Forensics – Better facial, voice recognition software on the horizon
  • 33. More Trends • More providers, may see increasing stratification between cost and quality / professionalism (as with other security services) • New technologies may allow better detection, or lower costs – or both – Augmenting human monitoring of online video with real- time “artificial intelligence” could allow for greater proctor- to-examinee ratios with even better security
  • 34. Lessons for Program Users • Do it right or be an easy target • Strong security is always needed for tests where cheating and theft occur, but not everyone has the same set threats or concerns • Different audiences may have different levels of technical sophistication, risk, time sensitivity, etc. • Online proctoring when done properly can be as capable as any other well-run proctoring method at reducing test security risks
  • 35. HANDBOOK OF TEST SECURITY • Editors - James Wollack & John Fremer • Published March 2013 • Preventing, Detecting, and Investigating Cheating • Testing in Many Domains – Certification/Licensure – Clinical – Educational – Industrial/Organizational • Don’t forget to order your copy at www.routledge.com – http://bit.ly/HandbookTS (Case Sensitive) – Save 20% - Enter discount code: HYJ82
  • 36. Questions? Please join our “Caveon Test Security” LinkedIn group and submit questions into the discussion box or contact richelle.gruber@caveon.com
  • 37. THANK YOU! - Follow Caveon on twitter @caveon - Check out our blog…www.caveon.com/blog - LinkedIn Group – “Caveon Test Security” Harry Layman Executive Director Digital Assessment Planning The College Board Dr. David Foster CEO Caveon Test Security Again, find our Online Proctoring Systems Compared paper at http://bit.ly/proctoring

Editor's Notes

  1. Introduce the paper; describe it in general terms. Next slide will provide specifics on structure and content.On this slide we say what the paper is FOR; on the next slide we can say a little bit more about “what it is”.
  2. This slide is just intended to set out the contents of the paper; it (and subsequent slides) will not replicate or include the contents – either the vendor list or the comparison tables.Harry, I changed the wording of the 4th bullet point only slightly to add in the concept of security threats early on.We need to focus this presentation on how online proctors and associated technology can affect test security during test administration.
  3. Might squeeze in: “We prefer the term online proctoring to remote proctoring: for these reasons.
  4. Other value propositions include: good view of testing environment; lockdown for a number of good reasons; proctor communication and control.Some more notes here might help.
  5. These are general features that help the online proctoring work or be effective. I would argue that they are not specific online proctoring features. Just general best practices. They are all related to the quality of security, but aren’t intended to detect and deal with threats.Not sure I believe what I just wrote. These are more enabling features.
  6. I wanted a way to show the entire set of threats, and then fade some away until only those relevant to what online proctoring can be effective against remain. About half of test security threats are not relevant to our discussion, but are relevant to a program’s security management plan. Something graphical or animated would work here.DFF: Some threats During Test Administration are not amenable to any kind of proctor detection or management. Use of pre-knowledge is a good example. Human proctors of any sort are not able to detect its use directly. But some data forensics models can do so in real-time (theoretically) and inform the proctor. Whether the proctor can or should do anything about it is anohter matter. We should probaby restrict this webinar to threats occurring during test administration, AND to threats the proctor can detect and manage.
  7. I wanted a way to show the entire set of threats, and then fade some away until only those relevant to what online proctoring can be effective against remain. About half of test security threats are not relevant to our discussion, but are relevant to a program’s security management plan. Something graphical or animated would work here.Test content protection is important every day, all day, all year; but online proctoring worries about content only during the test administrationCheating or inappropriate assistance is a multi-faceted threat that includes everything from electronic cheating, ‘traditional’ cheating (writing on your hand, pre-staging materials, etc) to impersonation / authentication threats.
  8. Relevance means that an online proctor (or onsite proctor) can detect the threat or attempted breach AND can do something about it.
  9. Describe collusion as someone helping the test taker during the exam. Proctors are individuals capable of collusion.Watching for a person leaving the testing room to perhaps communicate with someone in another room.Can’t deal with collusion that occurs through hidden ear mic (unless examinee vocalizes things ?) Can tell if the person is talking with someone in the room or on a micDFF: Squeeze this in. It will help later on. “In some situations, Proctors often collude with test takers. Being unable to see the test taker screen helps.”==========a) the camera itself is important; where it is positioned -- field of view -- is important. worst case are simple cameras built into monitors that offer a narrow view of the examinee's; as will be discussed later, side-views are better, and required by some services and big users.b) sounds are important queues. and individual mics included with camera units are one area where OLP clearly improves on a human proctor -- particularly one in a larger area monitoring many test takers. setup and validation procedures prior to exam start will insure the sound system meets requirements to monitor even the quietest talking -- as in talking to a concealed microphone.c) if there is suspecisous activity, a remote proctor's ability to pause a session and ask a test taker to explain -- a noise, sound of voices, reflections, etc. (especially when they have been logged electronically), and with the authority to suspend a test session, allows incidents to be dealt with quickly and forthrightly.d) can notify test taker if security breach (or potential breach) was recorded and noted int he logs. test taker may withdraw, or exercize greater caution if behavior was truely incidental.e) not allowing proctors to see test content is simply a zero cost, zero effort incremental security measure. while collusion is unlikely, there may be tests where an enterprising proctor might be tempted to monitize access to potential valuable content.
  10. Separate from cheating by getting help from another person (collusion) is the possibility of getting help from access to unauthorized devices, services and materials – books, notes, post-its, web sites, etc.Cheating aids can include calculators, cell phones, books, papers, notes, post-its, computer, Internet, etc.Camera angle will help proctor to detect if a person leaves the room to use a cheating aid.Some of these devices of course may be allowed by program.I would note here that for both collusion with others or using any sort of cheating aid, one factor impacting these risks arises from the examinee's access to and control of the testing facility / room pior to and after the exam session. So often the threats you are guarding against here is the use of materials or devices pre-staged by the test taker. I good "startup routine" where the proctor has the test taker pan the room and surrounds with the camera, does a full voice and video quality check, etc. will help deter (and detect) cheating attempts of this kind.
  11. Some improvement (over human proctoring with no logging / video of the actual test taker) arises from having video and audio of the test taker at the start of the test – to compare with something (potentially something submitted earlier by the test sponsor), and to have that available with test results. Includes: fingerprint readers, facial recognition software, voice recognition software, keystroke pattern recognition, photo matchingContinuous refers to occasional, perhaps random, requests for authentication. Automated procedures are amenable to continuous authentication. For longer exams where students have breaks and my leave the area and return, re-authentication after breaks, random re-authentication prompts at multiple points during an exam, might be warranted.Weak: include the use of ID, particularly as verified through a webcam. A “picture of a picture” is not as robust as a good, high resolution face picture to compare with an externally provided (by test sponsor or results recipient) face picture. ID’s can be faked and can’t be verified as genuine through a webcam.I’ve read about more cloud-based “software as a service” deployment of machine learning and AI tools and algorithms, and of “homeland security” contractors trying out facial recognition software and algorithms delivered to desktops via AWS – so this technology might be incorporated in more secure delivery system sooner than expected. (you still need something to compare a “face” taking a test “with”).( DFF: Facial recognition software can be used early and later.)
  12. This is not a big issue but I would leave the slide on for completeness. Note that this applies only if more than one test taker is allowed in the same room – not the typical use case. Online Proctoring is mostly used in circumstances where examinees are alone in their room. In other cases, tests may start at different times and questions may be randomized. DFF: You sent me a picture last week of an online proctor viewing an entire room of chinese school children. This is probably a good example of what this means. Do you want to add that slide?
  13. Good news and bad news!This is a big problem, because proctors can really only detect this behavior from stupid test takers that utilize an obvious cell phone or camera. Many photography devices are meant to be hidden (button, pen, watch, glasses, etc.) and are realistically not detectable by the proctor whether through a webcam or even if in the room. Methods to foil this type of theft are not proctor relevant and is the subject for a different webinar.
  14. Test taker can write down the questions/answers on a pad of paper or tablet, or orally capture on digital recording device.These are fairly easily detected by an online proctor.Need to make sure that online proctor cannot view test content.
  15. This is a summary page of the best practices listed on the several slides before.
  16. Weaknesses: Can see all areas in a room, even if the room is panned. (under desk, in closet, in drawers, in a hall. After panning cheating devices and helpers can return.
  17. Lockdown is complicated to create and maintain and requires a download by student/test taker (but which might be “automatic” / zero-touch (if it works)); will result in tech support calls, etc. But it’s advantages are clear. Plus, it will make it unnecessary for the proctor to view the test taker screen, which is a huge security problem.This may be a separate download from some other software required to administer a test or it may be integrated. Some approaches to this are problematic – and can have browser, operating and hardware-based issues and dependencies. But for many sorts of exams, prohibiting the user from access other programs from the testing workstation during testing, is important and best accomplished with this sort of solution.Specific behaviors desired or disallowed may vary between test programs – depending on whether or not, for example, their test delivery software provides “right mouse-button”, context-specific action menus, or allows “cut-and-past” during a test section that includes a performance task or long form constructed response type of question.
  18. It’s well-known in most high-stakes testing scenarios that the proctors or test administrators are the ones doing the cheating. Often proctors have a stake in how the test turns out.Security measures must consider the possibility of collusion and test theft with proctors.
  19. Having the ability to interact with the examinee directly during a test session, and their own interaction with the remote proxy during a “test preparation and startup” exercise (which validates the integrity of the test delivery system and the functioning of the remote proctoring solution), can serve as a significant deterrent to undesirable test taker behavior. Being able to question an examinee, say if there is an off-screen interaction, or suspicious movement, allows the test taker to address the issue in real time, provides greater evidence for the test taking event log, and can help insure that fair and practical decisions are made with regard to such incident. An added benefit is to provide a channel for technical support issues if they are sufficiently minor and specific to the test delivery system operation which could result in salvaging test sessions that might otherwise have to be re-scheduled.The form of the interactions can include free-form dialog with users, as you might find on more and more shopping sites, or by using canned scripts and pre-formatted messages, which can help insure a level of quality control, professionalism and standardized behavior by a test delivery organization (particularly if the canned messages are part of an over-arching test proctoring protocol which specified specific messages and content for use in dealing with the most frequent test taking proctor-involvement issues. Specific scripts and messages could be pre-codified for dealing with such things as “off-camera conversations with third parties”, movement of test taker from camera view, temporary occlusion of camera view, and the like.
  20. With such control a proctor can elicit test taker cooperation. Pausing a test and covering or removing the current test question on the screen puts the proctor in control of the cheating or theft situation. Ultimately the proctor can simply suspend or cancel the session if compliance isn’t forthcoming.I interviewed a manager of an online proctoring service using these features. She told me that in literally tens of thousands of such interactions NOT ONE test taker has refused to acknowledge the warning and comply with proctor instructions.When a test is suspended, it can be restarted at the same place the test was suspended.
  21. Note the strong methods involve biological-driven factors (fingerprint, facial recognition, voice recognition), and deeply rooted physiological patterns (keyboard analytics) that are both robust ways to authenticate examinees and quite difficult to “fake”, provided a secure mechanism has been used to capture the subject’s identity in the first place (or for validation of test results against a particular user profile even after the fact). (DFF: this involves a complex topic of authentication versus identification; it’s okay and maybe even desirable to simply collect this “biometric” or pattern when registering in a program, and make sure that those same steps are repeated (authentication) before the test begins and maybe during it. I would avoid the concept of identification for this webinar.)As noted in the authentication discussion above, “don’t use a “picture of a picture” – for anything. And do re-validate in circumstances where examinees can leave a monitored testing area and return for additional work.As noted above in the “authentication risks” of the “using proxy test takers” section: Weak: include the use of ID, particularly through a webcam. A “picture of a picture” is not as robust as a good, high resolution face picture to compare with an externally provided (by test sponsor or results recipient) face picture. (DFF: Fake ID’s are easy to obtain and impossible for an untrained proctor through a camera or in person to verify.)I’ve read about more cloud-based “software as a service” deployment of machine learning and AI tools and algorithms, and of “homeland security” contractors trying out facial recognition software and algorithms delivered to desktops via AWS – so this technology might be incorporated in more secure delivery system sooner than expected. (you still need something to compare a “face” taking a test “with”).
  22. It is critical for decision-making and taking action to have recorded the test security incident, stored it for a specified period of time, and easily used to inform discussion and decisions. Proctor logs of timing and communications and decisions are an important companion set of data.If a question is asked: No, except perhaps for comparative research purposes, I can see no reason to store video records for test sessions where no incident occurred.
  23. Technology advances continue to improve features / functions
  24. Very nice slide. As I said earlier, I’m not all that sure about the point made in the second bullet point.Second bullent: Feel free to delete or change. My thought was for example some programs may not care about content theft – they may release all their tests every year (psych 101 at purdue, calculus 101 at a community college, etc) as they are not that different from “problem sets” assigned during the course. Others might allow “open notes” so don’t care about monitoring if the kid has a cheat sheet – must more concerned about texting a fellow student or running IM on the test computer. Maybe for “speeded tests” there is even less concern about consulting notes or contacting others as there is just not enough “time” to really get help. Say long reading passages, a few short MCQs, with 10 passages and 40 questions in 40 minutes (this is the ‘reading comprehension’ section of the SSAT); much more concern here about securing the material before the test, and ability to look at suspecious timing data to see if someone already had access to the passages etc.Maybe I am just wrong !!
  25. Why should we worry about test quality?