Test security has been emerging as a cohesive discipline for the past ten years. There are no college courses that teach test security. And, even if there were, many practitioners don't have time to take those classes. How do you stay abreast of current developments? How do you train your staff in latest best practices if you don't know about them? Are there resources out there, and how do you find them? In this webinar, Caveon will host several special guest practitioners from various industries. These test security veterans have had to answer these very questions. They will address how continuing education will help you improve test security in your organization.

1. Caveon Webinar Series
www.caveon.com 1
June 17, 2015
Aimée Rhodes, CFA Institute
Joe Kamell, American Board of Radiology
Dennis Maynes, Caveon Test Security
Learning About and Teaching Best
Practices in Test Security;
A Panel Discussion

2. Why this topic?
www.caveon.com 2
Training and education
are essential for
improving test security
Respond
Protect
DetectImprove

3. Agenda for Today
• Educational Experiences
• Educational Resources
• Panel Discussion
• Summary
www.caveon.com 3

4. Educational Experiences
www.caveon.com 4
Pioneer: One who is among the first in a field of endeavor

5. Aimée Rhodes
Director of Exam Security at CFA Institute, Aimée is tasked
with maintaining the public confidence in CFA-sponsored
designations by ensuring that:
1. exam content is secure;
2. no candidate has an unfair
advantage over another;
3. only those individuals truly
qualified to pass the exam do
so; and
4. the risks inherent in our exam
programs are identified and
addressed appropriately.
www.caveon.com 5

6. Aimée’s Story
• A lawyer by education, Aimée served on active
duty in the US Army JAG Corps before entering
private practice.
• Aimée moved to ACT as Director, Test Security in
2005.
– She oversaw the security of all of ACT’s resident and
contract programs
• Joined CFA Institute in 2008 and has built their
exam security program from scratch, including:
– A data forensic program
– Web patrol program
– Unusual candidate behavior tracking
– Site visit program
www.caveon.com 6

7. Joe Kamell
1. exam content is secure,
2. Exams are delivered in a
consistent and fair exam
environment,
3. all security related events are
investigated, and,
4. risks to the security of all ABR
exams are identified and
addressed
www.caveon.com 7
As a manager of exam delivery and security at the American
Board of Radiology, Joe is tasked with maintaining the
integrity of ABR Certification by ensuring that:

8. Joe’s Story
• BS – Health Education / MBA in Technology Mgmt.
• Several years in retail loss prevention
• Training in interviewing and interrogation
techniques (Wicklander-Zulawski)
• As a result of a public event in 2011, ABR added
the responsibilities of exam security to my duties,
where the program was built from the ground up,
including:
– Web patrolling
– Data forensics
– Site visit program
– Investigative process
www.caveon.com 8

9. Dennis Maynes
Chief Scientist
Dennis oversees the statistical analysis
of exams for detecting potential test
fraud and strengthening the security of
client exams.
Other responsibilities:
1. Present and discuss test security
topics at conferences, workshops,
and webinars
2. Provide expert testimony support as
necessary
3. Conduct research for furthering the
state of the art of data forensics
www.caveon.com 9

10. Dennis’ Story
• With several others, founded Caveon in
2003
• Developed data forensics methodologies
• Presented in dozens of conferences
• Testified as an expert witness
• Favorite tag lines:
– The answer’s in the data!
– You can’t manage what you don’t measure!
www.caveon.com 10

11. Educational Resources
www.caveon.com 11

12. Reference Works
• https://cespcert.org/resources
• Handbook of Test Security
– CCSSO TILSA Guidebook
– Operational Best Practices for Statewide Large-
Scale Assessment Programs
– Standards for Educational and Psychological
Testing
– The ITC Guidelines on Test Security
www.caveon.com 12

13. Topic-specific Monographs
• ATP Test Security Committee
– ATP Guidelines for Legal Protection of
Assessment Content,
– ATP Online Piracy Enforcement Strategies:
Guidelines and Best Practices,
– ATP Security Survey Report 2013,
– Assessment Security Options: Considerations
by Delivery Channel and Assessment Model
www.caveon.com 13

14. Journals and Publications
• Journal of Educational and Behavioral
Statistics
• Applied Measurement in Education
www.caveon.com 14

15. Conferences
• ATP – www.testpublishers.org
• COTS - http://cete.ku.edu/conference-test-security
• ICE - http://www.credentialingexcellence.org
• NCSA - http://www.ccsso.org/ncsa.html
www.caveon.com 15

16. Workshops
• Caveon Test Security Workshop
• http://www.caveon.com/were-taking-test-
security-on-the-road/
• Various specialty webinars offered by ICE
and Caveon
www.caveon.com 16

17. CESP
• Certified Exam Security Professional
• Training and Credential
• www.cespcert.org
www.caveon.com 17

18. Other Online Resources
• Blogs: Caveon
• Candidate Agreements
• Test Security Procedures
www.caveon.com 18

19. Panel Discussion
www.caveon.com 19

20. Q1
www.caveon.com 20
• How do you stay current with test security
developments and ideas?

21. Q1: How do you stay current with test
security developments and ideas?
• Conferences are a good source of
information
• Benchmarking with colleagues in the
industry
– Vendors
– Testing partners
– Other Directors, Exam Security
• Test security is ALWAYS in the news!
www.caveon.com 21
Aimée Rhodes

22. • Warning – blatant plug!
– Caveon newsletters provide perspective and
news on current issues
– Caveon often discusses current issues in
webinars
• Get engaged on Social Media
– Current trends are often discussed on LinkedIn
– Some folks regularly “tweet” on test security
news
www.caveon.com 22
Q1: How do you stay current with test
security developments and ideas?
Dennis Maynes

23. Q2
www.caveon.com 23
• Is there a special vocabulary for test security? If
so, how do I learn it?

24. • Vocabulary and language are very important
• You need to know the differences between
– Vulnerabilities, Threats, Risks
– Anomalies, Irregularities, Violations, Breaches
• You need to be able to talk about “loss”
• Watch how you talk about potential test fraud
– Are you worried about “cheating” or “validity?”
• Learn how to precisely discuss issues without
stereotyping or implying malfeasance
www.caveon.com 24
Q2: Is there a special vocabulary for test
security? If so, how do I learn it?
Dennis Maynes

25. Q3
www.caveon.com 25
• What advice would you give for someone who is
new to this area and needs to create a test
security program from scratch?

26. www.caveon.com 26
• NETWORK!
• Reach out to other exam security professionals
and spend some time learning from them.
• Ask for examples of various documentation
(policies, letters, etc.).
• Conduct audits to understand where you are and
attack the major issues first.
• Attend training sessions, webinars and
conferences as much as you can.
Q3: What advice would you give for someone who is new to
this area and needs to create a test security program from
scratch?
Joe Kamell

27. • Build relationships with folks who have been there
before!
• Conduct a security audit or peer review to identify
and prioritize your needs
• Identify resources from within your organization
that you can rely on for help:
– Exam Development - Exam Administration
– Legal - Risk Management
– Internal Audit - Professional Conduct
www.caveon.com 27
Q3: What advice would you give for someone who is new to
this area and needs to create a test security program from
scratch?
Aimée Rhodes

28. Q4
www.caveon.com 28
• What things can you do to network with others
who are learning and teaching about test security?

29. www.caveon.com 29
• The most important thing is to pick up the
phone and don’t be shy asking questions.
• Partner with companies such as Caveon
and see what services are available that
would be useful to you. Realize that you
may not need the full menu of options they
provide.
Q4: What things can you do to network with
others who are learning and teaching about test
security?
Joe Kamell

30. • Find the right contacts among your vendors
• Attend conferences
• Seek out others who are in your position
• Establish peer groups
– Non-disclosure agreements
– Regular meetings
– Share information and experiences
www.caveon.com 30
Q4: What things can you do to network with
others who are learning and teaching about test
security?
Aimée Rhodes

31. Q5
www.caveon.com 31
• How can I get involved in promoting test security
education and providing training materials and
experiences for others in test security?

32. www.caveon.com 32
• Be willing to speak up and share your
experiences. Every testing program is
unique.
• Regardless of expertise, there is always
an opportunity to learn from others how
they have handled different security-
related events.
Q5: How can I get involved in promoting test security education
and providing training materials and experiences for others in test
security?
Joe Kamell

33. Q6
www.caveon.com 33
• What has been your biggest challenge and how
have you handled it?

34. www.caveon.com 34
• Educating myself and others what constitutes a
complete or comprehensive test security
program
• The field is still developing and resources are
not unified
• I read a lot. I read from other disciplines and I
listen to what others are saying.
• I think about how writings like “The Art of War”
apply to our discipline.
Q6: What has been your biggest challenge
and how have you handled it?
Dennis Maynes

35. Summary
• Education is critical – an uneducated
individual is one who can’t make choices
• We need to learn from others if we do not
wish to repeat the mistakes that others
have made.
www.caveon.com 35

36. Thank You!
www.caveon.com 36
Aimée Rhodes
Director, Exam Security
aimee.rhodes@cfainstitute.org
Dennis Maynes
Chief Scientist
dennis.maynes@caveon.com
Follow Caveon on twitter @caveon
Check out our blog
www.caveon.com/blog
LinkedIn Group “Caveon Test Security”
Joe Kamell
Exam Delivery and
Security Manager
jkamell@theabr.org