1. Liz Burns, Juniper Networks
Jennifer Semko, Baker & McKenzie
Jack Terry, NBEO
Rachel Schoenig, ACT
Dennis Maynes, Caveon Test Security
Lessons Learned from Using
Statistics to Invalidate Scores
June 19, 2014
Caveon Webinar Series presents:
1

2. Agenda for Today
• Question/Answer Format
• Please use the chat
window to ask questions
• Conclusions
2

3. To Kick Off the Discussion…
Let’s chat about why an
organization would use
statistics to invalidate scores.
3

4. Q1
If a candidate hires
legal counsel and
contests a score
invalidation, what
are the three most
important things you
should do?
5

5. Q1 - Answer
1. Don’t second guess yourself.
Remain confident.
2. Be prepared to provide legal
counsel with a high level
summary.
3. Bring your counsel on board in a
timely manner.
If a candidate hires legal counsel and contests a score invalidation,
what are the three most important things you should do?
6

6. Q2
In your opinion, which statistics
provide the most credible
information concerning potential
test fraud?
7

7. Q2 - Answer
• Old/New or EVT* Items
• Trojan Horse Items
• Value
• Consistent measurement
• Identify candidates who benefit
• Can be automated
• Easy to explain/understand
In your opinion, which statistics provide the most credible
information concerning potential test fraud?
*EVT = Embedded Verification Test
8

8. Q3
When and how do
you believe one
should
recommend score
invalidations to
your board or
governing body?
9

9. Q3 - Answer
When? When you have a good faith
basis for questioning the validity of a
score
How? Provide a full and frank summary;
give your Board the tools they need
When and how do you believe one should recommend score invalidations
to your board or governing body?
10

10. Q4
Describe at least three things
that should be avoided when
using statistics to invalidate
scores?
11

11. Q4 - Answer
1. Using complicated statistical
explanations as rationale
2. Using complex charts or graphs
to illustrate your point
3. Using the word ‘cheat’, ‘cheater’,
or other emotionally-charged,
accusatory language
Describe at least three things that should be avoided when using statistics
to invalidate scores?
12

12. Q5
Based on your
experience, what
are pros and
cons to using
statistics to
invalidate
scores?
13

13. Q5 - Answer
Based on your experience, what are pros and cons to using
statistics to invalidate scores?
Pros
• Statistics are powerful and
illustrative.
• Statistics can be applied
consistently.
• Statistics can provide you with a
“targeted action” approach.
14

14. Q5 - Answer
Based on your experience, what are pro’s and con’s to using
statistics to invalidate scores?
Cons
• Staying ahead of breaches to your
program is challenging.
• Statistics are not a “silver bullet.”
• Exam security requires a
comprehensive plan that is
constantly evolving.
15

15. Q6
Describe a time when you were
successful in using statistics to
invalidate a test score (or
scores). What factors
contributed to yoursuccess? Have you ever used only statistics
to successfully invalidate a score (i.e., there
was no physical or other corroborating
evidence of cheating besides the statistical
results)?
16

16. Q6 - Answer
Describe a time when you were successful in using
statistics to invalidate a test score.
• Studied similarity data, performed
analysis
• Reviewed seating charts
• Identified index values >7.5
• Created a report and referred to
board
17

17. Q7
Describe a time when you had
statistical evidence of aberration,
but were unsuccessful in using
that evidence to invalidate the
score (e.g., you lost a legal
dispute or you decided not to
pursue the case further for some
reason). Explain why you were
not successful.
18

18. Q7 - Answer
Describe a time when you had statistical evidence of aberration, but
were unsuccessful in using that evidence to invalidate the score. What
happened?
• Lack of corroborating evidence
• Unable to convey the compelling
nature of the statistical evidence
19

19. Q8
If a program has a solid testing
agreement in place with proctors and
candidates, how much time, effort, and
money do you estimate is required to
invalidate one test score based on
statistics?
20

20. Q8 - Answer
If a program has a solid testing agreement in place with proctors and
candidates, how much time, effort, and money do you estimate is required to
invalidate one test score based on statistics?
This varies based on your organization and
processes followed.
• Two examples from our panelists
• Example 1: Three primary areas of
expense
• Example 2: Internal, automated process
21

21. Q9
What internal
challenges and
obstacles did you
confront in instituting
your invalidation
program? Were they
operational? Legal?
Political? Please
describe.
22

22. Q9 - Answer
What internal challenges and obstacles did you confront in instituting your
invalidation program? Were they operational? Legal? Political? Please
describe.
• Political – Invalidated scores belonged to
customers and partners. Stakeholders
had to know what to expect and why this
was beneficial.
• Legal – Is process legally defensible? Do we
have all the right documentation in place?
• Operational – Continually evolving and
staying on top of technologies is key.
23

23. Q10
Have you instituted any KPIs or
other measurements around
your use of statistical analyses
and invalidations to measure
program impact and success?
What are they?
26

24. Q10 - Answer
Have you instituted any KPIs or other measurements around your use of
statistical analyses and invalidations to measure program impact and
success?
• Metrics re: cases opened, closed,
cancel rates
• Metrics re: resolution options
selected by examinee
• KPIs to measure effectiveness of
program
27

25. Q11
What legal agreements or legal
foundation need to be in place in
order to consider invalidating
based on statistics?
28

26. Q11 - Answer
What legal agreements or legal foundation need to be in place in order to
consider invalidating based on statistics?
• Clear, comprehensive, signed
candidate agreements
• Ethics policies in place
• Have a plan, be comprehensive
• Court will look to your agreements
29

27. Audience Questions
30

28. Audience Questions
“We see a lot of exam takers who score
very high in very short amounts of time.
We know that it would be impossible to
even read the questions in the short time
spans, but it’s hard to prove cheating.
Is there a guideline on the amount of time
taken per question that we can use to
take these cheaters down?”
31

29. Audience Question - Answer
• Some reading rates exceed 1,000
wpm (trained speed reading)
• Average adult reading rates are 250-
300 wpm on non-technical content
• Reading rates should be verified
• High-speed, erratic reading strongly
indicates braindump usage
Is there a guideline on the amount of time taken per question that we
can use to take these cheaters down?
32

30. Conclusions
“I would urge us to reframe our concerns
about test data integrity not as cheating
concerns, but as a validity issue.”
“I would first strongly caution everyone
NOT to do any analyses at all until first
developing and adopting policies and
procedures about what to DO with the
results of any analyses.”
– Gregory Cizek, NCSA, 2013
http://www.caveon.com/tilsa-test-security-guidebook-next-steps/
33

31. Conclusions
“Perhaps the worst situation one could be in would
be a situation where analyses have been conducted
and it must be admitted publically that nothing has
been done with the results.”
“It is important to treat each similarly situated case
the same way, and a coherent, comprehensive set
of policies and procedures, uniformly applied, is
essential.”
– Gregory Cizek, NCSA, 2013
http://www.caveon.com/tilsa-test-security-guidebook-next-steps/34

32. Thank you!
- Follow Caveon on twitter @caveon
- Check out our blog…www.caveon.com/blog
- LinkedIn Group – “Caveon Test Security”
Liz Burns, Juniper Networks
Rachel Schoenig, ACT
Jennifer Semko, Baker & McKenzie
Dr. Jack Terry, NBEO
Dennis Maynes, Caveon Test Security
35