The document provides an overview of adverse impact, how it is defined, measured, and evaluated. It discusses the legal framework around adverse impact from the Griggs v. Duke Power Co. case and the three phases of disparate impact litigation. It then focuses on how adverse impact is measured, discussing both statistical significance tests like the Z-test, Fisher's Exact Test, and practical significance measures like the 80% rule and standard deviation difference test. The document stresses the importance of evaluating both individual stages and cumulative adverse impact of a selection process. It also notes several factors that influence adverse impact and that it is a complex issue without single solutions.

1. Adverse Impact: A Primer on
What You Need to Know
John M. Ford, Ph.D.
Chris W. Hornick, Ph.D.
Kathryn A. Fox, M.A.
IPAC 2014 Pre-Conference Workshop

2. Overview
• Introductions
• Adverse Impact (AI) Overview
• How is AI Measured?
• Evaluating the AI of Your Process
• Importance Considerations Regarding AI
• Practical Advice to Addressing AI
2

3. Adverse Impact
Overview
3

4. Adverse Impact Defined
“A substantially different rate of selection in
hiring, promotion, or other employment
decision which works to the disadvantage of
members of a race, sex, or ethnic group.”
--EEOC Uniform Guidelines on Employee
Selection Procedures
4

5. Griggs v. Duke Power Co.
• Even though they may appear facially neutral, many
traditional assessment tools typically have significant and
large adverse impact on minority groups.
• “Good intent or absence of discriminatory intent does not
redeem employment procedures or test mechanisms” that
disproportionately impact a protected group (Griggs v.
Duke Power Co., 1971).
5

6. Why Should We Be Concerned With
Adverse Impact?
6

7. Three Phases of Disparate
Impact Litigation
• Phase 1: Plaintiff must demonstrate adverse impact.
• Phase 2: Employer Defense
– Defendant must show that the employment practice is:
• Job Related (i.e., Valid)
OR
• Consistent with Business Necessity.
• Phase 3: Alternatives
– Plaintiff can argue that an alternative employment
practice that is equally valid but has less adverse impact
could have been used.
7

8. 8
Valid Alternatives
“Where adverse impact still exists, even
though the selection procedure has been
validated, there continues to be an
obligation to consider alternative
procedures which reduce or remove that
adverse impact if an opportunity presents
itself to do so without sacrificing validity.”
--EEOC Questions & Answers

9. 9
What Could Be a Valid
Alternative?
• A similar selection procedure with similar validity but less
Adverse Impact.
• A different selection procedure with similar validity but
less Adverse Impact.
• Adding or removing a selection procedure to a test battery.
• A different weighting scheme for a test battery.
• A different method of using a series of tests (change order,
compensatory vs. multi-hurdle, etc.).
• A different presentation medium or response methodology.

10. How is Adverse Impact
Measured?
10

11. Measuring Adverse Impact
• No universally agreed upon method for
assessing adverse impact.
• In fact, practitioners, enforcement agencies
(such as the EEOC and U.S. Department of
Justice), the courts, and researchers often
rely on different techniques or methods of
evaluating adverse impact.
11

12. AI Measurement Perspectives
• Most approaches to measuring Adverse
Impact can be categorized into two general
perspectives:
– Statistical Significant Tests
– Practical Significance Measures
12

13. Statistical Significance Tests
• Focus is on determining the likelihood that the differences
between groups are the result of chance variability in the
data (such as sampling error).
• Differences between groups that are unlikely to be due to
chance are considered to be “real” and indicative of
adverse impact.
• Addresses question of “Is there a real disparity?”
• Commonly used by the courts and practitioners.
13

14. Z Test (or 2 Standard
Deviation Test)
• Established by Supreme Court in landmark Hazelwood
School District v. United States (1977).
• Is the minimum standard that many courts have relied on
(Siskin & Trippi, 2005).
• Compares expected selection rates for two groups to actual
or observed selection rates.
• When the difference between the expected and observed
selection rates are more than 2 (or sometimes 3) standard
deviations, the difference is considered to be “real”.
• Results can be misleading when samples are either very
large or small.
14

15. Fisher’s Exact Test (FET)
• Similar to Z Test in that it compares selection rates to
observed selection rates.
• It does not require a minimum sample or cell size.
– More appropriate when samples are smaller than 30 individuals or
there are less than 5 members in either comparison group.
• Adverse impact is indicated when p < .5.
• In larger samples, the results are typically equivalent with
Z Test, but FET is more accurate in small samples.
• FET is the most used statistical test by practitioners
(Cohen, Aamodt, & Dunleavy, 2010).
15

16. Statistical Significance Tests
Concerns
• Many Human Resource Professionals do not have the
technical expertise to conduct and interpret statistical
significance tests.
• Results are very difficult for a lay person to understand.
• Statistical Power is generally low in small samples.
• “Real” differences may be trivial or may not be practically
significant in large samples.
• Statistical significance tests do not provide information
regarding the actual magnitude of the difference.
16

17. Practical Significance
Measures
• Focus is on whether differences between two
groups are of sufficient magnitude to be
considered meaningful.
• Addresses question of “Is disparity large enough
to be meaningful?”
• Commonly used by enforcement agencies and
researchers.
17

18. Adverse Impact Ratio
(4/5th Rule or 80% Rule)
• Most widely used standard for evaluating adverse impact.
• A practical rule of thumb adopted by the Uniform
Guidelines on Employee Selection Procedures. It has also
been adopted by the Department of Labor, the Department
of Justice, and the Office of Personnel Management.
• Indicates that a selection rate for a protected group which
is less than four-fifths (or eighty percent) of the rate for the
group with the highest passing rate will generally be
regarded as an indication of adverse impact.
18

19. How to Calculate AI Ratios
86.
700
600

73.
300
220

Selection rate for group with
highest selection rate
Selection rate for protected
group
AI Ratio
85.
86.
73.

19
≥ .80 = No Adverse Impact
< .80 = Adverse Impact

20. Four-Fifths Rule Concerns
• Arbitrary Standard (why not the Three-Fourths Rule or
the Five-Sevenths Rule?)
• Does a poor job of demonstrating that the employment
practice “caused” the adverse impact.
• Some feel that it suggests a permissible level of
discrimination.
• Although courts often use the Four Fifths Rule in their
decision making, neither the Supreme Court nor any
federal circuit has adopted this standard. In fact, some
courts have ignored or rejected it.
• High error rate in small samples.
• Highly sensitive to selection rates.
20

21. Evaluating Adverse Impact in
Small Samples
• One-Person Rule
– Determine expected number of minorities to be selected.
• Overall selection rate multiplied by the number of minority
candidates rounded down to nearest whole number.
– Compare the actual number of minority hires to the
expected number.
– If the difference is 1 or more, this “rule” suggests
adverse impact and analysis should proceed to…
• N of 1 (or “Flip-Flop Rule)
– If the organization hired one less majority group member and one
more minority group member, would the order of selection ratios
be reversed? If so, adverse impact is generally not thought to have
occurred. 21

22. Standard Deviation Difference
(d-statistic or SD Difference Test)
• Involves comparing group mean scores using a
standardized scale (standard deviation units).
• Using the d-statistic to describe group mean differences
is a popular method for assessing adverse impact among
researchers because it results in a standard method of
comparison across groups or selection methods.
• A d-statistic of 0.00 indicates no difference between
groups, while higher d-statistic results indicate greater
differences between groups.
• No widely accepted benchmark or standard for
determining when adverse impact exists.
• Is not impacted by selection rates or cut scores.
22

23. 23
Protected Group Majority Group
Mean-ProtectedGroup
Mean-MajorityGroup
1 Standard Deviation
d = 1.00
Meets4/5thsRule
Fails4/5thsRule
Comparison of Adverse Impact Ratio and d-
statistic

24. Adverse Impact Is Not Interpreted Solely
on Statistics of Current Process
• Recruitment Practices
– Do they encourage or discourage minority
applicants?
• Sample Size
– When sample size is small, it is appropriate to
consider:
• Data from similar jobs.
• Data from the same job over time.
24

25. How should HR Professionals
Measure Adverse Impact?
• No universally agreed upon standard for measuring
adverse impact.
• Each method of assessing adverse impact has strengths
and weaknesses.
• Each method answers a different question.
• Unfortunately, methodologies do not always agree.
– Both perspectives are sensitive to the size of the
samples.
– The Four-Fifths Rule is highly sensitive to the
magnitude of the selection rates.
25

26. Example – Impact of Sample Size
on Different AI Methodologies
• Applicant Pool:
– 10 White, 10 Black
• Hired:
– 4 White (40%)
– 3Black (30%)
• AI Ratio = .75
– AI - Yes
• FET: p = 0.18
– AI - No
• Applicant Pool
– 1,000 White, 1,000 Black
• Hired:
– 400 White (40%)
– 325 Black (33%)
• AI Ratio = 0.81
– AI - No
• FET: p = 0.00
– AI - Yes
26

27. 27
Example – Impact of Selection Rate on
Different AI Methodologies
Pass Total SR
White 100 4100 2.4%
Black 30 2030 1.5%
Difference in SR = < 1%
AI Ratio = 0.61
AI - Yes
Pass Total SR
White 2400 4100 58.5%
Black 950 2030 46.8%
Difference in SR = 11.7
AI Ratio = 0.80
AI - No
FET: p = .01
AI - Yes
FET: p < .01
AI - Yes

28. Important to Utilize Multiple
Methodologies
• Professional best practices suggest that multiple methods
should generally be used when measuring adverse
impact.
– Free Adverse Impact Calculator
• adverseimpact.org/AdverseImpactAnalysis.xls
• The use of multiple methodologies increases the
confidence that decision makers, lawyers, judges,
politicians, and scientists will have in the results of the
analyses.
28

29. What If Your Process Has
Adverse Impact?
• Selection Methods Must Be Validated
• Test Fairness
• Consider Suitable Alternative Selection
Procedures
• Attempt to Reduce Adverse Impact of
Process
29

30. Fairness
• Uniform Guidelines (1978) admits that the concept
of fairness is a developing concept in the
professional literature.
• Principles (2003) – “Fairness has not single
meaning and, therefore, not single definition,
whether statistical, psychometric, or social.”
• Standards (1999) identifies four possible
meanings, two of which are applicable to selection:
– Equitable treatment for all examinees
– Lack of Predictive bias
30

31. Example – Fairness Analysis of Next
Generation Firefighter Test
31
Comparison of Regression Lines
For Black and White Incumbents
in Next Generation Validation Sample
Test Score
10090807060504030
3
2
1
0
-1
-2
-3
Legend
Black
White

32. Evaluating the Adverse
Impact of Your Process
32

33. 33
Typical Selection Process
Application
Minimum
Qualifications
Test 1 (Written Test)
Test 2 (Physical Ability
Test)
Test 3 (Oral Interview)
Background Check
Offer of Employment
Psychological Exam
Medical Exam
Hire

34. 34
Individual Stage Level AI
• Many people track and evaluate AI on the
basis of the individual stages of the
selection process
– In other words, all one needs to do is meet
the 4/5ths rule at each stage of the process in
order to meet the legal requirements

35. 35
Example: Individual Stage vs.
Cumulative
0.4
0.5
0.6
0.7
0.8
0.9
1
AdverseImpactRatio
MQ Screen Written Test Assessment
Center
Oral
Interview
Selection Stage
Individual Stage Cumulative
1. MQ Screen
W = 600, B = 220
2. Written Test
W = 230, B = 70
3. Assessment Center
W = 80, B = 20
4. Oral Interview
W = 33, B = 7
Total Applicants
W = 700, B = 300

36. 36
Example – Police Department Selection Process
Black-White Adverse Impact Ratios
For Each Assessment Stage
0
0.2
0.4
0.6
0.8
1
Written Classification Polygraph PAT Medical Psych Background Cumalative
0.79 0.83
0.74
0.89
0.88
1.01 0.95
0.27
AdverseImpactRatio
Selection Stage
Assessment Stage
2004 – 2007

37. 37
Important to Evaluate
Cumulative Adverse Impact
• The EEOC Guidelines indicate that the
AI of the overall selection process is
evaluated first.
– In other words, calculating AI only at the
stage level can get you in trouble!

38. Evaluating Individual Components
• If the total selection process demonstrates adverse
impact, the individual components should be
evaluated for adverse impact.
• If the total selection process does not demonstrate
adverse impact, the user is typically not expected
to evaluate the individual components.
– Exceptions:
• When selection process influences pattern of
assignments.
• When the weight of court decisions or other evidence
suggests that an individual component is not job
related.
38

39. Several Individual Selection Methods
Have Been Rejected by the Courts
• As a result of adverse impact, courts
have invalidated:
–Written Tests
–Physical Ability Tests
–Height and Weight Requirements
–Subjective Evaluation Processes
39

40. 40
Hiring in Rank Order Also
Impacts Adverse Impact
• Have you ever looked at the rank structure
of your hiring lists?
• EEOC is more concerned with the actual
selections than the pass points
– Who is actually hired?
– When are they hired and how does this effect
future opportunities?

41. 41
Example: Rank Order
Selection Process Results
10 of 70 W pass
4 of 30 B pass
W pass ratio = 14.3 %
B pass ratio = 13.3 %
AI Ratio = 0.93
Rank Score Race
1 92 W
2 88 W
3 87 W
4 86 B
5 81 W
6 80 W
7 79 W
8 78 B
9 77 W
10 76 B
11 75 B
12 72 W
13 71 W
14 70 W
Hires: 7 W, 3 B
Hire ratios: W = 10%, B = 10%
Hire ratio AI = 1.0
Conclusion: No Adverse Impact
Hires: 4 W, 1 B
Hire ratios: W = 5.7%, B = 3.3%
Hire ratio AI = 0.58
Conclusion: Adverse Impact

42. Important Considerations
Regarding Adverse Impact
42

43. 43
Adverse Impact Is Complex
• No single decision or policy is likely to
eliminate Adverse Impact.
• A variety of decisions throughout the selection
process influence the level of Adverse Impact.

44. 44
A Variety of Decisions Throughout the Selection
Process Influence the Level of AI.
How candidates are recruited
How candidates apply for the position
Minimum Requirements
Predictors—Which, How many, & How utilized
Cut Scores
How final decisions are made

45. 45
Adverse Impact Is Not Easy to
Predict
• AI is impacted by factors unrelated to group
differences
– Total Sample Size
– Number of Minorities in Sample
– Selection Ratio(s)
– Correlation between Predictors

46. 46
Example: Selection Ratio Drift and AI
Adapted from Roth, Bobko, & Switzer, 2006
Company A
White SR = 0.70
Black SR >= 0.56
No AI
Black SR ratio can drift 14
percent below White ratio
before AI is indicated
Company B
White SR = 0.20
Black SR >= 0.16
No AI
Black SR ratio can drift 4
percent below White ratio
before AI is indicated
All things being equal, Company B is more likely to violate the 4/5ths
rule than Company A simply because the selection ratio is lower.

47. 47
Example: AI Ratios From a Single-Hurdle
Selection System
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.9 0.7 0.5 0.3 0.1
ProbabilityofViolatingthe4/5Rule
Selection Ratio
12% Minority
20 % Minority
N = 200
d = 0.00
Roth, Bobko, & Switzer, 2006

48. 48
Example: AI Ratios From a Two-Hurdle
Selection System
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.1 0.3 0.5 0.7 0.9
ProbabilityofViolatingthe4/5Rule
Selection Ratio of Second Predictor
12% Minority
20 % Minority
N = 200
d = 0.00
Intercorrelation = .00
Selection Ratio = .10
Roth, Bobko, & Switzer, 2006

49. 49
Adverse Impact Is Not Easy to
Interpret
• AI ratios are difficult to interpret
– False Positive
– Real Group Differences
– Biased Predictors
– Discrimination
– Combination of the above

50. 50
Adverse Impact Is Not Easy to
Correct
• Many suggestions for addressing Adverse
Impact don’t always work out the way one
would intuitively expect them to.

51. 51
Example: Standardized Group Differences (d)
When Combining Predictors into a Composite
d1 = 1.0
d2 = 0.0
d3 = 0.0
d4 = 0.0
d5 = 0.0
Sackett & Ellingson, 1997
0
0.2
0.4
0.6
0.8
1
1 2 3 4 5
StandardizedGroupDifference(d-
statistic)
# of Uncorrelated Predictors
Expected
Observed

52. 52
Validity Is Not a Justification
To Ignore AI
• Although validity is a legal defense to AI,
ignoring AI can have serious negative
consequences:
– Even a successful defense to an EEOC investigation or court
case can be extremely costly and time-consuming
– Negative political, social, and organizational implications
– Tarnished organizational image
– Fewer top quality minority candidates
– Still vulnerable when there are valid alternatives

53. 53
Adverse Impact Should Be
Evaluated on a Continuum
• All AI is not created equally.
– Although they both violate the 4/5ths rule, an AI ratio
of .70 is preferable to .20.
– Similarly, 1.00 is preferable to .80.
• Higher AI ratios provide a variety of results:
– More diversity in your organization
– Greater likelihood of meeting the 4/5ths rule in
individual samples
– Lower likelihood of grievances, EEOC investigations,
lawsuits, and bad press

54. 54
Adverse Impact Analyses Results Are
Imperfect Indicators of Adverse Impact
• The 4/5ths rule is not Adverse Impact. It is an
indicator of potential Adverse Impact.
– “The 4/5ths rule merely establishes a numerical
basis for drawing an initial inference and for
requiring additional information” (Uniform
Guidelines, Questions & Answers)
• Statistical Significance Tests are based on
probability.

55. 55
Adverse Impact Analyses Sometimes
Provide Erroneous Conclusions
• Adverse Impact analyses often result in
false positives and false negatives.
• For example, AI ratios can vary
substantially over different
administrations.

56. 56
Example: One Client’s AI Ratios Over Multiple
Administrations
1.41
0.84
0.69
0.67
0.87
0.78 0.72
0.44
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1 2 3 4 5 6 7 8
AdverseImpactRatio
Administrations
Combined Sample = .75

57. Example – Simpson’s Paradox
57
Male Female
Applicants Hired
%
Hired
Applicants Hired % Hired
Year 1 150 30 20% 40 16 40%
Year 2 200 70 35% 50 35 70%
Year 3 100 15 15% 50 15 30%
Year 4 50 2 4% 600 48 8%
Total 500 117 23% 740 114 15%
Adapted from Bobko & Roth (2004)

58. 58
It May Be Inappropriate to Evaluate the
Adverse Impact of an Individual
Administration
• Adverse Impact Analyses should be interpreted
extremely cautiously in very small and large
samples.
• National/regional samples, aggregated samples
over time, and meta-analyses provide a better
estimation of Adverse Impact compared to
individual samples.

59. 59
Changing the Presentation
Medium Is Not a Panacea
• Research on the use of video or computer-based tests is
inconsistent.
• Research suggests that reduced AI may be the result of
something other than the change in medium:
– These tests are assumed to reduce AI by reducing the
reading comprehension demands of a test. However, this can
hurt validity if the acquisition of job knowledge or classroom
training is required.
– These formats may expand (or reduce) the domain of
constructs being measured
– Reduced AI may be result of reduced reliability.
• Computer-based and video-based tests should be evaluated just
like any other test—in terms of validity, reliability, AI, cost, and
practicality.

60. Practical Advice to
Addressing Adverse Impact
60

61. 61
Addressing AI Does Not
Require Lowering Standards
• The default assumption often seems to be that efforts to
increase diversity are equivalent to lowering standards.
• Although lowering the selection ratio, reducing
minimum qualifications, or evaluating skills at a lower
level can sometimes be justified and effective, it can also
have serious political, performance, and safety
implications.
• There are methods that have been shown to reduce AI
while increasing or maintaining standards and
requirements.

62. 62
Selection Processes Have
Many Goals
Diversity
Cost Perception
Legality
Quality
Selection
Goals

63. Write Clear and Specific
RFPs
• Define and Prioritize Your Organization’s Goals
with Regard to Testing.
• Clearly Define the Project.
• Define Your Evaluation Criteria.
• Determine Appropriate Weights for Evaluating or
Scoring Tests.
• Give Yourself Adequate Time to Review Tests.
• Don’t Be Afraid to Ask Follow-Up Questions.
63

64. Ask the Right Questions When Evaluating
the Adverse Impact of Potential Tests
• What was the size of the sample(s) used to evaluate adverse
impact?
• Was the sample similar to our applicant pool?
• Is the adverse impact sample representative of our labor
market?
• Was adverse impact evaluated in an incumbent or
applicant sample?
• What are the adverse impact ratios for relevant protected
groups at pass points we are likely to use?
• Is the pass point at which adverse impact is reported
similar to the pass point we will use?
• What is the SD Difference between relevant protected
groups. 64

65. 65
Evaluate Your Recruitment
Efforts
• Addressing AI through selection is just part
of the solution
– Applicant pools can sometimes have a greater
influence on AI outcomes than selection
practices
• Even the best selection system cannot
overcome serious deficiencies in
recruitment!

66. 66
• Show diversity in advertisements
• Use minority recruiters
• Emphasize EEO and diversity values
• Be specific about the job requirements
– Focus on characteristics that link to success on the job
– Focus on characteristics that align with the goals and
philosophy of the organization
• Describe the application and selection process
Recruitment Strategies that Increase the
Number of Qualified Minorities in the
Applicant Pool

67. 67
Use Both Formal and Informal
Recruitment Methods
• Use a variety of Formal & Informal methods
• Informal methods
– Word of Mouth
– Phone calls to people who have expressed interest
– Personal contact
• Formal methods
– Want Ads in Local Newspapers / News Stories
– Job Fairs
– Community Events
– Ads in Minority / Women Publications
– Radio Ads Minority Audience / PSAs
– Cable/Local Access TV Ads / PSAs

68. 68
How Applicants Learned About
Fire Fighter Job Openings
42
40
36 36
39
23
4
15
7
15
7
12
0
5
10
15
20
25
30
35
40
45
50
Current
Member
Friend or
Family
Website Radio Called
Dept.
Want Ads
White %
Black %

69. 69
Expand the Scope of Your
Job Analysis
• Reducing group differences in selection
begins with the job analysis.
– Focus on a multi-dimensional approach to
job analysis
– Ensure that minorities are adequately
represented and contribute to the job
analysis results

70. 70
Traditional Job Analysis
• Typical job analysis overemphasizes
cognitive aspects (ability, knowledge,
tasks)
• Traditional Job analyses are often so
heavily weighted toward traditional
cognitive aspects that multiple dimensions
get buried.

71. 71
Example - Results of a Multiple Dimensional
Approach to Firefighter Job Analysis
4.24
4.17
4.15
3.81
3.95
4.35
Critically
Important
Physical Ability
Mechanical Aptitude
Educational Skills
Interpersonal Skills
Practical Skills
Emotional Outlook
Very
Important
Importance Ratings for Fire Fighter

72. 72
Example – Police Officer Job
Analysis Linkage Results
55%
85%
13%
70%
23%
0% 25% 50% 75% 100%
Basic Educational
Skills
Emotional Outlook
Interpersonal Skills
Practical Skills
Physical Skills
% of Relevant Job Duties Linked to KSAOs

73. 73
Ensuring Multiple Dimensions in the
Job Analysis
• Expand the Job Analysis Domain.
– Increase the job analysis domain to capture the full range of
KSAOs.
– Include non-task elements that employees need to be successful.
• Group technical and cognitive KSAOs similarly to
how non-technical and non-cognitive KSAs are
grouped.
– Over emphasis on “knowledge of” will drown out other critical
elements of the job.

74. 74
Reevaluate How Your Organization
Defines Job Performance
• For many years, job performance was operationalized
as a unidimensional construct focused on the
performance of tasks.
• However, more recent research supports a broader
conceptualization of job performance

75. 75
Multidimensional Conceptualizations of
Job Performance
0
0.05
0.1
0.15
0.2
0.25
Task Contextual Social Skills
0.21
0.13
0.07
StandardizedGroupDifference
McKay & McDaniel, 2006

76. 76
Additional Benefits of Broader View
of Job Performance
• More complete, accurate performance evaluations for
incumbents.
• More equal career advancement opportunities in your
organization.
• Non-task related aspects of performance may become
more important in the future as the need for customer
service, adaptability, and team-based performance
increases (Borman & Motowidlo, 1997)

77. 77
Add Low-Impact Predictors
To Your Process
• Predictors that may provide incremental validity over
cognitive ability while reducing group differences:
– Non-cognitive Skills
– Structured Oral Interviews
– Assessment Centers
– Personality Tests
– Integrity Tests
– Practical Intelligence
– Biodata

78. 78
Example—Next Generation
Firefighter/EMS Test
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Validity
Basic
Educational
Skills
Combined
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
d-statistic
Basic
Educational
Skills
Combined
Combined includes Interpersonal Skills, Emotional Outlook, &
Practical Skills

79. 79
Cautions To Adding
Additional Predictors
• Not all alternative predictors are relevant to all jobs. Make
sure you can defend the use of each of your predictors.
• Your choice of predictors should go hand-in-hand with your
operational definition of job performance and your job
analysis.
• In some cases, adding alternative predictors can reduce
validity coefficients and increase AI.
• Adding additional predictors can significantly increase the cost
of your process.

80. 80
Evaluate the Weighting of
Your Predictors
• Alternative weighting schemes can provide similar
validity with much less AI (Hornick & Axton,
1998)
• Valuing non-task related aspects of performance
will support giving increased weight to low-
impact predictors

81. 81
Example—Differential
Weighting Models
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Validity
Regression
Equal-
Weighted
Rational
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
d-statistic
Regression
Equal-
Weighted
Rational
Hornick & Axton, 1998

82. 82
Evaluate the Sequence of
Your Predictors
• Many agencies Use a Multiple Hurdle or
Multi-stage Approach
• Predictors that are inexpensive, less time
consuming, and amenable to large-group
administrations are often given first
• The order in which you administer your
predictors can have a substantial impact on the
AI of your process

83. 83
Research on Sequencing of
Predictors
De Corte, Lievens, & Sackett, 2006
• With two predictors of roughly equivalent
validity, it is generally better to administer the
high impact predictor first.
– The sequencing has little to no effect on the quality
of the candidates selected.
– Administering the high impact predictor first
results in lower AI as long as it is not applied more
selectively than the second predictor.

84. 84
Example—Sequencing of Predictors
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Average Criterion Score
Cognitive/Structured Interview
Structured Interview/Cognitive
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Black AI Ratio
Cognitive/Structured Interview
Structured Interview/Cognitive
De Corte, Lievens, & Sackett, 2006
Cost
Cognitive/Structured Interview--$79,721
Structured Interview/Cognitive--$115,653

85. 85
Measure a Broad Range of
Skills Early in the Process
• Some agencies administer a cognitive test in the first
stage and measure non-cognitive skills in a later stage
(e.g., interview).
• This strategy will screen out many individuals who
have strong non-cognitive skills and would make
more complete candidates.
• Measuring a broad range of skills can increase the
validity (i.e., the quality of the candidate pool) and
minimize the AI of your first stage (as well as your
total process).

86. 86
Example—Which Candidate
Would be the Best Hire?
Cognitive Interpersonal
Emotional
Outlook
Practical
Candidate
A
87 60 60 60
Candidate
B
85 70 70 70
Candidate
C
83 90 90 90

87. 87
Example—Advantage of Measuring a
Broad Range of Skills Early in Process
Selection
Ratio
AI Ratio-
Cognitive
Screen
AI Ratio-
Complete
Model
% of Top
Candidates
Screened Out by
Cognitive Screen
.20 .32 .32 68%
.40 .37 .49 35%
.60 .52 .65 23%
.80 .63 .85 12%

88. 88
Consider Innovative Ways to
Measure Relevant Constructs
• Situational judgment tests
• Conditional reasoning tests
• Items and exercises with multiple correct
answers
• Relevant Physical Ability Events that favor
smaller-bodied candidates.
• Oral Assessment Center Exercises (such as
In-Basket)

89. Evaluate Your Minimum Qualifications
and Background Check Criteria
• These processes can sometimes substantially
reduce diversity before you administer your
first test.
• There is very little research investigating
the validity and adverse impact of many
background check criteria.
• Make sure you can defend the criteria you
use.
89

90. 90
Example: Investigating Credit
History During Background Check
• EEOC has publicly urged organizations to avoid
using credit history in making pre-employment
screening decisions.
• Credit history has substantial AI
– Freddie Mac study found that the race-credit
correlation is stronger than the income-credit
correlation.
• Bad credit histories are often the result of factors
unrelated to job performance (such as income loss,
medical problems, family breakup).
• Women sometimes have poor credit histories due
to ex-partners who have failed to live up to their
legal obligations (child support, alimony).

91. 91
Using Credit History as a Disqualifier in
Background Investigations Is Difficult to
Defend in Most Cases
• There is no evidence that credit history is related
to job performance.
• There is no evidence linking poor credit history to
theft, fraud, or criminality.
• There is no evidence to guide creating criteria for
rejecting someone due to poor credit history (how
much bad credit history is too much?)
• Credit records are notoriously susceptible to
errors.

92. 92
Regularly Reevaluate Your
Selection Process
• New research is conducted on predictors, job performance, and
AI on a consistent basis.
• You could become legally vulnerable if you are unaware of
new, superior alternatives.
• Conferences, Journals, IPAC Listserv are excellent avenues for
keeping abreast of current state of science and practice.
• Get advice from professional consultants or trusted colleagues
who have experience with the issues you are facing.