Public Safety Hiring Tutorial

1. Public Safety Hiring: A Tutorial on
Improving Your Selection Process From
the RFP to the First Day of the Academy
John M. Ford, Ph.D.
Chris W. Hornick, Ph.D.
Kathryn A. Fox, M.A.
IPAC 2014

2. Important Considerations
Before You Start Developing
(or Evaluating) Your Process
2

3. Factors That Should Drive Public
Safety Hiring Decisions
Hiring
Decisions
Professional
Standards
Legal
Guidelines
Scientific
Literature
Practical
Experience
Best Interests
of
Organization
Best Interests
of
Community
3

4. Factors That Should Not (But Often Do)
Drive Public Safety Hiring Decisions
4
Hiring
Decisions
Political
Pressure
Past
Precedence
Archaic or
Misguided
Rules
Self
Interest
Reactive
Emotional
Responses
Rumors
Fear

5. Define Your Selection Process Goals
5
Hiring
Process
Goals
Quality
Candidates
Legal
Defensibility
Candidate /
Public
Acceptance
Cost &
Required
Resources
Diversity

6. 6
Evaluate the Scope of Your Job
Analysis
• Evaluate the extent to which your job analysis
encompasses the complete job domain.
– Ensure that your job analysis captures the full range of
KSAOs relevant to the job.
– Identify the broad range of non-task elements that
employees need to be successful.
• Ensure that technical and task-related KSAOs are grouped
similarly to how non-technical and non-task KSAOs are
grouped.
– Over emphasis on “knowledge of” and task-related
abilities will drown out other critical elements of the
job.

7. Example – Importance Ratings of a
Firefighter Job Analysis with a
Consortium of Fire Departments
Very Important
Essential for successful
performance of the job
Critically Important
Failure to perform results in
extreme negative consequences
3.8
4.3
4.3
4.3
4.4
3.8
4.3
4.3
3 3.5 4 4.5
Basic Educational Skills
Emotional Outlook
Interpersonal Skills
Practical Skills
Physical Skills
Mechanical Aptitude
Firefighting Skills
EMS Skills

8. Example – Frequency Ratings of a
Firefighter Job Analysis with a
Consortium of Fire Departments
3.6
4.2
4.3
4.2
4
3.4
3.6
4.2
3 3.5 4 4.5
Basic Educational Skills
Emotional Outlook
Interpersonal Skills
Practical Skills
Physical Skills
Mechanical Aptitude
Firefighting Skills
EMS Skills
Fairly Often
Approximately once per shift
Very Often
More often than once
per shift

9. Example – Necessity Ratings of a
Firefighter Job Analysis with a
Consortium of Fire Departments
3.8
4.2
4.1
4
4.3
3.1
2.7
3.3
2.5 3 3.5 4 4.5
Basic Educational Skills
Emotional Outlook
Interpersonal Skills
Practical Skills
Physical Skills
Mechanical Aptitude
Firefighting Skills
EMS Skills
Required at a minimum
performance level
at time of hire
Required at a full
performance level
at time of hire

10. 10
Example – Police Officer Job
Analysis Linkage Results
55%
85%
13%
70%
23%
0% 25% 50% 75% 100%
Basic Educational
Skills
Emotional Outlook
Interpersonal Skills
Practical Skills
Physical Skills
% of Relevant Job Duties Linked to KSAOs

11. RFPs / Evaluating Proposals
11

12. Take Control of Your Proposal
Evaluation Process
• The way many proposal evaluation
processes currently function is not
conducive to making informed decisions.
– Don’t let test providers decide what
information you should consider.
• Evaluating tests requires professional
judgment.
– You must ask the right questions and evaluate
the evidence.

13. Write Clear and Specific
RFPs
• Define and prioritize your organization’s goals with regard
to testing.
• Clearly define the project (or clearly specify when you
want guidance).
• Allow consultants to demonstrate their strengths and the
value they can provide.
• Define your evaluation criteria based on your values.
• Determine appropriate weights for evaluating or scoring
tests.
• Give yourself adequate time to review tests.
• Don’t be afraid to ask follow-up questions.
13

14. Proposal Evaluation
• Select proposals that utilize assessment tools that are
consistent with your job analysis results.
• Evaluate the strength of the validation evidence.
• Pay attention to samples.
• Don’t compare corrected validity coefficients to
uncorrected validity coefficients.
• Evaluate adverse impact in a manner that is consistent
with how you will use the test.
• Recognize that adverse impact in applicant samples will be
larger than in incumbent samples.
• Has the test been involved in litigation?
• Don’t just evaluate the tests, evaluate the consulting
company / test provider. 14

15. Remember Hidden Costs
• Poor performance
• Increased training/remedial
training/retraining
• Lawsuits
• Turnover
• Grievances
• Disciplinary problems
• Accidents

16. Hiring Guidelines & Policies
16

17. Some Agencies Have Policies that
Function as a Bureaucratic
Straightjacket
• Many agencies have policies or agreements
that hinder their ability to develop an
effective hiring process.
– Mandatory Cut Scores
– Mandated Selection Tools
– Mandated Preference Points
– Mandated Weighting of Components
– Banding Requirements
– Mandated Minimum Requirements
17

18. Strategies for Changing Policies & Agreements
• Build long-term relationships.
• Establish your credibility.
• Demonstrate fairness and lack of bias.
• Show that you have their best interests at heart.
• Learn the organization (organizational structure, key
decision makers and potential advocates, available
resources, culture and values, lingo, etc.).
• Identify areas of agreement.
• Frame communications around strategic goals, belief
systems, and areas of agreement.
• Provide information that reduces fear.
• Measure! Measure! Measure!
• Trumpet successes. 18

19. Minimum Qualifications
19

20. Important Considerations Regarding
Minimum Qualifications (MQs)
• Represent the minimum competence level needed
to perform the job.
• MQs are often highly subjective.
• MQs often result in adverse impact or less
diversity in your candidate pool.
• MQs sometimes screen out outstanding
candidates.
• Processing applications can be very time
consuming – evaluate when in your process you
should review applications.
20

21. Hiring Announcements
21

22. Best Practices for Hiring
Announcements
• Emphasize EEO and diversity values.
• Describe the application and hiring process.
• Be specific about the job description and required
qualifications.
• Provide information on the benefits and available
career paths.
• Advertise through want ads, minority
publications, television (including Cable/Local
Access), and radio stations.
22

23. 23
How Applicants Learned About
Fire Fighter Job Openings
42
40
36 36
39
23
4
15
7
15
7
12
0
5
10
15
20
25
30
35
40
45
50
Current
Member
Friend or
Family
Website Radio Called
Dept.
Want Ads
White %
Black %

24. Recruitment
24

25. Recruitment Best Practices
• Involve women and minorities in the recruitment process.
• Show diversity in ads brochures, recruiters, media
coverage, etc.
• Ensure that women and minorities in the department have
a visible presence.
• Encourage “word of mouth” recruiting among women and
minorities.
• Go where the women and minorities are!
• Have a presence at cultural events and job fairs.
• Build positive relationships with local minority community
leaders and minority groups.
• Provide mentoring and training/tutoring for the tests.
25

26. Targeted Recruiting Do’s &
Don’ts
• Do not exclude non-minorities from any of your
recruitment initiatives or opportunities.
• Work with leaders in the minority community to
encourage minorities to take advantage of
recruitment initiatives or opportunities.
• Do not use quotas.
• Be consistent and fair to every candidate in your
hiring process.
• Hire highly qualified candidates.
26

27. Assessment Methodology
27

28. Best Practices for Making Decisions
about Your Assessment Process
• Don’t make assumptions – Investigate and
Measure.
• Recognize that there is no perfect process or
perfect test.
• Conduct a validation study or transportability
study.
• Be careful about making decisions based on single
administrations or small samples.
• Understand sampling error.
28

29. 29
Example: One Client’s AI Ratios Over Multiple
Administrations
1.41
0.84
0.69
0.67
0.87
0.78 0.72
0.44
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1 2 3 4 5 6 7 8
AdverseImpactRatio
Administrations
Combined Sample = .75

30. Example – Simpson’s Paradox
30
Male Female
Applicants Hired
%
Hired
Applicants Hired % Hired
Year 1 150 30 20% 40 16 40%
Year 2 200 70 35% 50 35 70%
Year 3 100 15 15% 50 15 30%
Year 4 50 2 4% 600 48 8%
Total 500 117 23% 740 114 15%
Adapted from Bobko & Roth (2004)

31. 31
Evaluate Testing Medium
Just as You Would Any Test
• Research on the use of video or computer-based tests is
inconsistent.
• Research suggests that reduced AI may sometimes be the result
of something other than the change in medium:
– These tests are assumed to reduce AI by reducing the
reading comprehension demands of a test. However, this can
hurt validity if the acquisition of job knowledge or classroom
training is required.
– These formats may expand (or reduce) the domain of
constructs being measured
– Reduced AI may be result of reduced reliability.
• Computer-based and video-based tests should be evaluated just
like any other test—in terms of validity, reliability, AI, cost, and
practicality.

32. Written Tests
32

33. Ask the Right Questions When
Evaluating the Validity of Potential Tests
• What does the test measure? Are those dimensions
consistent with your job analysis results?
• Does the test have a adequate validation report?
• Was the test based on a job analysis of the position you are
hiring for?
• Is the recommended use for the test consistent with your
process?
• Where was the test validated?
• Is the validation sample representative of your position?
• What was the criterion used for the validation?
• Does that criterion represent critical performance
dimensions of your position? 33

34. Ask the Right Questions When Evaluating
the Adverse Impact of Potential Tests
• What was the size of the sample(s) used to evaluate adverse
impact?
• Was the sample similar to our applicant pool?
• Is the adverse impact sample representative of our labor
market?
• Was adverse impact evaluated in an incumbent or
applicant sample?
• What are the adverse impact ratios for relevant protected
groups at pass points you are likely to use?
• Is the pass point at which adverse impact is reported
similar to the pass point you will use?
• What is the SD Difference between relevant protected
groups. 34

35. 35
Measure a Broad Range of
Skills Early in the Process
• Some agencies administer a cognitive test in the first
stage and measure non-cognitive skills in a later stage
(e.g., interview).
• This strategy will screen out many individuals who
have strong non-cognitive skills and would make
more complete candidates.
• Measuring a broad range of skills can increase the
validity (i.e., the quality of the candidate pool) and
minimize the AI of your first stage (as well as your
total process).

36. 36
Example—Which Candidate
Would be the Best Hire?
Cognitive Interpersonal
Emotional
Outlook
Practical
Candidate
A
87 60 60 60
Candidate
B
85 70 70 70
Candidate
C
83 90 90 90

37. Physical Ability Tests
37

38. 38
Legal Background of PATs
• Many Title VII standards have never been
interpreted by the Supreme Court, and lower
court decisions have been inconsistent and
contradictory (Brooks, 2001).
– “It is not an overstatement to say that no other aspect of
the personnel selection process is fraught with more
difficulty or clouded with more ambiguity than physical
ability testing.” Larry Hoover, “Trends in Police
Physical Ability Selection Testing”
– A 1997 survey of court reviewed police and fire PATs
found that less than 10% were successfully defended.
– Most valid PATs will demonstrate at least some adverse
impact with regard to Female candidates.

39. 39
PAT Recommendations
• Use a PAT that utilizes job simulation exercises.
– They tend to have lower adverse impact.
– They tend to be highly correlated with physical job
performance.
– They are the easiest to defend legally as being job-
related.
• In addition to measuring strength, including events
emphasizing balance, flexibility, and endurance can
increase validity while reducing adverse impact.
• Use a compensatory model.
• Ensure that administrators and proctors understand that
making changes to the test can invalidate the test.

40. More PAT Recommendations
• Make sure individual events are supported by the job
analysis.
• Record data for individual events.
• If women are underrepresented in the department, recruit
women from colleges and gyms to run through the PAT
before setting a cut score.
• Communicate the importance of physical fitness to the
candidates early in the process.
• Show a video of your PAT to the candidates and put the
video on your website.
• Provide mentoring and practice to candidates.
40

41. Oral Interviews
41

42. Oral Interview Best Practices
• Use Structured Interview format.
• Ask questions that assess situational
judgment and past behavior.
• Create scoring benchmarks with Subject
Matter Experts.
• Use mock candidates to allow assessor
panels to practice working as a team.
• Use a multiple hurdle or compensatory
model with written test scores. 42

43. Accommodating Disabilities
43

44. Best Practices in
Accommodating Disabilities
• Require a letter from a doctor with a
diagnosis and a recommended
accommodation.
• Establish a time frame for the letter from
the Doctor.
• Remember that accommodations need to be
reasonable – the candidate must be able to
reasonably perform the job.
44

45. Preference Points
45

46. Preference Points
Considerations
• Make sure your preference points don’t
overwhelm the variability in your scores.
• Measure the impact of your preference
points on the distribution of your scores and
your hiring process.
– Do they prevent the top candidates from being
hired?
– Do they reduce the diversity at the top of your
rank-order list?
46

47. Background Checks
47

48. Evaluate Your Background Check
Criteria
• There is very little research investigating
the validity and adverse impact of many
background check criteria.
• There is very little consistency in the
background check criteria used throughout
the country.
• Make sure you can defend the criteria you
use.
48

49. 49
Example: Investigating Credit
History During Background Check
• EEOC has publicly urged organizations to avoid
using credit history in making pre-employment
screening decisions.
• Credit history has substantial AI
– Freddie Mac study found that the race-credit
correlation is stronger than the income-credit
correlation.
• Bad credit histories are often the result of factors
unrelated to job performance (such as income loss,
medical problems, family breakup).
• Women sometimes have poor credit histories due
to ex-partners who have failed to live up to their
legal obligations (child support, alimony).

50. 50
Using Credit History as a Disqualifier in
Background Investigations Is Difficult to
Defend in Most Cases
• There is no evidence that credit history is related
to job performance.
• There is no evidence linking poor credit history to
theft, fraud, or criminality.
• There is no evidence to guide creating criteria for
rejecting someone due to poor credit history (how
much bad credit history is too much?)
• Credit records are notoriously susceptible to
errors.

51. Final Hiring Decision
51

52. Final Considerations
• Remember that Rome was not built in a day –
making changes may take a substantial amount of
time.
• Focus on the changes you can make.
• Continue your professional development and stay
aware of current research, litigation, and best
practices.
• Continue to measure!!!
• Periodically re-evaluate your process.
52