How to Build the Ideal Nurse Hiring Assessment

How to Build the Ideal Nurse Hiring Assessment March 23, 2012

How to Build the Ideal Nurse Hiring
Assessment

Friday, March 23, 2012, 9:00 AM & 10:00 AM
Session Number: CS-3

Presenters:

James Ostmann Sr, RN, MBA, Chief Nursing Officer,
Prophecy Healthcare, High Point, NC

Dan Biddle, PhD, President CEO,
Biddle Consulting Group, Inc., Folsom, CA

Prophecy Assessment Contributors
Prophecy Healthcare, Inc.
A leading provider of healthcare assessments: over 1.5 million exams administered
Standardized competency testing focuses on screening skill levels pre- and post-hire
Rigorous and comprehensive validation program
Over 100 nursing, allied, and home healthcare job knowledge competency exams
Over 50 self-assessment clinical-skills checklists

Biddle Consulting Group, Inc.
Leading U.S. EEO consulting firm
Nationally-recognized experts in test development and validation
Experience in 200+ state and federal cases involving EEO compliance & testing

PeopleClues
International test publishing firm that provides modern & validated behavioral
assessments
Online platform specifically designed for the commercial market
Assessments used by thousands of companies to hire, train, and promote

Prophecy Healthcare // AONE Annual Meeting & Expo 1


The Climate…
• In June 2011, Wanted Analytics reported that employers and
staffing agencies posted more than 121,000 new job ads for
Registered Nurses
• Typical annual turnover rates for RNs average 14% (KPMG
survey)
• Bureau of Labor Statistics reports “the health care industry added
428,000 jobs throughout the 18-month recession from December
2007 until June 2009, and has continued to grow at a steady rate
since the end of the recession.”
• Hospitals, long-term care facilities, and other ambulatory care
settings added 37,000 new jobs in March 2011, the biggest
monthly increase recorded by any employment sector.
• The BLS confirmed that 283,000 jobs have been added in the
healthcare sector within the last year.

The Question…
• With such high growth rates expected, how do
healthcare institutions hire the most qualified
“best fitting” nurses for their openings?
• What knowledges, skills, abilities, and personal
characteristics (KSAPCs) should hiring
authorities pay most attention to when building
a robust hiring system for nurses?
• Finally, how can these “key KSAPCs” be
accurately measured in the shortest testing
session possible?

The Presentation
• We’ll review the answers we found to these
questions while working with over 900
professionals, including:
– 492 nurse staff from Saint Francis Medical
Center (470 nurses and 22 nurse supervisors)
– 384 nurse staff from Frederick Memorial
Hospital (367 nurses and 17 nurse supervisors)
– 13 test development professionals and
industrial-organizational psychology consultants
– 30 film professionals



While Reviewing our Process…
• Keep in mind… we want you to replicate,
borrow, steal, or just learn from our process
• We don’t claim to have the “best” process,
but we have uncovered some of the key
traits to target in your testing program
• We’ve also learned how your hiring program
might benefit from using different tests in
different ways for hiring nurses

Our testing program resulted in a 3-pronged assessment strategy called
“Prophecy”
Our goal was to create a stronger alignment between the nurse job
requirements and nurse testing programs

• A typical nurse hiring process
measures only limited clinical Nurse Job
background (sometimes through a Performance:
written test) and interpersonal What makes nurses
competence (through interviews) perform they way
they do on the job?
• Prophecy measures cognitive,
Smarts? Drive?
interpersonal, and behavioral
Interpersonal skills?
– Intentionally
– Thoroughly
– With limited overlap

What Makes Up Nurse Job Performance?

But how should
test skills be Interpersonal
tested? Job Skills
Clinical Job
Knowledge Performance
How should
they be
Personality
weighted? Cognitive
Factors
Ability
Which skills are
overlapping?
We tried to answer these Experience/
questions while developing Background
Prophecy…



The Prophecy Development Process

Prophecy Overview

Project Phase 1: Job Performance

Project Phase 2: Situational
2008-2011
Validation
Process Project Phase 3: Clinical

Project Phase 4: Behavioral

Project Phase 5: Combining the Assessments

Prophecy is the aggregation of three
assessment tools. The three pieces work
together to form the most comprehensive
employment predictor for the nursing field.

Twelve Clinical Assessments were
included in the study, each having 30-
70 items measuring clinical specialty
job knowledge (e.g., Medical-Surgical,
Emergency Care, etc.)



The Situational Assessment included
21 video-based situational judgment
scenarios designed to measure the
applicants interpersonal competence
using a variety of complex situations
that emerge in hospital settings.

A 70+ item behavioral/personality test
bank was included in the study. Twenty
(20) of the most predictive items were
distilled into a “Nurse Effectiveness
Scale” which demonstrated strong
validity results.

Prophecy Overview
Hiring Nurse Job Performance
Process
Comparison Typical Typical
Process: Process:
Interpersonal Cognitive
Skills Ability

Result = 4-6% Job Performance
Predicted

Result = 16% Job Performance Predicted



A job analysis process was used to distill 19 job performance
dimensions that met the following criteria:

• Observable aspects of job performance that were likely to
relate to traits measured by the Situational Assessment;
• Represented qualifications that were needed on the first day
of the job;
• Not geographically limiting;
• Not likely change over time; and
• Differentiated “best performers” in the nursing profession.
• Emphasis on soft skills

Each performance dimension was phrased in “observable aspects of
job performance”

Job Performance Rating Survey Dimensions
Honest & Conscientious Problem Solving

Verbal Communication Patient Care (calm & competent)

Administering Medications Report Transitioning

Assertiveness Continuous Observation

Change Adaptation Developing Patient Relationships

Conflict Resolution Patient Care Plan Management

Accountability Patient Customer Service

Critical Thinking New Technique Application

Multitasking Delegating/Managing Patient Care

Following Clinician Instructions

• Both Hospitals participated in the nurse rating
process:
– Saint Francis Medical Center:
• 470 nurses were rated by
• 22 nurse supervisors
– Frederick Memorial Hospital:
• 367 nurses rated by
• 17 nurse supervisors

• A total of 888 nurses were rated on 19 dimensions
of job performance by 39 supervisors



Project Phase 2: Situational Assessment Build

• “Situational Judgment Tests” (SJTs) pose hypothetical situations to
applicants in a job-related context and then provides several
plausible alternatives on the most effective and least effective way to
handle the situation
• “Most effective” and “least effective” keys were determined using
panel of 50 nurse supervisors
• Why SJTs?
– SJTs measure valuable skills that are typically not measured in hiring
processes, or are not measured very accurately or systematically.
– Applicants perceive context-rich tests as more job relevant
– SJTs are scored consistently and reliably, compared to when the
same skills are measured using interviews
– Because they are based on “consensus scoring,” they are more
objective than interviews+

Step 1: Job analysis research was conducted at four medical/employment
centers and 19 of 169 key KSAPCs were selected
Step 2: Developed SJT Scenarios (based on 19 key competencies)
Step 3: Refined SJT Scenarios & Converted to 50-Page Script
Step 4: Staged Sets & Filming
Step 5: Multi-point Keying Process was developed based on level of
expert consensus
Step 6: Validation Study was conducted by correlating test scores to 19
job performance dimensions
Saint Francis Medical Center: 27 Nurse Supervisors rated 191 nurses
Frederick Memorial Hospital: 17 Nurse Supervisors rated 235 nurses

Criterion-related Validity: Demonstrated by empirical data showing
that the selection procedure is predictive of, or significantly
correlated with, important elements of work behavior
Criterion-Related Study
Performance Measure

70
60
50
40
30
20
10
0
0 20 40 60 80 100
Test Score

Job Performance Rating Test Score
21



Interpreting Criterion-related Validity
U.S. Department of Labor Guidelines for Interpreting Validity
Coefficients

Percentage of Job
Coefficient Value Interpretation Performance Explained
(Coefficient Squared)

Above .35 Very beneficial 12.3%+

.21 - .35 Likely to be useful 4.4% - 12.3%

.11 - .20 Depends on circumstances 1.2% - 4%

Below .11 Unlikely to be useful <1.2%

Source: U.S. Department of Labor (2000), Testing and Assessment: An Employer’s Guide to Good Practices (p. 3-10).


How well does the Situational Assessment
Predict Job Performance?
Theoretical Expectancy Bands

Likelihood of Receiving
Band Low High Cumulative % of Nurses “Above Average” Supervisor
Score Score Scoring in Band Ratings
FAIL 0 69 <5% <42%
E 70 82 20% 43%
D 83 86 40% 54%
C 87 90 60% 60%
B 91 94 80% 67%
A 95 109 100% 75%
Note: The “base rate” for the expectancy table was set at 60% because this portion of the distribution received ratings >=5, indicating they were
rated as the median (or higher) of job performance. The theoretical (corrected) validity used for this analysis as .30 (set roughly between the two
corrected validities observed in the study).



Situational Assessments Validity
Results Interpretation
Assuming 60% of the nurse applicants “show up qualified” (i.e.,
would receive average JP ratings) and the Situational
Assessment is used to select highest scoring 20%...
• 67% of this “top 20% group” will exceed performance
expectations on the job (opposed to only 60% by using no
assessment)
• This is a 7% improvement in overall workforce qualifications
(raised from 60% to 67%)
• In terms of people, say for instance your hospital hires 100
nurses per year, 7 additional nurses would be
expected to exceed performance expectations (be
successful on the job) than without using the
assessment.

Project Phase 3: Clinical Assessment Build

• The Clinical Assessments are designed to measure critical job
knowledge domains that are necessary for clinical practice
areas (e.g., Medical-Surgical, Emergency, etc.)
• They are developed and validated using a content validation
methodology, which means:
– Test content is mapped to a job analysis for the relevant
practice area
– Tests are balanced with respect to the questions measuring the
specific aspects of each practice area
– Cutoffs are established using a “Modified Angoff” process,
where nurse experts set the cutoff score by evaluating the
difficulty of each item on the test

• The Clinical Assessments are “Mastery Based” or “Criterion-
Referenced” tests. This means:
– Designed to distinguish between who “can” vs. who “cannot” perform
the job
– Test items focus on baseline competency skills and knowledge
domains
– Cutoff scores are set using nurse experts with respect to the “minimum
competency level necessary to perform the job on the first day”
• Without using Clinical Assessments, hospitals stand the risk of hiring
unqualified and/or under-qualified nurses who may not possess the
key knowledge and skill competencies and may create liability.
• Notwithstanding this focus, the Clinical Assessments still
demonstrated statistically significant correlations to job performance.




Critical Thinking
Giving Meds…
Con

New Techniques…
scie

Confl
Chan
n

ic
tiou

t Res
ge A
s

olutio
d
apta

n
onti

Amount of Job Performance Variance Predicted by Ranked list of the Job Performance
the Clinical Assessment (Ranked)(1,2) Dimensions predicted by the
Percentage Clinical Assessments (after
Performance Dimension (%) controlling for the variance of the
Administering Medications 35% other two tests)
Problem Solving 34%
Report Transitioning 33% HARD SKILLS &
HARD SKILLS &
Critical Thinking 33% “SOLO” PERFORMANCE
“SOLO” PERFORMANCE
Multitasking 32%
Assertiveness 30% The test correlated to the JP
Following Clinician Instructions 29% dimensions just as one would
Patient Care Plan Management 27% expect, with the highest in the
Continuous Observation 25% “hard skills” requiring “solo
Patient Care (Calm & Competent) 22% performance” and lowest with
New Technique Application 22%
“soft skills” involving team
Patient Customer Service 21%
performance.
Accountability 19%
Delegating/Managing Patient Care 18%
Developing Patient Relationships 17%
SOFT SKILLS AND
SOFT SKILLS AND
Verbal Communication 16%
TEAM PERFORMANCE
TEAM PERFORMANCE
Change Adaptation 16%
Honest & Conscientious 14%
Conflict Resolution 13%

Project Phase 4: Behavioral Assessment Build
• 344+ Nurses from Saint Francis Medical Center completed
a 70-item personality/behavioral scale (scored using a 1-5
agreement scale) that measured:
– Conscientiousness
– Tough-Mindedness
– Conventional
– Extroversion
– Stability
– Teamwork
– Good Impression
• 70-item scale was reduced to a 20-item “Nurse
Effectiveness Scale” using a split-half validation
methodology.




176 Nurses 168 Nurses
“Calibration” “Validation”
Sample 344 Nurses
Sample
•Build the test •Re-evaluate
•Maximize
correlations
correlations

0.3
0.4
0.35 0.25
0.3
0.2
0.25
0.2 0.15
0.15 0.1
0.1
0.05 0.05
0 0
Average Criterion…
Honest &…

Administering…

Continuous…
Developing Patient…
Patient Care Plan…
Patient Customer…
New Technique…
Patient Care (calm &…

Average Criterion…

Administering…

Developing Patient…
Patient Care Plan…
Patient Customer…
New Technique…
Honest &…

Patient Care (calm &…

Continuous…
Assertiveness

Critical Thinking
Multitasking

Problem Solving

Report Transitioning
Verbal Communication

Change Adaptation
Conflict Resolution
Accountability

Assertiveness

Accountability
Verbal Communication

Change Adaptation
Conflict Resolution

Critical Thinking
Multitasking

Problem Solving

Report Transitioning


Behavioral Validity Coefficients
New Techniques
0.3
Conscientiousness Accountability Delegation/Mg Patients
0.25
Correlations

0.2

0.15

0.1

0.05

0

• The tests were weighted
using Structural Equation
Modeling (SEM), allowing
them to be combined in
the most efficient way
possible.
• After correlating the three
tests with avg. job
performance, the
remaining contribution of
cognitive ability was non-
significant.
• This was because the
Clinical and Situational
Assessments were highly
correlated with the
cognitive ability test (r =
.16 and .29).



Percentage (%) of Job Performance Predicted/Explained by Assessment

Clinical Situational Behavioral
70%
% of Job Performance Explained

60%

50%

40%

30%

20%

10%

0%

s

re
ns

n
ng
n
on

ng

t
hip

on
lity

e
n
nt)
g
us

g
ce

n
s

tio

en
tio

Ca
ic
tio
es

kin
tio

in
tio

ki
ati

ni

a ti
io
an

te

em

rv
ns
uc
bi

lv
en

olu

va
in
ica

io
pta

nt
nt

tas
ic

pe

lic
rm

Se
nta

So

tio
str

ag
tiv

Th

sit

tie
er
cie

un

es
ed

m
da

ti-

pp
r fo

la
In

an
ou

er
bs
an
er

m

Pa
tR

Co
m
ns

al
M

ul
A

Re

eA
m
ss

sO
le
Pe

M
cc

ian

Tr
om

itic
Co

M
lic

g
ge

sto
g

A

ob

&
A

nt

in
qu
in

n
ge

ic

ou
rt
lC

nf

Cr
an

Pla
Pr

tie

Cu
t&

alm

ag
er

po
in

ni
ra

Co

nu
Ch
ba

ist

Pa

an
Cl

Re

ch
ve

es

re

nt
(C

nti
er

in

M
Te
on
A

tie
Ca
ng

ng
Co
V

dm

re

g/
H

Pa
wi

pi

ew
Ca

nt

tin
A

elo
llo

tie

N

ga
nt
Fo

ev

Pa

ele
ti e

D
Pa

D

Job Performance Dimension


Top Six Ranked Performance Dimensions Predicted by
Each Assessment(1)
Clinical Situational Behavioral
Administering Medications Patient Care (Calm & Competent) Accountability
Problem Solving Developing Patient Relationships Honest & Conscientious
Report Transitioning Verbal Communication Conflict Resolution
Critical Thinking Patient Customer Service New Technique Application
Multitasking Patient Care Plan Management Delegating/Managing Patient Care
Assertiveness Conflict Resolution Change Adaptation

Note (1): After controlling for the predictive variance of each of the other two tests.

• The final battery
collectively
measures each
job performance
dimension

• The tests load on
different aspects
of job performance
in a balanced way



How Does This Translate to Actual Hiring Benefits?

Practical Effectiveness of Using Prophecy Assessments(1)

Base Rate of Applicant-to- % Qualified Hiring Process
Qualified Nurses Hire Ratio Nurses Hired Improvement Rate

80% 50% 89% 9%

80% 30% 92% 12%
80% 10% 95% 15%

50% 50% 63% 13%

50% 30% 69% 19%
50% 10% 77% 27%
Notes: (1) Assumes an average job performance of 50 and a SD of 10.

Practical Impact of Using Prophecy Assessments in a Nurse Hiring Process
(Assuming 50% "Qualification Base Rate")
100%
Likelihood of Hiring an "Above

90% Likelihood of Hiring an "Above Average" Nurse
Using Prophecy Assessments
Average" Nurse

80%

70%
27%
17% Benefit
60%
9% Benefit
Benefit
50%

40%
Ability to Predict Hiring an "Above Average" Nurse Using No Assessments (or invalid assessments)
30%
100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%

Selection Ratio (% of Applicants Screened in Using Prophecy Assessments)

Scenario Results
Nurse Workforce Job Performance Improvement Levels Using Prophecy
Assessments (.45 Validity Coefficient, 20% Selection Ratio)

0.045
Using the Prophecy 24% non-overlap
0.04 Assessments creates a 6.3% between the tested
0.035 average performance gap and untested groups
between the tested and
0.03 untested group
0.025
6.3%
0.02

0.015

0.01

0.005

0
15 20 25 30 35 40 45 50 55 60 65 70 75 80 85

Nurse Job Performance Levels



Running Through the Math…
• Making the following assumptions…
– Hospital with 720 nurses
– Job Performance levels in the corporate nurse workforce is normally
distributed, with half “above average” and half “below average”
– Turnover is 13.9% (per American Association of Colleges of Nursing/Bernard
Hodes Group)
– No growth (to be conservative), although most research reports cite nurse
vacancy rates exceeding 8% and the Bureau of Labor Statistics (BLS)
projected that more than 581,500 new RN positions will be created through
2018, which is a 22% increase in the RN workforce
• Let’s run through 3 scenarios:
– Using no test (or an invalid test) for 3 years
– Using Prophecy Assessments with a 50% cutscore for 3 years
– Using Prophecy Assessments with a 20% cutscore for 3 years

Changing the Workforce One Hire at a Time
Job Performance Levels of the Workforce
After 3 Years of Not Using a Valid Test
70%
60%
50%
40%
30%
20%
10%
0%
Year 1 Year 2 Year 3
Percentage of Above-Average Nurses in the Workforce
Percentage of Below-Average Nurses in the Workforce

Job Performance Levels of the Workforce After 3
Years of Using Prophecy (w/50% Selection Ratio)
70%
60%
50%
40%
30%
20%
10%
0%




Job Performance Levels of the Workforce After 3
Years of Using Prophecy (w/20% Selection Ratio)
70%
60%
50%
40%
30%
20%
10%
0%

Wrap-up
• The most important factor in building and using a valid assessment process is
measuring the whole person (otherwise, valuable competencies are not
considered in making critical employment decisions)
• The next most important factor is measuring the various competencies of that
“whole person” using tests that have a proven, scientific connection to those
various skills and competencies.
• The next most important factor is measuring those various competencies in
reliable (consistent) and efficient (fast) ways.
• The final consideration is properly balancing (weighting) the various tests in a
way that matches the job
• Prophecy does all four.


How to Build the Ideal Nurse Hiring Assessment

Recommended

Recommended

More Related Content

Similar to How to Build the Ideal Nurse Hiring Assessment

Similar to How to Build the Ideal Nurse Hiring Assessment (20)

Recently uploaded

Recently uploaded (20)

How to Build the Ideal Nurse Hiring Assessment