This document summarizes two case studies on using new selection methods for high-stakes roles: selecting doctors in the UK NHS and private bankers at RBS. For doctors, job analysis identified key competencies like empathy and problem-solving. Situational judgement tests were developed and validated, improving predictive validity and reducing failures. For bankers, job analysis explored tensions between capabilities like profit and trust. Situational tests were also used, showing early signs of improved validity for assessing capabilities. The implications discussed developing role-specific frameworks and using evidence-based selection tailored to organizational contexts.

1. Evaluating new methods &
approaches for high stakes selection:
Implications for policy & practice
Professor Fiona Patterson
Work Psychology Group & University of Cambridge
University of Athens, 2011

2. Overview
• The selection process & best practice
• Case studies of new selection methods & approaches
in high stakes selection
Case Study 1: Selecting Doctors
Case Study 2: Selecting Private Bankers
• Implications for policy & practice

3. The Selection Process
Create
Job competency Attract pool of
Analysis model applicants
Tasks, roles, Create person Choose selection Selection
responsibilities specification methods
Knowledge, Interviews, Select out
skills, abilities, Simulations, unsuitable
Required level characteristics Psychometric tests applicants
of competence Reliability, validity,
Identify utility, fairness Select in (rank)
selection suitable
criteria applicants
Validate selection Evaluate Make
decisions candidate selection
reactions decisions

4. Case Study 1.
Selecting doctors in the UK National
Health Service (NHS)

5. Organisational context
• High profile, strong public interest
• Large applicant numbers - security risks
• Legal scrutiny of selection processes
• Applicants above average intelligence
• Strong professional ‘Trade Union’
• Independent coaching firms - sole purpose is to help
applicants be successful in selection

6. Selection in medicine through the ages… “It isn’t an
interview – just
“ Fill out the an informal
“Work for me, son – I application chat, sweetie.
knew your father.” form for HR Just a
and the job is formality.”
yours, mate.”
1970 1980 1990

7. • 23,000
applicants for
8,000 medical
school places Help!
• 8,000 medical
students apply
for their first
post
• 10,000 speciality
applicants
• 24,000 +
interviews
• Weeks of
offering,
rejecting,
cascading 1000s Consultant hours

8. Key questions
What attributes are important to be an effective
clinician?
What selection methods are available to test these
attributes?
Given the costs, beyond some basic assessment, is a
lottery the best option?

9. Why not use a lottery system?

10. Selecting doctors in the UK
• 8,000 applicants per year for 3,250 training posts in a
centrally coordinated recruitment office
– job analysis
– design of situational judgement tests (SJTs)
– validation studies

11. Job analysis
Three independent studies (Patterson et al, 2000):
• interviews with doctors
• observations studies
• interviews with patients
1) Empathy & sensitivity (sensitive to patient's emotions & feelings)
2) Communication skills (active listening, clarity of explanation)
3) Problem-solving (identifies root cause & decision-making)
4) Professional integrity (respect, vocational enthusiasm)
5) Coping with pressure (calm under pressure)
6) Clinical expertise (clinical process awareness, identifying options)

12. Job Role
“…establish immediate empathic rapport…define nature,
history of problems, the aetiology, patient ideas, concerns,
expectations, effects of problems….consider other continuing
problems/at-risk factors…reach shared understanding of
problems with patient….choose appropriate action for each
problem…use time/resources appropriately” (Pendleton et al, 1984)
• Role requires rapid decision-making in an interpersonal
environment, involving complex & significant cognitive/
non-cognitive demands of the individual

13. Using Situational Judgement Tests
(SJTs) to select doctors

14. What’s a Situational Judgement Test?
• Situational Judgement Tests (SJTs) are a measurement
method designed to assess judgement in work-relevant
situations:
– Present challenging situations likely to be encountered at work
– Make judgements about possible responses
– Scored against expert responses
• Validity evidence is well-established

15. Example SJT item
You are reviewing a routine drug chart for a patient with
rheumatoid arthritis during an overnight shift. You notice that
your consultant has inappropriately prescribed methotrexate
7.5mg daily instead of weekly.
Rank in order the following actions in response to this situation (1= Most
appropriate; 5= Least appropriate)
A Ask the nurses if the consultant has made any other drug errors
recently
B Correct the prescription to 7.5mg weekly
C Leave the prescription unchanged until the consultant ward round the
following morning
D Phone the consultant at home to ask about changing the prescription
E Inform the patient of the error

16. Validation studies
• Cost-effective to develop, administer & score
• Validation studies show good predictive validity
• Favourable applicant reactions
• Number failures reduced from 8% to less than 1%
• Significant savings in human cost (to doctors & patients)
• Significant savings in financial cost

17. Case Study 2.
Selecting Private Bankers

18. Resolving Tensions
between Growing Profit &
being a Trusted Advisor
for Private Bankers at RBS

19. Private Bankers

20. Key questions
What attributes are important to be an effective
private banker?
What are the job role requirements?
What selection methods are available to test these
attributes?

21. Job analysis of Private Bankers
• Provide investment services to customers across a number
of services including investment & portfolio management;
tax, estate planning & stock broking
• Capabilities include:
– Build excellent client relationships & looks after the
client’s long-term interests
– Generating income
– Act as a part of a team
– Follow regulations & manage risk

22. Private Banker Tension / Potential
Capability Dynamics Conflict
GROWING Positive interaction
PROFIT
BEING A TRUSTED BUILDING CLIENT
ADVISOR RELATIONSHIPS
ORGANISATION, TEAM
COMPLIANCE & INVOLVEMENT
MANAGING RISK

23. Using Situational
Judgement Tests (SJTs) to
select bankers

24. Example Situational Judgement Test item
You have a meeting with a potential new client scheduled for later today. His
name is Ioannis, he is 49 years old and a Director of an company called
Computers-R-Us. Computers-R-Us banks with your company and Ioannis
holds personal banking accounts with Barclays (a competitor). Ioannis is
planning to sell his business and is likely to have substantial money to invest
as a result.
Rank the importance of information you wish to elicit during the meeting (1= Most
appropriate and 5 = Least appropriate)
A An understanding of how much income Ioannis has – now and in the future both in
terms of what he earns and spends
B Details of the intended sale of Computers-R-US – when this will take place, how
much it is likely to be sold for, who are the key decision makers etc
C What types of personal banking services Ioannis already uses and any areas
where your bank could enhance this service
D Details about Ioannis’s personal/family situation
E Any interests (business or otherwise) Ioannis has outside of Computers-R-Us.

25. Example Situational Judgement Test item
You have a meeting with a potential new client scheduled for later today. His
name is Ioannis, he is 49 years old and a Director of an company called
Computers-R-Us. Computers-R-Us banks with your company and Ioannis
holds personal banking accounts with Barclays (a competitor). Ioannis is
planning to sell his business and is likely to have substantial money to invest
as a result.
Rank the importance of information you wish to elicit during the meeting (1= Most
appropriate and 5 = Least appropriate)
A An understanding of how much income Ioannis has – now and in the future both in
terms of what he earns and spends
B Details of the intended sale of Computers-R-US – when this will take place,
how much it is likely to be sold for, who are the key decision makers etc
C What types of personal banking services Ioannis already uses and any areas
where your bank could enhance this service
D Details about Ioannis’s personal/family situation
E Any interests (business or otherwise) Ioannis has outside of Computers-R-Us.

26. Implications
• Job analysis results used to explore the dynamics between
capabilities - which may help make better selection decisions
• Understanding how someone manages the tensions in their
role
• Gives applicants a realistic job preview
• Evaluation shows early signs of improved predictive validity &
added value
• Challenge – there is reluctance from some stakeholders to
expose the tensions in the role requirements

27. Implications for policy & practice
• Importance of bespoke job analysis in high stakes selection
• Selection methods in high stakes settings must reflect the
capability dynamics & tensions in the job role
• SJTs useful & valid method for assessing important
professional attributes
• Strong business case for the added value of bespoke
evidence-based measures, although persuading client groups is
often challenging
• Evidence-based approach is the key to stakeholder buy-in
where results are tailored to the organisational context

28. Thank you
fcp27@cam.ac.uk
f.patterson@workpsychologygroup.com

29. Generic competency frameworks
Benefits?
• Defines what is required of all employees and/or leaders
• Seen to act as ‘glue’ that binds people together
• Provides sense of what is important (values, culture)
• Drives consistency in performance management, development, induction,
appraisal, succession planning, etc
Limitations?
• Fails to identify role specific requirements that are different eg. HR
consulting model, motivational qualities associated with call centre tenure
• Comprehensiveness is overwhelming & difficult to use by line managers
• Inhibits use of business specific priorities
• Difficult to refresh – scale of change

30. Designing a new selection system for
Private Bankers
• Move to develop role specific capability frameworks:
– Off-the-shelf competency frameworks may prove invalid and unreliable
without situational customisation (Raelin & Cooledge,1995)
• Focus on core attributes that underpin success
• Describe in term of
– Motivation, ability, interpersonal characteristics
• Simplify to enable change to take place ‘just in time’
• Build for the line, not HR / psychologists
• Provide complexity where needed and to those who need it
• Understand the interplay between capabilities (dynamic model)