AGR CONFERENCE 2013 Time to put graduate selection and testing to the test

Time to put graduate selection and
testing to the test
Are the futures of thousands of young people being consigned
to the scrap heap just to make the lives of graduate recruiters
easier?
AGR Conference: Tuesday 9 July 2013

Us
Martin Pennington
Education, Careers & Employability specialist. AGCAS appointee to Graduate Success
Project. Former Head of CAS and member of AGCAS Board. Started in careers1984.
Simon Howard
Founder of Work, former Director of SHL Group. Managed first Graduate campaign 1980.

Nota Bene
Thought Leadership
Inspiring thought for reflection on the future direction of the industry
i.e. we don’t have to provide all the answers (!)
Thought Leadership
A fact is a fact. But a collection of facts is an opinion.

Agenda
1. Newport, we have a problem
2. How we got here
3. The AGR/AGCAS Graduate Success Study points to problems
4. Why using tariff points, the 2:1 and targeting produce the wrong results
5. Why online testing doesn’t produce the right or a fair outcome
6. Why we need to think again

Newport: Here’s the problem
300,000 graduates – 23,700 jobs: 92% need not apply
The fixation with the Top 100 Employers – budgets blown on 30 campuses, 91 others mostly
ignored
UCAS tariff point thresholds – not a proxy for ability, just a marker for exclusion
The 2:1 bar – not a predictor of job success, but changing campus life for the worse
Online testing – not up to the job; its use is neither fair, reliable or valid
It adds up to exclusion on a grand scale

“ First-rate men are uncommon, they are like the cream of the
cream...there is still the grade A product which for the purpose of this
simile represents the average university graduate...the ordinary
decent chap, carefully selected because of his high intelligence which
is brought out, matured and broadened by university life…but the first
rate man cannot be kept down and they were not born equal in
personality”.
Ewart Escritt, Head of Oxford University Appointments Committee
“In praise of the pass man” 1948

How we got here
• The inheritance problem - the top 10%
• The HE system and ‘any degree’ careers
• The search for proxies
• GRM vs CRM
• The internet opened the door…
• …and a page cost nothing
• …and a test hardly anything
A system designed for the elite has morphed
into a system for the masses

Some lessons from the AGCAS/AGR
Graduate Success Project
Martin Pennington Consulting
Education, careers and employability consultancy

AGCAS/AGR Graduate Success
Project
• To identify the factors influencing successful graduate transition to the graduate job
market, to inform the promotion of graduate recruitment good practice, and to raise
awareness of the HEAR
• To provide students, parents, careers advisers and employers with an in-depth
insight into how people from a variety of social and economic backgrounds make the
transition to the graduate job market

Method Numbers Outcomes
Online survey of 2011 and
2012 graduates
1653 responses Analysis and report
Online survey of large and
small employers
205 responses Analysis and report
In-depth telephone
interviews with graduates
31 graduates Graduate case studies on
website
Filmed interviews with
graduates
23 graduates Graduate film clips on
website
Project methodology and outcomes

The graduate survey
• Background data collected on all graduates
• Graduates asked to indicate views on 56 statements about career
choice, planning, job seeking etc.
• Responses to statements analysed against:
Background data:
• pre-HE education
• type of HE
institution
• parental HE
participation
If working:
• in a graduate job
• in a non-graduate
job
Current situation:
• working and/or
studying
• not working or
studying

Identified three factors indicating social/cultural ‘advantage’
1. Attendance at selective school/college prior to HE (23%)
2. Attendance at Sutton Trust 30 HE institution (37%)
3. Either parent participated in HE (45%)
Two groups identified:
Advantaged – if any of above factors applied
Non-advantaged – if none of above factors applied
Graduate groups

Background data collected on all employers
Employers asked to indicate views on 32 statements about entry requirements,
work experience/extra-curricular activities, institutional relationships etc.
Responses to statements analysed by size of employer:
Large (more than 250 employees)
SME (fewer than 250 employees)
Employer survey

Graduate survey: broad outcomes
Employed/further study
Younger (21-25)
Female
White
A levels
>320 UCAS points
1st or 2:1
Unemployed
Older (>25)
Male
Non-white
BTEC/Access
<320 UCAS points
2:2 or lower

Graduate survey: type of job
Graduate job
Male
Selective education
A levels
>320 UCAS points
ST30 institution
1st or 2:1
Non-graduate job
Female
Non-selective education
BTEC/Access
<320 UCAS points
Non-ST30 institution
2:2 or lower

Employer ratings: institution
0.00
2.00
4.00
6.00
8.00
10.00
12.00
Importance when recruiting graduates
Mean
Relative importance of different selection criteria Motivation and Interest
Fit between applicant and
organisation
Relevant work experience
Skills developed
Degree classification
Subject knowledge
Degree discipline
Other work experience
Extra-curricular work
experience
University attended
School attended

57% of employers targeted specific institutions
Institutional targeting
3
15
3328
20
Reasons for targeting Reduces volume of
applicants
To attract
applicants from
local institutions
Institutions offer
degree disciplines
preferred
Always get good
applicants from
targeted institutions
Other

‘Due to budgets and resources we can necessarily only target a certain number
of universities with regard to careers fair and other campus activity. We
nevertheless welcome applicants from any institution.’
‘We target a select number of institutions which we have had success with
previously. We would like to do more of this kind of activity - however,
resources restrict us.’
‘We do target specific universities with attraction strategies but will hire the best
applicant for the role whatever university they are from. Work experience is
more valuable than a 1st or 2:1 degree.’
‘Although we have targeted universities, this just means our presence on
campus is much more and we hold events etc. With non-targeted universities
we still accept and review applications.’
Employers said…

91% of employers said they recruited from any institution
regardless of any targeting strategy
91% of employers said that an applicant’s institution would not
affect their view of them
BUT:
70% of employers said that they took particular notice of
students attending careers fairs and other events on campus
64% of employers said that completing an internship with them
gave graduates the best chance of a job with them
Confusing messages?

Advantaged graduates were more likely to:
Think their choice of university would be helpful in getting a job after
graduation
Consider they had been encouraged by their institution to aim for a
graduate job
Have attended careers fairs and other employer-led events
Know that large employers recruit graduates in their final year and know
which to apply to
Think that most graduate jobs were with large employers
Understand what was involved in graduate assessment centres and
interviews
Graduate attitudes

Non-advantaged graduates were more likely to:
think that getting a job was the important thing, not necessarily a
graduate job
decide just to get any job after university rather than hold out for a
graduate job
Graduate attitudes

Employer ratings: degree classification
0.00
2.00
4.00
6.00
8.00
10.00
12.00
Mean
organisation
Skills developed
Subject knowledge
Degree discipline
experience
University attended
School attended

51% of employers set a 2:1 as a minimum entry requirement
75% of employers felt that getting at least a 2:1 should be a priority for
university students
29% of employers also stipulated a minimum UCAS points total as a selection
criterion
BUT:
74% of employers felt that a graduate with a 2:2 and relevant skills and
experience was likely to be more successful than one with a 2:1 and limited
experience
84% of employers felt that a graduate’s skills and motivation were more
important than their degree content
Confusing messages?

79% of graduates felt that getting a 2:1 would be more important for their
career prospects than other factors
92% of graduates felt that work experience would be important for their
career prospects
74% of graduates felt that taking part in extra-curricular activities would be
important for their career prospects
Many students face difficult choices about using their time
Graduate attitudes

Advantaged graduates were more likely to:
realise that extra-curricular activities would be important for
their career prospects and to have been involved in such
activities
Non-advantaged graduates were more likely to:
not have had the time to get involved in extra-curricular
activities
Graduate attitudes

Unemployed graduates and those in non-graduate jobs were more
likely not to have had the time to take part in work experience or extra-
curricular activities
Those in non-graduate jobs were more likely to have prioritised
academic work over career planning
Those in non-graduate jobs were more likely to have had a part-time
job at university
Impact on graduate outcomes

‘I had no time to take part in any extra-curricular activities or relevant
work experience/ voluntary work due to having to do paid work during
the day I wasn’t at uni.’
‘I only realised that experience was important when I finished
university. Universities should do more to provide internships.’
‘None of my lecturers explained clearly to us the importance of work
experience during the course, therefore I focused on my academic
work, and now as a graduate I feel that I have missed out on important
experience.’
Graduate views

Employer ratings: extra-curricular activity
0.00
2.00
4.00
6.00
8.00
10.00
12.00
Mean
organisation
Skills developed
Subject knowledge
Degree discipline
experience
University attended
School attended

54% of employers said that graduates had to have extra-
curricular experience to be considered seriously by them
36% of employers felt that students should take unpaid
relevant work experience in preference to paid non-relevant
work experience
Employer attitudes

Targeting: can employers have their cake and eat it?
2:1 & UCAS points: crude and arguably counterproductive?
Students are influenced by prevailing institutional culture
Students pick up mixed messages
Students do not have similar advantages…
…but they may have similar potential
The survey results do not appear to indicate diversity and
inclusion
Conclusions

Graduate recruitment practices which
produce the wrong results

Graduate recruitment practices that
1. Campus
targeting
TOP 30 (T30) Campuses
The 30 campuses selected by High Fliers as the most
visited by employers – and consequently the 30
campuses polled for the Times Top 100 Employers
SCHOOL
63% of all privately educated students attend the T30
23% of all T30 students are privately educated
Versus only 7% at all other campuses
SOCIAL CLASS
67% of all T30 grads come from the top two social classes
Compared to 52% at all other campuses
40% of all students from top two social classes attend T30
ETHNICITY
87% of all T30 graduates are white
Only 23% of all non-white students attend a T30 campus
Versus 35% of all white students
SUMMARY
By targeting your expenditure and presence on High Fliers campuses, you are
addressing a more middle class, more white and more privately educated
audience.
Source: RGCC analysis of HESA data

SCHOOL
62% of privately educated entrants have 360+
Versus only 37% of state school entrants
SUBJECT
360+ INCLUDES 66% of Languages students
But EXCLUDES 58% Maths & Computer Science students
and EXCLUDES 73% of Business Studies students
SOCIAL CLASS
47% of entrants from top two classes gain 360+
Versus 34% of entrants from all other social classes
65% of all 360+ entrants are from top two social classes
ETHNICITY
42% of white entrants have 360+
But only 31% of non-white entrants have 360+
86% of all 360+ entrants are white
2. Tariff points
SUMMARY
A levels (UCAS tariff points) are a poor predictor of academic achievement or
performance in a career and a bad proxy for intelligence. However they are
reliable markers of class, ethnicity and education background, as well as
favouring languages students at the expense those in Maths and Computer
Science.

SCHOOL
72% of privately educated students get a 2:1+
64% of state educated students get a 2:1+
SUBJECT
2:1 bar INCLUDES 77% of Languages graduates
and INCLUDES 76% of History & Philosophy graduates
but EXCLUDES 42% of Maths, Computer Science & Law
SOCIAL CLASS
70% of grads from the top two social classes get 2:1+
62% of grads getting 2:1+ are from all other social classes
58% of 2:1+ come from top two social classes
ETHNICITY
68% of white students get a 2:1+
but only 49% of non-white students get a 2:1+
and only 14% of 2:1+ graduates are non-white
3. Degree class
SUMMARY
Any employer committed to diversity or social inclusion would be hard pressed
to justify an indiscriminate 2:1 bar

88
62
87
140
71
240
300
360
420
480
540
600
CCC
BBB
AAA
AABB
AAAA
Average
entry grades
Brunel 2:1s and
1sts
Aberdeen 2:1s and
1sts
Imperial 2:2s and
3rds
Bath 2:2s and
3rds
Bristol 2:2s and
3rds

Online testing – fair, reliable and
valid?

Counting in or cutting out?
“We have a suite of
online tests which will
help you reduce
applications”
“It’s like turning off a
tap. We just increase
the cut-off to cut down
numbers”
“Since setting an 80th
percentile cut-off,
numbers have
become a lot more
manageable”
“We can’t trust the
exam system any
more, so we use
numeracy tests instead”
“The past 5 years fighting
to get decent A levels and
a good degree came down
to a 20 minute online test”

The issue of cut-offs
“Percentiles, being rank order scores, are not suitable for use as cut-offs. Being better
than 60% of other people who have taken the test is not the same as scoring 60%.”
“Rarely do we see sufficient validation – whether in the choice of test, the selection the
norm group or the application of a cut-off score.”
“Differences in the raw score lead to only small variations in the percentile at the extremes
and small raw score differences in the average band (anywhere between 31%ile and
69%ile) can lead to larger variations in the percentile score. So, for example, the cut-off
might be set at 60%ile but someone performing at 50%ile may only have got two more
answers incorrect. (which in real, practical terms will not mean very much).”
“Too often Numerical Reasoning is used in a manner disproportionate to its relevance to
the role”

The issue of cut-offs
“Where fewer skills are required and more training is available [i.e. Graduate Recruitment],
or where group differences have been observed [viz discriminatory pre-selection], lower
cut-off scores are more appropriate. ….Where testing is used to screen applicants for a
second round of assessment (interviews and assessment centres) much lower cut-offs
should be used”. Level A Occupational Testing Course Notes (SHL)
“The lower the cut-off is set, … the less disparate impact the test will have if there are
score differences between different groups”. Level A Occupational Testing Course Notes
(SHL)
“Given that we’re talking about screening applicants for graduate roles with abundant
training and which require rounded attributes to be successful, good practice suggests
that setting a high cut-off score is wrong, and runs a high risk of discrimination.”
“The fact that someone has performed at an above average level on a particular ability
test tells you more about their performance in that particular skill but not very much about
their potential performance on all the other criteria that are important for success. For
example, someone might be a numerical genius but not perform very well at other, critical
aspects of the role but these are never assessed.”

The issue of cheating
In a recent survey 72% said that they know of fellow students who have cheated in online
tests and of them 40% quoted specific instances.
“Everyone I know gets
their parents or someone
good at Maths to do the
test for them”
“I have many friends who
went to assessment
centres and met people
who had help with the
tests”
“People cheat on the
Arithmetic tests as they
they can be quite difficult”
“I know of people who
have had friends take the
numerical aptitude tests
for them”
“I get paid to do other
people's numerical and
verbal tests”
“These tests were never designed to
be used in an unsupervised manner
as an absolute disqualifier.
Employers should always test and
re-test.”

The issue of SJTs
“A measure designed to test non-cognitive professional attributes. In making judgements
about response options, candidates need to apply relevant knowledge, skills and abilities
to resolve the issues they are presented with”
Koczwara, Patterson et al 2012
“Using SJTs will favour those candidates with experience and exposure to professional
environments and behaviour. As a first stage screening tool candidates who have
potential but little or no experience will miss out”.
“It’s difficult to believe that an SJT can be sufficiently reliable to be used as an absolute
disqualifier, let alone validated for a range of graduate schemes”. (Me!)

All tests from reputable publishers are reliable in themselves. However:
When and how they are used should be reviewed
Cut-offs are inappropriate and often too high
Norm groups not always appropriate to a ‘45% graduate population’
Consistent levels of cheating disadvantage other candidates
High levels of anxiety
SJTs favour candidates from professional backgrounds
..and not all test are from reputable publishers.
Online testing – fair, reliable and
valid?
Objective assessment is fundamental to
successful graduate recruitment. But it must be
used fairly and objectively.

We need to think again
Constantly erecting more hurdles to entry is unfair, lazy and doesn’t work (unless we
revert to unalloyed elitism) (and ditch all commitments to diversity and inclusion).
Start again. Review the whole graduate recruitment process. Ditch the milkround. Ask
ourselves what do we need to know, when? Embrace multiple entry points. Ditch the 2:1
bar, UCAS tariff points and targeting just to get votes in the Top 100.
Work harder and be honest. If an undergraduate commits four hours of their life
submitting an application, then spend some time reading it. Would you tell your Chairman
that you couldn’t be bothered reading their Grand-daughter’s/Grandson’s application?
Create a fairer test for all. A universal test might just be a start to levelling the playing
field.
A system designed for the elite has morphed
into a system for the masses. It must now
change.

AGCAS Graduate Success report:
www.agcas.org.uk/assets/1519-Graduate-Success-Project-
downloads
Graduate Success website:
www.graduate-success.org.uk
Further information

SCHOOL
63% of all privately educated students attend the T30
23% of all T30 students are privately educated
Versus only 7% at all other campuses
TOP 30 (T30) Campuses
The 30 campuses selected by High Fliers as the most
visited by employers – and consequently the 30
campuses polled for the Times Top 100 Employers
SOCIAL CLASS
of all T30 grads come from the top two social
classes
Compared to 52% at all other campuses
all students from top two social classes attend
T30
ETHNICITY
87% of all T30 graduates are white
Only 23% of all non-white students attend a T30 campus
Versus 35% of all white students
produce the wrong results 1. Campus
targeting
SUMMARY
By targeting your expenditure and presence on High Fliers campuses, you are
addressing a more middle class, more white and more privately educated

SCHOOL
of privately educated entrants have 360+
rsus only 37% of state school entrants SUBJECT
360+ INCLUDES 66% of Languages students
But EXCLUDES 58% Maths & Computer Science studen
and EXCLUDES 73% of Business Studies studentsSOCIAL CLASS
entrants from top two classes gain 360+
% of entrants from all other social classes
60+ entrants are from top two social classes
ETHNICITY
42% of white entrants have 360+
But only 31% of non-white entrants have 360+
86% of all 360+ entrants are white
produce the wrong results 2. Tariff points
SUMMARY
A levels (UCAS tariff points) are a poor predictor of academic achievement or
performance in a career. However they are reliable markers of class, ethnicity
and education background, as well as favouring languages students at the
expense those in Maths and Computer Science.

SCHOOL
72% of privately educated students get a 2:1+
64% of state educated students get a 2:1+
SUBJECT
2:1 bar INCLUDES 77% of Languages
and INCLUDES 76% of History & Philosop
but EXCLUDES 42% of Maths, Computer S
SOCIAL CLASS
grads from the top two social classes get 2:1+
of grads getting 2:1+ are from all other social
classes
% of 2:1+ come from top two social classes
ETHNICITY
68% of white students get a 2:1+
but only 49% of non-white students get a 2:1+
and only 14% of 2:1+ graduates are non-white
produce the wrong results 3. Degree class
SUMMARY
Any employer committed to diversity or social inclusion would be hard pressed
to justify an indiscriminate 2:1 bar

AGR CONFERENCE 2013 Time to put graduate selection and testing to the test

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (6)

Similar to AGR CONFERENCE 2013 Time to put graduate selection and testing to the test

Similar to AGR CONFERENCE 2013 Time to put graduate selection and testing to the test (20)

More from EmmaAGR

More from EmmaAGR (20)

Recently uploaded

Recently uploaded (20)

AGR CONFERENCE 2013 Time to put graduate selection and testing to the test