Your SlideShare is downloading. ×
AGR CONFERENCE 2013 Time to put graduate selection and testing to the test
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

AGR CONFERENCE 2013 Time to put graduate selection and testing to the test

179

Published on

Published in: Education, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
179
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Time to put graduate selection and testing to the test Are the futures of thousands of young people being consigned to the scrap heap just to make the lives of graduate recruiters easier? AGR Conference: Tuesday 9 July 2013
  • 2. Us Martin Pennington Education, Careers & Employability specialist. AGCAS appointee to Graduate Success Project. Former Head of CAS and member of AGCAS Board. Started in careers1984. Simon Howard Founder of Work, former Director of SHL Group. Managed first Graduate campaign 1980.
  • 3. Nota Bene Thought Leadership Inspiring thought for reflection on the future direction of the industry i.e. we don’t have to provide all the answers (!) Thought Leadership A fact is a fact. But a collection of facts is an opinion.
  • 4. Agenda 1. Newport, we have a problem 2. How we got here 3. The AGR/AGCAS Graduate Success Study points to problems 4. Why using tariff points, the 2:1 and targeting produce the wrong results 5. Why online testing doesn’t produce the right or a fair outcome 6. Why we need to think again
  • 5. Newport: Here’s the problem 300,000 graduates – 23,700 jobs: 92% need not apply The fixation with the Top 100 Employers – budgets blown on 30 campuses, 91 others mostly ignored UCAS tariff point thresholds – not a proxy for ability, just a marker for exclusion The 2:1 bar – not a predictor of job success, but changing campus life for the worse Online testing – not up to the job; its use is neither fair, reliable or valid It adds up to exclusion on a grand scale
  • 6. 2. How we got here
  • 7. “ First-rate men are uncommon, they are like the cream of the cream...there is still the grade A product which for the purpose of this simile represents the average university graduate...the ordinary decent chap, carefully selected because of his high intelligence which is brought out, matured and broadened by university life…but the first rate man cannot be kept down and they were not born equal in personality”. Ewart Escritt, Head of Oxford University Appointments Committee “In praise of the pass man” 1948
  • 8. How we got here • The inheritance problem - the top 10% • The HE system and ‘any degree’ careers • The search for proxies • GRM vs CRM • The internet opened the door… • …and a page cost nothing • …and a test hardly anything A system designed for the elite has morphed into a system for the masses
  • 9. Some lessons from the AGCAS/AGR Graduate Success Project Martin Pennington Consulting Education, careers and employability consultancy
  • 10. AGCAS/AGR Graduate Success Project • To identify the factors influencing successful graduate transition to the graduate job market, to inform the promotion of graduate recruitment good practice, and to raise awareness of the HEAR • To provide students, parents, careers advisers and employers with an in-depth insight into how people from a variety of social and economic backgrounds make the transition to the graduate job market
  • 11. Method Numbers Outcomes Online survey of 2011 and 2012 graduates 1653 responses Analysis and report Online survey of large and small employers 205 responses Analysis and report In-depth telephone interviews with graduates 31 graduates Graduate case studies on website Filmed interviews with graduates 23 graduates Graduate film clips on website Project methodology and outcomes
  • 12. The graduate survey • Background data collected on all graduates • Graduates asked to indicate views on 56 statements about career choice, planning, job seeking etc. • Responses to statements analysed against: Background data: • pre-HE education • type of HE institution • parental HE participation If working: • in a graduate job • in a non-graduate job Current situation: • working and/or studying • not working or studying
  • 13. Identified three factors indicating social/cultural ‘advantage’ 1. Attendance at selective school/college prior to HE (23%) 2. Attendance at Sutton Trust 30 HE institution (37%) 3. Either parent participated in HE (45%) Two groups identified: Advantaged – if any of above factors applied Non-advantaged – if none of above factors applied Graduate groups
  • 14. Background data collected on all employers Employers asked to indicate views on 32 statements about entry requirements, work experience/extra-curricular activities, institutional relationships etc. Responses to statements analysed by size of employer: Large (more than 250 employees) SME (fewer than 250 employees) Employer survey
  • 15. Graduate survey: broad outcomes Employed/further study Younger (21-25) Female White A levels >320 UCAS points 1st or 2:1 Unemployed Older (>25) Male Non-white BTEC/Access <320 UCAS points 2:2 or lower
  • 16. Graduate survey: type of job Graduate job Male Selective education A levels >320 UCAS points ST30 institution 1st or 2:1 Non-graduate job Female Non-selective education BTEC/Access <320 UCAS points Non-ST30 institution 2:2 or lower
  • 17. Employer ratings: institution 0.00 2.00 4.00 6.00 8.00 10.00 12.00 Importance when recruiting graduates Mean Relative importance of different selection criteria Motivation and Interest Fit between applicant and organisation Relevant work experience Skills developed Degree classification Subject knowledge Degree discipline Other work experience Extra-curricular work experience University attended School attended
  • 18. 57% of employers targeted specific institutions Institutional targeting 3 15 3328 20 Reasons for targeting Reduces volume of applicants To attract applicants from local institutions Institutions offer degree disciplines preferred Always get good applicants from targeted institutions Other
  • 19. ‘Due to budgets and resources we can necessarily only target a certain number of universities with regard to careers fair and other campus activity. We nevertheless welcome applicants from any institution.’ ‘We target a select number of institutions which we have had success with previously. We would like to do more of this kind of activity - however, resources restrict us.’ ‘We do target specific universities with attraction strategies but will hire the best applicant for the role whatever university they are from. Work experience is more valuable than a 1st or 2:1 degree.’ ‘Although we have targeted universities, this just means our presence on campus is much more and we hold events etc. With non-targeted universities we still accept and review applications.’ Employers said…
  • 20. 91% of employers said they recruited from any institution regardless of any targeting strategy 91% of employers said that an applicant’s institution would not affect their view of them BUT: 70% of employers said that they took particular notice of students attending careers fairs and other events on campus 64% of employers said that completing an internship with them gave graduates the best chance of a job with them Confusing messages?
  • 21. Advantaged graduates were more likely to: Think their choice of university would be helpful in getting a job after graduation Consider they had been encouraged by their institution to aim for a graduate job Have attended careers fairs and other employer-led events Know that large employers recruit graduates in their final year and know which to apply to Think that most graduate jobs were with large employers Understand what was involved in graduate assessment centres and interviews Graduate attitudes
  • 22. Non-advantaged graduates were more likely to: think that getting a job was the important thing, not necessarily a graduate job decide just to get any job after university rather than hold out for a graduate job Graduate attitudes
  • 23. Employer ratings: degree classification 0.00 2.00 4.00 6.00 8.00 10.00 12.00 Importance when recruiting graduates Mean Relative importance of different selection criteria Motivation and Interest Fit between applicant and organisation Relevant work experience Skills developed Degree classification Subject knowledge Degree discipline Other work experience Extra-curricular work experience University attended School attended
  • 24. 51% of employers set a 2:1 as a minimum entry requirement 75% of employers felt that getting at least a 2:1 should be a priority for university students 29% of employers also stipulated a minimum UCAS points total as a selection criterion BUT: 74% of employers felt that a graduate with a 2:2 and relevant skills and experience was likely to be more successful than one with a 2:1 and limited experience 84% of employers felt that a graduate’s skills and motivation were more important than their degree content Confusing messages?
  • 25. 79% of graduates felt that getting a 2:1 would be more important for their career prospects than other factors 92% of graduates felt that work experience would be important for their career prospects 74% of graduates felt that taking part in extra-curricular activities would be important for their career prospects Many students face difficult choices about using their time Graduate attitudes
  • 26. Advantaged graduates were more likely to: realise that extra-curricular activities would be important for their career prospects and to have been involved in such activities Non-advantaged graduates were more likely to: not have had the time to get involved in extra-curricular activities Graduate attitudes
  • 27. Unemployed graduates and those in non-graduate jobs were more likely not to have had the time to take part in work experience or extra- curricular activities Those in non-graduate jobs were more likely to have prioritised academic work over career planning Those in non-graduate jobs were more likely to have had a part-time job at university Impact on graduate outcomes
  • 28. ‘I had no time to take part in any extra-curricular activities or relevant work experience/ voluntary work due to having to do paid work during the day I wasn’t at uni.’ ‘I only realised that experience was important when I finished university. Universities should do more to provide internships.’ ‘None of my lecturers explained clearly to us the importance of work experience during the course, therefore I focused on my academic work, and now as a graduate I feel that I have missed out on important experience.’ Graduate views
  • 29. Employer ratings: extra-curricular activity 0.00 2.00 4.00 6.00 8.00 10.00 12.00 Importance when recruiting graduates Mean Relative importance of different selection criteria Motivation and Interest Fit between applicant and organisation Relevant work experience Skills developed Degree classification Subject knowledge Degree discipline Other work experience Extra-curricular work experience University attended School attended
  • 30. 54% of employers said that graduates had to have extra- curricular experience to be considered seriously by them 36% of employers felt that students should take unpaid relevant work experience in preference to paid non-relevant work experience Employer attitudes
  • 31. Targeting: can employers have their cake and eat it? 2:1 & UCAS points: crude and arguably counterproductive? Students are influenced by prevailing institutional culture Students pick up mixed messages Students do not have similar advantages… …but they may have similar potential The survey results do not appear to indicate diversity and inclusion Conclusions
  • 32. Graduate recruitment practices which produce the wrong results
  • 33. Graduate recruitment practices that produce the wrong results 1. Campus targeting TOP 30 (T30) Campuses The 30 campuses selected by High Fliers as the most visited by employers – and consequently the 30 campuses polled for the Times Top 100 Employers SCHOOL 63% of all privately educated students attend the T30 23% of all T30 students are privately educated Versus only 7% at all other campuses SOCIAL CLASS 67% of all T30 grads come from the top two social classes Compared to 52% at all other campuses 40% of all students from top two social classes attend T30 ETHNICITY 87% of all T30 graduates are white Only 23% of all non-white students attend a T30 campus Versus 35% of all white students SUMMARY By targeting your expenditure and presence on High Fliers campuses, you are addressing a more middle class, more white and more privately educated audience. Source: RGCC analysis of HESA data
  • 34. SCHOOL 62% of privately educated entrants have 360+ Versus only 37% of state school entrants SUBJECT 360+ INCLUDES 66% of Languages students But EXCLUDES 58% Maths & Computer Science students and EXCLUDES 73% of Business Studies students SOCIAL CLASS 47% of entrants from top two classes gain 360+ Versus 34% of entrants from all other social classes 65% of all 360+ entrants are from top two social classes ETHNICITY 42% of white entrants have 360+ But only 31% of non-white entrants have 360+ 86% of all 360+ entrants are white Graduate recruitment practices that produce the wrong results 2. Tariff points SUMMARY A levels (UCAS tariff points) are a poor predictor of academic achievement or performance in a career and a bad proxy for intelligence. However they are reliable markers of class, ethnicity and education background, as well as favouring languages students at the expense those in Maths and Computer Science. Source: RGCC analysis of HESA data
  • 35. SCHOOL 72% of privately educated students get a 2:1+ 64% of state educated students get a 2:1+ SUBJECT 2:1 bar INCLUDES 77% of Languages graduates and INCLUDES 76% of History & Philosophy graduates but EXCLUDES 42% of Maths, Computer Science & Law SOCIAL CLASS 70% of grads from the top two social classes get 2:1+ 62% of grads getting 2:1+ are from all other social classes 58% of 2:1+ come from top two social classes ETHNICITY 68% of white students get a 2:1+ but only 49% of non-white students get a 2:1+ and only 14% of 2:1+ graduates are non-white Graduate recruitment practices that produce the wrong results 3. Degree class SUMMARY Any employer committed to diversity or social inclusion would be hard pressed to justify an indiscriminate 2:1 bar Source: RGCC analysis of HESA data
  • 36. 88 62 87 140 71 240 300 360 420 480 540 600 CCC BBB AAA AABB AAAA Average entry grades Brunel 2:1s and 1sts Aberdeen 2:1s and 1sts Imperial 2:2s and 3rds Bath 2:2s and 3rds Bristol 2:2s and 3rds Graduate recruitment practices that produce the wrong results Source: RGCC analysis of HESA data
  • 37. Online testing – fair, reliable and valid?
  • 38. Counting in or cutting out? “We have a suite of online tests which will help you reduce applications” “It’s like turning off a tap. We just increase the cut-off to cut down numbers” “Since setting an 80th percentile cut-off, numbers have become a lot more manageable” “We can’t trust the exam system any more, so we use numeracy tests instead” “The past 5 years fighting to get decent A levels and a good degree came down to a 20 minute online test”
  • 39. The issue of cut-offs “Percentiles, being rank order scores, are not suitable for use as cut-offs. Being better than 60% of other people who have taken the test is not the same as scoring 60%.” “Rarely do we see sufficient validation – whether in the choice of test, the selection the norm group or the application of a cut-off score.” “Differences in the raw score lead to only small variations in the percentile at the extremes and small raw score differences in the average band (anywhere between 31%ile and 69%ile) can lead to larger variations in the percentile score. So, for example, the cut-off might be set at 60%ile but someone performing at 50%ile may only have got two more answers incorrect. (which in real, practical terms will not mean very much).” “Too often Numerical Reasoning is used in a manner disproportionate to its relevance to the role”
  • 40. The issue of cut-offs “Where fewer skills are required and more training is available [i.e. Graduate Recruitment], or where group differences have been observed [viz discriminatory pre-selection], lower cut-off scores are more appropriate. ….Where testing is used to screen applicants for a second round of assessment (interviews and assessment centres) much lower cut-offs should be used”. Level A Occupational Testing Course Notes (SHL) “The lower the cut-off is set, … the less disparate impact the test will have if there are score differences between different groups”. Level A Occupational Testing Course Notes (SHL) “Given that we’re talking about screening applicants for graduate roles with abundant training and which require rounded attributes to be successful, good practice suggests that setting a high cut-off score is wrong, and runs a high risk of discrimination.” “The fact that someone has performed at an above average level on a particular ability test tells you more about their performance in that particular skill but not very much about their potential performance on all the other criteria that are important for success. For example, someone might be a numerical genius but not perform very well at other, critical aspects of the role but these are never assessed.”
  • 41. The issue of cheating In a recent survey 72% said that they know of fellow students who have cheated in online tests and of them 40% quoted specific instances. “Everyone I know gets their parents or someone good at Maths to do the test for them” “I have many friends who went to assessment centres and met people who had help with the tests” “People cheat on the Arithmetic tests as they they can be quite difficult” “I know of people who have had friends take the numerical aptitude tests for them” “I get paid to do other people's numerical and verbal tests” “These tests were never designed to be used in an unsupervised manner as an absolute disqualifier. Employers should always test and re-test.”
  • 42. The issue of SJTs “A measure designed to test non-cognitive professional attributes. In making judgements about response options, candidates need to apply relevant knowledge, skills and abilities to resolve the issues they are presented with” Koczwara, Patterson et al 2012 “Using SJTs will favour those candidates with experience and exposure to professional environments and behaviour. As a first stage screening tool candidates who have potential but little or no experience will miss out”. “It’s difficult to believe that an SJT can be sufficiently reliable to be used as an absolute disqualifier, let alone validated for a range of graduate schemes”. (Me!)
  • 43. All tests from reputable publishers are reliable in themselves. However: When and how they are used should be reviewed Cut-offs are inappropriate and often too high Norm groups not always appropriate to a ‘45% graduate population’ Consistent levels of cheating disadvantage other candidates High levels of anxiety SJTs favour candidates from professional backgrounds ..and not all test are from reputable publishers. Online testing – fair, reliable and valid? Objective assessment is fundamental to successful graduate recruitment. But it must be used fairly and objectively.
  • 44. We need to think again Constantly erecting more hurdles to entry is unfair, lazy and doesn’t work (unless we revert to unalloyed elitism) (and ditch all commitments to diversity and inclusion). Start again. Review the whole graduate recruitment process. Ditch the milkround. Ask ourselves what do we need to know, when? Embrace multiple entry points. Ditch the 2:1 bar, UCAS tariff points and targeting just to get votes in the Top 100. Work harder and be honest. If an undergraduate commits four hours of their life submitting an application, then spend some time reading it. Would you tell your Chairman that you couldn’t be bothered reading their Grand-daughter’s/Grandson’s application? Create a fairer test for all. A universal test might just be a start to levelling the playing field. A system designed for the elite has morphed into a system for the masses. It must now change.
  • 45. AGCAS Graduate Success report: www.agcas.org.uk/assets/1519-Graduate-Success-Project- downloads Graduate Success website: www.graduate-success.org.uk Further information
  • 46. SCHOOL 63% of all privately educated students attend the T30 23% of all T30 students are privately educated Versus only 7% at all other campuses TOP 30 (T30) Campuses The 30 campuses selected by High Fliers as the most visited by employers – and consequently the 30 campuses polled for the Times Top 100 Employers SOCIAL CLASS of all T30 grads come from the top two social classes Compared to 52% at all other campuses all students from top two social classes attend T30 ETHNICITY 87% of all T30 graduates are white Only 23% of all non-white students attend a T30 campus Versus 35% of all white students Graduate recruitment practices that produce the wrong results 1. Campus targeting SUMMARY By targeting your expenditure and presence on High Fliers campuses, you are addressing a more middle class, more white and more privately educated
  • 47. SCHOOL of privately educated entrants have 360+ rsus only 37% of state school entrants SUBJECT 360+ INCLUDES 66% of Languages students But EXCLUDES 58% Maths & Computer Science studen and EXCLUDES 73% of Business Studies studentsSOCIAL CLASS entrants from top two classes gain 360+ % of entrants from all other social classes 60+ entrants are from top two social classes ETHNICITY 42% of white entrants have 360+ But only 31% of non-white entrants have 360+ 86% of all 360+ entrants are white Graduate recruitment practices that produce the wrong results 2. Tariff points SUMMARY A levels (UCAS tariff points) are a poor predictor of academic achievement or performance in a career. However they are reliable markers of class, ethnicity and education background, as well as favouring languages students at the expense those in Maths and Computer Science.
  • 48. SCHOOL 72% of privately educated students get a 2:1+ 64% of state educated students get a 2:1+ SUBJECT 2:1 bar INCLUDES 77% of Languages and INCLUDES 76% of History & Philosop but EXCLUDES 42% of Maths, Computer S SOCIAL CLASS grads from the top two social classes get 2:1+ of grads getting 2:1+ are from all other social classes % of 2:1+ come from top two social classes ETHNICITY 68% of white students get a 2:1+ but only 49% of non-white students get a 2:1+ and only 14% of 2:1+ graduates are non-white Graduate recruitment practices that produce the wrong results 3. Degree class SUMMARY Any employer committed to diversity or social inclusion would be hard pressed to justify an indiscriminate 2:1 bar

×