Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence” Afternoon Presentations

224 views

Published on

With the Teaching Excellence Framework being implemented across England, a lot of higher education institutions have started to ask questions about what it means to be “excellent” in teaching. In particular, with the rich and complex data that all educational institutions gather that could potentially capture learning gains, what do we actually know about our students’ learning journeys? What kinds of data could be used to infer whether our students are actually making affective (e.g., motivation), behavioural (e.g., engagement), and/or cognitive learning gains? Please join us on 22 January 2018 in lovely Milton Keynes at a free OU- and HEFCE-supported event on Using data to increase learning gains and teaching excellence.

14.00-15.00 Measuring learning gains with (psychometric) questionnaires

Dr Sonia Ilie, Prof Jan Vermunt, Prof Anna Vignoles (University of Cambridge, UK): Learning gain: from concept to measurement
Dr Fabio Arico (University of East Anglia): Learning Gain and Confidence Gain Through Peer-instruction: the role of pedagogical design
Dr Paul Mcdermott & Dr Robert Jenkins (University of East Anglia): A Methodology that Makes Self-Assessment an Implicit Part of the Answering Process
15.00-15.45 Measuring employability learning gains

Dr Heike Behle (University of Warwick): Measuring employability gain in Higher Education. A case study using R2 Strengths
Fiona Cobb, Dr Bob Gilworth, David Winter (University of London): Careers Registration Learning Gain project

Published in: Education
  • Be the first to comment

  • Be the first to like this

22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence” Afternoon Presentations

  1. 1. 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence”. https://twitter.com/LearningGains #learninggainsOU https://abclearninggains.com/
  2. 2. Presentations in the afternoon Sonia Ilie University of Cambridge Heike Behle Warwick University Paul Mcdermott University of East Anglia Fabio Arico University of East Anglia Fiona Cobb University of London
  3. 3. Learning gain: from concept to measurement LEGACY – Cambridge Strand – Jan Vermunt, Anna Vignoles & Sonia Ilie
  4. 4. LEGACY aims Develop a context-appropriate theoretical understanding of learning gain Develop and test an instrument to measure learning gain: reliability, validity and at-scale usability Test a longitudinal model of learning gain in relation to student background characteristics, contextual factors, and existing measures of academic success
  5. 5. Learning Gain Complex concept Multidimensional A wide variety of definitions Requires theoretical grounding Difficult to measure Student engagement Data quality
  6. 6. Conceptual framework Developed based on: Qualitative student interviews Review of the theoretical literature Review of available measurement tools Focus on: Comprehensiveness, and Practicality
  7. 7. Developing measures Twelve measurement instruments used in the survey newly-developed, from qualitative work adapted from other sources unchanged from other sources Piloting before full administration
  8. 8. Developing measures Piloting: main problem: length Reliability: overall good, but issues already identified: 11 of the 12 measures adequately reliable 1 measure: epistemological beliefs: low reliability additional scale used for Round 2 Administration of measures: online
  9. 9. Longitudinal design Round 1 6,275 Round 2 3,189 Round 3 … Data HESA matching Oct 2016 June 2017 Feb-Mar 2018
  10. 10. Example #1 Relating and Structuring (from ILS, Vermunt & Vermetten, 2004, with minor adaptations) Item M(SD) Factor loading 1 I try to construct an overall picture of a course for myself 3.06(1.22 ) 0.47 2 I compare the conclusions drawn in different academic sources. 3.03(1.17 ) 0.61 3 I try to see the connection between the topics discussed in different academic subjects 3.25(1.08 ) 0.75 4 I try to discover similarities and differences between theories. 3.04(1.11 ) 0.70 5 I try to combine subjects that are dealt with separately into a whole. 3.18(1.16 ) 0.56 6 I relate specific facts to the main issue in a chapter or article. 3.16(1.08 0.62
  11. 11. Example #3 Reasoning ability (individual items from ICAR, 2015, own assembly) Item % Correct Factor loading Item % Correct Factor loading 1 Rotation 1 57.95% 0.57 7 Letters and Numbers 1 52.36% 0.47 2 Rotation 2 46.19% 0.49 8 Letters and Numbers 2 65.60% 0.47 3 Rotation 3 34.84% 0.51 9 Letters and Numbers 3 67.50% 0.53 4 Verbal 1 88.21% 0.46 10 Matrix 1 71.79% 0.46 5 Verbal 2 61.92% 0.36 11 Matrix 2 72.69% 0.37 6 Verbal 3 80.82% 0.47 12 Matrix 3 68.99% 0.45 Final model fit: RMSEA=0.043, CFI=0.944, SRMR=0.028, TLI=0.929, α=0.769
  12. 12. Lessons learned Theoretical framework essential Using qualitative work to inform scale development is useful Adapting measures tricky, and not always successful Measurement quality an issue: e.g. epistemology Student participation: important, but not trivial to achieve Quality of measurement scales fundamental to assessing change
  13. 13. Looking ahead: further analysis Students’ self-regulatory behaviours, learning patterns, engagement levels, attitudes towards research, and other abilities and attitudes may change over time learning gain
  14. 14. Thank you. Contact: learning-gain@educ.cam.ac.uk
  15. 15. Learning Gain and Confidence Gain Through Peer-instruction: the role of pedagogical design Fabio R. Aricò @FabioArico Open University Jan 2018
  16. 16. YOUR PRESENTER Fabio Aricò Senior Lecturer in Macroeconomics National Teaching Fellow 2017 School of Economics – University of East Anglia, UK Research fields • Higher Education policy and practice (widen. access, satisfaction) • Technology Enhanced Learning • Self-Assessment and Academic Self-Efficacy Twitter: @FabioArico 20
  17. 17. ACKNOWLEDGEMENTS HEFCE Piloting and Evaluating Measures of Learning Gain UEA Students, Alumni, and Research Assistants HEA – Teaching Development Grant Scheme 21
  18. 18. OUTLINE Part 1 Introduction to main concepts • Peer-instruction, Self-efficacy & Self-assessment, Learning Gain. Part 2 Description of my active learning pedagogy Introduction to my research questions • Operationalising learning/confidence gain Part 3 Empirical methodology and results • Regression outputs and discussion. 22
  19. 19. 1. Introduction to main concepts 23
  20. 20. FLIPPED CLASS and PEER-INSTRUCTION • Flipped classroom & Peer-Instruction  pre-reading + student interaction  Mazur (1997)  Henderson and Dancy (2009)  well-developed research in Physics and STEM. • Learning analytics for Peer-Instruction  Learning gain: Mazour Group - Bates & Galloway (2012)  Student satisfaction: Hernandez Nanclares & Cerezo Menendez (2014). • There is not much literature on the links with self-assessment skills Open field, with many unanswered questions  e.g. role of demographics, language, previous background  Pedagogically: self-assessment blends with flipping and Peer-instruction. 24
  21. 21. SELF-EFFICACY and SELF-ASSESSMENT Academic Self-Efficacy = confidence at performing academic tasks and/or attaining academic goals. Bandura (1977) 1. Mastery of experiences 2. Vicarious experiences 3. Verbal persuasion 4. Environment and settings See also: Pajares (1996) and Ritchie (2015). Idea: Students should develop their self-efficacy to master their learning experience. Measure learning gain along with increased self-efficacy: ‘confidence gain’. 25
  22. 22. FRAMEWORK 26 Active Learning Peer-instruction Self-Assessment Learning Gain Confidence Gain TEL learning analytics
  23. 23. 2. A description of my active learning pedagogy Research questions 27
  24. 24. ACTIVE LEARNING ENVIRONMENT Introductory Macroeconomics (2015-16 & 2016-17) • year-long module (compulsory 1st year) • 250 students (about 250 over past 2 years) • 22 lectures (2hrs per week) • 8 seminars (every second week) • 8 workshops (every second week) Students endowed with individual Audience Response Systems (clickers)  continuous data collection facilitated by technology;  comprehensive ethical approval obtained beforehand. 28
  25. 25. WORKSHOPS – teaching algorithm 29 Round 1 - formative question - 4 choices - no information - no answer Self-Assessment 1 - confidence question - 4 level Likert-scale - information shared Peer-Instruction - students talk - compare answers - explain each other Round 2 - formative question - Identical to R1 - information shared - correct answer Self-Assessment 2 - confidence question - 4 level Likert-scale - information shared
  26. 26. RESEARCH QUESTIONS 30 1. Is the pedagogy developing good self-assessment skills? Are students self-assessing correctly over Round 1/2? 2. Is peer-instruction able to generate learning/confidence gain? How does gain relate to initial knowledge/confidence (Round 1)? 3. Is learning gain associated to a confidence gain? Is this association affected by the structure of the teaching algorithm? 2016 Vicarious of Experience Scenario (VES) only 2017 Mastery of Experience Scenario (MES) contrasted with VES (4 sessions each).
  27. 27. WORKSHOPS – contrast 2 teaching algorithms 31 Round 1 - formative question - 4 choices - no information - no answer Self-Assessment 1 - confidence question - 4 level Likert-scale - information shared Peer-Instruction - students talk - compare answers - explain each other Round 2 - formative question - Identical to R1 - information shared - correct answer Self-Assessment 2 - confidence question - 4 level Likert-scale - information shared VES MES
  28. 28. 3. Empirical methodology and Results 32
  29. 29. DATASETS AND CODING Student Q1 Q2 Q3 …  1 0 1 1 2 1 0 0 3 1 1 … …  performance per question confidence by question performanceperstudent confidencebystudent 33 Formative questions 1 = correct 0 = incorrect Confidence questions 1 = strongly/agree 0 = strongly/disagree
  30. 30. OPERATIONALISING TWO GAINS For each 1st and 2nd response to formative assessment questions: % correct R2  % correct R1 Normalised Learning Gain (NLG) = 100%  % correct R1 For each 1st and 2nd response to self-assessment questions: % confident R2  % confident R1 Normalised Confidence Gain (NCG) = 100%  % confident R1 34
  31. 31. CHANGE IN CONFIDENCE: rough descriptives 35 0% 20% 40% 60% 80% 100% 1 3 5 7 2 4 6 8 Week 2016: only VES C.score K.score 0% 20% 40% 60% 80% 100% 1 3 5 7 2 4 6 8 Week 2017: VES & MES C.score K.score 1st 2nd MESVES Average confidence levels per session  Confidence gain is always positive  Confidence gain higher with VES (vicarious) and lower with MES (mastery)
  32. 32. EMPIRICAL METHODOLOGY Regression analysis at class-level N=140 (2016) and N=136 (2017) Dependent variables: %confident responses, NLG, NCG Independent variables: %correct responses, NLG Workshop Group dummy for 2 Workshop Session dummy for 8 VES/MES Scenario dummy for 2 Functional form: checking for polynomial forms Robustness checks: Robust-regression with White-correction (ML estimation of standard errors) 36
  33. 33. 37 % confident Round 1 % correct Round 1 RESULT – Self-assessment Skills Round 1 2016 VES =0.429*** R²=0.56 0.33 0.44 2017 VES =0.367*** R²=0.59 2017 MES =0.09***
  34. 34. 38 % confident Round 2 % correct Round 2 RESULT – Self-assessment Skills Round 2 2016 VES =0.282*** R²=0.26 0.57 0.60 2017 VES =0.322*** R²=0.64 2017 MES =1.44***
  35. 35. RESULT – Normalised Gains in 2016 39 NLG % Correct/Confident in Round 1 2016 0.37 0.48 0.85 0.52 NCG NLG R²=0.39 NCG R²=0.19
  36. 36. RESULT – Normalised Gains in 2017 40 NLG % Correct/Confident in Round 1 0.40 0.42 0.67 NCG NLG R²=0.21 NCG R²=0.42 =0.132**
  37. 37. RESULT – Learning Gain & Confidence Gain 41 NCG NLG 2016 R²=0.16 MES =0.125** -1 0.31 0.28 2017 VES =0.19*** R²=0.39
  38. 38. SUMMARY of RESULTS • In both Round 1&2, confidence levels are lower under MES. However, performance and confidence are consistently positively correlated  students self-assess correctly, irrespectively of MES/VES scenario. • Peer-instruction generates higher learning/confidence gain when knowledge/confidence in the classroom is neither too high or too low  ‘sweet-spot’ pattern does not depend on MES/VES scenario. • Under VES scenario confidence gain is lower, compared to MES scenario. However, confidence gain is consistently positively correlated to learning gain  students develop more confidence as they learn, irrespectively of MES/VES scenario. The pedagogy appears to be robust to teacher intervention  Active Learning! 42
  39. 39. 43 F.Arico@uea.ac.uk @FabioArico “Promoting Active Learning Through Peer-Instruction and Self- Assessment: A Toolkit to Design, Support and Evaluate Teaching”, Educational Developments, SEDA, 17.1, 15-18. STAY IN TOUCH!
  40. 40. Learning Gain and Confidence Gain Through Peer-instruction: the role of pedagogical design Fabio R. Aricò @FabioArico Open University Jan 2018
  41. 41. 22/01/2018 45 Self-Efficacy “Confidence is the Pivot to Success” Techsavvywomen.net Dr Paul McDermott, School of Pharmaceutical Sciences, UEA Dr Robert Jenkins, Norwich Business School, UEA
  42. 42. 22/01/2018 HEA Annual Conference 2017 46 Warm Up GMYMTNOLMERERO LONG TERM MEMORY
  43. 43. Self-Assessment 22/01/2018 Closely aligned to the construct of self-efficacy Self assessment can be defined as information about the learners provided by the learners themselves4. Good self assessment will provide ACCURATE data about: • The learner’s abilities • The progress they think they are making • What they think they can or cannot do with the material they have covered in the course The greater a learner’s self assessment ability in relation to a task the more likely it is they will develop a feeling of mastery over that task (self-efficacy). (4). Blanche, P., & Merino, B. Language Learning, 1989, 39, 313-340 “I can’t” “I know” - Suzanne Fergus
  44. 44. Self-Assessment 22/01/2018 Closely aligned to the construct of self-efficacy There have been many reported studies in the education research literature for the measurement of learners’ self-assessment. The most popular methodology employed across these studies is a multiple choice quiz followed by a confidence tier questionnaire. Many of which display the Dunning-Kruger effect (5). Ehrlinger, J., et al, Organizational Behaviour and Human Decision Processes., 2008, 105, p98-121.
  45. 45. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Our Hypotheses: 1). Previous research methodology doesn’t necessarily account for the differing motivations a student will experience when answering conceptual questions and then reporting their own confidence in a separate process. 2). If we were able to bring these two processes into one function we would have a method that allowed us to see through the fog of subjectivity and irrational optimism.
  46. 46. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Our Methodology (Taken from TBL): This format has been designed to allow For partial credit when marking MCQ’s Students distribute 4 marks across the answer options in a strategic manner to gain the best possible score. To mark the answer grid an acetate is placed over the answer sheet and the values in the clear boxes are written in the points column. Teambasedlearning.org
  47. 47. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Data analysis: We assigned a code to each answer strategy and ranked each strategy in order of increasing entropy S0 - the strategy profile that student has entered is 0,0,0,0 (invalid response) S1 - the strategy profile that student has entered is 1,1,1,1 S2a - the strategy profile that student has entered is 2,2,0,0 S2b - the strategy profile that student has entered is 2,1,1,0 S3 - the strategy profile that student has entered is 3,1,0,0 S4 - the strategy profile that student has entered is 4,0,0,0
  48. 48. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Data analysis: • 4 points on one answer (S4) corresponds to the entropy minima • 1 mark in each option (S1) is the entropy maxima 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Q1 Q2 Q3 Q4 Strategies used (%) 4003-summative Strategies by Student Quartile S1 S2a S2b S3 S4 Correlation r = -0.889 P value = 0.000 r2 = 0.362 Entropy Index: Where: k = number of categories fi = relative frequency of class i         k i f i k i f i i i fk f e 1 1 1 1
  49. 49. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Data analysis: This pattern held up across a range of different assessments (both summative and formative) Name Correlation Pvalue Rsquared Intercept Questions Students Data collection Formative course test -0.639 0.000 0.314 0.556 15 83 April 2016 Formative course test -0.480 0.000 0.185 0.572 17 101 December 2016 Summativ e course test -0.889 0.000 0.362 0.698 18 114 January 2017 Aromatic formative -0.871 0.000 0.302 0.750 10 102 January 2017
  50. 50. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Data analysis: This pattern held up across a range of different assessments (both Summative and formative) Name Correlation Pvalue Rsquared Intercept Questions Students Data Collection Endocrinology 1 -0.997 0.000 0.371 0.656 10 98 November 2015 Endocrinology 2 -0.671 0.002 0.095 0.537 9 96 November 2015 Endocrinology 1 -1.049 0.000 0.384 0.808 10 75 November 2016 Endocrinology 2 -1.359 0.000 0.264 0.813 9 77 November 2016
  51. 51. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Data analysis: This pattern held up across a range of different assessments (both Summative and formative) Name Correlation Pvalue Rsquared Intercept Questions Students Bipolar affective disorder -0.717 0.000 0.192 0.592 18 82 February 2016 Anxiety and depression -0.787 0.000 0.155 0.605 18 90 February 15/16 Schizophrenia -0.357 0.051 0.04 0.494 17 83 March 2016 Depression and Anxietiy -0.434 0.000 0.189 0.701 20 119 February 2017 Bipolar -0.697 0.000 0.242 0.735 20 119 February 2017 Schizophrenia -1.359 0.000 0.264 0.813 9 77 March 2017
  52. 52. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world The students do not know they are giving this self efficacy data They have a different “motivation” as they are focussed on an effort to maximise their grade.
  53. 53. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world • This data does not (necessarily) follow a Dunning Kruger pattern • Very clearly we can see that lower performing students use higher entropy strategies which we can attribute to lower confidence in their answers • Higher performing students use lower entropy strategies which we can attribute to higher confidence in their answers • The significant correlation values show that this is accurate self assessment data with the potential for a number of applications Data Analysis:
  54. 54. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Results 97.22% 1 2 3 4 5 6 7 8 9 4 4 4 4 4 4 4 4 4 10 11 12 13 14 15 16 17 18 4 4 4 4 2 4 4 4 4 2
  55. 55. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Results 72.22% 1 2 3 4 5 6 7 8 9 4 4 2 4 4 2 4 4 4 4 10 11 12 13 14 15 16 17 18 1 1 2 4 2 2 4 4 2 3 4 4 1 2
  56. 56. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Results 29.17% 1 2 3 4 5 6 7 8 9 1 4 4 2 2 3 2 2 3 2 1 1 4 4 1 10 11 12 13 14 15 16 17 18 2 1 2 1 1 3 1 2 1 1 4 1 1 2 2 3 1 3 2 1 1
  57. 57. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Results 10 11 12 13 14 15 16 17 18 1 4 4 1 4 4 4 4 4 1 4 1 1 2 3 4 5 6 7 8 9 4 4 4 4 4 4 4 4 4 29.17%
  58. 58. Confidence Measure 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Question number 1 2 3 4 5 Mark Grade Student ID Answer Key X X X X X Ms A. N. Nonymous A 2 1 B 4 1 2 1 1 11 55.00 C 3 2 1 1 0.00 D 1 0.00 4 1.5 1 0.666667 0.25 1.48 55.00 =MAX(C3:C6)/COUNTIF(C3:C6,">0") =AVERAGE Average = 55.4%
  59. 59. Confidence Measure 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world Average = 64.2% Average = 66.7% Average = 61.7% 2016-17 (SUMMATIVE): 2015-16 (FORMATIVE): Average = 51.2% Average = 52.3% Average = 45.4%
  60. 60. Confidence Measure 22/01/2018 A bit more… Calculated    yx ii i ss yyxx c   for each student y x ci positive ci positive ci negative ci negative Max = 3.86 Min = −1.94
  61. 61. Confidence Measure 22/01/2018 A bit more… Calculated    yx ii i ss yyxx c   for each student y x Above average performance Below average performance
  62. 62. Confidence Measure 22/01/2018 A bit more… Calculated    yx ii i ss yyxx c   for each student y x
  63. 63. Confidence Measure 22/01/2018 Interpretation/student feedback loop.    yx ii i ss yyxx c   Max = 3.86 Min = −1.94 Close to zero indicates close to mean results in confidence or grade or both. Green, positive: above average in both, closer to 4 more confident high marks. Green, negative: above average in grade, below average confident closer to -2 high performing student, low confidence. Red, negative: below average in both, closer to 4 low confidence, low marks. Red, positive: below average in grade, above average confident closer to -2 high confidence, low scores. Inform student of this, their score and means give reflective opportunity.
  64. 64. Self-Assessment 22/01/2018 METRICS have come into sharp focus as we move into a new REF-TEF world • This gives us a self-efficacy metric that instantly correlates to actual student performance • This methodology can be seamlessly incorporated into a vast range of teaching and assessment settings and the data is quickly and easily accessed • We propose that this methodology allows us to gain insightful self- assessment data through the direct measurement of subjective confidence (2nd order construct) Conclusion:
  65. 65. 22/01/2018 TBL Conference 2015 69 THANK YOU
  66. 66. Using data to increase learning gains and teaching excellence Milton Keynes Measuring employability gain in Higher Education. A case study measuring the impact of R2 Strengths on final years’ employability Dr Heike Behle, LEGACY, Warwick Institute for Employment Research (IER), University of Warwick 22 January 2018
  67. 67. @HeikeBehle Measuring employability gain • Employability NOT employment • Student Outcomes and Learning Gain (TEF) “acquisition of attributes such as lifelong learning skills and others that allow a graduate to make a strong contribution to society, economy and the environment” “progression to further study, acquisition of knowledge, skills and attributes necessary to compete for a graduate level job that requires the high level of skills arising from higher education” (DfE, 2017, p. 24) • Employability – ability to find, keep and progress in graduate employment • Requirements for a framework - Holistic - Sustainable - Cover HEIs interventions - Identify limits
  68. 68. @HeikeBehle The employability framework Individual Factors (demographics, health, skills, competences, knowledge, personality) Enabling Support Systems (public and private labour market intermediaries and support agencies) Individual Circumstances (household, work culture, resources, networks) Labour Market (demand on the local, national and international labour markets, operations and norms, regulations and institutional factors) Employability
  69. 69. @HeikeBehle Interventions to enhance students’ employability? • Skills, knowledge, credentials • International experiences • Work experiences (internship, sandwich course, visits to work places) • Extra-curricular activities • Volunteering • Reputation of HEI • Careers guidance, training for job search, R 2 Strength
  70. 70. @HeikeBehle Realise 2 Strength (R 2 Strength)  Dynamic assessment  Recognises the role of changing contexts in the use of strengths  Holistic and integrated understanding of an individual profile  Different to skills and knowledge • Strengths the student enjoys, is good at and has the opportunity to use Realised Strengths • Strengths the student enjoys and is good at but doesn’t use often Unrealised Strengths • Activities the student is neither good at nor enjoys Weaknesses •Strengths which, whilst the student is proficient in, they are not energised by them Learned Behaviours
  71. 71. Enabling Support Factors Individual Circumstances Individual Factors Local & National Labour Market Factors Employability Degree Life/ Work experience Extra-curricular communities of practice Awareness Transferable Skills Personal Qualities, Efficacy, Self-beliefs Meta-Cognition Skilful Practice Subject Understanding
  72. 72. @HeikeBehle Pilot Study: R2 Strength Two Research Questions  Does R2Strengths impact on employability of students and graduates?  How can we document the potential impact of R2 Strengths? Employability Method Impact of R2 Strengths
  73. 73. @HeikeBehle Mixed Methods Design Survey 6 Russell group universities Each University: 96 final year Home and EU Undergraduates (realised sample 524 before, 400 after) 3 groups of students: Group R2 Strengths Profile One-to-One 1 X X 2 X 3 Control group Qualitative Interviews 36 qualitative semi- structured interviews with group 1 participants Self- selecting participants Common interview guide Timeframe of interviews: March – May 2017 35-50 minutes in length
  74. 74. @HeikeBehle R2 Strengths Findings: Survey 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Before After Before After Before After Group1Group2Group3 I am aware of my strengths Strongly disagree Disagree Slightly disagree Neither agree or disagree Slightly agree Agree Strongly agree Source: R2 Strengths Survey (respondents to both surveys only, n = 400)
  75. 75. R2 Strengths Findings: Qualitative interviews “I thought I was someone who knew what I was good at and always thought I knew I’m good at this or that but I don’t think I reflected on a wider sense. It was more on specific skills of what I can do. That’s not how strengths were articulated in the project. It was more of a broader thing.” [Alexa] Tangible impact “It has helped me with attaching words to the strengths I have without being restricted by my own vocabulary. [Lucas]” Intangible impact More confident. Definitely. They're more quantified, and they're more visible, realisable, tangible. Especially my persistence, because I am very persistent. And my work ethic. [Philip] Confidence It was more like "these actually are my strengths." Talking about it made me seem more engaged […] it legitimised it. [Philip]
  76. 76. @HeikeBehle Summary • Holistic Employability Framework • Employability is not the same as employment rates. “Employability = ability to find, keep and progress in graduate employment” • TEF mainly focusses on employability skills, however, a wider focus is needed. • Case Study: R2 Strengths. • Positive employability impacts for students, both in tangible terms (application processes) but also in intangible terms (increased self- confidence and self-efficacy) could be evidenced. • R2 Strengths made a considerable difference to many students in encouraging awareness of their strengths, thus helping them to articulate these with increased self-confidence. • The quantitative survey did not capture evidence of students’ development in the way the qualitative interviews did. Students were less likely to agree that they had increased their knowledge, use and ability to articulate their strengths and their career-readiness over the life of the project
  77. 77. @HeikeBehle Final remarks Follow us on @LegacyLGproject ; @HeikeBehle Heike.Behle@warwick.ac.uk For further information www.legacy.ac.uk The LEGACY employability briefing can be downloaded here www.legacy.ac.uk Click: Resources Or here: www2.warwick.ac.uk/services/aro/dar/quality/legacy/hp- contents/employability_behle_h_uow_2016.pdf Report in preparation Wilson, Behle and Tassinari et al., (2018) R2 Strengths - Measuring employability gain
  78. 78. 82 Copyright © The Careers Group, University of London Careers Registration Measuring work-readiness learning gain Fiona Cobb Careers Registration Research Project Coordinator fiona.cobb@careers.lon.ac.uk @Careersgroup Photo by Tom Barrett on Unsplash Using data to increase learning gains and teaching excellence
  79. 79. 83 Copyright © The Careers Group, University of London Aims of this session  Discuss findings from the Careers Registration learning gain research project  Explore Careers Registration as an evidence based approach for employability support in Higher Education.  Consider wider implications of work readiness learning gain for HE strategic planning, policy and practice
  80. 80. 84 Copyright © The Careers Group, University of London Defining Learning Gain “the 'distance travelled', or the improvement in knowledge, skills, work-readiness and personal development demonstrated by students at two points in time.” [HEFCE, 2015]
  81. 81. 85 Copyright © The Careers Group, University of London Measuring Learning Gain “an attempt to measure the improvement in knowledge, skills, work-readiness and personal development made by students during their time spent in higher education” [HEFCE, 2017]
  82. 82. 86 Copyright © The Careers Group, University of London Where are you right now? Image: CC0 Public Domain Free for commercial use
  83. 83. 87 Copyright © The Careers Group, University of London  3-year project investigating Careers Registration (CR) as a measure of learning gain in work readiness  Led by The Careers Group, University of London Careers Registration Learning Gain project Image: CC0 Public Domain Free for commercial use
  84. 84. 88 Copyright © The Careers Group, University of London Project members Aberystwyth University City, University of London Goldsmiths, University of London King’s College London Lancaster University Liverpool John Moores University Queen Mary, University of London Royal Veterinary College School of Oriental and African Studies St George’s University of London St Mary’s University Ulster University University College London University of Bristol University of Edinburgh University of Exeter  16 Institutions  13 fully implemented CR to date  2 partially implemented by Sept 17  Representation from England, Scotland, Northern Ireland & Wales
  85. 85. 89 Copyright © The Careers Group, University of London What is Careers Registration?  Employability-related questions included in student registration each year — Current & Connected  Completed by all new and all re-enrolling students — Comprehensive & Consistent  Both cross-sectional and longitudinal data — Comparable
  86. 86. 90 Copyright © The Careers Group, University of London Core CR questions Career readiness - Self-assessed readiness to engage with career management - Select from 12 statements, e.g.  I am not ready to start thinking about my career yet — (Decide)  I have a career in mind & intend to gain relevant work experience — (Plan)  I am ready to apply for graduate level / professional opportunities — (Compete)  I have a job, further study or my own business plan confirmed — (Sorted) Employability experience - Self-reported experience - Choose from list of activities, e.g.  a placement year  a summer internship  volunteering  position of responsibility in a club or society  full time work prior to my course  self-employment / running my own business  no work experience to date Other questions on sector preference & experience, future plans & enterprise
  87. 87. 91 Copyright © The Careers Group, University of London Planning for success Most important factors in determining if graduates were in professional or managerial roles or non professional roles: 1. knowing exactly what they wanted to do or having a good idea about types of jobs and careers upon completing university. 2. Having a very targeted approach to job applications 3. Having undertaken unpaid work experience  7,500 students drawn from 27 institutions.  Study combined data from the 6 month DLHE survey with follow up data from a survey conducted two years later  Aimed to understand the relative importance of different behaviours, characteristics and factors in determining graduate outcomes
  88. 88. 92 Copyright © The Careers Group, University of London Project aims To assess whether a small number of questions in student enrolment can:  track development of student employability during their time in HE (distance travelled/learning gain)  evaluate the effectiveness of employability strategies and interventions. Image: CC0 Public Domain Free for commercial use
  89. 89. 93 Copyright © The Careers Group, University of London Research priorities Analysis •Cross-sectional ‘Snapshot’ CR •Longitudinal ‘tracked’ CR Comparison •DLHE •Employability health check •HEAR •NSS •Self Efficacy (CDSE tests) •Individual – Academic records/retention Impact •Impact evaluation of existing implementations of careers registration Application •Widening participation •Interventions •Consultants use of data •Engagement: Careers services, Academics, Students Recommendations
  90. 90. 94 Copyright © The Careers Group, University of London Careers Registration Learning Gain Project Year two findings
  91. 91. 95 Copyright © The Careers Group, University of London Year Two Analytics Measure What have we focused on? Career thinking response ratios year of study, discipline groups, Socio- demographics (undergraduates only) Compete Growth Dif. between yr 1-2, yr 2-3 of Undergraduate programmes, by discipline groups, WP characteristics Career thinking movement Between statements, and within and between categories Benchmarking (in progress) Constructing Career thinking benchmarks by discipline group and WP characteristics
  92. 92. 96 Copyright © The Careers Group, University of London Compete Growth
  93. 93. 97 Copyright © The Careers Group, University of London Compete growth  How many students are in the ‘compete’ category of career thinking by year 3 of study vs year 1 of study…  Data collected from 2 institutions who implemented CR in 2014 -15, and have collected data through to 2016/17  16,587 UG student responses analysed
  94. 94. 98 Copyright © The Careers Group, University of London Undergraduate Compete growth tracking - Year 1 compete stage career thinking 1.52% - Year 3 compete stage career thinking 19.8% - Increase of 18.28% in students career thinking patterns overall as students progress through their course of study.
  95. 95. 99 Copyright © The Careers Group, University of London 2014/5 and 2016/7 – Tracking Undergraduate students (Year 1 & 3)
  96. 96. 100 Copyright © The Careers Group, University of London Subject area Non Science subjects - Year 1: .96% compete - Year 3: 24.10% compete - Compete growth: 23.14 % Science subjects - Year 1: 2.7% compete - Year 3: 15.51% compete - Compete growth: 13.34 %
  97. 97. 101 Copyright © The Careers Group, University of London Subject area compete growth
  98. 98. 102 Copyright © The Careers Group, University of London Fee status Compete growth Overseas students 26.1% Home/EU students 17.36%
  99. 99. 103 Copyright © The Careers Group, University of London Mode of study Full time - Year 1: 1.52% compete - Year 3: 20.26% compete - Compete growth: 18.74% Part time - Year 1: 1.43% compete - Year 3: 5.41% compete - Compete growth: 3.94% % More part time students in ‘sorted’ category to begin with….
  100. 100. 104 Copyright © The Careers Group, University of London Tracking Compete growth – Part Time (Years 1 & 3 )
  101. 101. 105 Copyright © The Careers Group, University of London Compete growth: WP characteristics Category Split Disability Disability/no disability Gender Male/female (other) Mature student Under 21/ mature student Polar 3 Yes/no (deprivation categories 4+5) Ethnicity Black, Asian, White, Other
  102. 102. 106 Copyright © The Careers Group, University of London WP Category Findings Disability More Year 3 students with a disability in the compete stage of their career thinking, 22.10% compared to 19.62% Year 3 students with no disability. Gender 1.14% difference in compete growth based on gender Mature student Bigger increase in growth for U21’s (more Mature students in compete to start with) Polar 3 low participation neighbourhood background less likely to be in the compete stage in their career thinking. 12.39% low participation, 18.8% rest of cohort. Ethnicity Asian students had the highest growth in compete responses by year 3 of study (22% increase)
  103. 103. 107 Copyright © The Careers Group, University of London Tracked individual statement selection for 2015/16 to 2016/17 From year 1 – year 2, and year 2 – year 3 Changes categorised as ‘positive’, ‘negative’, or ‘no movement’
  104. 104. 108 Copyright © The Careers Group, University of London Career thinking movement by Subject Area
  105. 105. 109 Copyright © The Careers Group, University of London WP Statement movement - Minimal differences between year 1 -2 for all WP characteristics analysed - Higher % positive movement between years 2 -3 for: - Mature students (34% vs 24% Undergraduate) - Polar 3 (low part. 21%, high participation, 26%)
  106. 106. 110 Copyright © The Careers Group, University of London Next steps for analytics - Interactions and correlations between variables - Multinomial logistic regression modelling - DLHE comparison - Careers staff readiness to work with the data survey Photo by Rodion Kutsaev on Unsplash
  107. 107. 111 Copyright © The Careers Group, University of London Careers registration in action Careers registration data is being used to:  inform strategic planning and academic department engagement  identify individual student careers and support needs  promote the services offered by careers departments to their student bodies.
  108. 108. 112 Copyright © The Careers Group, University of London Strategic uses of the data I Academic departmental engagement • partnership agreements drawn up on the basis of CR data • CR data used by careers consultants to inform strategy for and planning of careers activities • data provided to personal tutors for careers discussions with students Strategic planning • data used in QAA reporting • data informing institutional level policy and KPIs Employer engagement • analysis of employer/sector data to identify gaps in employer engagement • data used to persuade new/existing employers of student interest in their sectors
  109. 109. 113 Copyright © The Careers Group, University of London Strategic uses of the data II Marketing and communications • targeted event marketing to students based on interests expressed in CR surveys Final year support programmes • targeted event for final year students who were identified as not knowing where to begin with their career planning Widening participation initiatives: • using CR data to promote relevant events and opportunities to WP students, (data held in student records system, responses can be linked with WP markers)
  110. 110. 114 Copyright © The Careers Group, University of London CR and The TEF Metrics based on DLHE ‘Employment or further study’ metric Benchmark factors: subject of study, entry qualifications, age on entry, ethnicity, sex, level of study …Graduate Outcomes…15 months to wait… Careers registration offers year on year tracking, data can be slices by benchmark factors
  111. 111. 115 Copyright © The Careers Group, University of London Fundamentals of Careers Registration Webinar series Date Webinar Title 21/9/2017 Designing, Planning and Delivering Careers Registration in record time 29/09/2017 Careers registration for postgraduates 31/10/2017 Using registration data in employer engagement 16/11/2017 Transforming careers practitioners professional practice with Careers Registration Data 30/11/2017 The power of dashboards: visualising careers registration data in your careers service 30/01/2018 Tracking the student journey: linking Careers Registration with an exit survey 06/02/2018 Careers Registration, DLHE and Graduate Outcomes 12/03/2018 Work readiness learning gain: what do the Careers registration findings tell us? https://www.eventbrite.co.uk/o/careers-staff-development-8361679885
  112. 112. 116 Copyright © The Careers Group, University of London Project website https://goo.gl/Ami6af Project Contact Fiona Cobb Fiona.cobb@careers.lon.ac.uk Twitter @FiCobb @careersgroup #learninggain
  113. 113. 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence”. https://twitter.com/LearningGains #learninggainsOU https://abclearninggains.com/
  114. 114. Big thanks Natalie Eggleston Rebecca FergusonMaryja Strickland Jekaterina Rogaten Rhona Sharpe University of Surrey
  115. 115. Open Discussion  Anyone has a question, idea, point to raise, concern, or anything else to share?
  116. 116. Open Discussion  Should we as universities buy-in to the narrative of learning gains? Are our students customers? Can we measure learning?  Is there a “best-way” to measure learning gains?  Is there a “best-way” to measuring learning gains across disciplines?  Is there a “best-way” to measuring learning gains across institutions?  What is the role of data protection/GDPR/ethics?

×