Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Predictors of postgraduate student
experience and engagement: a multilevel
analysis of postgraduate survey data.
Daniel Mu...
Background
• HEI from measures of process quality to
outcome measures
• Outcome measures include student
satisfaction
– Na...
This study
• The United Kingdom Engagement Survey (UKES), 2014 and
2015
– In 2014 32 institutions and 25000 students took ...
Questions
• What proportion of the variance in experience
and engagement is explained by student and
institutional charact...
Analytical approach
• Data preparation: Confirmatory Factor Analyses
used to test the factor structure
• Multilevel modell...
Models
• Null model
• Model 1 including institutional factors
• Model 2 including student factors
Variables in data sets
Dependent variables:
• sub-scales and overarching components
Independent variables: differs per dat...
PRES descriptives
The PRES has 7 scales:
supervision (SV), resources (RE),
research culture (RC), progress
and assessment ...
PRES variance in %
2013 SV RE RC PA RP RS PD RSP
D
PARP Q17a OVERA
LL
HEI 0.5 2.6 1.6 1.7 1.0 0.5 0.5 0.6 1.5 0.7 1.0
Disc...
PTES
The 7 scales for Teaching
and Learning,
Engagement, Assessment
and Feedback, Dissertation
or Major Project,
Organisat...
PTES variance in %
2014 TL EN AF DMP OM RSS SD OVERA
LL
HEI 1.5 1.4 2.4 0.9 1.8 2.7 1.3 2.0
Discipli
ne 3.7 3.1 5.7 2.2 4....
UKES structure
• Core set of scales (4 scales: Higher
Order Learning (HOL), Course
Challenge (CC), Collaborative
Learning ...
UKES variance in %
2014 HOL CC CL AI Overall
Level: HEI 1.9 1.8 17.4 2.9 7.4
Level:
Discipline 5.9 3.0 6.4 5.3 3.9
Level:
...
Overall conclusions
• Overall satisfaction levels are high
• By far the largest part of variance is explained at
the stude...
Overall conclusions
• Being disabled in general has a significant impact on
student experience and student engagement.
• G...
Overall conclusions
Given the low variance at the institutional level and
the significant predictors for all three surveys...
Limitations
• Census approach, not random sample
• Non-response and missing data
• Likert scales
• Large N and significanc...
• Report finalised and appearing soon
• Questions?
– Daniel Muijs: D.Muijs@soton.ac.uk
– Christian Bokhove: C.Bokhove@soto...
Selected references
• Cheng, J.H.S & Marsh, H.W. (2010). National Student Survey: are differences
between universities and...
Upcoming SlideShare
Loading in …5
×

Society for Research into Higher Education conference presentation

507 views

Published on

Predictors of postgraduate student experience and engagement: a multilevel analysis of postgraduate survey data. PTES, PRES and UKES.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Society for Research into Higher Education conference presentation

  1. 1. Predictors of postgraduate student experience and engagement: a multilevel analysis of postgraduate survey data. Daniel Muijs Christian Bokhove Alex Buckley SRHE conference 2015 10 December (International Human Rights Day)
  2. 2. Background • HEI from measures of process quality to outcome measures • Outcome measures include student satisfaction – National Student Survey (Marsh & Cheng 2008, Cheng & Marsh 2010) • Role student engagement – (Trowler & Trowler, 2010)
  3. 3. This study • The United Kingdom Engagement Survey (UKES), 2014 and 2015 – In 2014 32 institutions and 25000 students took part in the survey. • The Postgraduate Taught Experience Survey (PTES), 2014 and 2015 – 100 HEI’s and almost 70000 students participated in the most recent (2015) survey. • Postgraduate Research Experience Survey (PRES), 2013 and 2015 – In 2013 this survey took place in 122 higher education institutions in spring 2013. 48,401 postgraduate researchers took part, forming 41.9% of eligible respondents.
  4. 4. Questions • What proportion of the variance in experience and engagement is explained by student and institutional characteristics? • Can the survey reliably distinguish between institutions, and between courses? • What student and organisational characteristics are related to student experience and engagement?
  5. 5. Analytical approach • Data preparation: Confirmatory Factor Analyses used to test the factor structure • Multilevel modelling to account for the hierarchical nature of the datasets, in which students are nested within disciplines, and disciplines within universities. – Adaptation of the general linear model for hierarchical datasets – Solves the problem of attenuation of standard errors in standard linear regression models
  6. 6. Models • Null model • Model 1 including institutional factors • Model 2 including student factors
  7. 7. Variables in data sets Dependent variables: • sub-scales and overarching components Independent variables: differs per dataset but includes: • at the institutional level: type of HEI, country • At the student level: age, gender, disability, country of origin, ethnicity, student status (FT/PT)
  8. 8. PRES descriptives The PRES has 7 scales: supervision (SV), resources (RE), research culture (RC), progress and assessment (PA), responsibilities (RP), research skills (RS) and professional development (PD). 2013 SV RE RC PA RP RS PD RSPD PARP Q17A OVERALL N 47631 47351 47264 47630 47541 47512 47406 47635 47825 47623 47857 Mean 4.3 4.1 3.7 4.0 4.0 4.2 4.0 4.1 4.0 4.1 4.1 SD 0.9 0.9 0.9 0.8 0.8 0.8 0.8 0.7 0.7 1.0 0.6 2015 SV RE RC PA RP RS PD RSPD PARP Q17A OVERALL N 53161 52972 52878 53254 53260 52983 52842 53130 53297 53101 53319 Mean 4.4 4.1 3.8 4.1 4.1 4.3 4.1 4.2 4.1 4.1 4.1 SD 0.9 0.9 0.9 0.8 0.8 0.8 0.8 0.7 0.7 1.0 0.6
  9. 9. PRES variance in % 2013 SV RE RC PA RP RS PD RSP D PARP Q17a OVERA LL HEI 0.5 2.6 1.6 1.7 1.0 0.5 0.5 0.6 1.5 0.7 1.0 Discipli ne 1.1 8.6 4.5 2.3 1.6 1.0 1.4 1.2 2.0 1.7 2.3 Individu al 98.3 88.9 93.8 96.0 97.4 98.4 98.1 98.2 96.5 97.6 96.6 2015 SV RE RC PA RP RS PD RSPD PARP Q17a OVE RALL HEI 0.5 2.5 1.9 1.6 1.0 0.5 0.7 0.6 1.5 1.1 1.0 Disci pline 0.8 9.8 4.2 2.1 1.9 1.0 1.2 1.0 2.0 1.5 2.3 Indiv idual 98.6 87.6 93.9 96.3 97.1 98.4 98.2 98.4 96.5 97.4 96.6
  10. 10. PTES The 7 scales for Teaching and Learning, Engagement, Assessment and Feedback, Dissertation or Major Project, Organisation and Management, Resources and Services and Skills Development, and the overall satisfaction scale .
  11. 11. PTES variance in % 2014 TL EN AF DMP OM RSS SD OVERA LL HEI 1.5 1.4 2.4 0.9 1.8 2.7 1.3 2.0 Discipli ne 3.7 3.1 5.7 2.2 4.2 1.8 2.0 3.3 Individ ual 94.8 95.5 91.9 96.9 94.1 95.4 96.7 94.7 2015 TL EN AF DMP OM RSS SD OVERAL L HEI 0.5 2.5 1.9 1.6 1.0 0.5 0.7 1.0 Disciplin e 0.8 9.8 4.2 2.1 1.9 1.0 1.2 2.3 Individu al 98.6 87.6 93.9 96.3 97.1 98.4 98.2 96.6
  12. 12. UKES structure • Core set of scales (4 scales: Higher Order Learning (HOL), Course Challenge (CC), Collaborative Learning (CL), and Academic Integration (AI)) and optional scales (6 scales: Reflective and Integrative Learning (RIL), Time Spent (TS), Skills Development 1 (SD1), Skills Development 2 (SD2), Engagement with Research (ER) and Formulating and Exploring Questions (FEQ). • Substantial changes in 2015
  13. 13. UKES variance in % 2014 HOL CC CL AI Overall Level: HEI 1.9 1.8 17.4 2.9 7.4 Level: Discipline 5.9 3.0 6.4 5.3 3.9 Level: Individual 92.4 95.2 76.2 91.8 88.7 2015 CT LO IS RC CC Overall Level: HEI 0.0 1.1 1.9 0.0 0.3 0.5 Level: Discipline 4.4 7.1 4.8 7.3 2.7 5.5 Level: Individual 95.6 91.8 93.3 92.7 97.0 94.0
  14. 14. Overall conclusions • Overall satisfaction levels are high • By far the largest part of variance is explained at the student level. Very little HE-level variance in general. • The introduction of institutional factors in the models does not have much explanatory power for the overall student satisfaction or engagement. • Some student characteristics significant, but explain limited variance.
  15. 15. Overall conclusions • Being disabled in general has a significant impact on student experience and student engagement. • Gender and age show different patterns for different scales. • On the whole, Black and Minority Ethnic students have a positive student experience and student engagement. • There where we could include ‘distance learning’ as a variable, for example in the UKES data set, it was a negative predictor for engagement. • For countries of origin, African and Asian students are more positive about student experience and student engagement across the board, than the reference category (UK students).
  16. 16. Overall conclusions Given the low variance at the institutional level and the significant predictors for all three surveys it seems pertinent to not aim for a university wide approach for student experience and student engagement. Rather, individual factors could be addressed by every institution individually. Institutional policies could be aimed at improving experiences and engagement for different gender and age groups, distance learning, disabled students and students from Australasia and North America.
  17. 17. Limitations • Census approach, not random sample • Non-response and missing data • Likert scales • Large N and significance • Low N for some subgroups
  18. 18. • Report finalised and appearing soon • Questions? – Daniel Muijs: D.Muijs@soton.ac.uk – Christian Bokhove: C.Bokhove@soton.ac.uk • @ProfDanielMuijs and @cbokhove respectively on twitter. Thank you!
  19. 19. Selected references • Cheng, J.H.S & Marsh, H.W. (2010). National Student Survey: are differences between universities and courses reliable and meaningful? Oxford Review of Education, 36(6), 693-712. • Dedrick, R. F., Ferron, J. M., Hess, M. R., Hogarty, K. Y., Kromrey, J. D., Lang, T. R., Niles, J., & Lee, R. (2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79, 69-102. • Marsh, H. and Cheng, J. (2008). National Student Survey of teaching in UK universities: Dimensionality, multilevel structure, and differentiation at the level of university and discipline – preliminary results (York, Higher Education Academy). Available at: https://www.heacademy.ac.uk/resource/national-student-survey- teaching-uk-universities-dimensionality-multilevel-structure-and • Reynolds, D., Sammons, P., De fraine, B., Van Damme, J., Townsend, T., Teddlie, C. & Stringfield, S. (2014). Educational effectiveness research (EER): a state-of-the-art review. School Effectiveness and School Improvement, 25(2), 197-230. • Trowler, V & Trowler, P (2010). Student Engagement Executive Summary. York: Higher Education Academy. For the reports see https://www.heacademy.ac.uk/

×