CHE Seminar presentation 16 January 2020, Alistair McGuire, Department of Health Policy, LSE. Evaluating the Healthy Minds program: The impact on adolescent’s health related quality of life of a change in a school curriculum

1. Healthy Minds: A Randomised Controlled Trial to
Evaluate PHSE Curriculum Development Impact on Health and Soft
Skills
Alistair McGuire LSE
Grace Lordan LSE

2. Talk
• Evaluate whether ‘Healthy Minds’, a new Personal, Health, Social and
Economic (PHSE) curriculum is effective
• To do so we conduct an RCT on introduction of a new PHSE curriculum
within secondary schools
• Outline the curriculum development
• Outline the RCT
• Present results
• Place in context

3. PHSE
• PHSE is a timetabled secondary school subject taught from year 7 to
year 10 to provide skills to become more self-aware, resilient to
negative peer-pressures and to make more informed life-choices
• Covers sex and drug education, behavioural skills, self-awareness…etc
• Currently curriculum is under review but timetabling was compulsory
from September 2019
• PHSE teach is currently sub-optimal with 40% of schools requiring
improvement in this area (HoC Educ. Comm., 2015; Ofsted, 2017)
• Recent governmental review and announcement to make it statutory from
2019
• Unclear what is the best form of curriculum

4. Background
• Numerous studies show that early intervention influencing the physical, mental,
emotional, and social functioning domains that PHSE seeks to affect has a long
arm into adulthood
• Numerous studies state childhood health affects educational attainment, adult
labour market outcomes and long-term health (particularly true for mental
health)
• (Almond, Currie and Duque, 2018; Conti, Heckman and Pinto, 2016; etc)
• But little known about adolescent interventions
• Whether these are effective or even more effective than early childhood interventions
• And little known about the cognitive (hard) versus non-cognitive (soft) impacts
• Cunha, Heckman and Schennach (2010) on this distinction

5. Health Minds PHSE curriculum
• Coleman, Hale and Layard (2011) systematic review of individual
modules that had evidence of “effectiveness”
• Found a number of individual programmes had been proven to be
effective (in some manner)
• Combined 11 of these individual programmes into a 4-year
curriculum
• The Healthy Minds curriculum

6. What is Healthy Minds?
• 120 hour intervention delivered over 4 years
• Covers a range of topics combining 11 modules
• Each module has been subject to past evaluation individually
• Curriculum designed by Coleman et al (2011)
• Taught by staff who receive training in each module
• The treatment includes the aggregate effects of 11 individual modules
and training

7. Healthy Minds Curriculum
Teacher training time:
Year 7 = 7 days
Year 8 = 7.5 days
Year 9 = 1 day *
Year 10 =3 days

8. Health Minds Trial
• Study based on 34 schools followed up over 4 years
• 39 school-year cohorts (including ‘wait list controls’)
• Mix of academies, academy convertors, foundation, community and faith schools
• Began 2013 or 2014 as schools entered study
• Randomisation based on trying to minimise difference across schools in terms of
certain characteristics
• % Free School Meals; % with 5 GCSE grades; single sex/mixed entrance
• Two-arm randomised cluster design
• Trial not blind
• Control (some “wait list” control) versus treatment schools
• 53% treated
• Total of 4,634 pupils with 1,711 pupils in control arm versus 2,589 in treatment arm
• Intent to treat study to allow for differences in compliance/adherence in the
treatment arm delivery
• Differences in adherence (Noisy approx.: 58% = 1, 25% ≥0.75, 10% ≥ 50%)
• Main model is ITT.
• Data collected every two years (t=0, t=2, t=4)

9. Sample size calculation
• Sample calculations based on a total
number of 30 schools
• Originally recruited 37 schools over
two years
• Lost 3 schools
• 30 schools allows detection of an
effect size of 0.28 change in standard
deviation in main outcome
• Applying conventional statistical
significance of 0.05 and power of 0.80,
assume intra-class correlation (ICCs) to
be 0.06
Sample size calculation
• Based on Minimum Detectable Effect Size (MDES)
P = proportion of schools (clusters) randomly allocated to treatment group (0.59)
n = number of pupils in each school (cluster) [100]
J = total number of randomised schools (clusters) [32]
M = multiplier based Type 1 error (5%) and power (80%) [2.8]
ρ = within-school (cluster) variance [0.06]
MDES = 0.265 [<0.3 so stated design is satisfactory]

10. Sample size

11. Health Minds Trial
• 34 schools (39 school-cohorts)
• Long term and intensive commitment required, so some participation & retention
problems, particularly on reaching year 10.
• Drop-out plus efforts to recover meant 35 school-year cohorts in final analysis
• Final analysis 3,569 with 1,309 in the control arm and 2,264 in treatment arm
• Makes this one of the biggest studies globally to consider personal, social, health
education aiming to influence Health Related Quality of Life (HRQoL) in
adolescents
• Follow-up study on impact of Healthy Minds on educational attainment run separately
reporting in future years

12. Timeline
Date Activity
January 2013 Approach potential participating schools
January 2013 through
September 2013
Recruitment of schools
April – September 2013
Allocation to Phase 1 schools (pupils enter study September 2013) and
Phase 2 schools (pupils enter study September 2014)
July 2013 Year 1 Teacher Training for Phase 1 schools
September 2013 HM teaching begins for Phase 1 schools
September 2013
Data (baseline) questionnaires administered to Phase 1 schools
(Treatment and wait-list control schools)
July 2014
Year 1 Teacher Training for Phase 2 schools and Year 2 Teacher
Training for Phase 1 schools
September 2014 HM teaching begins for Phase 2 schools
September 2014
Data (baseline) questionnaires administered to Phase 2 schools
(Treatment, wait-list treatment schools and control schools)
July 2015
Year 2 Teacher Training for Phase 2 schools and Year 3 Teacher
Training for Phase 1 schools
May/June 2015
Data (interim) questionnaires administered to Phase 1 schools
(Treatment and wait-list control schools)
July 2016
Year 3 Teacher Training for Phase 2 schools and Year 4 Teacher
Training for Phase 1 schools
May/June 2016
Data (interim) questionnaires administered to Phase 2 schools
(Treatment, wait-list treatment and control schools)
July 2017 Year 4 Teacher Training for Phase 2 schools
May/June 2017
Data (endline) questionnaires administered to Phase 1 schools
(Treatment and wait-list control schools)
May/June 2018
Data (endline) questionnaires administered to Phase 2 schools
(Treatment, wait-list treatment and control schools)
August/September 2018 Data released and analysis undertaken

13. Trial flow
• Average cluster size
• Protocol = 100
• Randomisation=121
• Nos. schools
• Protocol = 15 (control) 15 (intervention)
• Randomisation=16 (control) 18 (intervention)
• School cohorts
• Randomisation= 16 (control) 23 (intervention)
• Nos. pupils
• Protocol = 1500 (control)
1500 (intervention)
• Randomisation= 1711 (control)
2589 (intervention)
• Final analysis = 1309 (control)
2264 (intervention)

14. Baseline school characteristics
School-level (categorical)
Intervention group Control group
n/N (missing) Count (%) n/N (missing) Count (%)
Academy
892/3012
(0)
4 (22%)
585/1613
(0)
5 (31%)
Academy convertor
904/3021
(0)
6 (33%)
902/1613
(0)
9 (56%)
Foundation
72/3021
(0)
1 (5.5%)
54/1613
(0)
1 (6.25%)
Community
830/3012
(0)
6 (33%) 0 0
Free school 0 0 72/1603 1 (6.25%)
Voluntary aided
182/3021
(0)
1 (5.5%) 0 0
Ofsted rating
Outstanding
Good
Requires improvement
2
11
5
1
10
4

15. Measured Outcomes
• 3 main validated questionnaires to measure outcomes
• CHQ-CF87 questionnaire - 87 items across 14 scales
• Measures physical and psychosocial health
• Main outcome CHQ-CF87 score of General Health
• Sample size and measure of statistically significant difference in effect of Healthy Minds course on
adolescents based on this measure
• Short Mood and Feelings questionnaire
• Mainly measures depression related issues
• SCARED questionnaire
• Issues of anxiety
• Takes 50 minutes to complete
• Can define outcomes across
• Externalised outcomes (health, behaviour, life skills and life satisfaction)
• Internalised outcomes (anxiety, self-esteem, mood)

16. Scoring the Main Outcome
• Summary component scoring for the child-completed form (CHQ-
CF87) is not currently available.
• Scoring is currently available at the scale level only
• So we have 1 primary outcome, and 12 secondary outcomes in CHQ-
CF87
• Scoring follows the CHQ-CF87 manual.

17. CHQ-CF 87 Questionnaire
Primary Outcome:
Global Health : “In general, how
would you say your health is?”

18. Methods
• Regression run across different outcomes
• Run with and without pupil fixed effects
• Also controlling for gender
• All t-tests are adjusted for multiple comparisons through a Benjamini
and Hochberg (1995) correction mechanism

19. Methods
• Little influence over treatment contamination (especially in wait-list
control schools) or adherence so identification is consistent with an
Intent-to-Treat design
• Conservative estimate
• Aim to recover the Average Treatment Effect (ATE)
• Calculate through a difference-in-difference estimator

20. Methods
• Also estimate an ATE in ITT identification strategy associated with
compliance
• Compliance based on counting 0.25 as completion per year (given 4-
year duration of trial)
• Essentially uses the trial allocation as an IV strategy

21. T tests at t=0
Outcome t=0 Outcome t=0
Global Health -0.009 Mental Health -0.077
(0.036) (0.042)
N 3789 N 3789
Physical Functioning -0.053 Past Heath N/A
(0.041)
N 3789 N
Emotional Difficulties -0.039 Self Esteem -0.081**
(0.040) (0.037)
N
3789 N 3789
Behavioural Difficulties 0.064 General Health -0.064*
(0.039) (0.037)
N 3789 3789
Physical Difficulties -0.036 Family Activities 0.034*
(0.035) (0.037)
N 3789 3789
Behaviour -0.157*** Family Cohesion 0.084**
(0.038) (0.037)
N 3789 3789
Global Behaviour -0.120*** Pain and Discomfort -0.026
(0.037) (0.036)
N 3789 N 3789

22. Global
Health
Physical Functioning
Emotional
Difficulties
Behavioural
Difficulties Self Esteem
Treatment 0.245** 0.162 0.052 0.123* -0.040
(0.110) (0.128) (0.086) (0.067) (0.078)
N 7252 7252 7252 7252 7252
Physical
Difficulties
Pain and Discomfort Behaviour
Global Behaviour
Mental Health
Treatment 0.297*** 0.206* 0.145** 0.140 0.067
(0.068) (0.116) (0.061) (0.135) (0.052)
N 7252 7252 7252 7252 7252
Past Health General Health Family Activities Family Cohesion
Treatment N/A 0.151** 0.118* 0.121
(0.070) (0.062) (0.101)
N 7252 7252 7252
Main Results

23. Healthy Minds and Primary Outcome
Robustness
Outcome Adjusted differences in means
Population (n)
Primary outcome: Global Health score from the CHQ-CF87
Baseline difference-in-difference estimates
0.245**
(0.110)
7,252
Including School effects
0.232*
(0.121)
7,252
Including Individual pupil effects
0.150*
(0.075)
7,252
Compliance Analysis
0.237*
(0.128)
5,533
Compliance Analysis with School Fixed Effects
0.145
(0.109)
5,533
Adjustment for balance
0.207
(0.135)
5,943
Adjustment for missing values (through mean imputation)
0.245**
(0.11)
7,252
Difference-in-difference estimates (interim results)
0.234**
(0.094)
5,821

24. Healthy Minds and Large Changes
Dependent Variable =1 if moved +1 standard deviation , 0 otherwise
Global
Health
Physical
Difficulties
Behaviour General Health Family Activities
Treatment 0.089*** 0.097*** 0.007 0.081** 0.027
(0.019) (0.022) (0.038) (0.032) (0.039)
N 2874 2793 2669 2839 2789
Dependent Variable =1 if moved +2 standard deviation , 0 otherwise
Global Health
Physical
Difficulties
Behaviour General Health Family Activities
Treatment 0.053** 0.039** 0.018 0.024 0.004
(0.017) (0.009) (0.017) (0.016) (0.024)
N
2874 2793 2669 2839 2789

25. Results - adjusted for multiple measures
• Primary outcome
• The ATE: General Health Score improves by 0.245 standard deviations
• What does this mean?
• Score of the average pupil in the Healthy Minds group is 0.245 s.d above the
average person in the control group
• So the average health after the programme for a Healthy Minds pupil exceed
the general health of 60% of those pupils not getting the teaching
• If you took the average pupil and taught the Healthy Minds course it would
improve their General Health score by 10 percentiles (e.g. from 50% - 60%)

26. Results in context - adjusted for multiple
measures
• Teaching Healthy Minds improves General Health score by an effect size of 0.245
• Comparable to Hattie (2015) meta-analysis effect sizes looking at academic
achievement for:
• Pre-school with at risk students 0.56 s.d.
• Reducing class size from 23 to 15 improves students’ test performance by 0.3 s.d.
• Effect of divorce or remarriage on pupil achievement 0.25 s.d.
• Summer schools 0.23 s.d.
• Extra-curriculum activity 0.21 s.d.
• Ability grouping 0.12 s.d.
• Depression -0.42 s.d.
• These do look mainly at educational achievement effects so they not directly
comparable to our results but they do provide some context

27. Additional discussion of results
• HRQoL cannot be measured with justice in 1 outcome
• The positive impact on General Health is robust to a number of different analyses
• Measures of effect size on aspects of behavior and external relations (family
relations)
• The Life Satisfaction effect size was 0.182 implying that average life satisfaction of those taught Healthy
Minds exceeds the general life satisfaction by 56% compared to those not taught.
• The impact on internalised emotions was generally negative
• But NOT significantly so
• Anxiety measures in particular showed some decreases.
• One possible explanation is that the teaching makes individuals more “self-aware” but this would need
further exploration.

28. Cost of Healthy Minds
• Cost involves 19 days teacher training
• Each training day costs £190 for the training and £160 for cover teacher
• Total cost per teacher is £6,650 over the 4 years
• Assuming a teacher teaches 3 classes of 30 pupils, including class materials
• COST per pupil over the 4-years is £100
• £25 per pupil per year (general spending on secondary school pupils is c£6,000 per year)
• The positive effect of the Healthy Minds is achieved at a very low cost

29. Conclusions
• Teaching Healthy Minds improves the primary outcome
• It is extremely cost-effective, providing good value for money
• IF Healthy Minds was compulsory the effect might be even greater (as the Intent
to Treat design incorporates non-compliance)
• The ‘Externalising’ effects are supportive of this primary effect
• One of the ‘Internalising’ effects (self-esteem) moves in the opposite direction
(not significant)
• We cannot be sure why but a plausible mechanism might be that teaching Healthy Minds is making pupils
more aware of how they are feeling and not hiding negative emotions?

30. Thank you!
• Questions/Comments…