Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Measuring the right outcomes in mental health


Published on

This talk presents the findings of an MRC study on whether the generic health measures of EQ-5D and SF-36 are valid in mental health. It uses mixed methods research (including interviews with service users) to show that these measures miss important ways in which mental health impacts on people's lives. It proposes 7 themes that seem to capture the important domains of recovery for people with mental health problems that provide the basis for a new generic outcome measure for mental health.

N.B. These slides were presented at the 20th Anniversary of the Centre for Mental and Physical Health Economics, 7th November 2013.

Published in: Health & Medicine
  • Be the first to comment

Measuring the right outcomes in mental health

  1. 1. Outcome Measurement in Mental Health John Brazier Director of the Economic Evaluation of Health and Care Interventions Policy Research Unit (EEPRU) School of Health and Related Research The University of Sheffield, UK CEMPH Conference 7 November 2013
  2. 2. Background Growing use of generic outcome measures in people with mental health problems: • Surveys (e.g. ESEMed, PMS in UK) • Use in clinical trials • Assessing cost-effectiveness to inform reimbursement decisions (e.g. NICE) • Routine outcome monitoring (e.g. IAPT, PROMS programme, PBR) Things have changed since the Gilbody review! 27/11/2013 © The University of Sheffield 2
  3. 3. 3 What should we be measuring? • Quality of life (QoL) or well-being are ill-defined and there are different ways of conceptualising them: functionings, capabilities, wellbeing etc. • The World Health Organization (1948) declared health to be “A state of complete physical, mental and social wellbeing, and not merely the absence of disease and infirmity” • QoL means different things to different people and this is why we need the views of mental health service users in developing and testing measures
  4. 4. Types of measure • Generic measures: those instruments designed for use on any population (EQ-5D, ICECAP etc.) • Condition specific measure (CSM): those instruments designed for use in a specific population (CORE-10, PHQ9, GAD etc.) Both types of measure are standardised and come with a scoring algorithm (that may or may not be ‘preference based’ for calculating QALYs)
  5. 5. Quality Adjusted Life years (QALY) • QALYs combines both quantity and quality of life into a single measure • QALYs can be used across all health care interventions for all patient groups
  6. 6. 6 By placing a tick in one box in each group below, please indicate which statements best describe your own health state today. Mobility I have no problems in walking about I have some problems in walking about I am confined to bed    Self-Care I have no problems with self-care • • • I have some problems washing or dressing myself Scored using UK TTO values ( Dolan et al,1997) with a range minus 0.54 (worst impairment) to 1 (full health) Preferred by NICE and used in DH PROMS programme 5 level version now available I am unable to wash or dress myself    Usual Activities (e.g. work, study, housework, family or leisure activities) I have no problems with performing my usual activities I have some problems with performing my usual activities I am unable to perform my usual activities    Pain/Discomfort I have no pain or discomfort I have moderate pain or discomfort I have extreme pain or discomfort    Anxiety/Depression I am not anxious or depressed I am moderately anxious or depressed I am extremely anxious or depressed   
  7. 7. How do we know if a measure like EQ-5D is measuring the right thing? • Does it cover the important dimensions of (mental) health related quality of life and relevant ranges (Content validity) • Does it reflect known group differences and correlate with other indicators of quality of life (Construct validity) • Does respond to known changes quality of life (Responsiveness) Assessing validity is problematic due to the absence of a gold standard and validity being a question of degree
  8. 8. 8 Overview of talk Based on the findings of an MRC funded study looking at the appropriateness of EQ-5D and other generic measures in people with mental health problems • Psychometric evidence – based on a systematic review and further analyses • Qualitative evidence – based on a review and indepth interviews with mental health service users • Implications for existing measures: EQ-5D, SF6D and the new kid on the block the SWEMWBS
  9. 9. Systematic review To assess the appropriateness of the EQ-5D and the SF36 family in terms of their validity and responsiveness in five mental health conditions: • Schizophrenia, Bipolar disorder, Personality disorders, Depression and anxiety • Ten health databases were searched • Studies were appraised and data extracted using a standardised template. • Performance: • Construct validity: known group differences with general population control or between severity groups and convergent validity • Responsiveness to known changes 27/11/2013 9
  10. 10. 10 PRISMA FLOW DIAGRAM Citations retrieved by search of electronic databases (n=7,095) Unique citations (n=4,115) Titles and/or abstracts potentially relevant (n=266) Studies satisfying inclusion criteria (n=154) PDs (n=9) Schizophrenia (n=33) 27/11/2013 © The University of Sheffield Bipolar disorder (n=22) Duplicate citations (n=2,980) Citations excluded after screening (n=3,849) Full papers excluded after screening (n=124) 12 references from reference list check Depressive & anxiety disorders (n=23 – not SF-36)
  11. 11. 11 Depression and Anxiety EQ-5D √√√√x √√ SF-6D √√ Known group: Casecontrol Convergent validity - √√ √√√√√ √√√ Responsiveness √√√√x√√√√√√√√√√ √√ Known group: Severity Where √ indicates results in support of validity or responsiveness and x indicates an inconsistent or non-significant results by test (and not an individual study);
  12. 12. 12 Depression and anxiety • EQ-5D and SF-6D: good known group and convergent validity, and responsiveness. • More highly correlated with depression scales in patients with anxiety than with clinical anxiety scales • Convergent validity in patients with anxiety may be driven by aspects of depression within anxiety disorder and presence of co-morbid depression. 27/11/2013 © The University of Sheffield
  13. 13. 13 Personality Disorders EQ-5D √ Known group: severity Known group: √ case-control Convergent validity √ Responsiveness √√√ SF-36 x SF-12 √√ SF-6D - √√ - - √ - - √ - -
  14. 14. 14 Personality disorders • EQ-5D: responsive, KGV and CV • Limited and mixed SF-36 evidence (related to two studies only) • Very little evidence on SF-12 and none for SF-6D
  15. 15. 15 Schizophrenia EQ-5D SF-36 Known group: Severity √ √ - - Know group: Case-control - √√√√√√√√√√ √ √ - √ √ √ x x √ x √ √ √√√√√√√√√ x x √√√√ x x xxxx - √ - x Convergent validity Responsiveness √ √√ x x √√√ x x x x SF-12 SF-6D
  16. 16. 16 Schizophrenia • KGV: Yes, but crude measures • CV & R: Mixed results • Clinical assessment of symptoms e.g. Positive and Negative Symptoms Scale (PANSS), functioning measures (e.g. Global Assessment of Functioning) and schizophrenia specific measures of HRQL (e.g. QLS). • Some evidence that EQ-5D reflected depression rather than other symptoms.
  17. 17. 17 Bipolar disorder EQ-5D SF-36 √√ √√√x√√ Known group: Severity Know group: Case- x control Convergent validity Responsiveness SF-12 SF-6D √ - √√√√√√√ √√x - √x√ x √ √ √ √ √ √ √ √ √ x √√√√x - -
  18. 18. 18 Bipolar Disorder • Majority evidence on the SF-36 • Crude KGV • CV good • EQ-5D (4 studies) • Mixed results for CV and R • EQ-VAS performed better than the index. • Properties demonstrated for depression but not mania 27/11/2013 © The University of Sheffield
  19. 19. Overview of quantitative evidence 19  EQ-5D and SF-36 achieved adequate levels of performance in depression and to some extent anxiety and personality disorder (but limited evidence)  Results were more mixed in schizophrenia and bi-polar with a suggestion that results in some studies may reflect differences in depression  These findings were supported by analyses of further data sets undertaken as part of the research But:  Need more evidence using better indicators for testing validity and responsiveness is required.  Need to assess content validity using qualitative research to better understand the findings 27/11/2013 © The University of Sheffield
  20. 20. Qualitative evidence
  21. 21. 21 Why Qualitative Research? • Content and face validity • Perspective of individual important • Required by FDA for measures being used to support labelling claims (and generally good practice) 27/11/2013 © The University of Sheffield
  22. 22. 22 Two research studies • Systematic Review and synthesis of qualitative research • Primary research – interviews of service users 27/11/2013 © The University of Sheffield
  23. 23. 23 Study 1: Systematic review • 13 studies • Canada/UK/Sweden/USA/Australia/NZ • Occupational Therapy (5); Nursing (4); Psychology (2); Psychiatry (1) Social worker (1) • Schizophrenia (3); Bi-polar (3); Panic Disorder (1); Mixed (6) • Framework analysis used to identify common and variable patterns of themes • Limitations: focus on severe mental health schizophrenia/bi-polar 27/11/2013 © The University of Sheffield
  24. 24. 24 Study 2: Service User Interviews • 19 interviews • Broad range diagnosis and severity • Severe and enduring (CMHT x 2) -Schizophrenia; Bipolar; Personality Disorder; -Severe Depression/Anxiety; PTSD • Mild to Moderate (IAPT) Anxiety/Depression • Themes from review made up initial themes of framework • Limitations – gatekeepers/diagnostic range 27/11/2013 © The University of Sheffield
  25. 25. 25 Conceptual Difficulties • Quality of life as a positive concept • Responses tend to focus on negative aspects of condition including clinical symptoms • Symptoms being separate from QoL • Review - overlap of QoL with other concepts • Recovery/lived experience • Overlap of domains within QoL • e.g. Self-stigmatization (Belonging or self perception) 27/11/2013 © The University of Sheffield
  26. 26. 26 Findings – QoL domains Autonomy Well-Being Ill-Being Self Perception Belonging Activity Hope Physical health 27/11/2013 © The University of Sheffield
  27. 27. 27 Well-Being – Ill-Being Positive – adds quality Negative – takes quality away Overall sense of well-being Overall feelings of distress Feeling calm/relaxed Anxiety/worry/fear Feeling safe Low mood/boredom Enjoyment Lack of energy/feeling tired (Happiness) Lack of concentration (If there was one thing you could change to improve your life, what would it be) I would lose this anxiety that I seem to be constantly carrying with me. I don’t know why, I don’t know how, although I can remember when me and my sister spoke she says, at that time, they always used to say I was very highly strung, it’s a term that you don’t really hear now, but I was always a bit like that when I was very young anyway and I seem to have carried this anxiety and nervousness with me ever since (IAPT Panic attacks)
  28. 28. 28 Physical Health • Physical health problems affect mental health • Mental health affects physical health • Presence of both makes life difficult to cope with it feels physical as well as mental …. my body aches and like I think I just become really tense and that is what makes my body ache and I feel like erm I feel like my chest is being crushed and erm I can’t breath and things like that and erm I just want to be asleep all the time to escape but I can’t sleep (CMHT Severe depression/Socio-affective PD) 27/11/2013 © The University of Sheffield
  29. 29. 29 Self Perception Positive – adds quality Negative – takes quality away Coherent sense of self Lack of self identity Positive self identity Low self esteem Self acceptance Lack of confidence Feelings of failure/uselessness The worse part is the lack of self worth and having to accept that one is disabled one has a stigma which stops you from doing anything else really other than being worthless (CMHT Schizophrenia) 27/11/2013 © The University of Sheffield
  30. 30. 30 Relationships and Belonging Positive – adds quality Negative – takes quality away Accepted and understood Lack of understanding Support Stigma Companionship/camraderie Rejection/exclusion Love and affection Loneliness/isolation Trust Abuse Feeling part of society Feeling alien to society I have feelings of erm not belonging to the human race, like, I feel very-, it’s not an outcast, I just don’t feel a connection erm I don’t know how else to describe that, it’s being like an alien, that’s the only way I can describe that, and I know that sounds weird but that’s the only way I can describe the feeling of it, I don’t feel akin with anybody, I am very guarded and things like that …I would just like to be supported by other people all working to a common cause err helping other people, that’s all I have ever wanted to do (CMHT Severe Depression/Anxiety)
  31. 31. 31 Autonomy Positive – adds quality Negative – takes quality away Choice Lack of choice Opportunity Lack of opportunity Control/self determination Control (excessive) Coping Not coping - overwhelmed I seem a bit of a control freak, I want everything to be worked out before I decide to do a certain thing, you know, I want everything to be fairly straightforward and I mean, you can’t, in a way, you can’t live life like that, and yet I still want to live life like that, do you know what I mean? … … it’s about the stress, erm having faith or taking this, stepping out of your comfort zone, whatever you want to call it, yeah (Schizophrenia) 27/11/2013 © The University of Sheffield
  32. 32. 32 Activity Positive – adds quality Negative – takes quality away Enjoyment – well being Stress (exacerbates symptoms) Rewarding - esteem Failure - esteem Meaning and purpose Belonging – social activity Structure Distraction from problems I went on a year’s course at engineering and I was absolutely scared about going on that, but I did it and I did it, you know, quite well … and at that time, I was really happy in my life and I thought well I’ve done something, I’ve achieved something here doing this … I think that’s because I had the drive and a purpose of getting up and going out every morning and doing what, you know, normal people do, sort of, you know, I got into a routine which was very good, so I was happy and more stable, I just felt a little bit more worthwhile, you know. I don’t like to feel useless really (Anxiety/panic attacks) 27/11/2013 © The University of Sheffield
  33. 33. 33 Hope and Hopelessness Positive – adds quality Negative – takes quality away Positive view of future Negative view of future Optimism Pessimism Dreams and goals Feelings of loss Thinking ahead Demoralization The one thing that I used to do a lot is not think about the future, I’d think a couple of days ahead and then not think about, you know, any further than that … cos now, one thing that’s different from when I started going through this process, is that I’m more willing to think further ahead, you know, I’m more willing to say, well in a year’s time I’d like to be at this place, before I did this I wouldn’t, but there’s still a long way to go, and still a lot of obstacles that I’ve got to overcome (Depression/trans-gender) 27/11/2013 © The University of Sheffield
  34. 34. 34 Overview of qualitative findings • Measuring QoL in mental health complex • QoL covers negative as well as positive aspects • Similar themes to ‘personal recovery’ – also service user driven: • Boardman review for DH • Connectedness, Hope and optimism, Identity, Meaning and Empowerment (CHIME) (Leamy et al, 2011) 27/11/2013 © The University of Sheffield
  35. 35. Discussion: implications
  36. 36. Content validity of EQ-5D Well-being/Ill-being Depression and anxiety - negative only, excludes calm, happiness, energy , safety etc Physical health Mobility, self care, usual activity and pain or discomfort Self-perception None Relationships and belonging Usual activities? - Excludes affection, trust, support and loneliness, stigma, abuse etc. (the quality of contact) Autonomy None Activity Usual activity (and perhaps self-care) - Nothing on stress/failure/meaning/ purpose etc. Hope and hopelessness None 27/11/2013 © The University of Sheffield 36
  37. 37. Content validity of SF-6D Well-being/Ill-being Depression and anxiety, energy - Calmness and happiness in SF-36 Physical health Physical functioning, role limitation, pain Self-perception None Relationships and belonging Social functioning - Excludes affection, trust, support and loneliness, stigma, abuse etc. (the quality of contact) Autonomy None Activity Role limitation Hope and hopelessness None - General health perception in SF-36 asks about future health expectation, but very limited 27/11/2013 © The University of Sheffield 37
  38. 38. 38 27/11/2013 © The University of Sheffield
  39. 39. 39 Adoption of SWEMWBS for PBR 27/11/2013 © The University of Sheffield
  40. 40. The Short Warwick-Edinburgh Mental Well-being Scale (S-WEMWBS) Below are some statements about feelings and thoughts. Please tick the box that best describes your experience of each over the last 2 weeks None of the time Rarely I’ve been feeling optimistic about the future 1 2 3 4 5 I’ve been feeling useful 1 2 3 4 5 I’ve been feeling relaxed 1 2 3 4 5 I’ve been dealing with problems well 1 2 3 4 5 I’ve been thinking clearly 1 2 3 4 5 I’ve been feeling close to other people 1 2 3 4 5 I’ve been able to make up my own mind about things 1 2 3 4 5 STATEMENTS Some of Often the time All of the time “Short Warwick Edinburgh Mental Well-Being Scale (SWEMWBS) © NHS Health Scotland, University of Warwick and University of Edinburgh, 2008, all rights reserved.” 27/11/2013 © The University of Sheffield 40
  41. 41. Content validity of S-WEMWBS Well-being/Ill-being Feeling relaxed, thinking clearly Physical health Self-perception Feeling useful Relationships and belonging Feeling close to other people Autonomy Make up my own mind about things Activity Dealing with problems Hope and hopelessness Feeling optimistic 27/11/2013 © The University of Sheffield 41
  42. 42. Implications for research 42 • Further testing of construct validity and responsiveness of EQ-5D and SF-6D using better indicators of HRQL • Comparative testing of SWEMWBS on populations with mental health problems • Extend qualitative research to conditions not well covered (e.g. OCD), recruit through different channels and extend to other countries and cultures • Develop a mental health specific generic preference-based measure?
  43. 43. 43 Developing a generic preferencebased mental health measure Questionnaire • Confirm dimensions • Develop items to reflect dimensions from qualitative work • Test properties of items in range of populations Scoring • Value using Time trade-off or discrete choice experiments with duration • General public and people with mental health problems 27/11/2013 © The University of Sheffield
  44. 44. 44 References: • Brazier JE, Connell J, Papaioannou D, Mukuria C, Mulhern B, O’Cathain A, Barkham M, Knapp M, Byford S, Gilbody S, Parry G. Validating generic preference-based measures of health in mental health populations and estimating mapping functions for widely used specific measures. Health Technology Assessment (in press) • Connell J, Brazier JE, O'Cathain A, Lloyd-Jones M, Paisley S. Quality of life of people with mental health problems: a synthesis of qualitative research Health and Quality of Life Outcomes 2012, 10:138. • Papaionnou D, Brazier JE, Parry G. How to measure quality of life for cost effectiveness analyses in personality disorders? A systematic review. Journal of Personality Disorder 2013; 27(3):383-401 • Papaionnou D, Brazier J, Parry G. How valid and responsive are generic health status measures, such as the EQ-5D and SF-36, in schizophrenia? A systematic review. Value in Health 2011, 14(6):907-920. 27/11/2013 © The University of Sheffield