Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Noelle E. Carlozzi, Ph.D.
Paulsen, JS, Stout, J, Nance, MA, Perlmutter, JS, Ross, CA, Goodnight, SM, Miner, JA, Dayalu, P,...
WHAT IS A PATIENT REPORTED OUTCOME (PRO)
MEASURE?
PRO MEASURES COMMONLY ASSESS
HEALTH-RELATED QUALITY OF LIFE
PHYSICAL EMOTIONAL SOCIAL
A PRO MEASUREMENT SYSTEM THAT IS
SPECIFIC TO HD
Carlozzi, N.E., et al. (2016):
• HDQLIFE: Development and assessment of he...
BACKGROUND
• PROs should be both reliable (i.e., repeatable) and valid (i.e.,
measure what was intended).
• Symptom progre...
HYPOTHESES
• Items on PROs should not exhibit item bias.
• PROs should demonstrate moderate relationships with observer
re...
SAMPLE CHARACTERISTICS
• N = 506 participants
• 38.8% prodromal
• 39.0% early-stage HD
• 22.5% late-stage HD
• Average age...
MEASURES
HDQLIFE PROs:
Chorea (34 items)
Speech Difficulties (27 items)
Swallowing Difficulties (16 items)
UHDRS clinician...
ITEM BIAS
• Item bias was assessed using differential item functioning (DIF)
both across HD stage and relative to cognitiv...
ITEM BIAS RESULTS
• Most items were free from DIF
• Chorea: no items consistently exhibited DIF; when DIF was present it w...
RELATIONSHIPS BETWEEN PROS AND
CLINICIAN-RATED SYMPTOMS
• Pearson correlations between self-report and associated
clinicia...
RELATIONSHIPS BETWEEN SELF-REPORT AND
CLINICIAN RATINGS
PRO Measures
Prodromal Early-HD Late-HD Combined
Composite
Scores
...
PSYCHOMETRIC RELIABILITY OF PROS
• Three separate sets of regression models were examined to
determine the psychometric re...
Table 4: Estimated PRO Reliabilities by HD Stage
Measure Prodromal Early Late
HDQLIFE Chorea 0.98 0.86 0.72
HDQLIFE Speech...
SIMPLE REGRESSION MODELS
Table 3: Simple Regression Models
PRO Measure beta R2
t
Chorea 0.97 0.94 83.22
Speech 0.92 0.92 7...
MODEL FIT RESULTS FOR PRO MEASURES
Model DF AIC BIC Chi-Square
HDQLIFE Chorea
Simple Regression 3 2291.75 2303.90
Heteroge...
-20
-15
-10
-5
0
5
10
15
0 50 100 150 200 250 300 350 400 450
Residual
Cognition Total Scores
Residual Plot: Chorea
Early ...
CUTOFF SCORES FOR RELIABILITY
Total Cognition Scores
(SDMT + Stroop)
PRO Reliability > 0.7
“adequate”
Reliability > 0.8
“g...
SUMMARY & CONCLUSIONS
• As HD progresses and cognition declines, high error variance and low
reliability can negatively af...
QUESTIONS?
ACKNOWLEDGEMENTS
Funding:
• National Institute of Neurological Disorders and Stroke: R01NS077946,
R03NS065194, R01NS040068...
CONTACT US
• Phone: 734 – 764 - 0644
• E-mail: PMR-HDStudy@med.umich.edu
• https://sites.google.com/site/codaresearch/
Upcoming SlideShare
Loading in …5
×

Understanding patient-reported outcome measures in Huntington disease: at what point is cognitive impairment related to poor measurement reliability

1,535 views

Published on

Understanding patient-reported outcome measures in Huntington disease: at what point is cognitive impairment related to poor measurement reliability, presented by Nicole Carlozo, PhD, University of Michigan, HSG 2016

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

Understanding patient-reported outcome measures in Huntington disease: at what point is cognitive impairment related to poor measurement reliability

  1. 1. Noelle E. Carlozzi, Ph.D. Paulsen, JS, Stout, J, Nance, MA, Perlmutter, JS, Ross, CA, Goodnight, SM, Miner, JA, Dayalu, P, McCormack, MK, Quaid, KA, Perlman, S, Hahn, EA, Lai, J-S, Downing, NR, Kratz, AL, Barton, SK, Ready, R, Frank, S, Cella, D, Gershon, RC, Shoulson, I, Marin, H, Geschwind, MD, Rao, SM, & Schilling, SG November 4, 2016 UNDERSTANDING PATIENT-REPORTED OUTCOME MEASURES IN HUNTINGTON DISEASE: AT WHAT POINT IS COGNITIVE IMPAIRMENT RELATED TO POOR MEASUREMENT RELIABILITY?
  2. 2. WHAT IS A PATIENT REPORTED OUTCOME (PRO) MEASURE?
  3. 3. PRO MEASURES COMMONLY ASSESS HEALTH-RELATED QUALITY OF LIFE PHYSICAL EMOTIONAL SOCIAL
  4. 4. A PRO MEASUREMENT SYSTEM THAT IS SPECIFIC TO HD Carlozzi, N.E., et al. (2016): • HDQLIFE: Development and assessment of health-related quality of life in Huntington disease (HD). Quality of Life Research, 25(10), 2441-2455. • New measures to capture end of life concerns in Huntington disease: Meaning and Purpose and Concern with Death and Dying from HDQLIFE (a patient reported outcomes measurement system). Quality of Life Research, 25(10), 2403-2415. • The development of a new computer adaptive test to evaluate chorea in Huntington Disease: HDQLIFE Chorea. Quality of Life Research, 25(10), 2429-2439. • HDQLIFE: The development of two new computer adaptive tests for use in Huntington disease, Speech Difficulties and Swallowing Difficulties. Quality of Life Research, 25(10), 2417-2427.
  5. 5. BACKGROUND • PROs should be both reliable (i.e., repeatable) and valid (i.e., measure what was intended). • Symptom progression in HD often includes cognitive decline, especially in the later stages. • Can we determine when cognitive impairment may preclude PRO responding (i.e., large error variance and low reliability)?
  6. 6. HYPOTHESES • Items on PROs should not exhibit item bias. • PROs should demonstrate moderate relationships with observer reports of similar constructs. • The variability and reliability for PROs should meet minimally acceptable standards.
  7. 7. SAMPLE CHARACTERISTICS • N = 506 participants • 38.8% prodromal • 39.0% early-stage HD • 22.5% late-stage HD • Average age 49.0 (SD = 13.2) • 58.5% female • 95.3% Caucasian
  8. 8. MEASURES HDQLIFE PROs: Chorea (34 items) Speech Difficulties (27 items) Swallowing Difficulties (16 items) UHDRS clinician-rated assessments: Total Functional Capacity Total Motor Score Stroop (Color Naming, Word Reading, and Interference) Symbol Digit Modalities Test
  9. 9. ITEM BIAS • Item bias was assessed using differential item functioning (DIF) both across HD stage and relative to cognitive performance. • In general, items should not exhibit DIF. Some degree of DIF is considered acceptable. Some types of DIF are more problematic than others.
  10. 10. ITEM BIAS RESULTS • Most items were free from DIF • Chorea: no items consistently exhibited DIF; when DIF was present it was minimal • Speech: 5 items exhibited DIF across cognitive tests and staging; no items consistently demonstrated non-uniform DIF • Swallowing: 4 items consistently exhibited DIF across cognitive tests and staging; no items consistently exhibited non-uniform DIF • Overall DIF was minimal
  11. 11. RELATIONSHIPS BETWEEN PROS AND CLINICIAN-RATED SYMPTOMS • Pearson correlations between self-report and associated clinician ratings were examined. • We expect moderate agreement between respondents (r’s between 0.40 and 0.60).
  12. 12. RELATIONSHIPS BETWEEN SELF-REPORT AND CLINICIAN RATINGS PRO Measures Prodromal Early-HD Late-HD Combined Composite Scores Chorea Speech Swallowing Chorea Speech Swallowing Chorea Speech Swallowing Chorea Speech Swallowing Clinician-rated Total Motor Score .40 .22 .31 .31 .21 .27 .22 .28 .07 .66 .54 .50
  13. 13. PSYCHOMETRIC RELIABILITY OF PROS • Three separate sets of regression models were examined to determine the psychometric reliability of the PROs 1. A simple linear regression model: split half reliabilities were compared 2. A heterogeneous variance model for HD stage: model was fit with different variances for each HD stage 3. A heterogeneous variance model for cognition: model was fit for variance in total cognition. Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 3 Item 5Item 4 Item 1 Item 2 Item 6 Split-Half Correlations .91
  14. 14. Table 4: Estimated PRO Reliabilities by HD Stage Measure Prodromal Early Late HDQLIFE Chorea 0.98 0.86 0.72 HDQLIFE Speech 0.98 0.85 0.69 HDQLIFE Swallowing 0.95 0.79 0.71 • Reliability standards: • ˂ 0.70 = unacceptable • 0.70 - 0.79 = acceptable • 0.80 – 0.89 = good • ≥ 0.90 = excellent
  15. 15. SIMPLE REGRESSION MODELS Table 3: Simple Regression Models PRO Measure beta R2 t Chorea 0.97 0.94 83.22 Speech 0.92 0.92 70.93 Swallowing 1.24 0.84 49.18 Note. all p <.0001
  16. 16. MODEL FIT RESULTS FOR PRO MEASURES Model DF AIC BIC Chi-Square HDQLIFE Chorea Simple Regression 3 2291.75 2303.90 Heterogeneous - Cognition 4 2112.89 2129.09 180.86* Heterogeneous - HD Stage 5 2096.93 2117.18 198.81* HDQLIFE Speech Difficulties Simple Regression 3 2330.83 2343.12 Heterogeneous -Cognition 4 2239.79 2256.19 93.03* Heterogeneous - HD Stage 5 2206.81 2227.30 128.02* HDQLIFE Swallowing Difficulties Simple Regression 3 2136.32 2148.63 Heterogeneous -Cognition 4 1993.47 2009.88 144.85* Heterogeneous - HD Stage 5 2038.47 2058.94 101.85* Note. * p < .0001 The heterogeneous models provide a better fit than the simple regression model
  17. 17. -20 -15 -10 -5 0 5 10 15 0 50 100 150 200 250 300 350 400 450 Residual Cognition Total Scores Residual Plot: Chorea Early HDHD Late HDHD variability variability variability Prodromal HDHD Total Cognition Scores
  18. 18. CUTOFF SCORES FOR RELIABILITY Total Cognition Scores (SDMT + Stroop) PRO Reliability > 0.7 “adequate” Reliability > 0.8 “good” Chorea <77 < 136 Speech N/A <109 Swallowing <134 <179 Note. M = 144.56 (SD = 77.31) for Total Cognition Scores for the combined sample
  19. 19. SUMMARY & CONCLUSIONS • As HD progresses and cognition declines, high error variance and low reliability can negatively affected the psychometric properties of PRO measures. • Although minimal standards for reliability on PRO measures was met for all HD groups, clinical cutoffs on cognitive tests can be used to maximize PRO reliability. • In cases where cognitive scores do not meet critical cutoffs, PRO measures should only be considered in conjunction with other assessments. • Recommended clinical cutoffs differed for different measures. This suggests that cognitive complexity may vary across PRO measures.
  20. 20. QUESTIONS?
  21. 21. ACKNOWLEDGEMENTS Funding: • National Institute of Neurological Disorders and Stroke: R01NS077946, R03NS065194, R01NS040068, & R01NS077946 • National Center for Advancing Translational Sciences UL1TR000433 • CHDI Foundation • HD Center Grant from the NJ Department of Health and Senior Services HDQLIFE Site Investigators and Coordinators: Praveen Dayalu, Amy Austin (University of Michigan, Ann Arbor, MI); Courtney Shadrick, Amanda Miller (University of Iowa, Iowa City, IA); Kimberly Quaid, Melissa Wesson (Indiana University, Indianapolis, IN); Christopher Ross, Gregory Churchill, Mary Jane Ong (Johns Hopkins University, Baltimore, MD); Susan Perlman, Brian Clemente (University of California -Los Angeles, Los Angeles, CA); Michael McCormack, Humberto Marin, Allison Dicke (Rutgers University, Piscataway, NJ); Joel Perlmutter, Stacey Barton, Shineeka Smith (Washington University, St. Louis, MO); Martha Nance, Pat Ede (Struthers Parkinson’s Center); Anwar Ahmed, Christine Reece, Lyla Mourany (Cleveland Clinic Foundation, Cleveland, OH); Michael Geschwind, Joseph Winer (University of California – San Francisco, San Francisco, CA); David Cella, Richard Gershon, Elizabeth Hahn, Jin-Shei Lai (Northwestern University – Chicago, IL)
  22. 22. CONTACT US • Phone: 734 – 764 - 0644 • E-mail: PMR-HDStudy@med.umich.edu • https://sites.google.com/site/codaresearch/

×