1. PROS IN CLINICAL CARE
Paul K. Crane, MD MPH
Associate Professor, Department of Medicine, School of Medicine
Adjunct Associate Professor, Department of Health Services, School of Public Health
University ofWashington
pcrane@uw.edu

2. Outline
 How did I come to this topic?
 Intro to PROMIS
 How did my wife come to this topic?
 Intro to CNICS
 PROMIS 2 research on depression
 PROMIS 2 network / Clinical Practice
Subcommittee
 Future forecast

3. Outline
 How did I come to this topic?
 Intro to PROMIS
 How did my wife come to this topic?
 Intro to CNICS
 PROMIS 2 research on depression
 PROMIS 2 network / Clinical Practice
Subcommittee
 Future forecast

4. My background
 UW med school (1997)
 Internal medicine internship (UW) and
residency (Barnes-Jewish) 1997-2000
 Hospitalist and Health Behavior Research
fellow 2000-01 (Washington U, St. Louis)
 General Internal Medicine Fellow, MPH, UW
(2001-03)

5. Interest in measurement
 Decided as a Health Behavior fellow to study
diabetes, depression, and health related
quality of life
 Determined that to know anything about
that topic, had to know something about how
to measure HRQL
 Lots of diabetes-specific HRQL scales, all of which
made claims that I did not understand
 And I thought I knew how to read the medical
literature!

6. Faries et al. paper
 A paper on the Hamilton Depression Rating
Scale looked at responsiveness of each item,
which they defined as differences between
placebo and active treatments with their drug
 Certainly didn’t like that way of defining things
 But it’s the item, not the scale!
Faries D et al.The responsiveness of the Hamilton Depression Rating
Scale. Journal of Psychiatric Research 2000; 34: 3-10.

7. GIM Fellowship: Psychometrics
 Worked with Gerald van Belle during fellowship
 We taught each other modern psychometrics
 Hambleton et al. (1991): Fundamentals of Item
ResponseTheory
 Embretson and Reise (2000): Item ResponseTheory for
Psychologists
 McDonald (1999):TestTheory: a Unified Treatment
 Would add to this list:
 Wainer et al. (2007): Testlet ResponseTheory and its
Applications.

8. Fellowship psychometrics
research - 1
 K08 proposal, “Improving cognitive tests with
modern psychometrics” – Alzheimer’s disease
specific 3-year K award
 Worked with item-level cognitive and PRO data
 Dan Mungas at UC Davis
 PROMIS I application on self-reported cognition
 Not discussed – “cognition is not important to HRQL”
 Became aware of a second UW PROMIS I
proposal from Dagmar Amtmann

9. Psychometrics Research – 2
 Dagmar’s project was funded, and she was
happy to have me involved as I wished during
my K award.
 So, I went.

10. PROMIS 1
 Large project (7 U01 projects, 1 StatisticalCoordinating Center,
each with an NIH Project Officer)
 One Danish physician (Jakob Bjorner) involved with one of the
projects
 No other practicing clinicians with modern psychometrics expertise
 No projects really integrated with clinical care
 I felt the need to speak up!
 Network structure; first set of domains built by the Network
 Depression;
 Anxiety;
 Alcohol abuse;
 Anger;
 Physical function;
 Fatigue impact / experience;
 Social role performance /satisfaction;
 Pain interference / quality / behavior

11. Pilkonis PA et al. Item banks for measuring emotional distress from the Patient-ReportedOutcomes Measurement
Information System (PROMIS®): depression, anxiety, and anger.Assessment 2011; 18: 263-283.

14. Some PROMIS strengths
 Extraordinary amount of attention, with
reasonable choices made
 Item QC led to items that are relatively easy to
read, simple to interpret, similarly worded, etc.
 Domain-to-domain look and feel is consistent
 Scores across the domains look similar
 Direction based on name of domain
 Scaled so 50 is mean of US, +/- 10 is 1 SD
 Attention to intellectual property
 Increasingly important consideration
 Proprietary items = $, risk for lawyer involvement

15. PROMIS Product: Short forms
 Brief group of items
 “Developed … based on simulations of CAT
results, item information, and item content”
 Candidate items identified based on psychometric
characteristics, and then reviewed by content
experts (Pilkonis 2011)

16. PROMIS product: CAT
 Efficient and brilliant use of computers
 You already know the CAT algorithm: pick a
number (“binary search”)
 Need an item bank, a scoring algorithm, and a
stopping rule
 Stopping rule can be composite
 Result: precise-enough scores for a domain after
a very few items (like 4 or 5)
 If IRT assumptions are appropriate, it’s very slick!
 Assumption: all items are equally useful

17. PROMIS 2
 1 coordinating center becomes 3 (technology
center, statistical center, network center
 12 PROMIS projects
 Structure different: no network projects (or
we’re all network projects)
 And one of the 12 PROMIS projects was ours!
 We’ll come back to this in a bit.

18. Outline
 How did I come to this topic?
 Intro to PROMIS
 How did my wife come to this topic?
 Intro to CNICS
 PROMIS 2 research on depression
 PROMIS 2 network / Clinical Practice
Subcommittee
 Future forecast

19. Heidi Crane, MD MPH
 UW undergrad, UW med school, Barnes-Jewish
Medicine Residency, UW ID fellowship
 K23 on body morphology disorder among people
with HIV
 Self-reported body morphology changes
 Tablets in the waiting room for people with long
waits ahead of them
 Other PRO domains on the assessment
 Crane HM et al. Routine collection of patient-reported
outcomes in an HIV clinic setting: the first 100
patients. Current HIV Research 2007; 5: 109-118)

20. Chart reviews for same-day
visits
 Depression: not identified by providers
 Substance use: not identified by providers
 Poor adherence: not identified by providers
 And alarmingly high prevalence of patients who told
the computer they were having problems, whose
providers documented “No problems with
adherence,” “Perfect adherence,” “Taking all meds”
 Reviewed these findings with clinic leadership
 Imperative to measure these things and make
sure providers have access to the findings at
the point of care

21. Integrating into routine care:
not trivial Instead of patients with long anticipated wait times
for research protocol, change to all patients
 Except not all patients; super frequent fliers for wound
care excluded
 So who?
 Offset the clinical day, so patients scheduled to
interact with tablet 20 minutes before provider
scheduled in the room
 Front desk implications, rooming staff / vitals
implications, …
 In short: a clinical change with an impact on patient
flow
 Importance of Clinic Leadership buy-in essential
 Other elements of case on succeeding slides

22. Source: http://www.improvingchroniccare.org/index.php?p=Chronic+Care+Model&s=124

23. Extension of Chronic Care Model
 clinical information system, delivery
system design, decision support
 “Listening to the patient’s voice in a systematic
standardized way”
 Delivering data to providers using 21st century
informatics tools
 Ultimate goals: Tailored, personalized,
evidence-based recommendations for clinical
actions

24. Patient-provider
relationship
 Devote time during the clinic session to elements
both patients and providers deem important
 Clarify patient concerns
 Patients more honest to CASI than they are to a
provider, less social desirability bias. More likely
to report to CASI poor adherence, substance
abuse, depression, risk behavior than to provider
 Fredericksen R et al. Integrating a web-based, patient-
administered assessment into primary care for HIV-
infected adults. Journal of AIDS and HIV Research

25. Figure 1A. Common situation in routine clinical care
Figure 1B. Situation with valid adherence measurement incorporated into clinical care
Poor
adherence
Adherence
not assessed
System not
aware No
intervention
Continued poor
adherence
Poor HIV
outcomes
Structural barriers
Provider barriers
Patient barriers to
assessment
Patient factors
*substance
abuse
*mental illness
*other
Poor
adherence
Patient factors
*substance
abuse
*mental illness
*other
Adherence
assessed
Intervention
Poor
adherence
Good
adherence
Better HIV
outcomes
Adherence
assessed
System aware

26. CNICS
 Madison Clinic part of UW Centers for AIDS
Research (CFAR)
 CFARs banded together to form CNICS, the
CFAR Network of Integrated Clinical Systems
 Initial partnership with the 1919 Clinic at
University of Alabama at Birmingham

27. http://www.positivethebook.com/

28. PROs at UAB
 Paper free clinic
 Never a feedback form on paper
 Touch screens provider room with feedback; monitors
in exam rooms
 Aspects of PROs at every visit (ROS)
 Patient flow is different – patients in a physical
“circuit” around the clinic
 Patient flow at Madison was much more waiting room
-> vitals -> back to waiting room -> exam room
 Addressed the “circuit” with “ticket numbers” so could
pick up the PRO Assessment where they left off
 Addressed these differences with personal visits

29. PROs in CNICS
 Extension to Fenway (Boston), UC San Diego
(large Latino population; Spanish essential),
others (UCSF, Hopkins, UNC, CaseWestern)
 Different EMR systems, different leadership / clinic
cultures, different patient groups, different patient
flow
 All PRO collection is local!

30. Our PROMIS 2 proposal
 CNICS infrastructure now with 30,000 PRO
Assessments from 8 CFARs around the country
 Extensive harmonized clinical data
 Uniquely situated in clinical care for PLWH
 Aim 1: PROMIS domains in clinical care
 Ask patients which domains they think are most
important
 Focus on groups of items patients see, not the whole bank
 Simulated CAT, PROMIS short form
 As of April 2012, 809 studies set up in Assessment Center;
only 2 administered an entire bank
 Other parts: 2 new domains; RCT on adherence;
active involvement with the Network

31. Outline
 How did I come to this topic?
 Intro to PROMIS
 How did my wife come to this topic?
 Intro to CNICS
 PROMIS 2 research on depression
 PROMIS 2 network / Clinical Practice
Subcommittee
 Future forecast

32. Depression domain
 Simulated CAT: 5 items for severe, moderate,
and mild depression
 Compared with short form content
Short Form CAT: Mild CAT: Moderate, Severe
Sad Sad Sad
Unhappy Unhappy Unhappy
Depressed Depressed Depressed
Helpless Helpless
Hopeless
Worthless
Like a failure
Nothing to look forward to
Discouraged about future Discouraged about future
Disappointed in self

33. Analyses of depression domain
 Qualitative analyses
 97 PLWH in 4 cities, stratified by depression severity
 “Repetitive,” “Redundant,” “Mas o menos lo mismo”
(both the Short Form and each of the simulated CATs)
 “What would a provider need to know to take great
care of a person with HIV?
 SUICIDALITY
 Providers: Distinct preference for PHQ-9 content
 Patients: Distinct preference for PHQ-9 content
 Quantitative analyses
 1299 PLWH in 4 cities
 PROMIS and PHQ-9 work fine, nothing to distinguish
either one

34. WSCD (“What Should CNICS Do”)?
• Administer PHQ-9, score using PROMIS item
parameters
 Gibbons LE et al. Migrating from a legacy fixed-
format measure to CAT administration: calibrating
the PHQ-9 to the PROMIS depression measures. Qual
Life Res 2011; 1349-1357.
 Best of both worlds
 Content providers and patients want, scores on
PROMIS metric, brief enough

35. Outline
 How did I come to this topic?
 Intro to PROMIS
 How did my wife come to this topic?
 Intro to CNICS
 PROMIS 2 research on depression
 PROMIS 2 network / Clinical Practice
Subcommittee
 Future forecast

36. PROMIS Clinical Practice Subcommittee
 Growing demand for PROMIS scales from clinicians
 Groundswell of understanding of need for PROMIS
focus on this issue
 “Sorry, I have to go, there goes my group, and I am their
leader!”
 Big initiatives to date: two papers (Broderick et al.
2013; Jensen et al. to be re-submitted) and EPIC
 Steering Committee vote:Work with EPIC to ensure
PROMIS content included
 Extensive discussion of which domains and
calibrations
 Initial build: short forms, scored using total scores
(not IRT scoring)
 Hopefully CATs in next build

37. PROMIS 2 Network
 Large number of additional domains being developed
(sexual function, self efficacy, substance use, GI symptoms)
 Bigger efforts in pediatric settings (CincinnatiChildren’s,
CHOP, UNC)
 Other initiatives: cancer, instantaneous assessment,
broaden physical functioning to address ceiling, …
 Less cohesive than PROMIS 1 (by design) in terms of
projects
 PROMIS standards document, domain framework, PROMIS
at NIH Clinical Center, PROMIS in National Children’s Study,
PROMIS in DoD care settings, ….
 3 Coordinating Centers transitioning to ??? In future
 PROMIS, NIHToolbox, NeuroQOL

38. RFA-CA-13-008
 The purpose … is to support the creation of a research resource
infrastructure for the administration of research investigations using person-
centered health outcomes … the Person-Centered Outcomes Research
Resource (PCORR).
 The PCORR will be expected to support the use and enhancement of the
following four measurement information systems, currently funded as
separate NIH programs:
 Patient Reported Outcomes Measurement Information System® (PROMIS®);
 the NIHToolbox for Assessment of Neurological and Behavioral Function (NIH
Toolbox);
 the Quality of Life (QOL) Outcomes in Neurological Disorders (Neuro-QOL;
 andThe Adult Sickle Cell Quality of Life Measurement Information System (ASCQ-
Mef).
 The main goal for the PCORR is to provide an integrated platform for
automated use of the four measurement information systems.
 This platform must be compatible with various modes of information collection
(including web/mobile-based entry, non-digital paper source data, and others).
 The PCORR platform must also be designed to allow resource users (i.e.,
external researchers and clinicians unaffiliated with the resource) to access
and use any of the four systems together or in isolation and tailor use to
meet the specific study needs…

39. PCORI funding
 Not surprising PCORI is interested in PROs
 Their default is that patients are the experts
 Atlanta meeting last fall on integrating PROs
in EHRs
 PROMIS measurement RFP now
 Additional input on PCORI priorities sought
 Additional funding initiatives likely
 Network-ness

40. Outline
 How did I come to this topic?
 Intro to PROMIS
 How did my wife come to this topic?
 Intro to CNICS
 PROMIS 2 research on depression
 PROMIS 2 network / Clinical Practice
Subcommittee
 Future forecast

41. “Prediction is very difficult,
especially about the future”
(N Bohr / Y Berra)
 Growing demand for PROs in clinical care
 ACOs, PCORI, Quality measures, IOM, Meaningful
use….
 Technological issues much less of a barrier
 Ubiquitous tablets and iPhones
 Initiatives such as PROMIS developing a lot of
content
 Where should this head?

42. 42

43. University of Utah Orthopedics
 PROMIS colleague Nan Rothrock got me
connected with Orthopedics faculty at U of
Utah
 Discussion with Dr. Darrel Brodke
 Chair interested in PRO collection x years
 ODI, NDI, SF-36, PROMIS PF, EQ5D
 Floor effects of ODI and NDI
 Data warehouse, also scores at point of care

44. Score interpretability
 Clinicians are not innumerate people!
 mmHg, HCT, chemistries, creatinine, saturations,
SNPs, omics, MRI physics…
 How do we get used to all these numbers and
different scales? The old fashioned way – we
use them!

45. Black box of score production
 Modern psychometrics, confirmatory factor
analysis, item response theory, graded response
model, polytomous data, computerized adaptive
testing, etc.
 There’s a whole science in there!
 But there is a whole science in producing the
creatinine value we use clinically too, and I don’t know
what’s in that black box
 I don’t think one needs expertise inside the black
box to use the output from the black box to take
great care of patients

46. Future will be longitudinal
 Power will come from integrating PRO data
alongside other clinical data
 Conceptualize PRO collection as an extension
of history taking
 Quantified history?
 Not the end of the discussion but a launching
point
 Launch from a deeper place than “So how’s your
depression been doing?”

47. Imagine this visit…
20
30
60
75
0
50
100
10
30
50
70
90
-18 -12 -6 0
SSRIdose(mg)
Depressionscore(mean50,SD10)
Time (months)

48. Next things to tackle
 One size fits all
 Great place to start
 Heterogeneity across patients within a clinical
setting
 Primary care may be the hardest
 Critical need to value patients’ time – can’t
possibly collect everything on everyone
 Measurement prior to visits vs. web-based
 More and more people are connected
 Critically important to reach those who are not!

49. Lessons learned
 Patient care is local
 Stakeholder buy-in is critical for clinical change to
survive
 Patients appreciate being asked
 The only way to study the data is to have the data
 Old questions of whether what we do makes a difference
 21st century technology to address those questions
 Personalized care is not just omics, it’s got a person
at the center
 Patient-provider relationship at the center of the Chronic
Care Model makes a ton of sense!

50. Too many great colleagues to thank
 Thanks for the invitation!
 Thanks to Joan Broderick and other PROMIS Clinical Practice
Subcommittee authors on the eGEMS paper, which caught your
eye and led to my talk today
 UW/MadisonClinic Colleagues
 UW PROMIS Colleagues
 CNICS investigators
 DagmarAmtmann and PROMIS 1
 Gerald van Belle, Dan Mungas, Eric Larson, EdWagner for
mentorship
 My local shop: Laura Gibbons, Shubhabrata
Mukherjee, Elizabeth Sanders
 Funding from NIH
 Patients