Based off of project findings, this presentation highlights the process of identifying KRIs for site quality for digestive disease studies and demonstrates the practical application of surrogate KRIs in risk-based monitoring.

1. 1
Therapeutic KRIs: Digestive Disease
Duncan Hall
CEO @TRI_OPRA_RBM

2. 2
The views and opinions expressed in the following
PowerPoint slides are those of the individual presenter
and should not be attributed to Drug Information
Association, Inc. (“DIA”), its directors, officers,
employees, volunteers, members, chapters, councils,
Communities or affiliates, or any organization with
which the presenter is employed or affiliated.
These PowerPoint slides are the intellectual property of
the individual presenter and are protected under the
copyright laws of the United States of America and
other countries. Used by permission. All rights
reserved. Drug Information Association, Drug
Information Association Inc., DIA and DIA logo are
registered trademarks. All other trademarks are the
property of their respective owners.
Disclaimer

3. 3
Illustrate the process of identifying KRIs for site
quality for digestive disease studies.
Share the results of the investigation into the
definition of KRIs in digestive disease.
Demonstrate the practical application of
surrogate KRIs in RBM.
Learning Objectives

4. 4
To determine if there is correlation in the
process of collecting primary clinical outcome
data and overall site quality in the area of
digestive disease.
Current state:
– 10 months into the project, 1 protocol, 21 countries,
169 sites, 647 subjects.
– Plans to extend into additional protocols in next
quarter.
Project Objectives

5. 5
You can conduct protocol risk assessment without conducting
risk based monitoring, but you cannot conduct risk based
monitoring without first conducting protocol risk assessment ...
Process of defining KRIs
Process Analysis Measurements
Data collection
process
definition
Process
Measurements
Risk
Identification
Critical Data
Critical
Process
KRI design
Protocol Risk
Assessment

6. 6
Risk Assessment and KRIs
Regulatory
Guidance
• Need to assess
protocol risks.
• Monitoring focus
should be on
protocol risk
factors.
• There should be a
relationship
between risk
assessment and
risk measurements
or risk indicators.
Standard KRIs
• Many KRIs used to detect site
quality risk will be standard.
• Standard KRIs are often
associated with safety and
performance, but they can also be
surrogate markers for quality risk
and should be monitored and
managed.
Study KRIs
• Designed to assess and monitor
the critical data and processes for
a study.
• KRIs may also change during the
course of a study, due to risk
factors changing as the study
progresses e.g. once enrollment is
complete.
Examples
AE rates
Inclusion / exclusion
deviations
Enrolment rates
Early term rates
Data Entry timeliness
Query rates
Image quality assessment
Dose changes

7. 7
Critical Data and Process
Common across all protocols
• Safety
• Compliance
• Data quality
Common across a therapy area/indication
• Endoscopy
• Mayo scoring system
Specific to the study
• Subject response to treatment

8. 8
Let’s work through one of the critical data
processes.
Endoscopy images
Process Analysis
Site
creates
test image
Site sends
test image
to central
reader
Image is
received
Image
quality is
assessed
Test image
is accepted
Site
conducts
subject
endoscopy
Site sends
image to
central
reader
Image is
received
Image is
assessed
Image is
reported
T1 Q1T2 T3
T4 T5 Q2

9. 9
Test images (pre FPI)
– T2-T1- time to test image receipt.
– T3-T1 – time to test image acceptance.
– Q1 – number of assessment cycles before test image
acceptance.
Subject images (post FPI)
– T5-T4 – time to image acceptance.
– Q2 – number of images not accepted.
Two processes:
1. Collection of critical data (Subject Images).
2. 3rd
party ‘objective’ assessment of site quality.
Intent was not to review the clinical assessment of the
image but the site process of collecting the image data.
Process Measurements

10. 10
1. Sites with a high number of non-reportable
images will present quality issues in other areas
2. Sites with a high turnaround time on image
data will present quality issues in other areas
Future assessment:
3. Sites that demonstrate higher than normal
time to test image acceptance and/or higher
than normal number of cycles to acceptance
will result in quality risks during study conduct.
Hypothesis

11. 11
Why did we select the measurements as defined?
There was a 3rd party objective and consistent assessment
of the sites process i.e. the central reader.
There was no reliance on the site entering data into EDC
therefore less delay in accessing the data.
Why?

12. 12
KRI Characteristics
What are we observing?
• Certain characteristics lend themselves to risk indication, yet others don’t
• This provides an opportunity to refine our risk indicators with each new study
Good Characteristics
• Data that is early and objective in nature
• Data that includes time:
• E,g, Time to process central data such as labs, images
/ Time for visit to CRF entry
• Data that includes quality assessments:
• E,g, central data assessments / evaluable lab samples
/ images/DM queries
• Data that involves 3rd party adjudication of a
subjective assessment
• E.g. Change in dose based on subject response
Poor Characteristics
• Data that is subjective in nature
• E,g Site Issues – very dependent on CRA process, not
consistently categorized. Useful for signal interpretation
once signal is identified, but not a good source of signal
• Data with a lot of variability, or automated data, this
may be as a result of collection process
• E.g.Total number of deviations, particularly from CTMS
as dependent on CRA process. But they do work well if
you can isolate by type, such as visit windows,
inclusion/exclusion criteria
• E.g EDC Change Hx, need to be able to remove system
generated information from the analysis
Howdowerefine?

13. 13
Data Analysis
Data processed and analysed using and RBM visualization tool
Data reviewed monthly with central monitoring team
Sites identified as showing risk with the imaging indicators were
further analysed with input from onsite monitors

14. 14
Image reporting timeliness
Time to image acceptance
plotted as % missing or reported
after 10 days of procedure
KRI Visualizations

15. 15
Hypothesis 1.
Sites with a high number of non-
reportable images will present quality
issues in other areas
Incidence rate of number of non-reportable
images currently too low to assess for statistical
significance or correlate to other site quality risk
factors
Results

16. 16
Hypothesis 2
Sites with a high turnaround time on image
data will present quality issues in other areas
Sites with higher than normal reporting times on the
images do rank in the higher risk scores for the sites,
top 26%
Corroborated by independent onsite monitor
assessments of sites with high number of significant
issues
Results

17. 17
Sites with higher than normal reporting times on images
also demonstrated elevated risk in the following areas
during the first 3-6 months
– AE rates
– Data entry timelines
– Query rates
– Query response times
– Deviations
– And correlated with another study indicator – high incidence of incorrect
subject response assessment
Results

18. 18
Results

19. 19
Interesting observation
After 6 months, the early data was removed from
the analysis, moving to a rolling 6 months.
The sites with elevated risk in months 1-3
presented a significantly reduced risk score once
the early data was removed
Sites that had started later, moved up the risk
ranking
Results

20. 20
Results
Dec 2014
Apr 2015

21. 21
Early data suggests that there is correlation between the
process of collecting image data and other site quality risk
factors, but it is too early to determine statistical
significance.
However, the removal of the early site data, indicates that
the site goes through a learning curve at the start of a
study, and it is important to identify those sites and apply
monitoring interventions early.
Conclusion

22. 22
Hypothesis 3.
Sites that demonstrate higher than normal
time to test image acceptance and/or higher
than normal number of cycles to acceptance
will result in quality risks during study
conduct
– Investigating the potential predictive nature of these 2 KRIs,
could they signify the sites propensity for lower than normal
data quality, before FPI?
– Are there other similar processes that occur prior to FPI that
could be investigated for correlation to data quality? e.g.
Contract approval process using eTMF and Start up data
Next Steps

23. 23
Duncan Hall
CEO
Triumph Research Intelligence
@TRI_OPRA_RBM
Join the conversation #DIA2015
Thank You