Risk based monitoring presentation into what Risk Based Monitoring (RBM) is all about. The value of getting it right, and the risks of getting it wrong. We look at the Triumph Research Intelligence (TRI) approach to RBM, and the TRI solution to RBM. An insight into the Future of Visual OPRA is given.

1. Risk Based Monitoring
Presenters:
Duncan Hall, CEO Triumph Research Intelligence
Tammy Finnigan, COO Triumph Research Intelligence
January 2014

2. Presentation Roadmap
1. Introductions and background
Slides: 1-3
2. What is RBM really all about?
4-6
3. The value of getting it right, the risks with getting it wrong
7-9
4. The TRI approach to RBM
5. The TRI solution to RBM
19-22
6. The Future of Visual OPRA
DH
10-18
23-27

3. Triumph Background
Triumph Consultancy Services
Formed in 2002
Purely life sciences focused consultancy
Specialists in clinical systems design and implementation,
business process definition, optimization and automation
Triumph Research Intelligence
Formed in 2013
Aim is to develop an operational platform (Visual OPRA) designed
specifically for the identification, management and reporting of
site quality
Provision of both platform and supporting services
DH

4. RBM: WHAT IS IT REALLY ABOUT?
Triumph Research Intelligence
TF

5. RBM – What Is It All About?
RBM
Risk
Monitoring
• A Forecast
• Signal
• Quantifiable
•
•
•
•
Informs
TF
Site based
Remote
Central
Statistical

6. RBM – What Are We Assessing?
BEHAVIOUR
An assessment of the sort of behaviour which is likely to result in risks to quality in one or
more of the following categories:
Reporting diligence
o What is the likelihood that an important event will be reported?
Data quality
o A measure of the variability of the data
o A measure of probability that the data is an accurate reflection of the real world
Protocol compliance
o The sites ability to comply with the protocol
o Direct or indirect
TF

7. RBM: VALUE OF GETTING IT RIGHT
Triumph Research Intelligence
DH

8. What Does Getting It Right Look Like?
SIGNAL AND NOISE
Getting it right means that we have isolated the signal from the noise. We have identified
those sites which exhibit behaviours likely to have a negative impact on quality.
Operationally
o We know which sites and even which patients to focus our monitoring effort on
o We know what behavioural aspects are impacting quality
o We know what corrective actions to take
Statistically
o We have an accurate measure of the site quality risk
o We are able to rank sites in order of risk to ensure effective prioritisation of action
Regulatory
o We have a record of our assessments of risk
o We have a record of the actions we took
DH
o We have evidence of the efficacy of our actions

9. What Does Getting It Wrong Look Like?
NOISE, LOTS OF IT
We have either failed to identify the true signal, and are simply creating noise, or we
create so much noise the signal is masked.
Operationally
o We spend a lot of time looking at data – some of it very pretty!
o We confuse performance with quality and start targeting low or high performers
o We give unclear messages to sites and see little or no sign of quality improvement
Statistically
o We look for outliers using inappropriate statistical methods (%, deviations, limits)
o We assess subjective or cleaned data sources
Regulatory
o Inadequate monitoring and oversight
o Higher chance of inspection findings
DH

10. RISK DETECTION & TECHNOLOGY
Triumph Research Intelligence
DH

11. Solution Process
Knowledge based used to
inform study design and
select M/RIs
Confirmation of
availability and quality of
data in warehouse
Visualisations generated
for each QRI, knowledge
based used to spot
patterns / behaviours
Sites ranked in terms of
risk and suggested
actions for breached
thresholds defined
through medical review
and knowledge base
Knowledge based and
quality oversight records
updated
DH
Study specific ETL built
Data consumed by
Quality Risk Engine and
optimised for
visualisation
CRAs / CDMG informed of
observations and
required actions
Intervention made with
site, and efficacy of
intervention measured
over agreed number of
assessment periods

12. Quality Risk Indicators
 Core set of Quality risk visualisations
defined
 Algorithms designed to represent QRIs on
funnel plots
 Each indicator allows data review as well as
visualisation (important for assessment of
data variability)
 Dynamic axis and funnel plot
materialisation
 Data series show / hide functionality
QUALITY RISK INDICATORS WHEN COMBINED ALLOW CREATION OF RISK SIGN
Quality risk indicators should:
•
•
•
•
•
DH
Indicate the probability that a site is a quality risk, NOT
site / CRA performance
Be based on objective data, not subjective (CRA /
operational data)
Be two dimensional to take account of data volume (error
rate is inversely proportional to volume)
Be specific and sensitive – separate signal from noise
Be easy to interpret
QRIs are not standalone, but
related and will give rise to specific
patterns of data which we call ‘risk
signatures’.
Risk signatures allow a more specific,
targeted response, allowing a more
valuable site / CRA interaction.

13. Data Volume – Dynamic Thresholds
Normalizing data by volume
0.7
0.6
Failure / 10 subjects
Combining
multiple risk
indicators per site
to create an
aggregate risk
score allows earlier
identification of
risk, with a lower
subject / visit
count
0.5
0.4
0.3
0.2
0.1
0
0
2
4
6
8
10
12
14
# subjects
Confidence increases with volume
DH
16
18
20

14. Risk Profiling – Medical History
Hovering over a site will
show a pop up with the site
identifier and data specific to
that site for easy risk
identification
DH

15. Risk Profiling – BP Number Preference
In the case of BP
readings, we are looking
for a low number
preference score. A high
score can indicate fraud
or bias. Either is a key
indication of risk
DH

16. Risk Profiling – AE Reporting
Both under and over
performing sites can be
an indications of poor
quality. These are shown
by the green and red
circled sites
DH

17. Activity Efficacy with Visual OPRA
Adverse Event Reporting Rates – Assessment period 2
Visual OPRA will allow different
assessment periods to allow historic
comparison with current state. This
view build evidence of quality
improvement or shows that
different actions need to be taken
Adverse Event Reporting Rates – Assessment period 4
Clear
indication of
reduction of
site quality
risk
DH

18. Turning Insight Into Activity
DH

19. RBM: QUALITY RISK DETECTION PROCESS
Triumph Research Intelligence
TF

20. Quality by Design
o RBM is only a piece of the puzzle
o Protocol design is critical to the success of any study
o “quality cannot be monitored into a study”
o Use risk profiles and root causes to inform:
− Protocol design process
− Site, patient and study team training
− Country specific (potentially site specific) monitoring plans
TF

21. Business Process
TF

22. RBM with Visual OPRA
Monitoring
Plan
Operational
Activities
Operational
Impact
TF
• Risk indicators (RIs) are used as a means of measuring site risk
• SDR is directed based on site risk profile
• Number of monitoring visits is reduced as less planned visits
• As RIs identify risk, likely root cause is predicted and suggested course of action
• Central monitors review subject data based on defined rules, to direct CRA to the patients that will reveal the
problems
• CRA confirms root cause and implements suggested course of action
• SDV is reduced significantly, and SDR is performed on targeted patients
• Site risk is reduced or decision is quickly taken to exclude site
• Data quality is improved as decisions can be taken quickly and are less dependent on CRA performance i.e. finding
the issues and taking the correct course of action
• Tracking root cause and actions linked to changes in site risk profile provides evidence of effectiveness and
oversight

23. RBM: THE FUTURE OF VISUAL OPRA
Triumph Research Intelligence
TF

24. Current state
Pilot
o We are currently piloting VisualOPRA with a mid-sized CRO on 2 studies
o Study A
− CV device study
− Risk analyses on 4 cuts of historic data
− Comparing the results from the risk profiles with the issues discovered during the statistical
analysis at the end of the study
− Determining if the issues discovered at the end of the study during the analysis phase could
have been identified earlier, and even prevented with early intervention
o Study B
− ‘Live’ CV study starting in Q2
− Using VisualOPRA to inform the monitoring visits
− Starting to relate risk profiles to root causes and corrective monitoring actions
o Case studies will be published in Q2 (Study A) and Q4 (Study B)
TF

25.  Further development of 4-6 QRIs planned
 Focus will be in treatment emergent QRIs
 Con meds
 Physical exams
 Visit window deviations
 QRI library by TA and geographic location will allow
appropriate application on per study basis
 Risk profiles linked to likely root causes and effective
actions
Failure / 10 subjects
Quality Risk Identification – Ongoing Development
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
# subjects
QUALITY RISK INDICATORS DEVELOPMENT
QUALITY METRICS
 Turning Quality metrics into 2
dimensional metrics as QRIs
8
6
Threshold or %
4
2
0
Site 1 Site 2 Site 3 Site 4 Site 5 Site x
Screen failures
Traditional BI tool view
TF





Missing data
Query resolution
Data entry timeliness
Screen failures
Early termination rates
20

26. Next steps
Demo
o If you would like to see VisualOPRA, contact us for a live demo, approx. 1hr
Data Analysis
o You provide the data for 1 study, we will use VisualOPRA to identify high risk sites and
compare to your study analyses
2 day workshop
o You provide the venue and we will facilitate a 2 day workshop on getting up to speed
with RBM
o The workshop will include:
−
−
−
−
−
TF
Comparison of the EMA and FDA guidance
Review of current RBM strategy (if available)
Examination of quality risk vs. performance measures
Identifying what is important to your organization
Process change required to turn RBM into reality

27. Summary
What IS RBM About?
o Proactive detection of behaviour likely to lead to poor quality
o Measurable level of risk to prioritise activity
Getting It Right
o Clear signal detected through the noise
o Probabilistic analysis if signal to rank significance
o Documented, repeatable evidence for regulatory authority
Getting It Wrong
o Lots of noise, pretty noise!
TRI Approach
o Funnel plots to normalise data and allow earlier detection
o Clear operational direction, managed through platform
Visual OPRA technology
o For purpose RBM platform, designed for study teams
o Proprietary Risk Engine and algorithms
o Dynamic reporting, review and analysis

28. Any Questions?

29. Presenters biographies and contact details
Duncan Hall, Founder and CEO
Duncan.hall@triumphconsultancy.co.uk
Duncan has over fifteen years of consultancy experience, thirteen of which have
been within clinical R&D in both CRO and Pharma businesses. Duncan started
Triumph in 2002 with the aim of building a global business which would be focused
in improving the delivery, quality and value of clinical systems. Duncan now takes a
primarily strategic role in Triumph, but still performs client delivery roles where
possible
Tammy Finnigan, COO, Triumph Consultancy Services
tammy.finnigan@triumphconsultancy.co.uk
Tammy’s entire career has been focused on clinical research, having worked in
project management and clinical operations for 10 years, with both large Pharma
and CRO businesses prior to joining Triumph. Her experience both in
monitoring, and managing clinical trials made her a significant hire for Triumph in
2007. Tammy’s experience, passion and eye for quality saw her promoted to Head
of EU Operations within her first year, and in 2011 she was appointed COO to take
over global operations responsibility.

30. TRIUMPH RESEARCH INTELLIGENCE IN THE US
7100 Six Forks Road
Suite 135
Raleigh
NC, 27615
USA
Tel: +1 919 239 4989
Fax: +1 919 239 4989
Email: info@tritrials.com
Web: www.triumphconsultancy.com
TRIUMPH RESEARCH INTELLIGENCE IN THE UK
Unit 6 Riverside Business Park
Stoney Common Road
Stansted Mountfitchet
Essex, CM24 8PL
UK
Tel: +44 1279 812 550
Fax: +44 1279 816 884
Email: info@tritrials.com
Web: www.triumphconsultancy.com