Authors: George Koustenis, FDA-CDRH David Breiter, Boston Scientific Roseann White, Abbott Vascular George Woodworth, Univ. of Iowa FDA/INDUSTRY STATISTICS WORKSHOP: Washington, D.C. Sept. 29, 2006

1. FDA/INDUSTRY STATISTICS WORKSHOP:
Washington, D.C. Sept. 29, 2006
Statistical Issues in Medical Device
Trials
George Koustenis, FDA-CDRH
David Breiter, Boston Scientific
Roseann White, Abbott Vascular
George Woodworth, Univ. of Iowa
2008/2/27 1

2. The Use of Objective Performance Criteria
(OPC) in Medical Device Evaluation
FDA/Industry Statistics Workshop
September 29, 2006
Washington, D.C.
George Koustenis
Division of Biostatistics
Office of Surveillance and Biometrics
2008/2/27
Center for Devices and Radiological Health 2
U.S. Food and Drug Administration

3. OBJECTIVE
The purpose of this presentation is
to provide a brief introduction to the
possible use of Objective
Performance Criteria (OPC) in the
evaluation of medical devices during
the regulatory approval process.
2008/2/27 3

4. DEFINITION OF OPC
Fixed Target(s)
Objective and Meaningful Standard
Provides Comparison in Evaluating Safety
and Effectiveness
Usually a Rate
Surrogate for Control Group
Benchmark for Minimally Acceptable
Values
Not a Control Group
2008/2/27 4

5. CONTROLS
Standard Comparison for an
Experimental Treatment
Group of Patients with Same
Condition, Demographics, &
Prognostic Values
By Controlling for as Many Variables
as Possible – Any Differences are
Presumed Due to New Intervention
2008/2/27 5

6. TYPES OF CONTROLS
Concurrent Randomized Controlled Trial
(RCT)
The Gold Standard
Effectively Minimizes Bias
Effectively Balances Demographics &
Unknowns
Supports Basic Assumptions of Standard
Statistical Methodology
2008/2/27 6

7. Historical Controls
Compares Current Therapy & Patients
Against Others Studied in Previous
Investigations
Not Randomized to Current Conditions
May Be Too Far Removed in Time
Very Difficult to Validate the Data from
Historical Trials
Tend to Produce More Positive Results
2008/2/27 7

8. ESTIMATES OF OPC’S
Necessarily Driven by Historical Data
Requires Appropriate Pooling of
Different Investigations
OPC’s Inherit All of Problems Seen
with Historical Controls
Even Sophisticated
Pooling/Analytical Techniques
Cannot Eliminate Basic Problems
2008/2/27 8

9. USE OF OPC’s In CDRH
Some Current Use of OPC’s
Cardiac Ablation Catheters
Replacement Heart Valves
Ophthalmics
Hip Replacement Systems
2008/2/27 9

10. USE OF OPC’s In DEVICE
TRIALS
Given that the use of OPC represents
a significant departure from the
standard scientific approach to the
design and analysis of medical
device clinical trials, the question
now becomes: when might it be
appropriate to use this type of non-
control comparison in the medical
device approval process?
2008/2/27 10

11. CITED ADVANTAGES Of OPC’s
Smaller Sample Size
Standard Value for All Sponsors
Save Time and Money
Easier to Execute
I.E. – “Least Burdensome”
2008/2/27 11

12. DISADVANTAGES Of OPC’s
All the Problems Associated with
Historical Controls
Problems with Validity of Data & Analysis
Problems with Advances in Practice of
Medicine
May be Disagreement on Final OPC Value
Problems with Single Arm Trials
Selection Bias
May Not See Pre-Post Benefits
2008/2/27 12

13. DISADVANTAGES Of OPC’s
May Not Be Least Burdensome
Smaller Sample Size? N=100 or 150?
Time and Resource Intensive to Develop
Sets a Minimum Standard
Superiority?
Older and Older Data
Protect and Promote the Public Health?
2008/2/27 13

14. WHEN MIGHT OPC’s Be USED?
History of OPC’s in Medical Devices
Non-Experimental or Quasi-
Experimental Designs May be Valid
in Certain Situations
2008/2/27 14

15. MINIMUM REQUIREMENTS
WHEN OPC’s MIGHT Be USED
Great Deal is Known About the
Natural History of the Disease or
Condition
Underlying Patient Population is Well
Described & Relatively Stable
Extensive Clinical History &
Experience with This Device
2008/2/27 15

16. MINIMUM REQUIREMENTS
WHEN OPC’s MIGHT Be USED
(cont.)
Stable and Well Known Standard of Care
Appropriate Ancillary Technology is
Relatively Stable
No Significant New Questions of Safety or
Effectiveness
Consensus Among FDA, Industry, Clinical,
Academic and Patient Communities
Expectation of Significantly Positive
Treatment Effect
2008/2/27 16

17. HOW SHOULD OPC’s BE
DERIVED?
Data Driven
Rigorous And Scientifically Valid
Methodologies
Valid Databases
Appropriate Statistical Modeling
Appropriately Designed and Powered
Pivotal Trial
Periodical Re-Evaluation and Updating the
OPC’s
2008/2/27 17

18. OTHER IMPORTANT OPC
ISSUES
All Parties Must Have a Clear
Understanding of the OPC in Medical
Device Trials – Success and Failure
OPC’s Already in Limited Use in Device
Trials
No Current General Policy of OPC Use in
CDRH
Need for Comprehensive, Coherent, and
Consistent OPC Policy for Device Trials
2008/2/27 18

19. WHAT SHOULD THIS POLICY
CONTAIN?
Clear Definitions of All Appropriate Terms
Established Guidance on Minimum Standards for
a Device or Device Class
Specific Roles Played by FDA, Industry, Clinical
Community, Academia and Other Interested
Parties
Provisions for Periodic Updating of OPC
Specific Guidance on Methodology to Derive an
OPC
Unambiguous Policy Regarding Failure to Meet
OPC
2008/2/27 19

20. SUMMARY
An introduction to the history and use of
Objective Performance Criteria (OPC) in medical
device evaluation
Advantages and disadvantages of OPC’s
When and how OPC might be used & developed
Discussed the need for developing a CDRH policy
regarding use of OPC in medical device trials
General outline of this policy is presented in an
effort to begin dialogue between CDRH, industry,
clinical community & other interested parties
2008/2/27 20

21. REFERENCES
Please see accompanying brief
commentary
Request electronic copy from
george.koustenis@fda.hhs.gov
2008/2/27 21

22. Q&A
2008/2/27 22