Why Design Validation is More Than Testing: How do we Validate our Validation?

Why Design Validation is More Than Testing:
How do we validate our validation?
or Not your grandmothers' approach to validation!
Presented by:
Michael Drues, Ph.D.
President, Vascular Sciences
Carlsbad, California
and
Adjunct Professor of Regulatory Science, Medicine
and Biomedical Engineering
George Washington University Graduate Dept. of Regulatory Science
Cornell University Graduate Dept. of Biomedical Engineering
GreenLight.Guru True Quality Summit (June 2, 2020)
https://virtual-summit.greenlight.guru/talks/why-design-validation-is-more-than-testing-how-do-we-validate-our-validation/
For questions or more information, call
(508) 887 – 9486 or e-mail mdrues@vascularsci.com
© Copyright 2020 by Vascular Sciences and Michael Drues, Ph.D. All rights reserved.

© Copyright 2020 by Vascular Sciences and Michael Drues, Ph.D. All rights reserved.
presented by: Michael Drues, Ph.D.
Description:
The “textbook” approach to design validation is intended to make sure we design the “right” device, i.e., the device meets user
needs and intended uses. But what exactly does that mean and how do we do it? More importantly, how do we know we did it
right? If we tick the validation checkbox, can we assume our validation is correct?
Validation usually involves a lot of testing. But what tests do we do? What tests do we not do? When can we use standard
tests and when do we have to develop new ones? More importantly, how do we know we’re doing the right tests? If a test is
“required” must we do it? In other words, how do we validate our validation? And how do we defend our validation when FDA
criticizes it?
Finally, the “textbook” approach to design validation is very myopic. What else needs to be validated? human factors plan?
reprocessing methods? post-market surveillance system? what else? This presentation will use the case study approach to take
a broad approach to validation in an interactive fashion including:
• What are we required to validate and how/when do we do it?
• How do we validate our validation? How do we defend our validation when FDA questions it?
• What are the consequences if our validation is wrong?
• If we change our device, when and how do we revalidate?
• What else needs to be validated?
• What are we not required to validate but we should anyway?
• How do we take a holistic approach to validation?
• What are the validation challenges for the future, i.e., 3-D printing, etc.?
In this presentation, participants will learn best practices to avoid timely and costly validation mistakes and creative ways to use
validation to their advantage!
Additional columns, articles, podcasts and webinars can be found:
Global Medical Device Podcast (GreenLight.Guru) here, Mike on MedTech (Medical Product Outsourcing) here, Medical Design
and Outsourcing here, Guerilla Regulatory Strategy (MED Device Online) here and Healthcare Packaging here or LinkedIn here.
Presenter Bio
Michael Drues, Ph.D., is a regulatory strategy consultant specializing in designing novel
regulatory strategies to bring new and innovative medical products to market and in developing
effective communication strategies between companies and regulatory agencies to minimize time
to market and avoid delays.
Dr. Drues received his B.S., M.S., and Ph.D. degrees in Biomedical Engineering from Iowa State
University. He works with leading medical device, pharmaceutical and biotechnology companies
ranging in size from start-ups to Fortune 100 companies. He also works on a regular basis for the
U.S. Food and Drug Administration, Health Canada, the US and European Patent Offices, the Centers for Medicare and Medicaid
Services and other regulatory and governmental agencies around the world.
Dr. Drues is an internationally recognized expert and featured keynote speaker on cutting-edge medical technologies and
regulatory affairs. He conducts seminars and short-courses for medical device, pharmaceutical and biotechnology companies,
the FDA, Health Canada, the US and European Patent Offices, CMS and other regulatory and governmental agencies around the
world.
Finally, Dr. Drues is an Adjunct Professor of Regulatory Affairs, Medicine and Biomedical Engineering at several universities and
medical schools. He regularly teaches graduate courses in Regulatory Affairs and Clinical Trials, Medical Device Regulatory
Affairs and Product Development, Combination Products and Pathophysiology.

1
or Not your grandmothers approach to validation!
For additional information, www.VascularSci.com,
call (508) 887-9486 or e-mail mdrues@vascularsci.com
© Copyright 2020 by Michael Drues, Ph.D. and
Vascular Sciences. All rights reserved.
GreenLight.Guru True Quality Summit (June 2, 20120
virtual-summit.greenlight.guru/talks/why-design-validation-is-more-than-testing-how-do-we-validate-our-validation/
1
© Copyright 2020 by Michael Drues, Ph.D.
and Vascular Sciences. All rights reserved.
Vascular Sciences
or Not your grandmothers' approach to validation!
presented by:
Michael Drues, Ph.D.
President, Vascular Sciences, Carlsbad, California
and
Adjunct Professor of Regulatory Science, Medicine and Biomedical Engineering
George Washington University Graduate Dept. of Regulatory Science
Cornell University Graduate Dept. of Biomedical Engineering
For questions or more information, contact me at
(508) 887 – 9486 or mdrues@vascularsci.com
Join me on LinkedIn at www.linkedin.com
How do we validate our validation? 2
Vascular Sciences
1
2

2
Vascular Sciences
Do you want more?
For columns, articles and podcasts… visit vvv.VascularSci.com
Global Med Dev Podcast (GreenLight.Guru) here
Mike on MedTech (Medical Product Outsourcing) here
Medical Design and Outsourcing here
Guerilla Regulatory Strategy (MED Device Online) here
Healthcare Packaging here
LinkedIn here
Vascular Sciences
Polling Questions
Do you validate your validation?
Do you justify not just what you do but what you don’t do?
Do you take your validation strategy to FDA prophylactically?
i.e., before there is a problem via a pre-sub?
Do you agree that even if a validation is “required,”
if its not realistic, it’s a waste of time and money?
Before we begin…
3
4

3
Vascular Sciences
Here’s what we’ll talk about…
✓ What are we required to validate and how/when do we do it?
✓ How do we validate our validation and how do we defend it to FDA?
✓ What are the consequences if our validation is wrong?
✓ If we change our device, when and how do we revalidate?
✓ What else needs to be validated?
✓ What are we not required to validate but we should anyway?
✓ How do we take a holistic approach to validation?
✓ What are the validation challenges for the future, i.e., 3-D printing, etc.?
✓ Anything else important?
In other words…
Answers are only as good as the questions we ask!
i.e., validations are only as good as the tests we perform!
Vascular Sciences
First, an important disclaimer...
I can’t make you an expert in a few minutes!
I’m not even going to try but…
Remember my philosophy of education:
To teach you how to think not what to think!
5
6

4
Vascular Sciences
Is it possible to think regulatory?
“Regulatory affairs is a way of
thinking much more than it is a body
of rules and regulations – or at least
it should be!”
Michael Drues (1964–)
Regulatory Strategist and Amateur Philosopher ☺
www.meddeviceonline.com/author/michael-drues
“Science is a way of thinking much
more than it is a body of knowledge.”
Carl Sagan (1934–1996)
American astronomer, author and science journalist
So how about this?
Maybe Carl Sagan would be proud!
Vascular Sciences
https://www.greenlight.guru/webinar/design-validation-how-to-get-right-product-to-market
Want more?
7
8

5
Vascular Sciences
What is design validation
How does verification differ from validation, i.e., V&V?
Vascular Sciences
Verification and Validation (V&V)
Design verification asks
“Did we design the device right?”
Design validation asks
“Did we design the right device?”
Not the same as ‘did we solve the right problem?’ or ‘did we ask the right question?”
Is this not simply ‘prudent’ engineering?
Design Control Guidance For
Medical Device
Manufacturers, March, 1997
What does this mean?
9
10

6
Vascular Sciences
Design Verification
Design verification asks
“Did we design the device right?”
or in more words…
comparing the output of each design process step
with the input to that step-in order to demonstrate that
the results are consistent with the goals
but…
Do we have the right goals? i.e.,
Are we asking the right questions?
Are we solving the right problems?
Vascular Sciences
Design Validation
Design validation asks
“Did we design the right device?”
or in more words…
test device against requirements (after verification)
Important reminder:
use devices as close to ‘final product’ as possible,
i.e., ‘manufactured’ using the same methods and procedures
expected to be used for production
11
12

7
Vascular Sciences
Product Validation vs. Process Validation
Design/Product Validation
• conforming to user and patient needs, i.e.
‘Does the device work right?’
Process Validation:
• manufacturing process meets predetermined specifications
Remember,
Design/Product Validation ≠ Process Validation
Regulation requires both – individually!
Should not separate them…
both are important!
Vascular Sciences
What does “validation”
really mean
Is it about ticking boxes on a form?
13
14

8
Vascular Sciences
Are we required to
“validate” we are solving
the right problem
Regrettably not… should we be?
Vascular Sciences
Case Study: Bare Metal Coronary Stent
Hint:
What's the biological problem (i.e., root cause)?
Remember…
If the only tool you have is a hammer…
all of your problems look like _____ ?
Transmyocardial
Revascularization (TMR/pTMR)
a.k.a. mechanical angiogenesis
Some reverse engineering…
What's the user need?
How does user need → design input?
Does this influence / limit / bias design?
True
Angiogenesis
How does that apply here? Hint: What solutions do we not have & why?
LOTS!
Why is this important? Can you say disruptive technology?
15
16

9
Vascular Sciences
What’s the real sham?
‘Unbelievable’: Heart Stents Fail to Ease Chest Pain
(New York Times, Nov. 2, 2017) here.
Background:
• >500K patients/yr (WW) have stents inserted to relieve
chest pain at cost $11K – $41K (US hospitals)
Method:
• 200 patients treated 6 weeks with drugs →
• procedure: “real or fake” insertion of stent
“This is one of the few studies in cardiology in which a sham
procedure was given to controls who were then compared
to patients receiving the actual treatment.”
“In both groups, doctors threaded catheter through the
groin or wrist and, with X-ray guidance, up to the blocked
artery. Once the catheter reached the blockage, the doctor
inserted a stent or, if the patient was getting the sham
procedure, simply pulled the catheter out.”
Conclusion:
“A procedure used to relieve chest pain in hundreds of
thousands of patients/year is useless for many of them.”
Is this a sham procedure or a sham device?
What do you think?
Vascular Sciences
Are we solving the
“right” problem
Case Study: Ventilators
17
18

10
Vascular Sciences
Want more?
https://www.greenlight.guru/webinar/covid-19-emergency-use-authorization-eua-medical-device-companies
Vascular Sciences
Always question the status quo!
BusinessWeek, May 29, 2006
“Treatments are based largely on rules and
traditions, not scientific evidence.”
something called “evidence-based medicine”
19
20

11
Vascular Sciences
We need more critical thinkers!
Einstein also said:
“Insanity is doing the same thing over and over again and
expecting different results.”
So
What do we get when we put both together?
Unfortunately,
The world is much more complicated than many would like to believe!
“The problems that exist in the world today
cannot be solved by the level of thinking
that created them.”
Albert Einstein (1879-1955)
German born American Physicist who developed the special and general theories of
relativity. Nobel Prize for Physics in 1921.
Vascular Sciences
What are the
consequences if our
validation is wrong
Case Study: da Vinci surgical robot?
Is this a usability validation failure?
Even if a validation is “required,”
if its not realistic, it’s a waste of time and money?
21
22

12
Vascular Sciences
Case Study: da Vinci surgical robot
Questions for Discussion:
1. What is the root cause of this problem? Hint: is lack of training the real root cause? How much training is enough?
2. Were these problems unforeseeable, i.e., anticipated risks? Were these problems new?
3. How does human factors / usability fit into this equation?
4. Who’s to blame? FDA: “does not have jurisdiction” / Manufacturer: “cannot require training” Is this a copout?
5. Speaking of blame… what happens when we get it wrong?
6. Does it make sense to separate the efficacy of the device from the skill level of the user?
7. How is this similar to Enteryx?
8. What are the lessons to be learned? What else is important?
At the end of the day…
Vascular Sciences
What are the additional
consequences if our
validation is wrong
Case Study: Bleeding Edge/Implant Files
Is the “root cause” of many problems improper validation?
23
24

13
Vascular Sciences
Public Perception
Article: New Documentary Film Is Scathing Indictment Of FDA's Regulation Of Medical Devices (Forbes, July 29, 2018) here or attached
Video: The Bleeding Edge Trailer on Netflix or YouTube
Video: We are guinea pigs: Filmmakers on the risks of medical devices (CBS News, July 27, 2018) on CBS or YouTube
Bleeding Edge Trailer (Netflix) (2 min)
Vascular Sciences
Implant Files
Implant Files Video: here Home Page here.
The regulatory ancestral network of surgical meshes here.
Implant Files (3 min)
25
26

14
Vascular Sciences
What else needs to be
validated
Case Study: Reprocessing
What if your device is reusable?
Vascular Sciences
Why is everyone talking about reprocessing?
UCLA Warns Nearly 180 Patients About Dangerous 'Superbug' Exposure (NBC News, Feb 18, 2015) (3 min)
So why now?
But this is NOT a new
problem…
and it is NOT unique to
duodenoscopes!
A canary in a coal mine?
27
28

15
Vascular Sciences
What types of devices are
subject to this guidance?
• Bronchoscopes (flexible and rigid) and accessories
• ENT endoscopes and accessories
• GI and Urology Endoscopes with elevator channels (not including accessories) [e.g.,
duodenoscopes for ERCP]
• Automated Endoscope Reprocessors (AERs)
• Colonoscopes (not including accessories)
• Neurological endoscopes (not including accessories)
• Arthroscopes and accessories
• Laparoscopic instruments and accessories
Note disclaimer at bottom:
“In the future this list may be updated as additional information regarding reprocessing medical
devices becomes available.” – recommendation: don’t consider this an exhaustive list!
My recommendation:
“A well-designed and properly validated cleaning procedure likely will be a requirement of most, if
not all, regulatory submissions in the future: 510(k)s, premarket approvals (PMAs), etc.” [Drues, 2015]
What about devices previously cleared or approved? – Don’t be surprised if you get a knock on your door…
Consult this guidance for all devices labeled for reuse!
Current reprocessing device list available here as of March, 2015
Vascular Sciences
Additional
Resources
What OEMs Should Learn From The Recent Endoscope
Reprocessing Incidents (And Their Fallout), MED Device Online
(April 17, 2015) available here.
Can We Design Medical Devices To Be Reprocessed
Without Killing People? (MED Device Online, April 30,
2015) available here.
29
30

16
Vascular Sciences
How early in the
development process
should we think about
validation
It’s never to early!
Start with validating that we are solving the right problem!
Vascular Sciences
How do we “validate”
what we don’t validate
Validating what we don’t do is just as important (if not
more) than validating what we do!
31
32

17
Vascular Sciences
When (not if) we change
our device while on the
market, when and how
do we revalidate
Case Study: Gynecological Mesh
Welcome to the world of change management… s510k vs. L2F
Reminder:
Most common cause of warning letters and 483’s!
Vascular Sciences
Case Study:
Gynecological Mesh
A common surgical implant has generated the largest multi-
district litigation since asbestos… one of the device's
manufacturers, Boston Scientific, now facing 48,000 lawsuits
CBS 60 minutes, May 13, 2018 here
Questions for Discussion:
1. Why did this problem happen, i.e., what was the root cause?
2. With all the "regulation" we have, all the regulatory and quality "professionals" in companies
like this and all the "oversight" from FDA, how did this happen? Could it happen
again? Could it happen to other companies? Why or why not?
3. If this was happening to you / your company, what would you do?
4. What are (should be) the lessons learned from this case, i.e., how can we prevent it from
happening again?
5. Who has a greater impact on medical device companies, FDA or product liability attorneys?
6. What else is interesting / important about this case?
Gynecological mesh (60 Minutes, May 2018) (14 min)
33
34

18
Vascular Sciences
Check out
www.greenlight.guru/webinar/medical-device-change-management-best-practices
Vascular Sciences
Design Changes and Process Changes
Very common source of problems down the road…
Most of which could be mitigated if not entirely avoided!
Reasons for Change:
• New/Improved features
• Fixing Problems
• New safety/regulatory/purchasing/etc. requirements
• Greater efficiency, i.e., cost reduction
• Lots of others…
Please…
Don’t use regulatory burden as a design criterion…
this practice is really holding us back!
35
36

19
Vascular Sciences
What are the challenges
for the future
Or put another way…
What's one of the most significant limitations of the Design Controls?
Fundamental tenant: meet the needs of your user! But…
Why assume the user knows what they really need?!?!?
Evolutionary vs. Revolutionary Product Development
horse → car, candle → lightbulb, etc.
How about a medical device example…
Vascular Sciences
Can we print a stent?
What kind of stent?
Better…
Can we print a customized bioabsorable stent?
Bare Metal Stents Stent Grafts Drug-Eluting Stents
Boring! Boring! Boring!
37
38

20
Vascular Sciences
How do we “validate” this?
Vascular Sciences
Can we 3DP a ‘customized’ vascular stent?
What’s next?
How about a personalized (i.e., 3-D printed) DES? – wicked cool! ☺
Advantages: choose drug (drugs), dosage, release rates, location, etc.
Should FDA regulate this? If so, how?
Northwestern Engineering News (Oct, 2016) available here.
Absolutely! Watch this…
Method:
micro continuous liquid interface production
(microCLIP)
↓
liquid photo-curable resin to print objects with light
↓
fabricate stent precisely matching patient anatomy
Advantages:
• Extremely high res (print features small as 7µm)
• Print up to 100 stents at a time →
faster/cheaper than traditional manufacturing
• Speed: print 4-cm stent in minutes
39
40

21
Vascular Sciences
Here’s what we’ll talk about…
✓ What are we required to validate and how/when do we do it?
✓ How do we validate our validation and how do we defend it to FDA?
✓ What are the consequences if our validation is wrong?
✓ If we change our device, when and how do we revalidate?
✓ What else needs to be validated?
✓ What are we not required to validate but we should anyway?
✓ How do we take a holistic approach to validation?
✓ What are the validation challenges for the future, i.e., 3-D printing, etc.?
✓ Anything else important?
In other words…
i.e., validations are only as good as the tests we perform!
Vascular Sciences
Good regulation is neither specific
nor rigid… nor should it be!
Flexibility of the QS Regulation (Preamble)
“The QS regulation embraces the same "umbrella'' approach to the CGMP regulation that was the underpinning
of the original CGMP regulation. Because the regulation must apply to so many different types of
devices, the regulation does not prescribe in detail how a manufacturer must produce a specific
device. Rather, the regulation provides the framework that all manufacturers must follow by
requiring that manufacturers develop and follow procedures and fill in the details that are
appropriate to a given device according to the current state-of-the-art manufacturing for that specific
device.
Manufacturers should use good judgment when developing their quality system and apply those
sections of the QS regulation that are applicable to their specific products and operations, 21 CFR
820.5 of the QS regulation. Operating within this flexibility, it is the responsibility of each manufacturer to
establish requirements for each type or family of devices that will result in devices that are safe
and effective, and to establish methods and procedures to design, produce, distribute, etc. devices that meet
the quality system requirements. The responsibility for meeting these requirements and for having
objective evidence of meeting these requirements may not be delegated even though the actual
work may be delegated.
FDA has identified in the QS regulation the essential elements that a quality system shall embody,
without prescribing specific ways to establish these elements. Because the QS regulation covers a
broad spectrum of devices, production processes, etc., it allows some leeway in the details of
quality system elements. It is left to manufacturers to determine the necessity for, or extent of,
some quality elements and to develop and implement specific procedures tailored to their
particular processes and devices.”
www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/PostmarketRequirements/QualitySystemsRegulations/default.htm
41
42

22
Vascular Sciences
The Philosophy of Design Controls
Design Controls should not be about a hard and fast set of
rules… it’s about understanding the intent and approaching
the process in a logical and systematic fashion.
In other words…
Don’t just follow the rules… think!
“Rules are mostly made to be broken
and are too often for the lazy to hide behind.”
General Douglas MacArthur (1880 –1964) was an American general in the US Army during the 1930s
and played a prominent role in the Pacific theater during World War II. He was one of only five men
ever to rise to the rank of General of the Army in the U.S.
44
Vascular Sciences
What are we really trying to accomplish?
There is a common adage in medicine:
The surgery went perfectly but the patient died anyway.
The regulatory spin:
We followed the regulation perfectly but the patient died anyway.
The engineering spin:
We designed the medical device perfectly but the patient died anyway.
The testing/validation spin:
We tested/validated the medical device perfectly but the patient died anyway.
Another common medical adage:
If you’re not prepared to act on the result of a test, don’t do the test.
[fatigue testing example]
Bottom line:
If we meet the requirements, have we done our jobs? Is that enough?
43
44

23
Vascular Sciences
What's the takeaway
Doing a validation is not enough!
Don’t just tick boxes on a form!
If you are not doing the right validation…
i.e., are you asking the right questions?
are you solving the right problems?
isn’t this a colossal waste of time and money?
Vascular Sciences
There are many regulatory consultants out there…
but there are surprisingly few good ones!
So how do you become a good one?
Learn when to follow and
more importantly…
when to lead!
45
46

Why Design Validation is More Than Testing: How do we Validate our Validation?

More Related Content

Similar to Why Design Validation is More Than Testing: How do we Validate our Validation?

More from Greenlight Guru

Recently uploaded

Why Design Validation is More Than Testing: How do we Validate our Validation?