Talk given to USC Undergraduates

1. Analytical Validation
-A House of Cards
E. Dennis Bashaw, Pharm.D

2. Disclaimer: The presentation today should not be
considered, in whole or in part as being
statements of policy or recommendation by the
US Food and Drug Administration.
The examples given in the presentation today are
based on actual situations seen by the presenter.
All identifying information has been removed to
protect the confidentiality of the applicants
involved.

3. Trends in Drug Discovery
Scannell, JW, et al “Diagnosing the decline in pharmaceutical R&D efficiency” Nature
Reviews Drug Discovery, 11:191-200 (March 2012)

4. 4
Between the birth of the world and 2003, there
were five exabytes of information created. We
[now] create five exabytes every two days. See why
it's so painful to operate in information markets?“
Eric Schmidt, Google
5 exabytes of information is the
equivalent of 250,000 years of
DVD quality video
A 21st Century Perspective on Information

5. 5
An Explosion of Information Yields
an Explosion of Questions

6. “You might very well think that;
I could not possibly comment!”

7. House of Cards
If you do not have confidence in the validation of the
analytical method, how can you have confidence in the
concentration values?
If you do not have confidence in the concentration values, how
can you have confidence in the derived pharmacokinetic
parameters?
Safety & Efficacy
Clinical
Pharmacology
Bioanalytical
Validation

8. Relevant FDA Guidances

9. The Importance of Methods Validation
to Clinical Pharmacology
Bioanalytical Methods Validation Guidance, pg 2

10. MDS Canada

11. Cetero

12. Bioanalytical Review Problems
Mis-labeled samples Improper Shipping Flawed Extraction
Analytical Problems Calculation Issues Analysis/Reporting

13. Case Study #1
The Case of the
“Come As You Are”
Standard Curve

14. Case Study #1
 IV drug, relative bioavailability study in
patients with varying degrees of renal
insufficiency
 Analytical plan called for daily standard
curves to be constructed at 2, 5, 10, 25,
100, 500, and 1000ng/ml using triplicate
samples.
 Analysis consisted of 5 runs over the
course of 3 weeks by the same analyst

15. Case Study #1
Target
(ng/mL) 2 5 10 25 100 500 1000
Run 1 1.76 4.2 10.2 22.3 120 515 1100
1.5 5.13 8.7 24.11 111 517.2 1070
2.13 4.8 11.4 27.2 102 505 1012
Mean 1.84 4.78 10.08 24.65 108.25 509.3 1045.5
Run 3 1.55 1005
2.42 1089
2.59 1070
Mean 2.19 1054.67
Run 4 1.76 4.76 10.35 21.45 114 535 1093
1.23 4.22 11.2 22.76 110 503 1125
1.16 3.98 10.22 31.22 102 487 1200
1.94 4.25 27.86 99 493 1103
1.87 5.15 22.4 1114
5.6 26.4
4.9
5.55
Mean 1.59 4.80 10.59 25.35 106.25 504.5 1127
NO DATA !
TRIPLICATE SAMPLES???

16. Case Study #1
 Analyst chose the number of standards he
would use to construct a standard curve
differently between each assay run.
 Worst case was two samples (low & high)!
 Assay was essentially out of control
 Over 5 runs over two weeks no two
“standard” curves were constructed the
same way
 Report was “signed-off” by analyst, lab
supervisor, Director of Analytical Services,
and Vice-President for Pharmaceutics!

17. Case Study #2
The Case of
History Repeating Itself

18. Case Study #2
 Initial assay developed in the mid 1980s using, for the time,
state of the art HPLC system
 Original assay validation report showed adequate accuracy,
precision, sensitivity, selectivity, etc.
 Drug approved, but due to poor absorption the sponsor
immediately began a series of formulation studies to improve
bioavailability
 Assay validation of the later studies was merely a copy of the
earlier report, a table of standard curve results but no
tracings.

19. Case Study #2
 Why didn’t the sponsor submit individual study validation
reports or tracings?
◦ A. They didn’t have them
◦ B. The assay conditions had changed
◦ C. The assay performance had changed
Due to changes in equipment/column the retention time
for the parent had gone from 3 minutes to 10 minutes.
While this is certainly manageable, the sponsor decided to
not disclose this.

20. Case Study #3
The Case of the
“U” Shaped
Concentration Time Curve

21. Case Study #3
“U” Shaped Plasma Concentrations
(mean of 12 subjects)
Not surprisingly, the calculation of half-life was “difficult” for Group 2!

22. Case Study #3
In two studies done in support of this
project more than a quarter of the subjects
in a treatment leg in both trials (assayed at
the same time, by the same analyst) showed
these results. The study site chose to accept
the half-life estimate without comment!
What this is, is the “black box phenomena”
of data collection and analysis.
A computer gets the output from the
detector, runs the statistical &
pharmacokinetic analysis modules, produces
standard tables that a report is written from.
But the report writer is not necessarily
exposed to the primary data.
Study A
Study B

23. Case Study #4
The Case of
“The Missing Reagent”

24. Case Study #4
 A study was conducted in 24 subjects 12 normal and 12 with
renal insufficiency.
 Due to irreversible protein binding, the extracted plasma
samples had to be acidified with 0.1N HCl within an hour of
extraction.
 Because of analytical problems at their prior lab, the company
elected to send their samples to Europe for analysis.
 A total of approximately 320 samples were shipped.

25. Case Study #4
 Upon analysis, out of 300+ samples
all concentrations were BLOQ!
 Subsequent investigation revealed
that the SOP did not specify an
acidification target and that the
stock bottle of HCl was sub-potent
(age) and contaminated.
 The cost of a bottle of acid and a
proper SOP was approximately $1
Million and six months of
development time.

26. Lessons
Case #1-Written SOPs are only effective when followed.
The fact that an analyst could change procedures on a daily
basis and yet all levels of management signed off on the
report should not be possible.
Case #2-Analytical methods need to be constantly
monitored for changes in performance and when found must
be investigated. Hoping the FDA will not ask questions is not a
risk management strategy.

27. Lessons
Case #3-Data analysis should include a program of
primary data examination. Over-reliance on the computer
to catch errors is totally dependent upon an exhaustive
programming of failure modes and is unlikely to ever be all
inclusive.
Case #4-Common laboratory reagents play a key role in
analysis. A seemingly small detail caused a costly delay.
Does your lab have an SOP on reagents and how well is it
followed? Last inventory?

28. Parallels From Other Industries
 NASA prepares a monthly
safety case study that looks at
system failures of many kinds.
 August 2012 issue dealt with
the loss of Air France Flight 447
 The plane was lost with all
hands due to a combination of
design and a lack of
understanding of failure modes
by the crew that led to
inappropriate control inputs.

29. The “Glass Cockpit” Problem
The ability to automation controls during flight is a huge benefit to the
safety of the flight during normal situations. However, there are some very
big problems that crop up.
Automation Bias: This is where the pilots use the automation, such as auto
pilot, as substitute to the gathering of information. They lose situational
awareness because the computers are doing it for them. It can go as far as
forgetting to ever check on the system and its reliability.
Over Trust: The pilots start trusting the systems because of the fantastic job
it does, and start no longer worry about the integrity of the systems and
allow them to do the job. Some times they believe the computers over the
other warning signs.
http://aviationknowledge.wikidot.com/aviation:glass-cockpits

30. The “Glass Cockpit” Problem
Over Confidence: With the ability to decrease the workload of
the pilot, it means they can now complete more complex tasks
during flight. However, this can create an illusion of "good
piloting". The question that needs to be asked is "could you do
this if the automation was off".
Reliance: The above problems often lead to reliance on the
automation system. It is now being seen that pilot are losing
their "flying skill" and its being replaced with supervising
computer systems. The problem is some pilots can no longer
fly the aircraft without the automated systems. (Skitka, 2000)
http://aviationknowledge.wikidot.com/aviation:glass-cockpits

31. Extrapolation to Bioanalytical
Methods
Although focused on aviation, these principles can be applied
to bioanalytical methods and data analysis where automation
has begun to replace the human element in analysis.

32. An Observation on Quality
 Quality is neither necessarily expensive nor time
consuming
Lack of Quality is always Costly
◦ Financial, Cost to re-do work
◦ Time, Delay to market
◦ Reputational, Client loss of confidence in ability
◦ Business, Loss of clients

33. Closing Thoughts
Would you stake your professional life
right now on the quality of your
bioanalytical work?
If, not then WHY do you tolerate it?
WHY do you think Your Professor/Client/
or a Regulatory Agency will?

34. “It is not enough to do your best;
you must know what to do,
and then do your best.”
W. Edwards Deming