Abstract To address the growing need for information on a therapeutic, besides information on safety and efficacy, and the increasing trend to extrapolate data from traditional randomized control trials (RCT’s) to influence clinical practice; an in-depth evaluation of the utility and practicability of RCT’s in influencing real-world clinical practice is evaluated. The pragmatic clinical trial (PCT) is discussed and introduced as a potentially viable means to influence clinical practice. The regulatory impact of this new adaptation is also explored. Concepts of study design, including concepts such as validity, generalizability, efficacy and effectiveness are discussed for both RCT’s and PCT’s.

1. PragmaticClinical Trials:
External Validity &
Generalizabilityfor Real
WorldEvidence
By: Jodean Campbell
Walden University
CLRA-6560-2, Capstone
8/21/2017

2. INTRODUCTION:REALWORLDEVIDENCE
❑Thiscapstone projectwas conducted as a means to address the growing
need for informationon a therapeutic,besidesinformationon safetyand
efficacy,and the increasingtrendof extrapolatingdatafrom traditional
randomized controltrials (RCT’s)to influenceclinical practice.
❑An in-depth evaluationof the utilityand practicability of RCT’sin
influencing real-worldclinicalpracticeis evaluated.
❑The pragmaticclinical trial(PCT) is discussedand introduced as a
potentially viable means to influenceclinical practice.
❑Concepts of study design, includingconcepts suchas validity,
generalizability, efficacy and effectivenessare discussed for both RCT’s and
PCT’s.

3. All fields of industry are taking advantage
the new paradigm of ‘big data’to
optimize efficacy, efficiency and
productivity.
Evidence Based Medicine
relates individual clinical signs, individual
clinical experience with the best scientific
evidences obtained bythe clinical
research.
Influence health care regulations and
service delivery.
Makerecommendations oncost-
effectiveness of atherapeutic.
WHY REAL WORLD EVIDENCE?

4. WHERE TOSTARTWHEN REAL WORLDEVIDENCE IS NEEDED?
In the next few slides, we’ll explore the rationale for pragmatic studies as the best study design
when collecting real world evidence in the setting of a clinical trial.

5. WHAT DO YOU WISH TO
EVALUATE?
Questions to ask regarding the nature of
your research questions

6. INTERNALOR EXTERNALVALIDITY?
ComparisonBetween Internaland External Validity
Type Internal External
Meaning Internal validity is the extent to which the
experiment is free from errors and any
difference in measurement is due to
independent variable and nothing else.
External validity is the extent to which the
research results can be inferred to world at
large.
Concerned with… Control Naturalness
What is it? It is a measure of accuracy of the
experiment.
It checks whether the casual relationship
discovered in the experiment can be
generalized or not.
Identifies How strong the research methods are? If outcome of the research be applied to the
real world?
Describes Degree to which the conclusion is warranted Degree to which the study is warranted to
generalize the result to other context.
Used to… Address or eliminate alternative explanation
for the result.
Generalize the outcome

7. WHICHIS BESTTO GATHERREALWORLD
EVIDENCE?
× •Internal Validity
✓ •External Validity

8. EFFECTIVENESSOR EFFICACY?
EFFICACY STUDY
❑Subjects placed under restrictive
treatmentgroups
❑Treatment is evaluatedunder ideal
conditions
❑EfficacyTrials
❑Explanatory/RCT in nature
❑Ideal to evaluateinternal validity of a
therapeutic (Kukull & Ganguli, 2012).
❑A phase III, randomized, double-
blinded, placebo controlled clinical trial
❑ Evaluating the safety and efficacy of a
therapeutic
EFFECTIVENESS STUDY
❑Subjects placed under non-restrictive
treatmentgroups
❑Treatment is evaluatedunder normal
conditions(Kukull & Ganguli, 2012).
❑Management trials
❑Pragmatic in nature
❑Best suited to evaluate the external
validity of a therapeutic
❑A pragmatic cohort study, alongside a
Phase II-III randomized, placebo
controlled double-blindclinical trial
study
❑ Evaluating the cost effectivenessof
the therapeutic

9. WHICHIS BESTTO GATHERREALWORLD
EVIDENCE?
× •Efficacy Trial
✓ •Effectiveness Trial

10. GENERALIZABLEOR SPECIFICRESULTS?
GENERALIZABLE
Representativeof general population
Subjective Analysis
Real World Evidence
SPECIFIC
Results valid only for sample
Objective Analysis/A Priori
Evidence Only SupportsEfficacy
UnrestrictiveSampling &
Randomization
Restrictive Sampling & Randomization

11. WHICHIS BESTTO GATHERREALWORLD
EVIDENCE?
✓ •Generalizable Results
× •Specific Results

12. CHARACTERISTICSOF RCT’s
Internal
Validity
Measures
Causal
Relationships
Efficacy
Superiority
Studies
Specific
Objective
Analysis/A Priori

13. RCT’s DONOTSUPPORTREAL WORLDEVIDENCE
Randomized Clinical Trials (RCT’s) are considered the
industry standard for scientific research and exploration
• Rigorously determine if a cause-effect relationship exists between a
treatment and an outcome
• Randomization results in a sample with comparable prognostic &
unknown factors
Strict Exclusion and Inclusion Criteria allows researchers
to control confounding
• Treatment evaluated under optimal conditions
• Issues when broad inferences are drawn from RCT’s
• Information yielded may not be representative of the population

14. CHARACTERISTICSOF PCT’s
External
Validity
Measures Accuracy
EffectivenessManagement Studies
Generalizable Subjective/Representative

15. PCT’s SUPPORTREALWORLDEVIDENCE
A pragmatic study is designed to yield information required to make
decisions, from sound scientificdata.
• “PCT’s address practical questions about the risks, benefits, and costs of an
interventionas they would occur in routine clinical practice, rather than in an ideal
setting (United States of America, U.S. Department of Health & Human Services,
Agency for HealthcareResearch and Quality (AHRQ), 2013).”
• PCT’s havehigh external validity & generalizability, which is ideal for informing
stakeholders on important applicability and utility of a therapeutic, otherwise not
obtainablefrom traditional RCT’s
Pragmatic clinical trials have many applications that cannot be
achieved by RCT’s alone, including the ability to
• Compare clinically relevantalternatives,
• Utilize a diverse study populationfor various sources, settings and locations,
• Measure a wide range of outcomes, and adjust to the local context of the therapeutic
being evaluated (Leung,2015).

16. WHICHIS BESTTO GATHERREALWORLD
EVIDENCE?
✓ •PRAGMATIC CLINICAL TRIAL
× •RANDOMIZED CLINICAL
TRAILS

17. SO WHYNOT
JUSTUTILIZE
META-
ANALYSISTO
GENERATE
REALWORLD
EVIDENCE?

18. METAANALYSISVS. PCT
•While meta-analysis can inform stakeholders on decisions,
it is dependent on data from RCT’s which are not designed
to provide generalizable information on the extrinsic
validity and applicability of a therapeutic.
•RCT’s are limited in their scope to provide information on
the external validity and applicability of a therapeutic
(Leung, 2015).
•Pragmatic clinical studies, however, are specifically
designed to provide information on the external validity
and applicability of a therapeutic, and thus should be the
standard to attain information for EBM (United States of
America, U.S. Department of Health & Human Services,
Agency for Healthcare Research and Quality (AHRQ), 2013,
March).

19. Example Pragmatic Clinical Trial:
“Oral and Topical Antibiotics for Clinically Infected Eczema in Children: A Pragmatic
Randomized Controlled Trial inAmbulatory Care,” by Zhu, et al, 2015
❑ ResearchQuestion: “Is oral or topical antibiotics necessary for children with mild
clinically affected eczema? Or are topical steroid and emollient treatment
sufficient therapy?”
❑ StudyDesign:Three armed, randomized, blinded study, in an ambulatory setting.
Children with clinically mildly infected eczema were randomized by control, oral
antibiotic and topical placebo, or topical and oral placebo for a week.
❑ Results:It was found there was no need to prescribe oral or topical antibiotics to
treat children with clinically mildly infected eczema; and that topical steroids and
emollient was therapeutically effective (Zhu, et al, 2015).
❑ Utility: This finding has great significance for the application of these therapies
and the standard of care for patients. In the light of the increasing treat of over-
prescribing antibiotics and the threat of resistance and superbug formation, this
finding is of great utility to all stakeholders
❑ Synthesis:This complex study design perfectly highlights the utility of the
pragmatic study, in which diverse outcomes can be assessed through various
treatment group assignments.

20. SUMMARY/CONCLUSION
❑We are currentlyin the genesis of
the union of clinical research and
clinical practice, an astonishingly
radical paradigm that will take a few
more decadesto fully materialize.
❑It is obviously of immense
importance as regulatorybodies
are in the initial stages of surveying
the utility of this new paradigm.
❑Ultimately, there is a great need
for sound scientific data to help
guide decisions in clinical practice.
❑While meta-analysis can inform
stakeholders on decisions, it is
dependenton data from RCT’s which
are not designed to provide
generalizable information on the
extrinsic validity and applicability of a
therapeutic.
❑RCT’s are limited in their scope to
provide information on the external
validity and applicability of a
therapeutic.
❑Pragmatic clinical studies, however,
are specifically designed to provide
information on the external validity
and applicability of a therapeutic, and
thus should be the standard to attain
information for real world applicability
of a therapeutic.

21. REFERENCES
Kukull, W. A., & Ganguli, M. (2012). Generalizability:The trees, the forest, and the low-hanging fruit.
Neurology, 78(23), 1886–1891.http://doi.org/10.1212/WNL.0b013e318258f812
Leung, L. (2015). Validity, reliability,andgeneralizability in qualitativeresearch. Journal of Family
Medicine and Primary Care, 4(3), 324–327. http://doi.org/10.4103/2249-4863.161306
Masic, I., Miokovic, M., & Muhamedagic, B. (2008). Evidence Based Medicine – New Approaches and
Challenges. Acta Informatica Medica, 16(4), 219–225.http://doi.org/10.5455/aim.2008.16.219-225
S, S. (2017, March10). Difference Between Internal and External Validity (with Comparison Chart).
Retrieved August 20, 2017, from http://keydifferences.com/difference-between-internal-and-external-
validity.html
Stang, A. (2011). RandomizedControlled Trials—an Indispensible Part of Clinical Research. Deutsches
ÄrzteblattInternational, 108(39), 661–662. http://doi.org/10.3238/arztebl.2011.0661
United States of America, U.S. Department of Health & Human Services, Agency for Healthcare
Research and Quality (AHRQ). (2013, March). Using PragmaticClinical Trials to test the Effectiveness of
Patient-CenteredMedical Home Models in Real-World Setting. Retrieved August 02, 2017, from
https://pcmh.ahrq.gov/sites/default/files/attachments/UsingPragmatic_032513comp.pdf
Zhu, Y., Cheng, Y., Luo, R.-C., & Li, A.-M. (2015). Aspirin for the primary preventionof skin cancer: A
meta-analysis. Oncology Letters, 9(3), 1073–1080. http://doi.org/10.3892/ol.2015.2853