Using primary care databases to
evaluate drug benefits and harms:
are the results valid and
replicable?
David A. Springate,...
Outline
1. Primary Care Database (PCD) study validity
2. PCD Replications
– Statins and Ischaemic heart disease
– β-blocke...
PCD studies are all the rage. . .
Number of UK PCD
publications is rapidly
increasing
1990 1995 2000 2005 2010
050100150
P...
Uses of Primary care databases. . .
• Prevalence / incidence studies
• Associations between conditions
• Harms and risks o...
BUT, There are still concerns about the
validity of PCD-based studies. . .
Threat Refs
Data quality Herrett 2009,
Khan 201...
PCD Replications
“Non-reproducible single
occurrences are of no
significance to science.”
—– Karl Popper (1959)
An approach...
Replicating studies in another, independent
PCD
• Agreement implies that conclusions are not dependent
on data source
• So...
Replications were performed in CPRD
• Largest UK primary care database (CPRD-GOLD)
• ˜ 14 million patients
• ˜ 650 practic...
Criteria for replication
1. Effectiveness studies
2. Different GP computer system from CPRD (not
Vision)
3. No practice over...
PCD replications
Hippisley-Cox and Coupland (2006) Effect of statins on the
mortality of patients with ischaemic heart dise...
Effect of statins on the mortality of patients
with ischaemic heart disease (QResearch)
Objective To measure the effect of s...
Effect of statins on the mortality of patients
with ischaemic heart disease
Summary statistics
Measure Analysis CPRD Qresea...
Mortality rates for patients with IHD
Age Comorbidity Sex
0
100
200
300
400
Total0−4445−5455−6465−7475−8485−94
95_plus
N
o...
Survival analyses for patients on Statins
QResearch
Adjusted HR 0.47 (0.41 to 0.53)
CPRD
0 2 4 6 8
Time since diagnosis of...
Case-control analysis: Odds ratios for effects
of Statins on mortality in IHD patients
Odds are relative to
patients not on...
Misleading pooled odds ratios
Combining effects of e.g. two drugs of the same BNF
chapter
Group Y N Odds ratio
Drug 1 Cases...
Case-control analysis: Adjusted OR for
duration of use of statins on survival
Odds are relative to
patients not on statins...
Summary — Statins study
1. Strikingly similar results in the two studies, despite
different GP computer systems (Vision vs ...
Does β-adrenoceptor blocker therapy improve
cancer survival?(DIN-LINK)
Objective To examine the effect of β-blocker treatme...
Comparison of summary statistics
Cohort size:
CPRD 11316
DIN-LINK 3462
BPLM = Blood pressure
lowering medicines
Deaths Ali...
Comparison of patient samples by cancer site
BPLM = Blood pressure
lowering medicines
0
10
20
30
40
breast
colon
lungoesop...
Survival analyses: comparison of hazard ratios
Cancer type
breast
colon
lung
oesophagus
ovarian
pancreas
prostate
renal
st...
Summary — β-blocker study
1. Different individual cancer HR’s and overall
conclusions
2. Important differences in some cohor...
Summary — β-blocker study
1. Different individual cancer HR’s and overall
conclusions
2. Important differences in some cohor...
”an experiment is reproducable until another
laboratory tries to repeat it.” — Alexander Kohn
”an experiment is reproducable until another
laboratory tries to repeat it.” — Alexander Kohn
• Artifact of differences in ...
”an experiment is reproducable until another
laboratory tries to repeat it.” — Alexander Kohn
• Artifact of differences in ...
Conclusions / recommendations
These replications add to the evidence
that PCD results are valid
Conclusions / recommendations
These replications add to the evidence
that PCD results are valid
• PCD Replication is hard!
Conclusions / recommendations
These replications add to the evidence
that PCD results are valid
• PCD Replication is hard!...
Conclusions / recommendations
These replications add to the evidence
that PCD results are valid
• PCD Replication is hard!...
Thanks. . .
Research team
David Reeves
Evan Kontopantelis
Ivan Olier
Darren Ashcroft
Authors of the original
studies
Iain ...
Upcoming SlideShare
Loading in...5
×

Using primary care databases to evaluate drug benefits and harms: are the results replicable and valid?

208

Published on

Databases of electronic medical records and in particular primary care databases (PCDs) are increasingly used in research. The largest PCDs contain full data on all primary care consultations by millions of patients over two or more decades. They provide a means for investigating important healthcare questions which cannot be practically addressed in a Randomised Controlled Trial. However, concerns remain about the validity of studies based on data from PCDs. Most work around validity has attempted to confirm individual data values within a dataset. We take a different approach and instead replicate published PCD studies in a second, independent, PCD. Agreement of results then implies that the conclusions drawn are independent of the data source (though this doesn’t rule out that such as confounding by indication are commonly influencing both).
We replicated two previous PCD studies using the Clinical Practice Research Datalink (CPRD). The first was a retrospective cohort study of the effect of Beta-blocker therapy on survival in cancer patients using DIN-LINK. The second was a nested case-control analysis of the effects of Statins on mortality of patients with ischaemic heart disease using QRESEARCH.
Our analyses produced several important quantitative differences compared to the original studies, altering conclusions. These could not be fully explained by either demographic differences in the patient samples or structural differences between the datasets. Our study highlights both the caution that needs to be applied when assessing the findings from analysis of just a single database and the difficulties in performing replications of existing PCD studies.

Published in: Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
208
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Using primary care databases to evaluate drug benefits and harms: are the results replicable and valid?

  1. 1. Using primary care databases to evaluate drug benefits and harms: are the results valid and replicable? David A. Springate, University of Manchester Centres for Primary Care/Biostatistics
  2. 2. Outline 1. Primary Care Database (PCD) study validity 2. PCD Replications – Statins and Ischaemic heart disease – β-blockers and Cancer 3. Lessons to be learned
  3. 3. PCD studies are all the rage. . . Number of UK PCD publications is rapidly increasing 1990 1995 2000 2005 2010 050100150 PCD articles in pubmed year Numberofarticles There is global interest in UK PCD research Institutions affiliated with UK PCD publications xx x x x xxxxxx x x x x xx x x x xx x xx x xxx xx xxx x xx xxxxxx xx xx x xx x x x xx x x x xx x x xxxxxx x x x x x x xx x xxx x xxxxx xxx xxx x x x x xx xxx xx x xx x x xxx xx x x x x x x x x x x xx xx x xxx x x x x x x x x xx xx x xx xxxx x x x x xx x xx x xxx x xx xx xx x x xx x xxxxxx x x x xx x xxx x x xxx x x x x xxxxx x xx x xx xxxxxx xx xx x x x x xx x xxx x x xx xx x xxxxx x xxxxx x xx xxx x x x xx xx xxx x x x x xx x xx xx x x x x xx xx xxxxxx x x x xx x x x x x x x xx x x x x x xx x x x xxx x xxxxx x x x xxxx x x xxxxx xx xx x x xxxxxxxx xxx x xxxxx x x xx xxx x x xx xxxx x x xxx xx x xx xxxxx x xx x x xx x x x xx x x x xxx x xx x xxxx xx xxx xx x xx x xx x xxxx xxx x x xxx x x xxxx x x x x x x x x x x x x x x xxxxxx xxxx xx x xxx x x x x x x x x xx x x x x x xxx x x x xx x xxx x x x x x x x xx x x x x x x xx xx xxxx x x x x xxx x x xx xxx x x xxx x x x xx xxxxxxxxx xx xx x xxxx xx x xxxx x x xx x x x x xx x xxx x xx xxxxxx xx x xx x x x xxx x x x xxxxx xx xx x x x x x x x x x x x xxxxx x xx xxx x xxx x x x x x x x xx x x x x x xxx xx x xxxxx x x xx xx x x x xxxxxxx xx x x xxxx xx xx x x x x xxxx xx xx xxx xxx x xx xx x xxx x x x x x xxx x x x x xxxx x x x x xxxx xxx xxxxxxxx x xx xx xx x xxxx x x x xxxx x x x xx xxxx xx x xx xxx xxx x x x xxx xxxx xxxx x xx x x x x xx x x x x xx xxx x x x x x x x x xxxxxxxx x xxx xx x xxx x xx xxxx xx x xxxxxxxx xxxxx x xx xx x xxxxxxx x x xx xxx x x xx x xx xx x x xx x x x xxx x x x xxx x x xx xx xx xxx x x x xx x x xxx x x x x xx x x xxx x xx xxxxxxxx x x x x x x xx x xxxxxx x x xxxx xxx x xxx x x x x xx x x xx x x x x x x x x x x x x x x xx x xx xx x x x xx x x x xxx x xxx x xx xx x x x x x xx xx xx x x xx x xxxxxxx x xxxxxxxxxxx xxxxxxxxxxx xxx x x x xxxx x xxxx xxxxxxxxxxxxxxxxxx xxxxx
  4. 4. Uses of Primary care databases. . . • Prevalence / incidence studies • Associations between conditions • Harms and risks of treatments • Comparative effectiveness • RCT comparisions / replications (and replacements?)
  5. 5. BUT, There are still concerns about the validity of PCD-based studies. . . Threat Refs Data quality Herrett 2009, Khan 2010, Jordan 2004 Data completeness Marston 2010, Delaney 2007, Collins 2010 Confounding Tannen 2008, Lewis 2007 Clinical coding www.ClinicalCodes.org
  6. 6. PCD Replications “Non-reproducible single occurrences are of no significance to science.” —– Karl Popper (1959) An approach to validity that asks whether flaws and differences in the data make any difference to the ultimate conclusions rather than looking at validity and completeness of the underlying individual data http://xkcd.com/242
  7. 7. Replicating studies in another, independent PCD • Agreement implies that conclusions are not dependent on data source • Some factors could influence both (such as confounding by indication) • First completely independent PCD replication (but see Vinogradova (BMJ 2013): relationship between bisphosphate exposure and cancer in QResearch and CPRD)
  8. 8. Replications were performed in CPRD • Largest UK primary care database (CPRD-GOLD) • ˜ 14 million patients • ˜ 650 practices across the UK • Uses the Vision GP computer system
  9. 9. Criteria for replication 1. Effectiveness studies 2. Different GP computer system from CPRD (not Vision) 3. No practice overlap with CPRD 4. Representative coverage 5. Primary Care Database (Not integrated primary/secondary/pharmacy)
  10. 10. PCD replications Hippisley-Cox and Coupland (2006) Effect of statins on the mortality of patients with ischaemic heart disease: population based cohort study with nested case-control analysis. Heart 92:752-758 (QResearch) Shah, Carey et al. (2011) Does β-adrenoceptor blocker therapy improve cancer survival? Findings from a population-based retrospective cohort study Br J Clin Pharmacol 72:157-161 (DIN-LINK) We then compare • summary statistics • mortality rates • model coefficients and standard errors
  11. 11. Effect of statins on the mortality of patients with ischaemic heart disease (QResearch) Objective To measure the effect of statins on mortality for community based patients with IHD Design Cohort survival analysis and nested case-control Setting 1.18 million patients in 89 practices Subjects Patients with first diagnosis of IHD between January 1996 and December 2003 Outcomes - Cohort: Adjusted hazard ratios (+/- 95%CI) for all-cause mortality - Case-control: Odds ratio (+/-95%CI) for current use of statins, previous use and duration of use
  12. 12. Effect of statins on the mortality of patients with ischaemic heart disease Summary statistics Measure Analysis CPRD Qresearch Number of practices Cohort 661 89 Number of patients Cohort 91589 13029 Cases 15591 2266 Controls 62356 9064 Median age Cases 80 80 Controls 79 80 Percent female Cases 45.5% 44.3% Controls 45.5% 44.3% Median followup (months) Cases 22.1 20.3 Controls 22.5 21 Percent on statins Cases 17% 19.6% Controls 23.6 25.4%
  13. 13. Mortality rates for patients with IHD Age Comorbidity Sex 0 100 200 300 400 Total0−4445−5455−6465−7475−8485−94 95_plus N o diabetes D iabetes N o hypertension H ypertensionN o C C F C C F Fem ale M ale Patient group Mortalityrate(per1000personyears) PCD CPRD QResearch
  14. 14. Survival analyses for patients on Statins QResearch Adjusted HR 0.47 (0.41 to 0.53) CPRD 0 2 4 6 8 Time since diagnosis of IHD (Years) Survival 0.000.250.500.751.00 Adjusted HR 0.43 (0.40 to 0.46)
  15. 15. Case-control analysis: Odds ratios for effects of Statins on mortality in IHD patients Odds are relative to patients not on statins. Dotted line represents 1:1 odds Previously on statins Currently on statins 0.0 0.5 1.0 1.5 2.0 AllStatins Atorvastatin C erivastatinFluvastatinPravastatinSim vastatin AllStatins Atorvastatin C erivastatinFluvastatinPravastatinSim vastatin Statin type Oddsratio PCD CPRD Qresearch
  16. 16. Misleading pooled odds ratios Combining effects of e.g. two drugs of the same BNF chapter Group Y N Odds ratio Drug 1 Cases 600 200 3/3 = 1 Drug 1 Controls 75 25 Drug 2 Cases 10 30 0.333/1 = 0.333 Drug 2 Controls 30 30 Pooled Cases 610 230 2.65/1.91 = 1.39 Pooled Controls 105 55
  17. 17. Case-control analysis: Adjusted OR for duration of use of statins on survival Odds are relative to patients not on statins. Dotted line represents 1:1 odds 0.00 0.25 0.50 0.75 1.00 0−12 13−24 25−36 37−48 49−60 >60 Duration (months) AdjustedOddsratio PCD CPRD Qresearch
  18. 18. Summary — Statins study 1. Strikingly similar results in the two studies, despite different GP computer systems (Vision vs EMIS) 2. As expected, narrower confidence intervals due to larger study 3. Original study was well designed (Matching, appropriate analyses etc.) 4. Given the results, pooling of “all statins” is questionable
  19. 19. Does β-adrenoceptor blocker therapy improve cancer survival?(DIN-LINK) Objective To examine the effect of β-blocker treatment on cancer survival Design Survival analyses for 9 cancer types Setting 3462 cancer patients on β-blocker or other antihypertensive therapy Subjects Patients 40-85 with first cancer diagnosis between 1997 and 2006 Outcomes - Adjusted hazard ratios (+/- 95%CI) for all-cause mortality in each cancer type - Pooled hazard ratio (random effects)
  20. 20. Comparison of summary statistics Cohort size: CPRD 11316 DIN-LINK 3462 BPLM = Blood pressure lowering medicines Deaths Alive at 1 year Diagnosed 1997−2001 Diagnosed 2002−2006 0 25 50 75 100 Beta−blockersO therBPLM Beta−blockersO therBPLM Beta−blockersO therBPLM Beta−blockersO therBPLM Drug combination Percentageofpatients PCD CPRD DIN−LINK
  21. 21. Comparison of patient samples by cancer site BPLM = Blood pressure lowering medicines 0 10 20 30 40 breast colon lungoesophagus ovarian pancreas prostate renal stom ach Cancer site Percentageofpatients PCD CPRD DIN−LINK
  22. 22. Survival analyses: comparison of hazard ratios Cancer type breast colon lung oesophagus ovarian pancreas prostate renal stomach Overall Hazard ratio 1.09 (0.8, 1.49) 1 (0.77, 1.3) 1.12 (0.89, 1.41) 1.05 (0.69, 1.6) 1.14 (0.63, 2.06) 1.88 (1.09, 3.25) 1.54 (1.13, 2.09) 1.14 (0.52, 2.52) 1.44 (0.76, 2.74) 1.18 (1.04, 1.33) % weight 14.85 20.93 25.7 8.17 4.27 4.91 15.19 2.37 3.61 100 0 1 2 3 4 all Beta−blockers vs controls DIN−LINK Cancer type breast colon lung oesophagus ovarian pancreas prostate renal stomach Overall Hazard ratio 1.19 (1.03, 1.37) 0.85 (0.74, 0.97) 1.04 (0.91, 1.19) 1.27 (1.01, 1.59) 1.05 (0.74, 1.5) 0.94 (0.74, 1.21) 1.03 (0.92, 1.15) 0.46 (0.26, 0.83) 1.03 (0.78, 1.36) 1.01 (0.91, 1.13) % weight 14.87 15.11 15.36 10.7 6.32 9.76 16.27 2.89 8.71 100 0 1 2 3 4 all Beta−blockers vs controls CPRD
  23. 23. Summary — β-blocker study 1. Different individual cancer HR’s and overall conclusions 2. Important differences in some cohort statistics 3. Differences remain after correcting to give the same patient:practice 4. Differences remain after reducing the size of the CPRD study 5. Databases appear to be demographically similar (Carey et al. 2004)
  24. 24. Summary — β-blocker study 1. Different individual cancer HR’s and overall conclusions 2. Important differences in some cohort statistics 3. Differences remain after correcting to give the same patient:practice 4. Differences remain after reducing the size of the CPRD study 5. Databases appear to be demographically similar (Carey et al. 2004) WHY?
  25. 25. ”an experiment is reproducable until another laboratory tries to repeat it.” — Alexander Kohn
  26. 26. ”an experiment is reproducable until another laboratory tries to repeat it.” — Alexander Kohn • Artifact of differences in GP computer systems? DIN-LINK uses Torex/iSoft systems (See Kontopantelis et al 2013)
  27. 27. ”an experiment is reproducable until another laboratory tries to repeat it.” — Alexander Kohn • Artifact of differences in GP computer systems? DIN-LINK uses Torex/iSoft systems (See Kontopantelis et al 2013) • Analysis methods? – No matching - potential confounding – No clustering by practice – Limited control for covariates – Is meta-analysis the most appropriate method (Assumes independence)? • Data quality?
  28. 28. Conclusions / recommendations These replications add to the evidence that PCD results are valid
  29. 29. Conclusions / recommendations These replications add to the evidence that PCD results are valid • PCD Replication is hard!
  30. 30. Conclusions / recommendations These replications add to the evidence that PCD results are valid • PCD Replication is hard! – Methods details are inadequate for replication – Clinical codes not provided with the original article – Relies on active cooperation of authors of original studies – Even then, ambiguity can remain
  31. 31. Conclusions / recommendations These replications add to the evidence that PCD results are valid • PCD Replication is hard! – Methods details are inadequate for replication – Clinical codes not provided with the original article – Relies on active cooperation of authors of original studies – Even then, ambiguity can remain • Publish full methods (in online appendix?) • Publish full clinical code lists... ClinicalCodes.org
  32. 32. Thanks. . . Research team David Reeves Evan Kontopantelis Ivan Olier Darren Ashcroft Authors of the original studies Iain Carey (St. Georges University, London) Carol Coupland (University of Nottingham) Contact: david.springate@manchester.ac.uk
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×