Contact for this presentation:
Alexander K. Rowe, MD, MPH
Malaria Branch, Division of Parasitic Diseases and Malaria, Cent...
Early results of a systematic review
of strategies to improve health care
provider performance in low- and
middle-income c...
Background
• Health care providers (HCPs) play key roles
in improving quality and coverage of health
interventions
• In lo...
Objectives
Conduct systematic review of strategies to
improve HCP performance and related health
outcomes in LMICs, produc...
Methods: inclusion and exclusion criteria
• Source of studies. Published and unpublished
• Timing. For electronic database...
Methods: inclusion and exclusion criteria
• Source of studies. Published and unpublished
• Timing. For electronic database...
• Types of outcomes
– Direct measures of HCP behavior (e.g., tasks related
to diagnosis, treatment, counseling)
– Patient ...
• Types of outcomes
– Direct measures of HCP behavior (e.g., tasks related
to diagnosis, treatment, counseling)
– Patient ...
Methods: classification of study design
• “Adequate” study designs for primary analysis:
– Pre-post with comparison (+/- r...
Methods: classification of study design
• “Adequate” study designs for primary analysis:
– Pre-post with comparison (+/- r...
Methods: literature search strategy
• Searched: 15 electronic databases (e.g., MEDLINE,
CINHL, EPOC specialized register, ...
Methods: data abstraction
• Double, independent abstraction with
20-page abstraction form
• Data entered into Access datab...
• Effect size in terms of %-point change
• If outcome is a percentage, effect size =
(%POST – %PRE)intervention
– (%POST –...
Early, illustrative results
(might not represent final dataset)
• >105,000 citations screened
• After removing duplicates: 2,430 reports
identified for abstraction (all study designs)
Re...
Early results: data abstraction
• From August 2007, studies with “adequate”
designs prioritized to be abstracted first
(85...
Early results: study designs (N=275)
• “Adequate” designs (223/275, or 81%)
− 145 randomized studies (54%)
• Other designs...
Early results: strategies tested (N=275)
• 390 intervention groups (i.e., strategies tested)
• 456 comparisons
− 177 inter...
Early results on effectiveness:
Analysis of 213 comparisons from
172 studies with adequate design
(all comparisons are:
in...
0
5
10
15
20
25
30
35
Magnitude of effect size (percentage-point change)
No.ofMES
-100
to
-109
-0.1
to
-9
0
to
9
10
to
19
...
0
5
10
15
20
25
30
35
Magnitude of effect size (percentage-point change)
No.ofMES
-100
to
-109
-0.1
to
-9
0
to
9
10
to
19
...
0
5
10
15
20
25
30
35
Magnitude of effect size (percentage-point change)
No.ofMES
-100
to
-109
-0.1
to
-9
0
to
9
10
to
19
...
-110
-60
-10
40
90
140
16 315No. comparisons
Trainingonly
Jobaids/printed
mat.forHCPonly
Training+jobaidsor
printedmat.for...
-110
-60
-10
40
90
140
16 315No. comparisons
Trainingonly
Jobaids/printed
mat.forHCPonly
Training+jobaidsor
printedmat.for...
-110
-60
-10
40
90
140
16 315No. comparisons
Median MES (%-pts) 10 7 2 13 13 6 10 17
10 17 18 79 14
Medianeffectsize(%-poi...
-110
-60
-10
40
90
140
Medianeffectsize(%-points)
1 2 3 4 5 >6
No. of components in the strategy
Does MES vary by the numb...
-110
-60
-10
40
90
140
Medianeffectsize(%-points)
1 2 3 4 5 >6
No. of components in the strategy
75th
percentile
median
25...
• Extreme heterogeneity among studies
– Settings
– Outcomes
– Interventions (e.g., not all “training” is same)
• MES crude...
1. Surprisingly large evidence base on effectiveness
of strategies, with many strategies tested in many
settings
2. Effect...
4. Evidence base seems fragmented
– Researchers not building on each others’ work
– Varying methods makes it difficult to ...
Acknowledgments
• Charity Akpala
• Tashana Carty
• Adrijana Corluka
• Didi Cross
• Bhavya Doshi
• Onnalee Gomez
• Meg Grif...
Thanks for your
attention!
“Results! Why, man, I have gotten a lot of results.
I know several thousand things that won’t w...
Upcoming SlideShare
Loading in …5
×

Presentation on the literature review of interventions to improve health care provider performance

2,077 views

Published on

This presentation was given in a Future Health System Consortium organised session at the Global Symposium on Health Systems Research in November 2010. The author is Alex Rowe from the Centers for Disease Control and Prevention.

Published in: Health & Medicine, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,077
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Distribution of 74 median effect sizes (1 per comparison) from 56 studies
  • Distribution of 74 median effect sizes (1 per comparison) from 56 studies
  • Distribution of 74 median effect sizes (1 per comparison) from 56 studies
  • Effect size as a function of number of components in strategy (N=74 comparisons)
  • Effect size as a function of number of components in strategy (N=74 comparisons)
  • Presentation on the literature review of interventions to improve health care provider performance

    1. 1. Contact for this presentation: Alexander K. Rowe, MD, MPH Malaria Branch, Division of Parasitic Diseases and Malaria, Center for Global Health Centers for Disease Control and Prevention Mailstop F22 4770 Buford Highway Atlanta, GA 30341 United States Telephone: 1-770-488-3588 Fax: 1-770-488-7761 Email: axr9@cdc.gov Presentation on the literature review of interventions to improve health care provider performance (for the First Global Symposium on Health Systems Research, Session on Evidence on Improving Health Service Delivery, November 18, 2010)
    2. 2. Early results of a systematic review of strategies to improve health care provider performance in low- and middle-income countries Alexander K. Rowe (Malaria Branch, Centers for Disease Control and Prevention [CDC]) Samantha Y. Rowe (CDC) David H. Peters (Johns Hopkins University) Kathleen A. Holloway (World Health Organization) John Chalker (Management Sciences for Health) Dennis Ross-Degnan (Harvard Medical School)
    3. 3. Background • Health care providers (HCPs) play key roles in improving quality and coverage of health interventions • In low- and middle-income countries (LMICs), however, HCP performance often inadequate • Most existing reviews* of strategies to improve performance are outdated or have important methodological limitations * Two new reviews on community/lay health workers
    4. 4. Objectives Conduct systematic review of strategies to improve HCP performance and related health outcomes in LMICs, produce: 1. Library & database of studies on improving HCP performance, for policy-makers and researchers 2. Detailed report and summary articles on effectiveness and costs of strategies 3. Evidence-based policy recommendations (i.e., guidance on implementing clinical guidelines in LMICs) and associated research agenda
    5. 5. Methods: inclusion and exclusion criteria • Source of studies. Published and unpublished • Timing. For electronic databases, studies by May 2006; for other methods (e.g., gray literature), no a priori end of search date (i.e., some studies after 2006) • Setting. LMICs, as per World Bank
    6. 6. Methods: inclusion and exclusion criteria • Source of studies. Published and unpublished • Timing. For electronic databases, studies by May 2006; for other methods (e.g., gray literature), no a priori end of search date (i.e., some studies after 2006) • Setting. LMICs, as per World Bank • Type of health condition. No exclusions; performance related to any health condition is acceptable • HCPs. Facility- or community-based health workers, pharmacists, shopkeepers who sell medicines, and private sector health workers; only exclusions were household providers (e.g., patient’s family)
    7. 7. • Types of outcomes – Direct measures of HCP behavior (e.g., tasks related to diagnosis, treatment, counseling) – Patient outcomes (e.g., mortality, health care utilization) – Other (e.g., economic outcomes, HCP knowledge) Methods: inclusion and exclusion criteria
    8. 8. • Types of outcomes – Direct measures of HCP behavior (e.g., tasks related to diagnosis, treatment, counseling) – Patient outcomes (e.g., mortality, health care utilization) – Other (e.g., economic outcomes, HCP knowledge) • Sample size. >20 observations per study group and time point • Language of the study. No exclusions Methods: inclusion and exclusion criteria
    9. 9. Methods: classification of study design • “Adequate” study designs for primary analysis: – Pre-post with comparison (+/- randomization) – Post-only with randomized comparison group – Interrupted time series (>3 data points before and after intervention)
    10. 10. Methods: classification of study design • “Adequate” study designs for primary analysis: – Pre-post with comparison (+/- randomization) – Post-only with randomized comparison group – Interrupted time series (>3 data points before and after intervention) • Despite interest in other study designs, due to very large number of reports, focus on studies with adequate designs
    11. 11. Methods: literature search strategy • Searched: 15 electronic databases (e.g., MEDLINE, CINHL, EPOC specialized register, etc.) • Bibliographies of 510 previous reviews & other articles • Document inventories and websites of 29 organizations involved with HCP performance (e.g., MSH, BASICS, Core Project, DFID, PAHO, QAP, USAID, World Bank) • INRUD Bibliography and WHO Rational Use of Drugs database • Asked colleagues for references & unpublished studies
    12. 12. Methods: data abstraction • Double, independent abstraction with 20-page abstraction form • Data entered into Access database • Queries to study investigators (clarifications, details on contextual factors, etc.)
    13. 13. • Effect size in terms of %-point change • If outcome is a percentage, effect size = (%POST – %PRE)intervention – (%POST – %PRE)control • Calculate effect size such that positive = success • If >1 outcome, take median effect size (MES) of primary outcomes Methods: analysis of effect size
    14. 14. Early, illustrative results (might not represent final dataset)
    15. 15. • >105,000 citations screened • After removing duplicates: 2,430 reports identified for abstraction (all study designs) Results: Literature search
    16. 16. Early results: data abstraction • From August 2007, studies with “adequate” designs prioritized to be abstracted first (851 reports on about 425 distinct studies) • To date, 463 reports double-abstracted on 275 distinct studies • Studies from >60 LMICs (50% low-income) • Wide range of health conditions addressed by HCPs: nutrition; antenatal care; treatment of ARI, diarrhea, or malaria; family planning, etc.
    17. 17. Early results: study designs (N=275) • “Adequate” designs (223/275, or 81%) − 145 randomized studies (54%) • Other designs (52/275, or 19%), e.g., pre-post study without controls (not primary focus)
    18. 18. Early results: strategies tested (N=275) • 390 intervention groups (i.e., strategies tested) • 456 comparisons − 177 intervention vs. “no intervention” control − 249 intervention A vs. intervention B • Most strategies had multiple components (e.g., training + supervision = 2 components) • Median of 4 components/strategy (range: 1–17) • Commonly tested strategies include: training, supervision or feedback, community activities, new guideline, printed materials, and job aids
    19. 19. Early results on effectiveness: Analysis of 213 comparisons from 172 studies with adequate design (all comparisons are: intervention vs. no intervention control) (might not represent final dataset)
    20. 20. 0 5 10 15 20 25 30 35 Magnitude of effect size (percentage-point change) No.ofMES -100 to -109 -0.1 to -9 0 to 9 10 to 19 -20 to -29 -30 to -39 -10 to -19 20 to 29 40 to 49 50 to 59 30 to 39 60 or higher Distribution of 213 MES from 172 studies with adequate design (all strategies) Min = -105 Median = 9 Max = 130 IQR: 3–23
    21. 21. 0 5 10 15 20 25 30 35 Magnitude of effect size (percentage-point change) No.ofMES -100 to -109 -0.1 to -9 0 to 9 10 to 19 -20 to -29 -30 to -39 -10 to -19 20 to 29 40 to 49 50 to 59 30 to 39 60 or higher Distribution of 213 MES from 172 studies with adequate design (all strategies) 50% are small (<10%-points) or negative Min = -105 Median = 9 Max = 130 IQR: 3–23
    22. 22. 0 5 10 15 20 25 30 35 Magnitude of effect size (percentage-point change) No.ofMES -100 to -109 -0.1 to -9 0 to 9 10 to 19 -20 to -29 -30 to -39 -10 to -19 20 to 29 40 to 49 50 to 59 Min = -105 Median = 9 Max = 130 IQR: 3–23 30 to 39 60 or higher Distribution of 213 MES from 172 studies with adequate design (all strategies)
    23. 23. -110 -60 -10 40 90 140 16 315No. comparisons Trainingonly Jobaids/printed mat.forHCPonly Training+jobaidsor printedmat.forHCP Median MES (%-pts) 10 7 2 13 13 6 10 17 10 17 18 79 14 Train+qualmgmt+ jobaids/printed materialsforHCP Community activities+other Non-community activitiesinvolving commodities+other Training+quality management Qualitymgmt techniqueonly Medianeffectsize(%-points) 75th percentile median 25th percentile
    24. 24. -110 -60 -10 40 90 140 16 315No. comparisons Trainingonly Jobaids/printed mat.forHCPonly Training+jobaidsor printedmat.forHCP Median MES (%-pts) 10 7 2 13 13 6 10 17 10 17 18 79 14 Train+qualmgmt+ jobaids/printed materialsforHCP Community activities+other Non-community activitiesinvolving commodities+other Training+quality management Qualitymgmt techniqueonly Medianeffectsize(%-points) 75th percentile median 25th percentile
    25. 25. -110 -60 -10 40 90 140 16 315No. comparisons Median MES (%-pts) 10 7 2 13 13 6 10 17 10 17 18 79 14 Medianeffectsize(%-points) 75th percentile median 25th percentile • All median values small/moderate (2–17 %-points) • Some groups with large variation (some effects large; potential for positive deviance analysis)
    26. 26. -110 -60 -10 40 90 140 Medianeffectsize(%-points) 1 2 3 4 5 >6 No. of components in the strategy Does MES vary by the number of components in the strategy?
    27. 27. -110 -60 -10 40 90 140 Medianeffectsize(%-points) 1 2 3 4 5 >6 No. of components in the strategy 75th percentile median 25th percentile Does MES vary by the number of components in the strategy?
    28. 28. • Extreme heterogeneity among studies – Settings – Outcomes – Interventions (e.g., not all “training” is same) • MES crude summary measure that can mask variation in effect (improved methods in development) • Precision very difficult to assess Limitations
    29. 29. 1. Surprisingly large evidence base on effectiveness of strategies, with many strategies tested in many settings 2. Effect sizes vary substantially: half have small effects, but some have large effects 3. Analyses to identify factors associated with high effectiveness Early conclusions (1)
    30. 30. 4. Evidence base seems fragmented – Researchers not building on each others’ work – Varying methods makes it difficult to put pieces together 4. Standardizing methods & outcomes would improve ability to summarize evidence base and develop effective and practical recommendations; perhaps coordinated research agenda is needed 5. Final results pending (end of 2012) Early conclusions (2)
    31. 31. Acknowledgments • Charity Akpala • Tashana Carty • Adrijana Corluka • Didi Cross • Bhavya Doshi • Onnalee Gomez • Meg Griffith • Karen Herman • Qing Li • Connie Liu • Earl Long • Eliza McLeod • Dawn Osterholt • Gabriel Ponce-de-Léon • Nancy Pulsipher • Atiq Rahman • Nirali Shah • Banafsheh Siadat • Sanja Stanojevic • Savitha Subramanian • Jeff Willis • Kindra Willis • Shannon Wood • Karen Wosje • Alicia Wright • Chunying Xie • Special thanks to investigators who responded to queries • Funding: Bill and Melinda Gates Foundation, CDC, World Bank
    32. 32. Thanks for your attention! “Results! Why, man, I have gotten a lot of results. I know several thousand things that won’t work.” Thomas A. Edison (1847–1931)

    ×