Starting with the setting of priorities on what gets solicited in federal RFAs, RFPs and other solicitations for research, and in foundation announcements of research grant programs… So it is truly a systemic problem that limits what gets researched, what gets published, what gets included in the systematic reviews that lead to guidelines, how much the guidelines miss the mark of what practitioners, planners and policymakers need, and therefore what they use and implement of the research that has been disseminated to them.
Limitations of the RCT
1. Copies of the order form available. Website for full text. 2.
No one has summarized all that I’ve said with greater simplicity and clarity than Gil Ramirez, a biostatistician, on the Community Preventive Services Task Force, formerly at UT Houston, then Drew Medical School, and now here at Florida International University’s new School of Public Health.
There is nothing truly new about these, but they are neglected in comparison with the emphasis on publishing and using as guidance for policy and practice the types of evidence drawn from randomized controlled trials.
Practice-based research, either “action research” or “participatory”
The first is the one that most justifies the investment in CBPR approaches. The second presents its limitation or trade-off. The third paradox presents the homophily-heterophily dilemma in trying to integrate the project into the community, the community into the project, and the community health worker caught in the middle…
Katz DL, Murimi M, Gonzalez A, Nijike V, Green LW. From clinical trial to community adoption: The Multi-site Translational Community Trial (mTCT). Am J Public Health. Published online ahead of print June 16, 2011: e1–e11. doi:10.2105/AJPH.2010.300104, 2011.
Designing for Dissemination and Implementation Making Research More Relevant & Actionable for Translation to Other Populations and Settings Lawrence W. Green
The Challenges & Opportunities• The two biggest challenges: – To close the gap between what policy makers, program planners, practitioners and communities need & what they are getting from our research – Reform some peer review & editorial tendencies• The two biggest opportunities – Extend CBPR principles to work with policy makers, program planners & practitioners in use of natural experiments—e.g., evaluation – Combine PR with multi-site RCT methods that expand the external validity of the results
Where Have All the Data Gone? Longtime Passing… 17 yrs ―It takes 17 years to turn 14 per cent of original [applied] research to the benefit of patient care‖ * *Balas & Boren, 2000. Reviews, guidelines, textbooks Unknown 0.5 year 0.6 year 0.3 year 6. 0 - 13.0 years 9.3 years Bibliographic databases Kumar, 1992 Kumar, 1992 Poyer, 1982 Antman, 1992Original research Implementation Acceptance Submission Publication18% Negative 46% Lack of Lack of 35% 50% results numbers, numbers, *Balas, 1995 Inconsistent Poynard, 1985 Dickersin, 1987 Koren, 1989 indexing Design issues Design issues
From the Levy Arrow to the Pipeline Fallacy of Producing & Vetting Research to Get Evidence-Based Practice* Basic Research The 17-year odyssey Guidelines for Practice - Research Evidence-Based Funding/patient or Publication Synthesis Practice Population needs, Peer Review Priorities & demands; local Of Grants Peer Review practice or policyPriorities for Evidence-based circumstances;Research Funding Medicine professional Academic appointments, movement discretion; promotion, & tenure credibility & fit of Impact Factor criteria the evidence. Scoring** Blame the practitioner*Green LW (2008). If it’s an evidence-based practice, where’s the or blame disseminationpractice-based evidence? Family Med 25 (suppl_1): 20-24;
CDC’s Model for Research to Practice& Policy The research to The CDC Wedge practice gap is here? Adapted from Green LW, Popovic T et al, CDC Futures Group on Research, Atlanta: CDC, 2004; Sleet DA et al. Health Promot Pract 2003;4:98-102; & Hanson et al PHR 2012
Mediating and Moderating Variables Mediator Intervention Outcome or Program Variable(s) Mediator Moderators ModeratorsGreen L & Kreuter M. Health Program Planning: An Educational andEcological Approach. 4th ed. New York: McGraw-Hill, 2005.
The Prevailing Standard of Evidence: The Randomized Controlled Trial Change in Mediating outcome variablesIntervention variable(s) expected totested by measured change, basedcomparison with and on previousa control compared evidence andcondition ? between theory experimental & control Context groups Context--Interventions highly standardized. --Comparison based on average--Interventions reduced to simplistic form change for each group--Everything else held constant. --Subgroup analysis discouraged--Clients randomized, no choice. --Dropouts discounted, ignored--Interventionists highly trained, restrained & --Cut-off date for outcomes oftensupervised; no discretion. too soon for change to occur
Problems Identified by IOM Report* (www.nap.edu)• Narrow focus: Lack of attention to larger systems context• Lacking details of implementation process• Lack of relevance to real world• Many studies focus on one intervention, but obesity may require a combination of interventions; in fact, some things appear not to work when tested alone, but are essential ingredients in a more comprehensive program*Institute of Medicine. Bridging the Evidence Gap in Obesity Prevention: A Framework toInform Decision Making. Washington, DC: The National Academies Press,2010. Full text online at www.nap.edu.
IOM Conclusions about Status of Evidence• The current evidence lacks the power to set a clear direction for obesity prevention across a range of target populations• This lack of evidence for effectiveness seen as a lack of effectiveness• It is difficult to fund, conduct & publish research on community, environmental, and policy-based obesity prevention initiatives• Assessing or reporting on generalizability of research results to other populations or settings has not been given priority
Scientists: Evidence-based Practice Is it valid? Is it important? Is it useful? High internal validity + Low external validity = DiminishedG. Ramirez,7/30/2009 relevance for practice or policy!
Paradigm Shift: Practice-based Evidence What’s useful? and important? What is sufficiently valid? High external validity + Relaxed internal validity = IncreasedG. Ramirez,7/30/2009 relevance for practice!
Types of Community-Engaged Evidence for Health Research• Participatory research evidence – Community-Based Participatory Research (CBPR) – Practice-based or action research• Surveillance evidence• Population diagnostic evidence• Program evaluation evidence – Multi-component – Continuous quality improvement – How context effects (moderates) outcomes
Change in Per Capita Cigarette ConsumptionCalifornia & Massachusetts vs Other 48 States, 1984-1996 5 0 Percent Reduction -5 -10 -15 -20 -25 Other 48 States California Massachusetts 1984-1988 1990-1992 1992-1996
The Spheres of Practice-Based, Community- Based, Academic & Participatory Research Practice- Participatory Based Research Research The sweet spot Highly for implementation CBPR Controlled Community- Academic Based Research Research
Definition and Standards of Participatory Research for Health* Systematic investigation… Actively involving people in a co-learning process… For the purpose of action conducive to health --not just involving people more intensively as subjects of research or evaluation*Green, George, Daniel, et al., Participatory Research…Ottawa: RoyalSociety of Canada, 1997. www.lgreen.net/guidelines.html
Three Paradoxes• The internal validity–external validity paradox – The more rigorously controlled a study testing the efficacy of an intervention, the less reality-based it becomes, so it cannot be taken to scale or generalized• The specificity – generalizability paradox – The more relevant and particular to the local context, the less generalizable to other contexts, yet the more it is perceived to be practice-based ―like us‖• The homophily–social distancing paradox – The effectiveness of indigenous health workers uses their commonalities with the community, but they seem to lose that with increased professionalization; – obverse paradox for scientists…
Number of Publications on CBPR Based on Scopus Search* Publications on CBPR 450 400 350 300 250 200 Publications 150 100 50 0 1987 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010*Based on unpublished Scopus review by Doug Brugge, Tufts U., 2011.
The Lenses of Scientists, Health Professionals and Lay People Subjective Indicators of HealthProfessional, LaypersonScientific Objective Indicators of Health
The Multi-Site Translational Community Trial (mTCT) Proposal*• Blends the internal validity advantages of – Cluster randomized trial or multi-site RCT – Fidelity to the function (but not the form) of an efficacy-tested intervention• With the external validity advantages of – Diversity of settings, cultures, circumstances – Adaptation of the form (not the function) of the efficacy-tested intervention• With some sacrifice of CBPR degrees of freedom*Katz DL et al. From controlled trial to community adoption…Am J PublicHealth. Published online ahead of print, June 16, 2011: e1–e11. .
Aligning Evidence* with (and deriving it from) Practice: Matching, Mapping, Pooling & Patching • Matching ecological levels of a system or community with RCT evidence of efficacy for interventions at those levels • Mapping theory to the causal chain to fill gaps in the evidence for effectiveness of interventions • Pooling experience to blend interventions to fill gaps in evidence for the effectiveness of programs in similar situations • Patching pooled interventions with indigenous wisdom and professional judgment about plausible causes & interventions to fill gaps in the program for the specific population*Green & Kreuter, Health Program Planning: An Educational and EcologicalApproach. 4th ed. NY: McGraw-Hill, 2005, Chap 5. Green & Glasgow, 2006.
• Take-home points of ―Designing for Dissemination and Implementation‖ – Consider the trade-offs between internal and external validity of original research and evaluation – If we want more evidence-based practice, we need more practice-based evidence – Extend CBPR principles to work with policy makers, program planners & practitioners in use of natural experiments—e.g., evaluation – Combine RCTs with CBPR in multi-site trials (Katz et al., AJPH, 2011)
Some Applications• Katz DL, Murimi M, Gonzalez A, Nijike V, Green LW. From clinical trial to community adoption: The Multi-site Translational Community Trial (mTCT). Am J Public Health. Published online ahead of print June 16, 2011: e1–e11. doi:10.2105/AJPH.2010.300104, 2011.• Green LW, Travis JW, McAllister RG, Peterson KW, Vardanyan AN, Craig A. Male circumcision and HIV prevention: Insufficient evidence and neglected external validity. Am J Prev Med. Nov 2010;39(5):479-82.• Green LW, Travis JW, McAllister RG, Peterson KW, Vardanyan AN, Craig A. Author responses. Am J Prev Med. 2011; 40(3): e9-e10 and e13-4.• Green LW, & Glasgow R. Evaluating the relevance, generalization, and applicability of research: Issues in external validation and translation methodology. Evaluation & the Health Professions 2006; 29(1): 126-153.• Green, L.W. Public health asks of systems science: To advance our evidence-based practice, can you help us get more practice-based evidence? American Journal of Public Health 96(3): 406-409, Mar. 2006.• Mercer, S.M, DeVinney, B.J., Fine, L.J., Green, L.W., Dougherty, D. Study designs for effectiveness and translation research: Identifying trade-offs. American Journal of Preventntive Medicine. 2007; 33(2): 139-154.• Sanson-Fisher, R.W., Bonevski, B., Green, L.W., D’Este, C. Limitations of the randomized controlled trial in evaluating population-based health interventions. American Journal of Preventive Medicine. 2007; 33(2): 155-161.
Some References• Glasgow RE, Green LW, Taylor MV, and Stange KC. An evidence integration triangle for aligning science with policy and practice. Am J Prev Med. 2012; 42: 646.• Garfield SA, Malozowski S, Chin MH, Naryan K M, Glasgow R, Green LW, Hiss RG, & Krumholz HM. Considerations for Diabetes Translational Research in Real-World Settings. Diabetes Care 26(9): 2670-2674., Sep 2003.• Green LW, Glasgow RE, Atkins D, Stange K. Making Evidence from Research More Relevant, Useful, and Actionable in Policy, Program Planning, and Practice: Slips ―Twixt Cup and Lip‖. Am J Prev Med. Dec 2009;37(6S1)S187- S191. Full text online: http://rwjcsp.unc.edu/resources/articles/S187-S191.pdf• Jagosh J, Macaulay AC, Pluye P, Salsberg J, Bush PL, Henderson J, Sirett E, Wong G, Cargo M, Herbert CP, Seifer SD, Green LW, and Greenhalgh T. Uncovering the Benefits of Participatory Research: Implications of a Realist Review for Health Research and Practice. Milbank Quar. 2012;90(2):311-346.