1. Developing an Instrument to Measure the
Reporting Quality of the QI Six Sigma
Projects in Healthcare
Anselmo Chung, DrPH, MBA, PMP, SSMBB
Johns Hopkins Bloomberg School of Public Health
October 19, 2016
2. However, quality of Lean Six Sigma projects are poor, thus not able to
provide evidence of its effectiveness
Lean Six Sigma has been used successfully to improve
Patient Safety and Quality
2
“There was no rigorous study design in the LSS article, therefore
it’s impossible to believe their reported outcomes”
“Difficult to understand which intervention aligned to what root
cause”
“No pre-post analysis on the primary outcome variable”
“No comparison group analysis”
“Lacks statistical analysis with P-value”
“Article was just poorly written with poor structure”
3. AGREE II can be configured to evaluate Lean Six Sigma projects
• AGREE II is a validated tool that evaluates Clinical Practice Guideline for quality
• There are 6 Domains with 24 total questions and two final overall assessment
questions
– Domain 1: Scope and Purpose
– Domain 2: Stakeholder Involvement
– Domain 3: Rigor of Development
– Domain 4: Clarity of Presentation
– Domain 5: Applicability
– Domain 6: Editorial Independence
– Final Overall Assessments
How do we properly evaluate LSS projects for quality?
3
http://www.agreetrust.org/agree-ii/
4. Apply the AGREE II-LSS version to assess the quality of LSS articles
Evaluating LSS project articles using AGREE II - LSS
4
Article selection approach
• Domain 1: Scope and Purpose
• Domain 2: Stakeholder Involvement
• Domain 3: Rigor of Development
• Domain 4: Clarity of Presentation
• Domain 5: Applicability
• Domain 6: Editorial Independence
• Final Overall Assessment
A comparison of the distribution of the 6 domain
scores for all assessed articles
Assessment Categories
AssessmentScores(%)
OverallDomain 6Domain 5Domain 4Domain 3Domain 2Domain 1
80
70
60
50
40
30
20
61.91
32.05
39.33
65.09
47.44
63.07
76.77
95% CI for the Mean
Distribution of Assessment Scores for All Assessed Articles
Scatterplot of overall article assessment score
vs publication year
Publish Year
OverallAssessmentScore(%)
2013201220112010200920082007200620052004
100
90
80
70
60
50
40
30
20
10
Overall Assessment Score vs Publish Year
Based on the AGREE II instrument, the LSS expert team of this study developed
the AGREE-LSS by slightly modifying some of the original AGREE II instrument
assessment questions to make them appropriate to review QI journal articles
rather than CPGs. The AGREE-LSS instrument mirrors the AGREE II tool and
comprises 24 items organized into the original 6 quality domains:
Domain 1. Scope and Purpose are concerned with the overall aim of the QI
project; the specific health care related problem statement questions that
frame the opportunity for the QI project; and the intended target population
that the interventions from the QI project will impact (items 1-4).
Domain 2. Stakeholder Involvement focuses on the extent to which the QI
project was developed by the appropriate QI practitioners and represents
the views of its intended stakeholders (items 5-7).
Domain 3. Rigor of Development relates to the approach and methods used
to gather and synthesize the root causes of health care challenges, the
methods to formulate the recommendations, the assessment of the
interventions benefits and potential risks, and control mechanisms to
update and revise interventions to ensure sustainability (items 8-15).
65
Domain 4. Clarity of Presentation deals with the language, structure, and
format of the QI project report (items 16-18).
Domain 5. Applicability pertains to the likely barriers and facilitators to
implement process improvements, strategies to improve stakeholder buyin,
and resource implications of applying and sustaining the improvements
(items 19-22).
Domain 6. Editorial Independence is concerned with the formulation of
recommendations not being unduly biased with competing interests (items
23-24).
