Your SlideShare is downloading. ×
0
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Using Designed Experiments to Improve Service Quality in a Customer Care Environment

108

Published on

This 2003 presentation for the Rocky Mountain Quality Conference explored the use of advanced quality improvement techniques for call center environments. The degree to which individual contributions …

This 2003 presentation for the Rocky Mountain Quality Conference explored the use of advanced quality improvement techniques for call center environments. The degree to which individual contributions vs. process factors caused outcomes, using Design of Experiments (DOE), and human factors in process improvement were covered.

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
108
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • This slide is an agenda slide. The PDCA format tends to tell a good story.
  • You may want to review the PDCA cycle with the audience. This is our approach to process improvement, based on commonly accepted process improvement practices.
  • You may want to note that we had new quality expectations and a new form in effect in January. While we had closed the gap, raising average quality scores 6%, we still had a 2-3% gap to close, and we were subject to contractual penalties that were material (about $40K/mo.).
  • This Pareto diagram was constructed from tallying the total lost points by question from a sample using the QA form.
  • The Test and Control groups were balanced by Coach group, QA score, and AHT at the outset to make them as identical as possible. Tenure was not a factor as Agents had all been on the project longer than 4 months. The 3% goal was chosen because we assumed we could cut lost points in Best Practices and Notes by half. The other solution, coaching improvement, is in progress with the Coach’s Checklist effort.
  • We executed according to plan.
  • This data was from weeks 5-8. The boxplot shows mean (red dot), median (horizontal line), middle 50% distribution (box) and range (whiskers). While Degrees of Freedom (DF), Sum of Square (SS), Mean Squares (MS), and F-ratio computations are shown, the p-value is the important, showing the degree of uncertainty in the measurement (1-p is the confidence that the result is non-random). N (number of samples), Means, and Standard Deviations of the groups are shown below. The lower diagram shows the overlap of the 95% Confidence Intervals, that is, the less the overlap, the less chance the result is random – this is a graphical view of the p-value.
  • This diagram confirms The Brain-EKP hit exactly the right areas, as expected. The other factors represent coaching improvement areas.
  • We executed according to plan.
  • Transcript

    • 1. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference Denver, June 2003 Ed Powers VP Corp. Planning and Development Quality Center Partners, Inc. Fort Collins, Colorado 80525
    • 2. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 2 of 62 Objective This presentation helps quality professionals better understand and apply Design of Experiments (DOE) principles in non-manufacturing or customer service environments. It describes Center Partners’ business challenges and how DOE techniques have helped determine the effectiveness and ROI of new software and training solutions.
    • 3. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 3 of 62 Agenda • Center Partners—Who We Are, What We Do • Client Expectations and Business Challenges • DOE Overview • Using DOE in a Service Environment • Example 1: Using DOE to Evaluate Performance Management Software on Quality and Average Handle Time Improvement • Example 2: Using DOE to Evaluate Monitoring Software and Coaching Effectiveness on Quality Improvement • New DOE Applications • Summary • Q&A
    • 4. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 4 of 62 About Center Partners • Founded 1997, in Fort Collins, Colorado, as a call center outsourcer specializing in high-touch customer care for complex products • 8000% growth in first five years to over $80M in 2002 billables • Purchased in 2001 by the WPP Group • Currently answering over 2 million calls a month on behalf of clients like Qwest Communications, Xerox, Agilent and Comcast • Hassle Free Contact Center Services. Done Right. On Time.
    • 5. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 5 of 62 Center Partners’ Basic Client Expectations • Meet contract metrics: – Service Level – Average Handle Time (AHT) – Quality – Sales/Retention Goals – Others
    • 6. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 6 of 62 About Service Level…
    • 7. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 7 of 62 About Average Handle Time…
    • 8. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 8 of 62 Center Partners’ Business Challenges • Meet or exceed contract metrics • Delight clients • Make money To meet these challenges, continuous service and process improvement is not optional! To meet these challenges, continuous service and process improvement is not optional!
    • 9. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 9 of 62 Design of Experiments Defined The arrangement in which an experimental program is to be conducted, and the selection of the versions (levels) of one or more factors or factor combinations to be included in the experiment. Source: ASQ Quality Press,Glossary and Tables for Statistical Quality Control, Second Edition, 1983, 160 pages
    • 10. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 10 of 62 General Process Model ProcessInputs Outputs Controllable Factors Uncontrollable Factors x1 x2 x3 z1 z2 z3 Source: Montgomery, D. C., Design and Analysis of Experiments, Fourth Edition, 1997, 704 pages
    • 11. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 11 of 62 Example: Golf Factor Level Driver Oversized or regular size Ball Balata or three-piece Conveyance Walk and carry clubs or use golf cart Refreshments Beer or water Time of day Morning or afternoon Source: Montgomery, D. C., Design and Analysis of Experiments, Fourth Edition, 1997, 704 pages
    • 12. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 12 of 62 Typical Experimentation • “One factor at a time” – What if something changes?? – Were there any interactions? • “Best guess” – What factor(s) caused the result?? – How do we know this is the best solution? DOE tests many variables at once, quickly and efficiently with more useful results DOE tests many variables at once, quickly and efficiently with more useful results Source: Montgomery, D. C., Design and Analysis of Experiments, Fourth Edition, 1997, 704 pages
    • 13. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 13 of 62 ANOVA—Workhorse of DOE • ANalysis Of VAriance: Statistical method to separate causes (factors) and effects (response) by accommodating systemic randomness (experimental errors) • Tests statistical hypotheses and provides confidence levels in conclusions.
    • 14. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 14 of 62 How ANOVA Works Group 1 Group 2 100 101 105 98 98 96 99 99 101 89 110 91 103 93 101 92 90 100 Compare means by analyzing variation within and between groups. Between Within
    • 15. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 15 of 62 Industrial Example • Engineer studying effect of varying cotton weight percent in synthetic fiber on tensile strength of cloth material • Randomized experiment with a single factor at multiple levels • Hypothesis (H1): tensile strength will be different for different percentages of cotton; “Null Hypothesis” (H0): there is no effect Source: Montgomery, D. C., Design and Analysis of Experiments, Fourth Edition, 1997, 704 pages
    • 16. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 16 of 62 Example Data Weight % Cotton Observed Tensile Strength (lb/in2 ) 1 2 3 4 5 Avg. 15 7 7 15 11 9 9.8 20 12 17 12 18 18 15.4 25 14 18 18 19 19 17.6 30 19 25 22 19 23 21.6 35 7 10 11 15 11 10.8 Variation Within Treatments Variation Between Treatments Source: Montgomery, D. C., Design and Analysis of Experiments, Fourth Edition, 1997, 704 pages
    • 17. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 17 of 62 ANOVA Computations Source of Variation Sum of Squares Degrees of Freedom Mean Square F0 P-Value Cotton Weight % 475.76 4 118.94 14.76 <0.01 Error 161.20 20 8.06 Total 636.96 24 H0 is rejected; cotton weight % DOES affect tensile strength Source: Montgomery, D. C., Design and Analysis of Experiments, Fourth Edition, 1997, 704 pages
    • 18. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 18 of 62 Experimental Design and Analysis • Fixed effects • Random effects • Regression • Analysis of Covariance • Randomized Complete Block • Latin Squares • Graeco-Latin Squares • Balanced Incomplete Block • Two-Factor Factorial • 2k , 3k • Confounding • ½ Fraction; ¼ Fraction • General 2k-p , 3k-p • Multi-Factor Factorial with Random Factors • Nested and Split-Plot • Multiple Regression
    • 19. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 19 of 62 Using DOE in a Services Environment • Fewer metrics; results often less tangible • Many more potential variables—many uncontrolled • Environments can be highly dynamic and may influence testing • Higher PEOPLE content
    • 20. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 20 of 62 Process Factors ProcessInputs Outputs Controllable Factors Uncontrollable Factors x1 x2 x3 z1 z2 z3 •People •Methods •Materials •Equipment •Environment •Information
    • 21. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 21 of 62 Continuum of Observed Performance Performance is Due to Chance Alone Performance is Due People Factors Alone Process Individual What % is the mix in our business?
    • 22. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 22 of 62 Detecting Non-Random Events What is the minimum number of times would you need to flip a coin to determine if it were not “fair”?
    • 23. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 23 of 62 Detecting Non-Random Events What is the minimum number of times would you need to flip a coin to determine if it were not “fair”? Solution: 7. Use the binomial distribution. Assume r = n (you get either all “heads” or all “tails”). To be >99% that the effect is non-random, determine n when y <=0.01: y = pr (1-p)n-r n! r!(n-r)! y n .5 1 .25 2 .125 3 0.0625 4 0.03125 5 0.015625 6 0.0078125 7 When n=r, y reduces to: y = pn With a perfectly balanced coin, there is less than 1% chance of getting 7 heads or 7 tails in a row. <1%>99%
    • 24. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 24 of 62 Identifying LIKELY “High Performers” Quality AHT Other- Save Rate? y n 0.25 1 .0625 2 .0156 3 y n 0.33 1 .1089 2 .0359 3 .0119 4 p=0.25 p=0.33 y n .10 1 .01 2 p=0.10 Example: 2 metrics both in upper 10% yields a 99.0% certainty of non- randomness y n .20 1 .04 2 .008 3 p=0.20
    • 25. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 25 of 62 Good Performance Two Months in a Row (One metric, July and August 2002, Population Average 55 Agents/Mo.) July and August QA AHT Upper 10% (E=1) Christy, Charles, Robert Christy, Andrea, Brian, Graham Upper 20% (E=2) Christy, Charles, Robert, Rebecca, Thomas Christy, Andrea, Brian, Graham, Nancy, Matthew Upper 25% (E=3) Christy, Charles, Robert, Rebecca, Thomas Christy, Andrea, Brian, Graham, Nancy, Matthew, Victor, Kyle Upper 33% (E=6) Christy, Charles, Robert, Rebecca, Thomas, Trula, Jami Christy, Andrea, Brian, Graham, Nancy, Matthew, Victor, Kyle, Rigoberto, Jennifer, Stephanie, Robyn In the Agent population, 13% exhibit non-random behavior for QA, 22% for AHT when considering top 1/3 of Agents In the Agent population, 13% exhibit non-random behavior for QA, 22% for AHT when considering top 1/3 of Agents
    • 26. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 26 of 62 Good Performance in the Same Month (2 metrics, July and August 2002, Population Average 55 Agents/Mo.) QA and AHT July August Upper 10% (E=1) Christy Christy Upper 20% (E=2) Christy Christy Upper 25% (E=3) Christy Christy Upper 33% (E=6) Christy, Stephanie, Robyn Christy, Sarah Less than 6% of Agents exhibit non-random behavior; we can be at least 99% sure Christy was a stand-out Less than 6% of Agents exhibit non-random behavior; we can be at least 99% sure Christy was a stand-out
    • 27. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 27 of 62 Results Agent Performance is Due to Chance Alone Agent Performance is Due Agent Factors Alone Process Individual Quality AHT At best, Agent factors account for about 50% of observed performance in Center Partners’ business At best, Agent factors account for about 50% of observed performance in Center Partners’ business
    • 28. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 28 of 62 More About People Factors • People VARY from person to person • People are HABITUAL • People RESPOND DIFFERENTLY when obvious supervision is present
    • 29. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 29 of 62 Side-By-Side vs. Remote QA Scores Analysis of Variance (jump start data.sta) Marked effects are significant at p < .05000 SS df MS SS df MS Effect Effect Effect Error Error Error F p SCORE 3802.597 1 3802.597 21562.79 98 220.0285 17.28229 .000069 Summary Table of Means (jump start data.sta) N=100 (No missing data in dep. var. list) SCORE R 78.29268 S 90.83051 All Grps 85.69000 Min-Max 25%-75% Medianvalue Box&WhiskerPlot:SCORE HOW SCORE 10 30 50 70 90 110 R S >99.9% confidenceThere is a real difference in quality scores between remote and side-by-side observation There is a real difference in quality scores between remote and side-by-side observation
    • 30. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 30 of 62 Example 1: Using DOE to Evaluate Performance Management Software on Quality and Average Handle Time Improvement
    • 31. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 31 of 62 Structure of a Simple DOE Study “Control” Group “Test” Group Production Group
    • 32. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 32 of 62 The Brain-EKP® Experiment • The PDCA Cycle • Plan: Issue, Measures, Causes • Do: Experiment • Check: Results • Act: Learning and Next Steps
    • 33. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 33 of 62 The Improvement Cycle Plan Do Check Act •Understand the issue •Understand the process •Define the measures •Uncover the root cause(s) •Determine the solution •Establish the goals •Plan the project •Execute the plan •Implement the solution •Compare results to goals •If successful, document, share knowledge and leverage solutions •If unsuccessful, revisit root causes and redo the cycle
    • 34. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 34 of 62 Plan: Situation Analysis (June, 2002) • Issue: Center Partners performance below quality goal • Process: Call Handling (Center Partners Key Business Process 6.4) • Measures: QA score goal 80%; current process average 76-78% and in a state of statistical process control
    • 35. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 35 of 62 Plan: Causal Analysis Wrong support structure assumptions Lack of sr. mgt. reinforcement QA scores not improving “Best Practices” issues Client scoring differences Agents not motivated Agents not knowledge -able Not a priority Too much time spent on other tasks Don’t know how Lack time management skills Lack of automation “Special” projects Lack of support “Emergencies” Competing responsibilities Too few support people Process defects Agent life issues Sporadic events/outages Low support productivity Don’t know job priorities Not a personal priority Lack of CSM reinforcement No BFTs Client politics/ structure Bias Bad calibration Complacency Not reinforced by Coaches Don’t know importance Conflicting metrics/rewards Don’t agree with forms Changes not communicated Training ineffective Coaches don’t communicate Changes frequently Not enough QA focus No Jump Start / Base Camp Bad data format No ownership Can’t find information Too much reliance on memory Difficult to navigate Time/AHT pressure Changes frequently Unaware of changes High point weighting Too ‘lazy’ to look Skills not habitual Good habits not formed Cause and Effect analysis identified Best Practices and Agent Motivation as potential primary causes Cause and Effect analysis identified Best Practices and Agent Motivation as potential primary causes
    • 36. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 36 of 62 Best Practices Complete/Accurate Notes Just Ask Proactive Education Assurance to help Hold/Silence Additional Assistance Recap Others 323 216 188 177 120 104 88 87 277 20.4 13.7 11.9 11.2 7.6 6.6 5.6 5.5 17.5 20.4 34.1 46.0 57.2 64.8 71.4 77.0 82.5 100.0 0 500 1000 1500 0 20 40 60 80 100 Defect Count Percent Cum % Percent Count LVD SOC June 1-24th Pareto Plan: Pareto Analysis Data validation showed Best Practices and Complete/Accurate Notes were 34%, coaching issues were 41% of the issue Data validation showed Best Practices and Complete/Accurate Notes were 34%, coaching issues were 41% of the issue
    • 37. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 37 of 62 Plan: Solution Design • Solution: – Improve coaching process (a separate project) – Address “Best Practices” issue with The Brain-EKP® • Goal: – Brain-EKP® : 3% improvement in average QA scores • Project Plan: – Designed experiment with equally balanced Test and Control groups – Treatment with/without The Brain-EKP® – Also study impact on Average Handle Time (AHT)
    • 38. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 38 of 62 Example 1 Experimental Model Call Handling Process Customer Calls Resolved Issues •QA Score •AHT Controllable Factor Uncontrollable Factors Brain-EKP® Coaching Client Changes
    • 39. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 39 of 62 Plan/Do Task Duration Completion Kickoff 1 day 6/27/02 Experimental Design 3 days 7/12/02 Installation and Testing 11 days 7/17/02 Knowledge Model Development 7 days 7/24/02 Training and Roll-out 20 days 7/26/02 Experimental Runs 8 weeks 9/20/02 Progress Checks Weekly Weekly Conclusions 1 week 9/20/02
    • 40. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 40 of 62 Control Test 0.7 0.8 0.9 1.0 Group QAScore Boxplots of QA Score by Group (means are indicated by solid circles) One-way ANOVA: QA Score versus Group Analysis of Variance for QA Score Source DF SS MS F P Group 1 0.04995 0.04995 7.84 0.006 Error 155 0.98816 0.00638 Total 156 1.03811 Individual 95% CIs For Mean Based on Pooled StDev Level N Mean StDev --+---------+---------+---------+---- Control 77 0.83519 0.07752 (--------*--------) Test 80 0.87088 0.08202 (-------*--------) --+---------+---------+---------+---- Pooled StDev = 0.07985 0.820 0.840 0.860 0.880 Check: QA Analysis 99.4% certainty of a 3.6% difference Boxplots and ANOVA show that the Test group outperformed the Control group Boxplots and ANOVA show that the Test group outperformed the Control group
    • 41. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 41 of 62 Others Confidence Vantive # Best Practices Used Systems Correctly Empathy Willingness to help Upsell Follow-up Greeting Pro-active education 62.5016.3022.0026.5026.7029.1033.7038.0039.7041.3356.10 15.94.25.66.86.87.48.69.710.110.514.3 100.084.179.974.367.560.753.344.735.024.914.3 400 300 200 100 0 100 80 60 40 20 0 Defect Count Percent Cum % Percent Count LVD SOC - Aug 19 - Sept 9 Brain Pilot Group Check: Pareto Analysis Best Practices was reduced to the #8 issue; Complete/Accurate Notes was now out of the top 10 Best Practices was reduced to the #8 issue; Complete/Accurate Notes was now out of the top 10
    • 42. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 42 of 62 Test Control 15 10 5 Group AHT Boxplots of AHT by Group (means are indicated by solid circles) One-way ANOVA: AHT versus Group Analysis of Variance for AHT Source DF SS MS F P Group 1 19.17 19.17 3.44 0.066 Error 145 808.46 5.58 Total 146 827.64 Individual 95% CIs For Mean Based on Pooled StDev Level N Mean StDev ------+---------+---------+---------+ Control 71 10.469 2.061 (----------*----------) Test 76 9.746 2.611 (----------*----------) ------+---------+---------+---------+ Pooled StDev = 2.361 9.50 10.00 10.50 11.00 Check: AHT Analysis 93.4% certainty of 43.4 second AHT reduction Boxplots and ANOVA show that the Test group outperformed the Control group Boxplots and ANOVA show that the Test group outperformed the Control group
    • 43. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 43 of 62 Act: Learning • The Brain-EKP® provided a better user interface for call handling than the one provided by our client, resulting in better QA scores and AHT • Technology alone is not sufficient—adequate coaching and floor support is needed to ensure tool usage and help modify Agent habits
    • 44. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 44 of 62 Example 2: Using DOE to Evaluate Monitoring Software and Coaching Effectiveness on Quality Improvement
    • 45. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 45 of 62 Plan: Situation Analysis (January, 2002) • Issue: Center Partners performance at quality goal but wanted incentive levels; quality monitoring software supplier made claims that their system would improve quality • Process: Call Handling (Center Partners Key Business Process 6.4) • Measures: QA at 80% and in a state of statistical process control; goal was to increase 5%
    • 46. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 46 of 62 Plan: Pareto Analysis Pareto of Failure Causes 0 5 10 15 20 25 30 35 40 H abit A ttitude/O ther Learning Category Count 0 20 40 60 80 100 120 Cumulative%
    • 47. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 47 of 62 Plan: Theory & Assumptions • Rewards programs have proven unsustainable in the past • To change a habit, people need to focus on the issue and receive feedback multiple times • Many more observations and feedback in a shorter time will help change the habits • Seeing audio and video playback of calls during coaching sessions will improve results • Using visual aids will help remind Agents to conform to QA expectations
    • 48. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 48 of 62 Plan: Solution Design • Solution: – Run a statistically designed experiment across three sites to test effects of quality monitoring software, intensive feedback, and visual aids on QA scores • Goal: – 10% improvement in average QA scores • Project Plan: – 23 design
    • 49. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 49 of 62 Example 2 Experimental Model Call Handling Process Customer Calls Resolved Issues •QA Score Controllable Factors Uncontrollable Factors Site Differences Client Changes Software FeedbackVisual Aids
    • 50. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 50 of 62 Experiment Outline • Identify 36 individuals to be part of the program. Individuals will be chosen based on average score – 80% +/- 5%. • 8 combinations (Feedback, Visuals, Software) tested 3 times each in Fort Collins (FTC) and Loveland (LVD), 4 combinations (Feedback, Visuals) tested 3 times each in Idaho Falls (IDF)
    • 51. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 51 of 62 Experimental Design (LVD, FTC) REPLICAT NAME FEEDBACK VISUALS SOFTWARE LOCATION CHANGE COACH 1 Low Low Low LVD 1 High Low Low LVD 1 Low High Low LVD 1 High High Low LVD 1 Low Low High FTC 1 High Low High FTC 1 Low High High FTC 1 High High High FTC 2 Low Low Low FTC 2 High Low Low LVD 2 Low High Low LVD 2 High High Low LVD 2 Low Low High FTC 2 High Low High FTC 2 Low High High FTC 2 High High High FTC 3 Low Low Low FTC 3 High Low Low LVD 3 Low High Low LVD 3 High High Low LVD 3 Low Low High FTC 3 High Low High FTC 3 Low High High FTC 3 High High High FTC 24 Agents total Examples of treatment: High – Low – Low = Multiple observations Low – Low – Low = Control Group Low – High – Low = Delivery of QA as normal leave visual reminder on their computer about areas they missed Low – Low – High = include viewing software as part of the “normal” feedback session
    • 52. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 52 of 62 Plan/Do Task Duration Completion Approval 1 day 1/11/02 Experimental Design and Planning 3 days 1/14/02 Kickoff 1 day 1/15/02 Training, Calibrations, and Roll-out 2 days 1/17/02 Experimental Runs 3 weeks 2/7/02 Analysis 2 days 2/11/02 Presentation of Conclusions 1 day 2/14/02
    • 53. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 53 of 62 Check: FTC/LVD Results Marginal Means (Unweighted); variable: DIFFERENCE Design: 2**(3-0) design NOTE: Std.Errs. for means computed from MS Error=.0062194 Pooled Overall Std.Err. Feedback Visual Witness Means Std.Dev. Std.Dev. N for Mean High Low Low .089333 .111142 .111142 3 .045532 High Low High .172967 .061224 .061224 3 .045532 High High Low .066667 .043716 .043716 3 .045532 High High High .043800 .118826 .118826 3 .045532 Low Low Low -.053800 .045048 .045048 3 .045532 Low Low High -.002467 .072750 .072750 3 .045532 Low High Low -.048333 .054243 .054243 3 .045532 Low High High .055333 .075755 .075755 3 .045532 The group exposed to high feedback and monitoring software but not visuals improved 17% The group exposed to high feedback and monitoring software but not visuals improved 17%
    • 54. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 54 of 62 Check: FTC/LVD Factor Analysis Effect Std.Err. t(17) p Mean/Interc. .040437 .016098 2.51198 .022392 (1)FEEDBACK .105508 .032196 3.27709 .004444 (2)VISUAL .022142 .032196 .68772 .500905 (3)WITNESS .053942 .032196 1.67543 .112144 1 by 2 .053775 .032196 1.67025 .113177 1 by 3 .023558 .032196 .73172 .474305 2 by 3 .013542 .032196 .42060 .679313 11% of the 17% gain can be attributed to feedback 11% of the 17% gain can be attributed to feedback
    • 55. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 55 of 62 Check: FTC/LVD Factor Analysis ParetoChartofStandardizedEffects;Variable:DIFFEREN 2**(3-0)design;MSResidual=.0062194 DV:DIFFEREN:=v9-v8 EffectEstimate(AbsoluteValue) -.420604 -.687721 .7317222 1.670252 1.675429 -3.27709 p=.05 2by3 (2)VISUAL 1by3 1by2 (3)WITNESS (1)FEEDBACK 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
    • 56. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 56 of 62 Check: Summary of All Sites • Out of the 17 agents who received high feedback, 13 showed improvement • Results show three out of four high feedback agents showed an average of 9.7% improvement in their scores in four days • Model shows about a 1% gain per high frequency feedback session on average • Subsequent to the experiment, much of the gains were lost in the first month!
    • 57. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 57 of 62 Act: Learning • High feedback is the primary cause of higher scores; software and visuals contributed a minimum amount —software supplier’s claims need refinement! • Habits are more difficult to change than originally anticipated
    • 58. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 58 of 62 Desire (Want to) Knowledge (What to do /Why) Forming Habits Skill (How to) Habit Source: Covey, S., The Seven Habits of Highly Effective People, 1990, 319 pages Creating or changing a habit requires work in all three dimensions Creating or changing a habit requires work in all three dimensions
    • 59. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 59 of 62 New DOE Applications • Training tactics and tools • New technologies • Incentive programs • Marketing tactics
    • 60. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 60 of 62 Summary • DOE is an efficient and effective means for understanding cause and effect relationships • Simple DOE techniques can be used effectively in non-manufacturing or service environments • PEOPLE factors tend to be significant—especially when changes require a change of habit
    • 61. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 61 of 62
    • 62. Using Designed Experiments to Improve Service Quality in a Customer Care Environment Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 62 of 62 Author Biographical Information: Ed Powers • VP Corporate Planning and Development for Center Partners, Inc. • 16 years of experience in sales, marketing, quality management, and consulting • Formerly with Hewlett-Packard, Sorcia • BSEE 1987 Illinois Institute of Technology • HP Quality Maturity System Reviewer, ASQ Certified Quality Manager, 2003 Baldrige Examiner • Published in AMA Marketing News, Call Center Solutions magazines

    ×