Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
THE HIDDEN BIAS IN CUSTOMER
METRICS
BOB E. HAYES, PHD
BUSINESS OVER BROADWAY
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
...
Upcoming SlideShare
Loading in …5
×

The Hidden Bias in Customer Metrics

2,702 views

Published on

This is a study that examines how customer experience professionals interpret the Mean Score and Net Promoter Score. Results show that estimations of customer segment sizes based on the Mean or Net Score result in large biases.

Published in: Business
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

The Hidden Bias in Customer Metrics

  1. 1. THE HIDDEN BIAS IN CUSTOMER METRICS BOB E. HAYES, PHD BUSINESS OVER BROADWAY
  2. 2. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com Purpose  Help understand how people interpret two different metrics:  Mean Score  Net Score  While these two metrics are equivalent (as one goes up, so does the other), do people see them differently?
  3. 3. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com If you know one metric, you know the others  These metrics are highly correlated with each other  They tell you the same thing – relatively speaking  If you have an NPS of - 30, you know your mean score will be about 6.2.  If you have an NPS of 40, you know your promoters will be about 57% and detractors will be about 17%. Net Promoter Score® Mean % 9-10 (Promoters) % 0-6 (Detractors) % 7-8 (Passives) % 6 or greater -100 4.1 -2.7 97.3 5.5 24.9 -90 4.4 -0.4 89.6 10.8 31.0 -80 4.7 2.2 82.2 15.6 36.9 -70 4.9 5.1 75.1 19.8 42.4 -60 5.3 8.3 68.3 23.4 47.8 -50 5.6 11.8 61.8 26.4 52.9 -40 5.9 15.6 55.6 28.8 57.7 -30 6.2 19.7 49.7 30.6 62.3 -20 6.5 24.1 44.1 31.8 66.7 -10 6.8 28.8 38.8 32.4 70.8 0 7.1 33.8 33.8 32.5 74.7 10 7.4 39.0 29.0 31.9 78.4 20 7.7 44.6 24.6 30.8 81.8 30 7.9 50.5 20.5 29.0 84.9 40 8.3 56.7 16.7 26.7 87.9 50 8.6 63.1 13.1 23.8 90.6 60 8.9 69.9 9.9 20.3 93.0 70 9.2 76.9 6.9 16.2 95.3 80 9.5 84.3 4.3 11.5 97.2 90 9.9 91.9 1.9 6.2 98.9 100 10.1 99.9 -0.1 0.3 100.5 Recommend question is on a 0 to 10 scale where 0 = Not at all likely and 10 = Extremely likely; sample to calculate other metrics from NPS consists of three independent studies on consumer attitudes toward PC manufacturers and wireless service providers.
  4. 4. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com Summary Metrics Paint Similar Picture  Each of the 48 data points in the graph represents a single company’s summary metric: mean of recommend and net score of recommend (NPS)  Two summary metrics (NPS and Mean) are strongly correlated (r = .97).  Looking at regression equation, we know comparable Mean and NPS values: NPS of 0.0 corresponds to a Mean of 7.0.  NPS of… Equals Mean of… y = 0.03x + 7.09 R² = 0.94 0 1 2 3 4 5 6 7 8 9 10 -100 -80 -60 -40 -20 0 20 40 60 80 100 Recommend-MeanScore Recommend - Net Score (NPS) Data are from three independent studies about consumer attitudes toward their: 1. PC manufacturer: Survey of 1058 US consumers from 2007. GMI (Global Market Insite, Inc.) 2. Wireless service provider: Survey of 994 US consumers from 2007. GMI (Global Market Insite, Inc.) 3. Wireless service provider: Survey of 5686 global consumers from 2010. Mob4Hire N = 48 brands, most brands represent different wireless service providers (N = 41) and a handful of PC manufacturers. Metrics were calculated for brands with 30 or more responses. -100 4.0 -50 5.5 0 7.0 50 8.5 100 10.0
  5. 5. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com Study Design  Participants invited via blog post about the study; post shared through social media connections, professional online communities and email list.  Blog post included hyperlink to web-based data collection instrument  CEM professionals were given 5 NPS values (-100, -50, 0, 50 and 100) and 5 comparable Mean values (4.0, 5.5, 7.0, 8.5 and 10. For each metric value, they were asked to make their best guess about the size of the following customer segments:  % of respondents who have a rating of 9 or 10 (Promoters / Top Box Score)  % of respondents who have a rating between 0 and 6, inclusive (Detractors / Bottom Box Score)  % of respondents who have a rating of 7 or 8 (Passives / Middle Box Score)  % of respondents who have a rating of 6 or greater (Satisfied / Top Box Score)
  6. 6. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com Sample Demographics B2B and B2C 42% B2C 3% B2B 55% Individual contributor of CEM Program 3% Manager of CEM Program 28% Director of CEM Program 9% Senior Executive of CEM Program 13% Outside consultant 22% Other 25% Beginner 10% Average 13% Proficient 45% Expert 32% What is your level of expertise in your company's CEM Program?What is your current role? What Best Describes Your Company?  Forty-one CX professionals participated in the study.
  7. 7. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com Sample: Expertise in Metric Beginner 24% Average 30% Proficient 21% Expert 24% Beginner 21% Average 37% Proficient 17% Expert 24% Level of expertise in using NPS in their CEM program Level of expertise in using Mean Scores in their CEM program  Wide range of expertise seen for both NPS and Mean Scores
  8. 8. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com Estimating % of Promoters From NPS and Mean Values  People are more accurate at estimating the size of Promoters (i.e., Top Box Score) using the NPS (except for slight understimation for NPS = 100.  When given a high Mean Score, people greatly underestimate the percent of Promoters. Estimated percent of promoters: CEM professionals’ best guess as to the percent of respondents who have ratings of 9 or 10 (Promoters) 0 10 20 30 40 50 60 70 80 90 100 4.0 5.5 7.0 8.5 10.0 PercentofCustomerswith aratingof9or10(Promoters) Mean Value Actual % Promoters Estimated % Promoters Actual percent of promoters: The actual percent of respondents who have a rating of 9 or 10 (Promoters) Confidence Interval: Sample statistic ± margin of error at a 95% confidence level for estimated percent of Promoters  CEM professionals were given different NPS and Mean values and asked to estimate the % of customers who are Promoters. For estimation using Mean Values, N = 26 to 30. For estimation using NPS Values, N = 33 to 41. 0 10 20 30 40 50 60 70 80 90 100 -100 -50 0 50 100 PercentofCustomerswith aratingof9or10(Promoters) NPS Value Actual % Promoters Estimated % Promoters
  9. 9. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com Estimating % of Detractors From NPS and Mean Values  When given a low summary metric, people greatly underestimate the percent of Detractors – more so for Mean values than for NPS values.  When given a high summary value, people slightly overestimate the percent of Detractors Estimated percent of Detractors: CEM professionals’ best guess as to the percent of respondents who have ratings of 0 - 6 (Detractors) Actual percent of Detractors: The actual percent of respondents who have a rating of 0 - 6 (Detractors) Confidence Interval: Sample statistic ± margin of error at a 95% confidence level for estimated percent of Detractors  CEM professionals were given different NPS and Mean values and asked to estimate the % of customers who are Detractors. For estimation using Mean Values, N = 26 to 30. For estimation using NPS Values, N = 33 to 41. -10 0 10 20 30 40 50 60 70 80 90 100 4.0 5.5 7.0 8.5 10.0 PercentofCustomerswith aratingof0or6(Detractors) Mean Value Actual % Detractors Estimated % Detractors -10 0 10 20 30 40 50 60 70 80 90 100 -100.0 -50.0 0.0 50.0 100.0 PercentofCustomerswith aratingof0to6(Detractors) NPS Value Actual % Detractors Estimated % Detractors
  10. 10. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com Estimating % of Passives From NPS and Mean Values  When given a low summary metric, people greatly underestimate the percent of Detractors – more so for Mean values than for NPS values.  When given a high summary value, people slightly overestimate the percent of Detractors Estimated percent of Passives: CEM professionals’ best guess as to the percent of respondents who have ratings of 7 - 8 (Passives) Actual percent of Passives: The actual percent of respondents who have a rating of 7 - 8 (Passives) Confidence Interval: Sample statistic ± margin of error at a 95% confidence level for estimated percent of Passives  CEM professionals were given different NPS and Mean values and asked to estimate the % of customers who are Passives. For estimation using Mean Values, N = 26 to 30. For estimation using NPS Values, N = 33 to 41. -10 0 10 20 30 40 50 60 70 80 90 100 -100.0 -50.0 0.0 50.0 100.0 PercentofCustomerswith aratingof7or8(Passives) NPS Value Actual % Passives Estimated % Passives -10 0 10 20 30 40 50 60 70 80 90 100 4.0 5.5 7.0 8.5 10.0 PercentofCustomerswith aratingof7or8(Passives) Mean Value Actual % Passives Estimated % Passives
  11. 11. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com  Greater downward bias in predictions when NPS is used. Estimating % of Positives (6-10) from NPS and Mean Values  For each NPS value, CEM professionals underestimated the percent of Positive respondents.  When given a mean, CEM professionals underestimated the percent of Positive respondents when mean was 5.5 or greater; overestimated percent of Positive respondents when Mean = 4. Estimated percent of positives: CEM professionals’ best guess as to the percent of respondents who have ratings of 6 - 10 (Positives) Actual percent of positives: The actual percent of respondents who have a rating of 6 – 10 (Positives) Confidence Interval: Sample statistic ± margin of error at a 95% confidence level for estimated percent of Positives  CEM professionals were given different NPS and Mean values and asked to estimate the % of customers who are Positives (6 – 10 rating). For estimation using Mean Values, N = 26 to 30. For estimation using NPS Values, N = 33 to 41. 0 10 20 30 40 50 60 70 80 90 100 -100 -50 0 50 100 PercentofRespondentswitha ratingof6orgreater NPS Value Actual % 6 or greater Estimated % 6 or greater 0 10 20 30 40 50 60 70 80 90 100 4.0 5.5 7.0 8.5 10.0 PercentofRespondentswitha ratingof6orgreater Mean Value Actual % 6 or greater Estimated % 6 or greater  Greater downward bias in predictions when NPS is used.
  12. 12. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com Study Findings  Some bias in customer metrics  CX professionals underestimate the size of important customer segments when only looking at Mean or Net Promoter Scores  CX professionals are better at estimating the size of NPS-specific segments based on the Net Promoter Scores than Mean Scores  CX professionals underestimate the size of Satisfied segment based on summary metric on both Mean or Net Promoter Scores (slightly worse for NPS)
  13. 13. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com Implications  Operations  Correct interpretation of results improves how leaders run their organizations  Present both aggregated metric (NPS or Mean) and segment-specific metric (Top/Bottom Box Score)  Minimizes misinterpretation of data  Human Resources  Data Education / Training on basics of data analysis  Data literacy is essential to Big Data Success  Visualization/Analytic Software  Pick vendors who allow you to summarize your data in different ways
  14. 14. Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com For More Information Bob E. Hayes, Ph.D. Email: bob@businessoverbroadway.com Web: www.businessoverbroadway.com Blog: www.businessoverbroadway.com/blog Twitter: www.twitter.com/bobehayes

×