Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

In a Word: The Customer Sentiment Index

3,231 views

Published on

I developed the Customer Sentiment Index, a measure based on a single word that customers use to describe a company. Using the following question, "What one word best describes this company?", Results show that the customer-generated words can be reliably scaled along a sentiment continuum and that these scores (CSI) are logically related to important customer experience metrics, like customer loyalty and satisfaction with the customer experience.

Published in: Marketing
  • Be the first to comment

In a Word: The Customer Sentiment Index

  1. 1. Developing the Customer Sentiment Index Bob. E. Hayes, PhD bob@analyticsweek @bobehayes
  2. 2. 2Copyright © 2015 Business Over Broadway and AnalyticsWeek Who I am Scientist / Blogger Author Consultant
  3. 3. 3Copyright © 2015 Business Over Broadway and AnalyticsWeek Business Over Broadway Solve problems, primarily business problems, through the use of the scientific method Owner Using data and analytics to help make decisions that are based on fact, not hyperbole. AnalyticsWeek Help businesses optimize their data/analytics Chief Research Officer Improving talent/technology recruitments, facilitating deeper community engagement with the power of online/offline channels and conducting research What I do
  4. 4. 4Copyright © 2015 Business Over Broadway and AnalyticsWeek Structured • Data generated to measure specific construct How satisfied are you with company? • Customer-generated Unstructured • Data not generated to measure specific construct emails, social media, support calls, movie reviews, tweet content, transcripts of comments • Algorithm-generated Approaches to Measuring AttitudesIntentSourceScore
  5. 5. 5Copyright © 2015 Business Over Broadway and AnalyticsWeek Intentionally Unstructured Measurement • Intentional measurement – Text needs to be generated for the purpose of measuring attitude • Sentiment lexicons – list of words that reflect an opinion on a negative- positive continuum • What one word best describes this company? • What one word best describes this company’s products / services?
  6. 6. 6Copyright © 2015 Business Over Broadway and AnalyticsWeek What one word best describes this company?
  7. 7. 7Copyright © 2015 Business Over Broadway and AnalyticsWeek Developing a Sentiment Lexicon • Judgment-based – Subject matter experts’ classification/scaling of words • Empirically-based – User-supplied textual metadata (e.g., star ratings on reviews)
  8. 8. 8Copyright © 2015 Business Over Broadway and AnalyticsWeek Judgment-Based • Two subject matter experts independently rate list of words from customer survey – 0 (negative sentiment) to 10 (positive sentiment) • High agreement between experts (rater) Lexicon Mean SD 1 2 3 4 1. Rater 1 First Rating 6.49 2.32 2. Rater 2 First Rating 6.44 1.93 .87 3. Rater 1 Second Rating 6.35 2.39 .98 4. Rater 2 Second Rating 6.42 1.95 .99 .90 5. Average Sentiment 6.39 2.11 .96 .96 .98 .97 N = 251. All correlations are statistically significant at the p < .01 level. Average sentiment: Based on based on the average second sentiment ratings of each rater. Bold correlations represent inter-rater agreement. Italic correlations represent intra-rater agreement. Descriptive Statistics Correlations
  9. 9. 9Copyright © 2015 Business Over Broadway and AnalyticsWeek Empirically-Based • Examine four corpora (a collection of written text) with accompanying ratings • Data from four review sites* – OpenTable, IMDB, Goodreads, Amazon/TripAdvisor * See Christopher Potts: http://web.stanford.edu/~cgpotts/talks/potts-wordnetmods.pdf
  10. 10. 10Copyright © 2015 Business Over Broadway and AnalyticsWeek Description of Four Lexicon Sources
  11. 11. 11Copyright © 2015 Business Over Broadway and AnalyticsWeek Calculating Sentiment for Each Word • Re-scale values from 1 to 5 -> 0 to 10 • Calculate sentiment of each word (adjective) – “excellent” sentiment value of 8.39 – “good” sentiment value of 6.69
  12. 12. 12Copyright © 2015 Business Over Broadway and AnalyticsWeek Distribution of Words’ Sentiment Values
  13. 13. 13Copyright © 2015 Business Over Broadway and AnalyticsWeek Descriptive Statistics and Correlations of Sentiment Values of Words
  14. 14. 14Copyright © 2015 Business Over Broadway and AnalyticsWeek Judgment and Empirical Approach • Classification/Scaling of words are comparable across methods Lexicon Mean SD N 1 2 3 4 1. OpenTable 7.07 1.48 164 2. AmazonTripadvisor 7.52 1.17 168 .72 3. Goodreads 6.48 1.29 168 .58 .60 4. IMDB 6.95 .82 169 .72 .62 .73 5. Expert 6.39 2.11 251 .71 .60 .52 .67 All correlations are statistically significant at the p < .01 level. Expert sentiment was the average of the experts' second/adjusted sentiment ratings. CorrelationsDescriptive Statistics
  15. 15. 15Copyright © 2015 Business Over Broadway and AnalyticsWeek Difference among Lexicons
  16. 16. 16Copyright © 2015 Business Over Broadway and AnalyticsWeek Reliability of Customer Sentiment Index • B2B Technology Company – Customer Survey – one word, ratings • Context is important Mean SD N 1 2 3 4 1. CSI - Expert 7.09 1.84 894 2. CSI - OpenTable 7.12 1.18 766 .77 3. CSI - IMDB 6.78 .86 786 .60 .78 4. CSI - Goodreads 6.30 1.23 757 .62 .74 .93 5. CSI - Amason/Tripadvisor 7.65 .97 623 .65 .83 .77 .68 Correlations among CSI scores Mean SD N CSI Expert CSI OT CSI IMDB CS GR Overall Satisfaction 7.60 1.99 1595 .57 .48 .33 .30 Recommend 7.91 1.96 1585 .56 .49 .35 .31 Purchase same / similar 8.22 2.26 1527 .34 .31 .24 .24 Purchase additional / different 6.32 2.83 1508 .18 .16 .10 .09 Expand use 6.80 2.59 1523 .22 .22 .18 .16 Renew service contract 7.90 2.53 1159 .30 .25 .19 .18 Ease of doing business 7.37 2.17 1204 .55 .49 .37 .35 Account Management 6.98 2.32 1189 .42 .35 .26 .25 Product Quality 7.93 1.89 1297 .50 .44 .27 .27 Service / Repair 7.67 2.15 1092 .41 .37 .29 .30 Technical Support 7.73 2.29 1253 .43 .37 .29 .29 Communications from Company 7.37 2.13 1282 .51 .45 .34 .32 Direction and future products/services 7.45 1.96 1165 .48 .41 .26 .26 All correlations statistically significant at the p < .05 level. All measures are on a scale from 0 (low loyalty/satisfaction) to 10 (high loyalty/satisfaction). Based on respondents (N = 1619) of annual customer survey of a B2B technology company. Loyalty Satisfactionwiththe CustomerExperience Correlations of CSI scores with Loyal
  17. 17. 17Copyright © 2015 Business Over Broadway and AnalyticsWeek Validity of Customer Sentiment Index Mean SD N CSI Expert CSI OT CSI IMDB CSI GR CSI A/TA Overall Satisfaction 7.60 1.99 1595 .57 .48 .33 .30 .43 Recommend 7.91 1.96 1585 .56 .49 .35 .31 .42 Purchase same / similar 8.22 2.26 1527 .34 .31 .24 .24 .21 Purchase additional / different 6.32 2.83 1508 .18 .16 .10 .09 .12 Expand use 6.80 2.59 1523 .22 .22 .18 .16 .18 Renew service contract 7.90 2.53 1159 .30 .25 .19 .18 .21 Ease of doing business 7.37 2.17 1204 .55 .49 .37 .35 .45 Account Management 6.98 2.32 1189 .42 .35 .26 .25 .36 Product Quality 7.93 1.89 1297 .50 .44 .27 .27 .42 Service / Repair 7.67 2.15 1092 .41 .37 .29 .30 .30 Technical Support 7.73 2.29 1253 .43 .37 .29 .29 .33 Communications from Company 7.37 2.13 1282 .51 .45 .34 .32 .39 Direction and future products/services 7.45 1.96 1165 .48 .41 .26 .26 .37 All correlations statistically significant at the p < .05 level. All measures are on a scale from 0 (low loyalty/satisfaction) to 10 (high loyalty/satisfaction). Based on respondents (N = 1619) of annual customer survey of a B2B technology company. Loyalty Satisfactionwiththe CustomerExperience Correlations of CSI scores with Loyalty/CX Metrics
  18. 18. 18Copyright © 2015 Business Over Broadway and AnalyticsWeek Relationship between CSI and Recommend B2B Survey B2C Survey
  19. 19. 19Copyright © 2015 Business Over Broadway and AnalyticsWeek Applications • Mobile Surveys • Extract more information from single word – Apply different lexicons (e.g., anxiety, strength) • Simplify feedback process – Shorter surveys benefit customers – Fewer dashboard metrics facilitate executive reports
  20. 20. 20Copyright © 2015 Business Over Broadway and AnalyticsWeek Bob E. Hayes, Ph.D. Email: bob@analyticsweek.com and bob@businessoverbroadway.com Web: www.analyticsweek.com and www.businessoverbroadway.com Blog: www.businessoverbroadway.com/blog Twitter: www.twitter.com/bobehayes

×