• Save
Quantifying the Buzz Effect: Integrating Social Media With Loyalty & Defection Models
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Quantifying the Buzz Effect: Integrating Social Media With Loyalty & Defection Models

  • 1,205 views
Uploaded on

Measuring Loyalty & Defection of Ford Owners Through Text Mining Social Media: How to identify and qualify comments made through social media outlets, focus on how to attribute those comments to a......

Measuring Loyalty & Defection of Ford Owners Through Text Mining Social Media: How to identify and qualify comments made through social media outlets, focus on how to attribute those comments to a customer, and ultimately quantify the relationship between those comments and future customer behavior.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,205
On Slideshare
1,192
From Embeds
13
Number of Embeds
2

Actions

Shares
Downloads
0
Comments
0
Likes
1

Embeds 13

http://192.168.6.179 12
https://twitter.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Quantifying the "Buzz" Effect:Integrating Social Media with Loyalty & Defection Models Marketing Associates: Keith Shields, Director, Decision Sciences Roni Leibovitch, Senior Consultant, Digital Intelligence Mindy Deatrick, Senior Consultant, Quantitative Solutions Ford Motor Company: Margaret Kishore, Performance and Metrics Manager
  • 2. About the Title…  “Buzz” refers to the amount of, and sentiment of, the Ford-related comments available through social media outlets.  The “Buzz Effect” refers to the increase or decrease in brand loyalty / defection (measured by repurchase) that occurs as a result of a change in the Buzz.  “Quantifying the Buzz Effect” means we want to put a number on the amount of that increase or decrease.  The advantage of this is that we can begin to put a dollar value on salient, publicly- known events…such as refusing to take government bailouts.  “Integrating Social Media With Loyalty / Defection Models” means that we will:  Extract signals of future vehicle purchase decisions from customer comments found through social media outlets AND  Capture those signals in the form of predictive variables to put into loyalty models.  Those variables and their associated model coefficients will quantify the buzz effect.  For the purpose of this analysis we focus our efforts on Twitter. 1
  • 3. Warnings and Disclaimers  We will do our best to reveal trends, patterns, and findings without showing actual numbers (but for some cases). Hiding / changing of numbers is done to protect the innocent (Ford Motor Company especially).  In the course of the presentation we will share many Ford-related “tweets”. These will be actual tweets. They will not be censored because their informal nature highlights a point we want to make about text mining. Please try not to be offended.  We use “off-the-shelf” techniques when it comes to categorizing sentiment.  Our expertise is in modeling and predicting customer behavior based on all available and relevant customer data.  We see social media as a potentially rich source of customer data, and those data just happen to be free-form text. 2
  • 4. One More Item of Note… 3
  • 5. Background on Ford’s Social Media Efforts…  Measuring the “Consumer Experience”  Alan Mulally and Apple…  The Dealership Experience: Sales and Service  The Ownership Experience  How do people share experiences? Traditionally by talking to each other. But how much today is done through Twitter, Facebook, Blogs?  By analyzing the comments and sentiment expressed through Social Media outlets can we glean meaningful insights about the Ford Consumer Experience?  Can we make inference about a consumer’s affinity for Ford…or an existing customer’s loyalty to Ford?  If no, then we’re probably not trying hard enough.  Examples next 2 slides. 4
  • 6. Google Twitter Search: Ford Comments Search: “My Ford Focus is great.”  I love my Ford Focus, but not so much Ford Service in Northampton Mass. Thieves.  Got my new computer yesterday and cant wait to get my new 2012 Ford Focus SEL in 4-6 weeks! 23 Apr  Am test driving Hondas and Fords 7 Apr We’d like to have a mechanism for intervening here. On April 7 this person indicated he was facing a choice between buying a Honda and buying a Ford. Does this mean we can simply scrape Twitter for the words “test drive”? Seems like it would be predictive of future behavior… 5
  • 7. Google Twitter Search: Ford Comments Search: “I don’t like my Ford Escort.”  The ford escort texting and driving, I really like my life and my car, please dont try and drive into us, twice. Close call!  My old 93 ford escort is running 130k and runs like a charm.... And my 2003 ford ranger truck has 80k without problems. Again this seems like something that, if captured and quantified in the form of a variable, would be predictive in the context of a loyalty model. 6
  • 8. Google Twitter Search: Ford Comments  Search: “Ford, government bailout”  This weekend my wife and I purchased a FORD. Why? Because they chose not to accept the government funded bailout.  Ford didnt accept the government bailout - thats pretty awesome.  Wait #Ford pulled the ad that was critical of the #Obama bailout but is now running one that jokes about drinking and driving?  GM CEO wants higher gas tax. Buy a Ford car or truck. Please RT  This is an example of how capturing “influencers” could be very important. This person happens to have 340 followers and routinely tweets about auto-related topics.  So the effort to mine Twitter for Ford sentiment extends beyond improving the loyalty and defection models…but the title of this presentation does not. That said, we will discuss how we are affecting marketing programs with our existing knowledge of influencers. 7
  • 9. Background on Ford’s Social Media Efforts…  Measuring the “Consumer Experience”  Alan Mulally and Apple…  The Dealership Experience: Sales and Service  The Ownership Experience  How do people share experiences? Traditionally by talking to each other. But how much today is done through Twitter, Facebook, Blogs?  By analyzing the comments and sentiment expressed through Social Media outlets can we glean meaningful insights about the Ford Consumer Experience?  Can we make inference about a consumer’s affinity for Ford…or an existing customer’s loyalty to Ford?  Yes! So what can we do capitalize upon good sentiment and reverse bad sentiment? 8
  • 10. Start With the Current Infrastructure The Ford Motor Company has a customer data warehouse that collects relevant data from all customer touchpoints, “customerizes” it, and applies a suite of predictive models that are used for targeted campaigns. More importantly the warehouse is connected to many customer-facing and dealer-facing operational systems, and it passes important information about customer behavior, both past behavior and predicted behavior, to operational systems when decisions regarding the customer have to be made in real time. Users & business interface Customer Data by Touch Point Dealers Call Center Click-thru Mail Call Center Data Creation Sales Dealer Service Customers Website Survey LOYALTY AND DEFECTION User-facing systems easily access MODELS APPLIED scores and strategies in real time. Customer Data “Customerization” Data Refinery Warehouse 9
  • 11. Fitting in to the Current Infrastructure… Social media is just another customer touch point. The text we mine from social media outlets is another set of data about the customer, just like the call center, website, or the customer surveys.  We’d like to use that data just like the rest of the customer data: to help us predict customer purchase behavior. In some sense social media provides a source of unsolicited surveys. Users & business interface Customer Data by Touch Point Dealers Call Center Click-thru Mail Call Center Data Creation Sales Dealer Service Customers Website Comments / Survey Twitter, FB, … Sentiment LOYALTY AND DEFECTION User-facing systems easily access MODELS APPLIED scores and strategies in real time. Customer Data “Customerization” Data Refinery Warehouse 10
  • 12. A Source of “Unsolicited Surveys”… Why do we survey customers? From the narrow perspective of someone who predicts customer behavior, the graph below is a big reason why.  What’s more compelling is that relationship between a customer’s opinion and loyalty holds up when we control for predicted loyalty.  The “Loyalty=1” group is the group that scores in the lowest 20% of a loyalty model…a model built without survey data.  The results remain consistent within each loyalty tranche so much so that customers within group 5 can have lower repurchase rates than those in group 3, depending on survey response. How much do we spend on surveys?  Whatever it is, our feeling is that if we can establish the above relationship with social media sentiment (use it as your X-axis), and cover more customers for less than what we currently spend on surveys, then we have the beginning of a business case for extracting sentiment from social media. 11
  • 13. A Quick Digression on Business Cases…  We believe that there are three “pillars” for the business case to actively engage consumers through social media outlets (specifically Twitter): 1. Conquesting new customers 2. Concern resolution 3. Voice of Customer STRATEGY: Introduce Social Media as an additional consumer touch-pointSUPPORTS THE 1. Conquest 2. Concern Res 3. VOCSTRATEGY: $XX mils per year $XX mils per year $XX mils per year 12
  • 14. How We Use Survey Data In the Models…  Let P = probability a Ford customer will repurchase another Ford upon disposing of any one of his current Fords.  Logistic regression is a very popular way to model and predict P.  ln[p / (1-p)] = b0 + b1*x1 + b2*x2 + … bn*xn  b0, b1, b2 are parameter estimates. They quantify the extent to which x1, x2, …, xn affect the probability of repurchase.  x1, x2, …, xn are explanatory variables, e.g. # of previous Ford purchase, time since most recent Ford purchase, miles from nearest Ford dealership, etc…  Now let s1 = 1 if “very likely”, 0 otherwise  Let s2 = 1 if “likely”, 0 otherwise …  Let s5 = 1 if “not at all very unlikely”, 0 otherwise  Refit the logistic regression:  ln[p / (1-p)] = b0 + b1*x1 + … bn*xn + bn+1*s1 + bn+2*s2 + bn+3*s3 + bn+4*s4 + bn+5*s5 Can be thought of as the VOC (Voice of Customer) Index, but it’s based on just survey data, which may only be available on 10% (roughly) of the customers. This is a nice metric, because it, by design, predicts loyalty. 13
  • 15. How We Use Twitter Data In the Models…  Let’s treat the Ford customer’s “tweets” the same way we treat survey data.  Go back to our logistic regression:  ln[p / (1-p)] = b0 + b1*x1 + … bn*xn + bn+1*s1 + bn+2*s2 + bn+3*s3 + bn+4*s4 + bn+5*s5  And let t1 = 1 if we can identify a “Ford positive” tweet for the customer, 0 otherwise.  Let t2 = 1 if we can identify a “Ford neutral” tweet for the customer, 0 otherwise.  Let t3 = 1 if we can identify a “Ford negative” tweet for the customer, 0 otherwise.  Refit the model:  ln[p / (1-p)] = b0 + b1*x1 + … bn*xn + bn+1*s1 + bn+2*s2 + bn+3*s3 + bn+4*s4 + bn+5*s5 + bn+6*t1 + bn+7*t2 + bn+8*t3 Can be thought of as the BUZZ INDEX, and it comes directly from what ford customers are saying on Twitter. This metric also, by design, predicts loyalty. So this is a quantification of the BUZZ EFFECT. 14
  • 16. Interpreting the “Twitter-Enhanced” Model…  ln[p / (1-p)] = b0 + b1*x1 + … bn*xn + bn+1*s1 + bn+2*s2 + bn+3*s3 + bn+4*s4 + bn+5*s5 + bn+6*t1 + bn+7*t2 + bn+8*t3  When we fit this model, we get an intuitive result:  bn+6 > bn+7 > bn+8 => good tweets lead to higher loyalty than do neutral tweets, neutral tweets lead to higher loyalty than bad tweets.  Not as intuitive (but interesting nonetheless):  All three parameters are greater than 0 (implying ANY tweeting is better than no tweeting).  bn+8 (the parameter for bad tweets) is NOT SIGNIFICANT. There is not a sufficient volume of bad tweets to support a significant result. The large majority of FLM tweets are good.  We think that the upshot of all of this is that tweeting about Ford, irrespective of sentiment, signifies a high level of customer engagement. This has implications beyond our efforts to better predict loyalty. 15
  • 17. The Buzz Variables Improve theLoyalty Model: So What?  More data and better data yield models that do a better job of “separating” loyalists and non-loyalists.  One way this manifests itself: ranking the population of customers with a better model will yield higher repurchase rates in the top decile (or demi-deciles…depending on how many groups you want to establish), and lower repurchase rates in the bottom decile.  So a marketing campaign that increases everyone’s likelihood of repurchase by 15% (not an uncommon number), does so on a larger base of loyalists within the top decile, and thus creates more incremental sales for the same amount of mailings.  Say the difference between these two bars is 200 bps.  Some of the incremental sales from the campaign Repurchase Rate Old Model noted in the bullet above (top decile only), are Model w/VOC & Buzz attributable to having a better model.  How many? .02 * .15 * top decile population  If the population of interest is 250,000 customers, then the impact of the better model is 750 incremental repurchases.  If a repurchase is worth $5,000 profit, then the case 1 2 3 4 5 6 7 8 9 10 for the “buzz variables” is substantial: $3.75 million. Model Decile Low Loyalty to High 16
  • 18. Why We Will Regularly Re-Fit the Buzz Index  We have several reasons to believe that results may change when we re-fit our enhanced loyalty model (twitter sentiment data being the enhancement): 1. The number of Tweeters is increasing all the time. Ford’s customer email capture isn’t great but it is improving, and there is evidence that Ford customers are, relatively speaking, very active on Twitter. 2. Attribution of tweets to customers is difficult and unsure; finding the Twitter names of Ford customers is difficult and painstaking. 3. Classifying the sentiment of Tweets is an imprecise exercise, especially when using off-the- shelf tools and software. 4. The content of the “Ford-tweeting” population leads to potentially biased results; it is biased toward a demographic that naturally tends to be less Ford loyal:  Females  Young  Used vehicle owners  Appear to be more service loyal, which is a good thing 5. The tweeting population also happens to be geographically biased, but this does not concern us as much as #4. 17
  • 19. Re-Fit the Buzz Index:The Increasing Number of Tweeters According to GIGAOM (http://gigaom.com/), Twitter had 175 million users in December 2010, and was growing by 370,000 new users every day. Also as of 12/2010:  65% of those users lived outside the US.  Roughly 6% of all Americans were active on Twitter. More recent studies indicate the number is 9%-10% (or 13% of internet users). Of the Ford customers active as of 12/2010 (who had a valid email), we were able to find 9% of them active on Twitter. Not surprisingly we have found the number of Ford related tweets to be increasing over time.  Good and neutral tweets have increased whereas bad tweets have stayed flat. Seems like a good thing, but we have some thoughts on comment classification. Frequency of Ford-Related Comments Found on Twitter Ford-Related Twitter Comments "Binned" 60,000 6000 60000 50,000 Negative 5000 50000 Positive 40,000 Postive, Negative 4000 Neutral 40000 Neutral 30,000 3000 30000 20,000 2000 20000 10,000 1000 10000 0 0 0 Aug-08 Apr-09 Aug-09 Apr-10 Aug-10 Jun-08 Sep-08 Jun-09 Jun-10 Oct-08 Nov-08 Dec-08 Feb-09 Sep-09 Oct-09 Feb-11 Nov-09 Dec-09 Feb-10 Sep-10 Oct-10 Nov-10 Dec-10 Sep-08 Sep-09 Sep-10 Nov-08 Nov-09 Nov-10 May-08 Jul-08 Jan-09 Mar-09 May-09 Jul-09 Jan-10 Mar-10 May-10 Jul-10 Jan-11 Mar-11 May-08 Jul-08 Jan-09 Mar-09 May-09 Jul-09 Jan-10 Mar-10 May-10 Jul-10 Jan-11 Mar-11 18
  • 20. Re-Fit the Buzz Index:Attributing the Tweets  Twitter names can be found if you can supply an email.  With an email address you can find, through the Twitter API, a Twitter name, a first name, and a last name associated with that email address. It will not return the email.  Ford has 11 million bought-new still-retained customers. We have emails on several million of them.  We cannot run several million emails through the Twitter API. Even if they could be processed, we would not be able to get back the email. We would only get back the thousands of Twitter names associated with, but not matched to the emails.  The only sure way to attribute emails to Twitter names is to go through the API one email at a time. Can we off-shore this? We can, but we still will not be done until 2013 if we go this route. So we had to be more clever…  Whatever method we choose, we need to recognize the Twitter name as useful customer data, and as such, store it in the data warehouse. Next slide… 19
  • 21. Re-Fit the Buzz Index:Attributing the Tweets – You Must Retain Data  What comes out of the attribution process is a table that looks like this: EMAIL TWITTER_NAME FIRST_NAME LAST_NAME kshields@marketingassociates.com Keith Shields rleibovitch@marketingassociates.com ronedog Roni Leibovitch mdeatrick@marketingassociates.com Mindy Deatrick  We have another process (using RADIAN6) that scours Twitter for comments that contain words in our “start list” (e.g. Ford, Lincoln, Mercury, Taurus, Mustang, Fusion, etc…). It produces a table that looks like this: TWITTER_NAME COMMENT SENTIMENT ronedog My new Ford Focus is also imported from Detroit. Good 20
  • 22. Re-Fit the Buzz Index:Attributing the Tweets – You Must Retain Data  We have to integrate this into existing data warehouse processes (which should be easy enough, if we’re treating this like just another source of customer data): Customer Touchpoint EMAIL TWITTER_NAME FIRST_NAME LAST_NAME Twitter Data Creation kshields@marketingassociates.com Keith Shields rleibovitch@marketingassociates.com ronedog Roni Leibovitch mdeatrick@marketingassociates.com Mindy Deatrick Customer Data TWITTER_NAME COMMENT SENTIMENT Warehouse ronedog My new Ford Focus is also imported from Detroit. Good 21
  • 23. Re-Fit the Buzz Index:Classifying the Tweets…  Prepackaged comment binning algorithms are not as accurate as we’d like…they result in a high instance of inappropriate comment binning (“Positive”, “Neutral”, or “Negative”). Here are some actual examples of inappropriately binned comments:  Positive: “Classic Car For Sale 2001 FORD EXPLORER - Mt. Royal NJ: Runs and Looks Great!!!”  Negative: “You have insulted my Ford Fiesta, shame on you.” AND “Just drove a Ford Fiesta getting 30 mph. Not bad!” AND “Just dropped my car off at the FORD Dealership . I want a FORD Fusion soooooo BAD.”  Neutral: “Ford Escape!!!” AND “Smart, easy, & fun ride with the new Ford Focus with Ford Syncs help!”  In order for us to get the intuitive results we showed on slide 12 we had to depart from sentiment classification algorithms and do a “brute-force” classification.  Interns, Cornerstone Schools, Detroit, Mi. (http://www.cornerstoneschools.org/)  There are other inexpensive ways…all of which we believe to be more accurate than existing “machine intelligence”…albeit not as scalable: • Existing call center personnel • Mechanical Turks (https://www.mturk.com/mturk/welcome) 22
  • 24. Re-Fit the Buzz Index:The Biased, But Changing, Population of Tweeters…  See the graph on the right. The fastest growing population of tweeters is 18-34 year olds. About 56% of tweeters are 34 years or younger.  The challenge for Ford: the median age for Ford customers is well above 34, despite some recent strong entries in the small car market.  The most “tweeted about” Ford is, not surprisingly, the Focus. 23
  • 25. Re-Fit the Buzz Index:Geographically-Biased Population of Tweeters…  The numbers represent how much higher the Twitter use per capita is in that state versus the nation as a whole.  For example: if the national usage rate is 10%, then Michigan is 11% lower than that: .10 - .11(.10) = 8.9%. 24
  • 26. Given the Worries About Attribution, Classification…  The real opportunity could lie in the business experts using intuition and common sense to tailor campaigns and programs to tweeters based on their most recent comments.  This would only rely upon a good mechanism for scraping relative comments from Twitter and reacting procedurally and appropriately.  If we look at the comments as unsolicited survey responses we see opportunities for customized offers and programs (no models needed – the comment reveals the customer’s intent):  Private Sales Offer and/or Pre-approval: “Just dropped my car off at the FORD Dealership. I want a FORD Fusion soooooo BAD.”  Rewards Program Offer: “My moms taking me to get this Ford explorer in the mornin tho i should have a new whip before July then im haulin ass to the A”  Offer for trade-in: “Ford focus sucks. Very uncomfortable vehicle.”  Offer for service discount / extended warranty: “My car is running rough and keeps blowing the injector and on plug coil fuses, its a 2006 3.0 V6 Ford Fusion. HELP!!” 25
  • 27. So We Revisit Our Three Pillars…  We believe that there are three “pillars” for the business case to actively engage consumers through social media outlets (specifically Twitter): 1. Conquesting new customers: pay attention to influencers 2. Concern resolution 3. Voice of Customer STRATEGY: Introduce Social Media as an additional consumer touch-pointSUPPORTS THE 1. Conquest 2. Concern Res 3. VOCSTRATEGY: $XX mils per year $XX mils per year $XX mils per year 26
  • 28. Conquest New Customers: Influencers Fact: 93.6% of Twitter users have less than 100 followers, while 98% of users have less than 400 followers. Meanwhile, 1.35% of users have more 500 followers, and only 0.68% of more than 1,000 followers. 27
  • 29. Conquest New Customers: Influencers Fact: As Twitter users attract more followers, they tend to Tweet more often. This is particularly evident once someone has 1,000 followers the average number of Tweets/day climb from three to six. When someone has more than 1,750 followers, the number of Tweets/day rises to 10. Fact: A small group of Twitter users account for the bulk of activity. Sysomos discovered that 5% of users account for 75% of all activity, 10% account for 86% of activity, and the top 30 account for 97.4% of activity. 28
  • 30. Conquest New Customers: The Opportunity  Find out who is expressing in-market sentiment and send them a targeted offer.  We estimate that through Twitter alone, roughly 35,000 customers per year express inclination to buy Ford.  Applying a result from an analysis of "handraiser campaigns", we assume 15% of the 35,000 will purchase FLM. This is 35,000 * 15% = 5,250 sales.  Assuming 20% lift from a targeted offer to in-market customers (not an uncommon number), we estimate that a conquesting campaign directed at in-market "social-media leads“. This is 5,250 * .2 = 1,050 incremental sales.  In 1Q2012 Ford will run a program that tests the opportunity we have framed in the above bullets. We will have the capability to offer differentiated treatment to influencers. Marketing Associates built the process to find the tweets and measure the back-end results.  INTEGRATION will be through the customer data warehouse and EXECUTION through the concern resolution center.  At this point we won’t trouble you with another infrastructure diagram. 29
  • 31. Some recommendations…  Treat social media as another source of relevant customer data.  Comments about your product are, in some sense, unsolicited surveys. They can, like surveys do, improve your ability to predict the behavior of your own customers.  Pay careful attention to the integration of social media data. Integration requires “customerization”, so subsequent customer behavior can be tracked.  Attributing comments to customers is tricky. It can also be painstaking. The good news is that it can be done cheaply.  Correct classification of comments is essential to understanding the true signal in the comments.  The most accurate means of classification may also be the least scientific: have an English speaker (who preferable understands colloquialisms) read the comments, and bin them.  Categories can be “good”, “bad”, “in-market”, “service issue”, or whatever aligns with the differentiated treatments and offers.  Retain data, and integrate intelligently into a data warehouse. Test and measure several tactical approaches to customers and prospects who are commenting about your products. 30
  • 32. Quantifying the "Buzz" Effect: Integrating Social Media with Loyalty & Defection Models Questions? Thanks for your time and attention. 30