SKIM at Sawtooth Software Conference 2012: Analyzing Menu-based Conjoint modeling data

1,665 views

Published on

At Sawtooth Software's 2012 Conference, our methodologists Gerard Loosschilder and Paolo Cordella presented two approaches to analyzing Menu-Based Choice modeling data on their predictive validity.

Published in: Business, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,665
On SlideShare
0
From Embeds
0
Number of Embeds
227
Actions
Shares
0
Downloads
23
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

SKIM at Sawtooth Software Conference 2012: Analyzing Menu-based Conjoint modeling data

  1. 1. expect great answersMenu-Based Choice modeling (MBC):a practitioner’s comparison of different methodologiesSawtooth Software Conference, March 2012Carlo Borghi, Paolo Cordella, Kees van der Wagt and Gerard Looschilder
  2. 2. Menu-based choice modeling – the next big thing Sawtooth Software has recently launched its new Menu Based Choice modeling software. Although the idea of build-your-own exercises has been around for a while, the launch of a new tool from Sawtooth Software usually causes a lot of excitement and uptake of use. As practitioners, we at SKIM want to be ready for the avalanche of projects, so we started to look into pros and cons of several analysis approaches.2
  3. 3. Look around Menu-based choices are everywhere and are becoming increasingly common Menu-based choice modeling – the next big thing3
  4. 4. ✔ ✔ ✔ Cheddar $0.50 American cheese $ 0.75 Curly fries $1.25 Whopper $3.50 California W. $ 4.50 ✔ ✔ Crispy Onions Bacon $1.50 $1.50 Omega3 $3.75 Chicken Deli $ 3.50 French fries $1.05 Total price $ 8.504
  5. 5. Menu-based choice exercises are found in areas wherecombining items matters• Menu optimization in fast food/branded restaurant chains• Telecom services bundling• BYO computers (e.g. Dell)• Optional features pricing optimization in automotive market• Add-on services in the financial and insurance services industry
  6. 6. Menu-based Choice Modeling exercises deliver item- level forecasts of performance in these markets It can deliver: • Demand curves on an item level among many items • Forecast revenue and find the optimal price for all items on the menu • Measure uptake and decide whether to add a new item to your portfolio • Cross-effects price sensitivity and cannibalization effects • Does decreasing the price of single items hurt full menu sales? • Most often chosen combinations and their prices • Suggesting which items to bundle • Insight in budget constraints • How many items can we stuff in a bundle before we exceed the decision maker’s budget?6
  7. 7. At SKIM, we’re practitioners. We would like to understand how MBC works in our practice In particular, we would like to better understand the analysis procedure. At first sight, we loved the beta version of the Sawtooth Software tool, but we wanted to investigate more. So we developed an alternative analysis approach, and applied it to a study into the consumer’s willingness to pay for features of a notebook computer. This presentation contains a comparison of results on aspects of internal validity.7
  8. 8. We apply the approach to a study into consumer features of computer notebook SKIM’s Menu Based Choice exercise8
  9. 9. Eye scanner High quality touch- Technical advancement has screen brought new vistas of safety and security and today it is No glare screen very easy to make your Easy keys laptops and notebooks safe and secure with technologies such as fingerprint readers, Universal plug for face recognition, eye External radio w/ US, EU, UK speakers scanners etc. Wireless speakers On-screen keyboard spotlight Eye recognition ensured only you can access your laptop through a fast and accurate scan of the retina. The laser scanner is conveniently positioned on top of the Gold-plated jack External battery screen, next to the webcam indicator 3D-ready HD DVORAK keyboard webcam9
  10. 10. This pilot application had the following specifications: 7 26 9 3 attributes Choice tasks price levels • 12 consumer 9 choice tasks: There are 3 price features • 7 random tasks levels per feature, (2 levels: On/Off) • 2 hold out tasks varied in accordance • 12 price attributes to estimate with an orthogonal (3 levels) predictive validity research design • 1 notebooks core attribute (3 levels) • 1 none option Sample size: 140810
  11. 11. There are various models to analyze MBC data: As presented in Bryan Orme’s paper “Menu-Based Conjoint Modeling Using Traditional Tools” : • Exhaustive Alternatives Model • Serial Cross Effect Model Both models have drawbacks that we thought we could solve using SKIM’s method: • Choice Set Sampling Model11
  12. 12. Exhaustive Alternatives Model All possible ways to choose options are included Drawbacks in the choice set. The number of possible • This model formally recognizes and predicts the combinations grows combinatorial outcomes of menu choices. exponentially with the • The dependent variable is the choice of a number of options (2ⁿ combination using a single logit-based (MNL) dichotomous choices), model transcending into a • All possible combinations of options are coded as problem of computational one attribute where each level is a combination : feasibility. • e.g. with 3 on/off options, this attribute would have 2^3=8 levels • Price: one price attribute for each option (or one total price attribute)12
  13. 13. Serial Cross-effect Model The choice of each option is modeled Drawbacks separately Only significant • The dependent variable is the single cross-effects should choice of a feature be included - meaning they have • N different logit models predicting the to be detected choice of option X as a function of: beforehand • Price of option X • All other significant cross effects13
  14. 14. We thought of solving it by introducing a ‘hybrid’ approach: Choice Set Sampling approach • Like in the Exhaustive Alternatives Model, we consider the full choice set with all possible combinations of options. However: • we code each feature and its price as separate attributes (instead of a unique attribute with all combinations as levels); • we use importance sampling* – we consider a random sample from the set of all chosen combinations • Similarly to the Serial Cross-Effect Models, we also consider whether a respondent chose an option at various price points. * See importance sampling Ben-Akiva and Lerman (1985)14
  15. 15. Coding the “sampling of alternatives” approach 1. In our model there are a total of 3*2^12=12888 possible combinations. However, “only” 4560 were chosen at least once. Combination # Core Feature 1 Price 1 Feature 2 Price 2 Feature 3 Price 3 ... Feature 12 Price 12 Choice 1 3 2 0 1 3 2 0 ... ... ... 0 2 1 2 0 2 0 1 2 ... ... ... 1 3 2 1 1 1 3 2 0 ... ... ... 0 ... ... ... ... ... ... ... ... ... ... ... ... 4560 3 2 0 1 3 2 0 ... ... ... 0 Note: » Each feature is either included in the combination (1) or not (2) » Option prices are alternative specific 2. We draw a random sample from this choice set. It is basically still a single logit-based (MNL) model where the dependent variable is the choice of the combination.15
  16. 16. Coding the “sampling of alternatives” approach 3. Each task is codified with 33 concepts/combinations drawn from the sub-sample, with: • The chosen alternative in each task • 32 combinations randomly sampled from the choice set of all chosen combinations CASEID Task# Concept# Core Feature 1 Price1 Feature 2 Price2 ... Response 1 1 1 1 1 2 1 3 ... 0 1 1 2 1 2 0 1 3 ... 1 1 1 3 1 2 0 2 0 ... 0 ... ... ... ... ... ... ... ... ... ... 1 1 33 2 2 0 1 3 ... 016
  17. 17. Coding the “sampling of alternatives” approach 4. In addition, our model is “hybrid” because we add extra dummy tasks for each respondent: • For each choice task, we add 12 dummy tasks, one per feature • We check whether a feature has been chosen at a specific price point • No explicit modeling of cross effects between features CASEID Concept# Core Feature 1 Price1 Feature 2 Price2 Feature 3 Price3 ... Response 1 1 1 1 1 2 0 2 0 ... 1 1 2 1 2 0 2 0 2 0 ... 0 1 1 1 2 0 1 3 2 0 ... 0 1 2 1 2 0 2 0 2 0 ... 1 1 1 1 2 0 2 0 1 1 ... 0 1 2 1 2 0 2 0 2 0 ... 1 • This coding contains the information that respondent 1 in task 1 chooses feature 1 at price points 1, while she does not choose feature 2 at price point 3, and so on. Therefore we embed a price barrier in our model which amplifies accuracy in price sensitivity estimation.17
  18. 18. Analysis steps of SKIM’s Choice Set Sampling approach • Using this setting we run HB estimation, so we can estimate utilities for: • Each feature (present/not present; 12 utility values and their mirrors) • Each price level for each feature (3*12 utility values) • None option (nothing is chosen; one utility value) • We build a simulator in Excel, based on either Share of Preference (SoP) or Share of First Choice (SoFC) with which we have: • Single feature choice prediction • Combinations choice prediction18
  19. 19. Serial-Cross effect model • Using Sawtooth Software’s MBC we build 12 different models for each feature choice • We could not find any significant cross-effects between the features, both using counts and aggregate logit • We use HB estimation and we simulate: • Single Feature Choice predictions using Draws and Point Estimates • Combinations choice predictions using Draws, Point Estimates and Weighted Draws.19
  20. 20. All approaches can be used to answer the same business question That’s why we compare the approaches to see: • Which approach delivers the highest validity? And because as practitioners, we often find ourselves dealing with demanding clients and strict deadlines, so that we don’t just need approaches that work but that are also efficient and as easy to apply: • Which approach is most efficient to a practitioner?20
  21. 21. The results - Choice Set Sampling vs Serial Cross-Effect model Which approach has the highest validity?21
  22. 22. Results 1: Single Features Choice Predictions The Hold-out choice tasks suggest a similar performance Hold-out 1 Hold-out 2 R-Squared MAE R-Squared MAE Serial HB, Point Estimates 0.991 0.9% 0.990 1.0% Cross-Effect Model HB, Draws 0.991 0.9% 0.992 1.1% HB, First Choice 0.987 1.6% 0.989 1.7% Choice Set Sampling Model HB, Share of Preference 0.984 1.5% 0.981 1.5%22
  23. 23. Both approaches have a very low MAE on the hold-out tasks 60.0% 50.0% Freq. of choice 40.0% 30.0% 20.0% 10.0% 0.0% Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option Option Option 10 11 12 Observed Serial cross-effect model (Draws) Choice set sampling model (SoP) Holdout task - 123
  24. 24. No structural consistency in errors Holdout task - 1 6.0% 5.5% 5.0% Absolute error 4.5% 4.0% 3.5% 3.0% 2.5% 2.0% 1.5% 1.0% 0.5% 0.0% Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option Option Option 10 11 12 Choice set sampling model (First Choice, MAE = 1.5%) Serial cross-effect model (Draws, MAE = 0.9%)24
  25. 25. The individual hit rate is almost the same across the two hold out tasks • Hit rate: % of respondents for which the choice on the option was predicted correctly • 2 holdout tasks x 1408 respondents = 2816 observations for the hit rate 100.0% 91.1% 90.9% 86.8% 86.5% 87.5% 86.6% 86.5% 89.0% 87.0% 87.2% 90.0% 84.4% 84.9% 90.7% 90.2% 88.9% 80.0% 86.4% 85.8% 86.5% 85.1% 87.0% 86.5% 86.8% 86.7% 84.1% 70.0% 60.0% Hit rate 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option Option Option 10 11 12 Choice set sampling model (First Choice) Serial cross-effect model (Weighted Draws)25
  26. 26. Result 2: Feature combination predictions Both models fit individual choices of combinations 100% 90% 83.5% 86.2% 80% 86.4% 83.6% 70% 66.2% 67.7% 63.6% 67.9% 60% 66.2% 63.2% 50% 40% 41.3% 41.0% 30% 20% 10% 0% All 12 option At least 11 At least 10 At least 9 choices At least 7 choices At least 8 choices choices predicted choices predicted choices predicted predicted correctly predicted correctly predicted correctly correctly correctly correctly Choice set sampling model (First Choice) Serial cross-effects model (Weighted Draws)26
  27. 27. So we can conclude that both approaches are viable tools for MBC analyses Both models • Are able to predict accurately hold-out choice tasks on aggregate level • Are extremely effective to predict individual choices of single options and combinations So both models are viable tools for analyzing MBC data. • But which one is the most effective for practitioners?27
  28. 28. So both approaches work and it comes down to efficiency. Which approach is most efficient to a practitioner?28
  29. 29. Choice Set Sampling approach – Benefits and Drawbacks Benefits Drawbacks • One model to estimate, one • Complex procedure: time model to simulate consuming set up for estimation • No need to make a call on • Simulations are computationally which cross-effects to intensive include • Simulators are not very handy • Explicitly predicts choice of tools for clients combinations29
  30. 30. Serial Cross Effect Model – Benefits and drawbacks Benefits Drawbacks • Dedicated software • Learning curve of available understanding how to interpret the significance of cross / • Explicit inclusion of cross- interaction effects and their effects in the model inclusion in the model – it • Easy simulation tools takes art and craft to build an accurate model • Once cross-effects are included in the model, they hold for all respondents30
  31. 31. To conclude: Sawtooth Software’s Serial Cross-effect model is the practitioner’s choice • We would recommend using Sawtooth Software’s Serial Cross-effect model and software package, • After the initial learning, it’s an easy to apply and time-effective solution, thanks to its dedicated software • One just needs to invest in the learning curve of making the call about the significance and meaning of interaction/cross effects • If you want to use the Choice Set Sampling model, be prepared to invest time to create dedicated tools31
  32. 32. contact us or follow us online!Carlo Borghi, Paolo Cordella, Kees van der Wagt and Gerard Looschilderwww.skimgroup.com | +31 10 282 3535 linkedin.com/ facebook.com/ twitter.com/ youtube.com/ skimgroup.com company/skim skimgroup skimgroup skimvideos

×