SKIM at Sawtooth Software Conference 2013: ACBC Revisited

911 views
770 views

Published on

At Sawtooth Software's 2013 conference, SKIM conjoint experts, Marco Hoogerbrugge, Jeroen Hardon, and Chris Fotenos presented on Adaptive Choice-Based Conjoint.

Choice-Based Conjoint (CBC) is a long known, well-established "conjoint brand". It is widely regarded as the default method for nearly any piece of conjoint research. Not many people, excluding specialist methodologists, are aware that Adaptive Choice-Based Conjoint (ACBC) has been on the market as well for several years, and its predictive validity is much better than CBC when researching complex markets, with many products, in different price tiers. Through our presentation, we helped to confirm this general finding and elaborate on the various options that are available within ACBC.

Published in: Business, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
911
On SlideShare
0
From Embeds
0
Number of Embeds
209
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

SKIM at Sawtooth Software Conference 2013: ACBC Revisited

  1. 1. expect great answers ACBC Revisited 2013 Sawtooth Software Conference October 2013 | Marco Hoogerbrugge, Jeroen Hardon, Chris Fotenos
  2. 2. ACBC Introduction 2
  3. 3. Respondents first complete a build your own (BYO) exercise to reveal their ideal product 3
  4. 4. Next they are shown a screening section where they can accept or reject several possible purchase options 4
  5. 5. Within the screening section, respondents are asked whether certain levels of attributes are ‘unacceptable’ or ‘must-have’ for them 5
  6. 6. Given some insight into the respondents’ preferences, they finally see a series of choice tasks more aligned with their consideration set 6
  7. 7. Reduced design spaces for each respondent • In highly complex categories with many possible concepts ACBC helps to collect data on the combinations that are most relevant to each respondent. Orthogonal designs are created with the emphasis on the relevant sphere for each respondent at the cost of D-efficiency of the design at individual level. Design Space Respondent 1 Attribute B • Design Space Respondent 2 Design Space Respondent 3 Attribute A 7
  8. 8. Aimed at getting more relevant choice data in big and complex markets • Big markets  say, you have a simulator with 50 products (e.g. 7 brands x 7 tiers) • In regular CBC tasks a respondent chooses between concepts of different brands and different tiers • In ACBC, respondents choose between brands within a tier, or between tiers within a brand, or even more refined • Complex markets • Multi-attribute products (not just one SKU attribute) • Optional: price dependent on all or most underlying product features 8
  9. 9. To assess and search for ways to improve ACBC, we tested several variations of it along with other methodologies in the US TV market. 9
  10. 10. Our thoughts behind also including CBC legs • What are possible success factors and potential weaknesses of ACBC? And can we apply them also within CBC? • Reduced design space per respondent • Increased utility balance in every choice task • If ACBC is (perhaps) too extreme in applying these properties, CBC + one of these properties might become more successful. • Reduce design space  Dynamic CBC (see later) • Increased utility balance  CBC with between concept prohibitions 10
  11. 11. Our thoughts behind improving ACBC Price Quality Actual products in the market simulator 11
  12. 12. Modern televisions contain many features which creates a wide price spectrum and more complex purchase decision for shoppers Brand Screen Size Screen Type LED, LCD, LED-LCD, Plasma Resolution 720p, 1080p Wi-Fi Capability Yes/No 3D Capability Yes/No HDMI Inputs 0 – 3 HDMI Connections Price 12 22”, 32”, 42”, 52”, 57”, 62”, 67”, 72” $120 - $3,500
  13. 13. To benchmark and improve ACBC, we tested it against 7 other methodologies, including slight variations of ACBC itself Methodology Description A CBC Standard CBC B ACBC Standard ACBC C ACBC ACBC with price included in the unacceptable questions D Dynamic CBC SKIM-developed version of CBC where the design is adapted based on previous answers E ACBC ACBC with a more narrow tested price range of 90% to 120% F ACBC ACBC with 4 attributes varied from the BYO choice in the generation of concepts G CBC Price balanced CBC H ACBC Each leg 300 resps, total 2400 resps ACBC without the screener section, giving every respondent 12 total choice tasks In all ACBC legs, brand was not included in the BYO task. All legs showed respondents a summed total price for each concept (according to ACBC logic) 13
  14. 14. Dynamic CBC shows respondents a full set of CBC-like choice tasks with an ACBC-like design that is adapted based on their responses Like standard CBC, Dynamic CBC starts with a near-D-efficient design space for all respondents. • Respondents see a full set of choice tasks. • After 1/3 and 2/3 of the choice tasks, the design space is narrowed down based on the respondents’ choices in order to collect more data on the concepts that are most relevant to them. Design Space: Tasks 1-3 Design Space: Tasks 4-6 Attribute B • Design Space: Tasks 7-9 Attribute A 14
  15. 15. Following the ACBC/CBC exercise, respondents were shown a series of three holdout choice tasks • • 15 The holdout tasks presented respondents with concepts that were similar in their features and total prices. With a large number of products in every holdout task, this is much more like a real simulator than a usual holdout task of three or four concepts!
  16. 16. Conjoint models can seldom predict one clear ‘winner’ in these holdout tasks • Example of a share of preference distribution (model prediction) for a respondent: concept1 1.48% 0.03% concept12 0.88% concept3 0.65% concept13 29.78% concept4 0.33% concept14 0.95% concept5 4.68% concept15 27.31% concept6 0.18% concept16 12.99% concept7 4.67% concept17 4.67% concept8 16 concept11 concept2 • 0.64% 2.40% concept18 0.17% concept9 1.75% concept19 4.28% This same problem may well occur with actual simulators with many products! concept10 0.07% concept20 2.11%
  17. 17. Success metric: mean SoP So we thought it better to look at the means of the predicted share of preference of the concepts that the respondent chose in the three holdout tasks. 17
  18. 18. First results: respondents enjoy taking a (well designed) conjoint experience 18
  19. 19. Respondents liked taking the ACBC and CBC exercises with not many finding the tasks too difficult to complete “Thank you so much for letting me know about different types of TVs. Now I have an idea what is the best TV I should buy within 6 months.” (CBC leg) Standard CBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC ACBC w/ price range of 90%-120% ACBC w/ 4 atts varied from BYO Price balanced CBC ACBC without a screening section 0% I liked filling in this questionnaire 19 20% 40% 60% 80% 100% It was difficult for me to fill in this questionnaire
  20. 20. Respondents liked taking the ACBC and CBC exercises with not many finding the tasks too difficult to complete “I liked the format used in this survey. It was not confusing and questions were based on answers that I had previously given rather than random repetition.” (ACBC leg) Standard CBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC ACBC w/ price range of 90%-120% ACBC w/ 4 atts varied from BYO Price balanced CBC ACBC without a screening section 0% I liked filling in this questionnaire 20 20% 40% 60% 80% 100% It was difficult for me to fill in this questionnaire
  21. 21. Main results 21
  22. 22. All ACBC legs outperformed all CBC legs though there is not a significant difference between different ACBC variations ACBC without a screening section ACBC w/ price range of 90%120% ACBC w/ 4 atts varied from BYO ACBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC Price balanced CBC CBC Standard CBC 0% 10% 20% Mean SoP of Selected Holdout Task Concept 22 30%
  23. 23. Intermezzo All legs showed respondents a summed total price for each concept (according to ACBC logic) • • 23 Summed price = (summed value of attribute levels) * random number •  Price becomes a continuous variable Consequently we applied piecewise utility estimation for price in all legs • Number of cutpoints to be defined before running HB. ACBC allows up to 12 cutpoints.
  24. 24. Intermezzo Point estimation Utility 3 2 1 0 -1 $0 -2 -3 -4 -5 24 $50 $100 $150 $200
  25. 25. Intermezzo Piecewise estimation Utility 3 2 1 0 -1 $0 -2 -3 -4 -5 25 $50 $100 $150 $200
  26. 26. Adding more cut points in the piecewise estimation (more granularity in price utilities) significantly increases the performance, especially of the ACBC legs! ACBC without a screening section ACBC w/ price range of 90%-120% ACBC w/ 4 atts varied from BYO ACBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC Price balanced CBC CBC Standard CBC 0% 10% 20% Mean SoP of Selected Holdout Task Concept Original Analysis with 12 cut points Analysis with 25 cutpoints 26 30%
  27. 27. We were thrilled about some of these findings but disappointed about others • + • Omitting the screening section in ACBC isn’t bad • A narrower price range in ACBC isn’t bad • More cutpoints in piecewise estimation improves ACBC • All models beat random choice by a factor 3-4 • +/• Dynamic CBC performs significantly better than standard CBC 27
  28. 28. We were disappointed about some of these findings but thrilled about others • • None of the alternative ACBC legs outperformed current ACBC » The flipside of the coin is that ACBC is actually very robust! • Price balanced CBC didn’t outperform standard CBC • The absolute value of the mean SoP is actually quite low (~20%) » This may well be the case in actual simulators with many products too! 28
  29. 29. Mean Squared Error in a holdout task MSE = ∑ (Aggregate actual share – Aggregate mean predicted share) 2 concepts Or √ MSE = √∑ ( (Aggregate actual share – Aggregate mean predicted share) 2 ) concepts 29
  30. 30. MSE as a success metric confirms previous conclusions ACBC without a screening section ACBC w/ price range of 90%120% ACBC w/ 4 atts varied from BYO ACBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC Price balanced CBC CBC Standard CBC 0% 1% 2% 3% 4% 5% 6% √(MSE), average of three holdout tasks 30
  31. 31. One concept in one holdout task causes a large amount of the high MSE scores • Actual holdout share (aggregate level): 43% • Predicted holdout share: • Standard CBC leg is worst (predicts 8%) • CBC with price balance predicts 17% • ACBC modules are better don’t come very close (avg prediction 27%) 31
  32. 32. Results with and without this one concept ACBC without a screening section ACBC w/ price range of 90%-120% ACBC w/ 4 atts varied from BYO ACBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC CBC Price balanced CBC Standard CBC 0% 1% 2% Excluding one concept 32 3% 4% 5% 6% √(MSE), average of three holdout tasks Original Analysis
  33. 33. A step aside Cluster analysis on SoP predictions leads to a conclusion but not to a solution 33
  34. 34. Cluster analysis across the legs leads to two layers in respondent grouping All respondents Respondents who are predicted to buy a ‘low end’ TV (1/3 of sample) Respondents who are predicted to buy a ‘mid/high end’ TV (2/3 of sample) Different respondent groups for whom Different respondent groups for whom a particular brand is predicted a particular brand is predicted 34
  35. 35. Example: the high end Vizio cluster Mean predicted SoP for holdout task1 52.7% Mean predicted SoP for holdout task2 51.9% Mean predicted SoP for holdout task3 50.6% LG Panasonic Samsung Sony Vizio Actuals of holdout task1 Actuals of holdout task2 Actuals of holdout task3 0% 35 20% 40% 60% 80% 100%
  36. 36. The actual shares in the holdout tasks are much lower for Vizio Mean predicted SoP for holdout task1 52.7% Mean predicted SoP for holdout task2 51.9% Mean predicted SoP for holdout task3 50.6% 30.1% Actuals of holdout task1 35.2% Actuals of holdout task2 25.0% Actuals of holdout task3 0% 36 20% 40% 60% 80% 100% LG Panasonic Samsung Sony Vizio
  37. 37. Price plays a much bigger role in the actual choices than predicted Mean predicted SoP for holdout task1 52.7% Mean predicted SoP for holdout task2 51.9% Mean predicted SoP for holdout task3 50.6% 30.1% Actuals of holdout task1 35.2% Actuals of holdout task2 LG Panasonic Samsung Sony Vizio 25.0% Actuals of holdout task3 0% 20% 40% 60% 80% 100% Panasonic has cheapest product within the highest tier of holdout 1, Samsung 2nd cheapest. Samsung has cheapest product within the lowest tier of holdout 3. 37
  38. 38. This same phenomenon applies to all clusters • • 38 Brand is apparently estimated as too important, at the cost of price. This is perfectly in line with the most extreme mismatch in a concept that we saw before in the MSE calculation: the share of the cheapest product was greatly underestimated in this concept.
  39. 39. Another analytical approach (regression) leads to the same conclusions Dependents: actual holdout answers Aggregate multinominal logistic regression Explanatory: SoP predictions for all holdout tasks This aggregate regression leads to: - Positive coefficients for ‘own’ SoP of a concept, as expected - Also positive coefficients for SoPs of concepts with the same TV of competing brands  in order to partially undo the brand emphasis in the own SoPs 39
  40. 40. Conclusions 40
  41. 41. Conclusions • 41 ACBC again outperforms CBC (in a complex market).
  42. 42. Conclusions • ACBC again outperforms CBC (in a complex market). • We’ve seen no significant differences within the tested ACBC variations • This indicates that the performance of ACBC is robust. • It seems like ACBC can be made more efficient (omitting screener) with the same performance. 42
  43. 43. Conclusions • • We’ve seen no significant differences within the tested ACBC variations • 43 ACBC again outperforms CBC (in a complex market). Dynamic CBC performs much better than standard CBC; further refinement may lead to results close to or better than ACBC.
  44. 44. Conclusions • • We’ve seen no significant differences within the tested ACBC variations • Dynamic CBC performs much better than standard CBC; further refinement may lead to results close to or better than ACBC. • 44 ACBC again outperforms CBC (in a complex market). Performance of ACBC improves quite a lot by increasing the number of price slopes >11.
  45. 45. Conclusions • • We’ve seen no significant differences within the tested ACBC variations • Dynamic CBC performs much better than standard CBC; further refinement may lead to results close to or better than ACBC. • Performance of ACBC improves quite a bit by increasing the cutpoints for piecewise uitility estimation >11. • 45 ACBC again outperforms CBC (in a complex market). Respondents like a (well designed) conjoint survey better than we had thought, including standard CBC.
  46. 46. Directions for future research • Finding solutions for the discrepancies that we found between actual holdout answers and conjoint predictions • Developing more variants of Dynamic CBC • Double checking on existing data with holdout tasks • Increase number of cutpoints in piecewise estimation • Cluster analysis approach with predicted SoP 46
  47. 47. contact us or follow us online! Marco Hoogerbrugge Research Director m.hoogerbrugge@skimgroup.com Jeroen Hardon Senior Research Consultant j.hardon@skimgroup.com Chris Fotenos Project Manager c.fotenos@skimgroup.com

×