Inspire 2014 – Delhaize America: Improving Cross-Organization Program Performance

1,415 views

Published on

Delhaize America, LLC operates over 1,500 food supermarkets in the eastern United States under the Food Lion, Hannaford, and Bottom Dollar Food brands. In this session you will learn how Alteryx can be used for customized, site-specific test & control operations (A/B testing) within the larger framework of Delhaize America. See how Alteryx and Delhaize have partnered together to engineer practical solutions to real-world business issues, and discover some of the inherent advantages of adopting Alteryx vs. other tools.

Andrew Jastrzembski, Shopper Insights Manager, Delhaize America
Michal Polanowski, Shopper Insights Analyst, Delhaize America

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,415
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Inspire 2014 – Delhaize America: Improving Cross-Organization Program Performance

  1. 1. #inspire14 Test and Control: Solution Development Andrew Jastrzembski, Jason Thomas, Michal Polanowski Delhaize America
  2. 2. Who Are We Andy Jastrzembski Shopper Insights Manager where he provides the voice of the shopper. Started as an Operations Researcher Officer in the US Air Force and then transitioned to retail. Has BS in Applied Mathematics and Psychology and a MS in Operations Research. Jason Thomas One of the founders of the team. Worked for the company for 13 years. Currently, oversees the development of Shopper and Basket-level algorithms and scripts for both ad hoc analyses and production analytics. Michal Polanowski With Delhaize America since 2012. Focus on the test and control support for various business strategy, marketing, and category initiatives. Prior, taught at the University of Georgia where he received his PhD.
  3. 3. #inspire14 Disclaimer • The views and opinions expressed in this presentation are solely those of the authors of the presentation. These views and opinions do not necessarily represent those of Delhaize America, Delhaize Group or any of our subsidiaries. • This case study utilizes financial and geographical references for educational purposes only. The values and results presented are fictional. Any numerical, geographical, or program similarities to actual Delhaize America programs are coincidental.
  4. 4. #inspire14 Delhaize America • Part of Delhaize Group with food supermarkets in 6 countries • Delhaize America consists of three banners with over 1,500 food supermarkets
  5. 5. #inspire14 Consumer Insights • Our mission is to deliver and drive the voice of consumer throughout the organization by providing actionable insights • Shopper Insights delivers insights based on transaction and customer level data • We support the analytical needs of strategy, marketing, pricing, category management, merchandising, finance, and legal
  6. 6. #inspire14 Agenda • Why are we at this conference? • What is Test and Control? • Research Problem • Test and Control Methodology • Meeting with Business Owners • Identify Stores to Be Excluded • Match Test and Control • Perform the Test and Evaluate the Results • Present Findings to Business Owners
  7. 7. #inspire14 First comprehensive test and control environment in Alteryx We want to share our methodology with the rest of the industry Why we decided to speak at #inspire14?
  8. 8. #inspire14 Test and Control Overview Total Population Test and Control Match Not Suitable Stores • Estimates the effectiveness of an initiative/change • Matches stores/households in the pre period (before the initiative) • Analyzes the behavior after the initiative
  9. 9. #inspire14 -25% 0% 25% 04/24/12 05/22/12 06/19/12 07/17/12 08/28/12 Performance Store Performance Lift: 15%, Impact: $5,600 Test Stores Control Stores Lift is the gap between the test and control store performance Test and Control Overview Lift Pre Period Test Period
  10. 10. #inspire14 -25% 0% 25% 04/24/12 05/22/12 06/19/12 07/17/12 08/28/12 Performance Store Performance Lift: 15%, Impact: $5,600 Test Stores Control Stores Lift is the gap between the test and control store performance Test and Control Overview Lift Pre Period Test Period
  11. 11. #inspire14 Lift Results Can Be Tricky Test and Control Overview Test Store Performance Control Store Performance + Lift - Lift + Lift
  12. 12. #inspire14 Food Lion has a Sunday flyer in small local newspapers Research: Effectiveness of the Sunday Flyers • Food Lion wanted to test the effectiveness of the flyer • Currently we spend $1.5M/week on Sunday flyers • A test market representative of all Food Lion stores • The test market will not receive a Sunday flyer
  13. 13. #inspire14 Test and Control Process Meet with Business Owner Performed in Alteryx Identify Stores to be Excluded Match Test and Control Stores Perform the Test Meet with Business Owner
  14. 14. #inspire14 Understand the business reasons for the test Meeting with Business Owners • Define criteria for success and failure • Success = an impact of at least $1.4K per store per week ($1.5M total) • Align on dates, stores, products that are involved • All stores belong to the same region and flyer covers products in all departments • Identify blackout periods and other business initiatives that can potentially impact the test or control results • No blackout periods and no other Food Lion activity
  15. 15. #inspire14 Test and Control Process Meet with Business Owner Performed in Alteryx Identify Stores to be Excluded Match Test and Control Stores Perform the Test Meet with Business Owner
  16. 16. #inspire14 Numerous events can skew the results Identify Stores to be Excluded • Competitive events • Store openings, closings, remodels • Seasonal stores with high sales volatility • Natural events High Volatility Low Volatility
  17. 17. #inspire14 We automated the process Competitive Events and Store Volatility Module Volatility= St.Dev(Sales) Mean(Sales)
  18. 18. #inspire14 Review stores with high volatility and competitive activity Volatile Stores Should Be Removed Remove These Stores Food Lion Store # Event Date Estimated Impact Volatility 5043 Lowes Foods 8/7/13 ($47,500) 0.65 5323 0.44 5011 Publix 4/11/13 ($22,500) 0.43 5554 Kroger 6/14/13 ($75,000) 0.09 5832 Wal-Mart 7/26/13 ($1,000) 0.08 5934 0.04 5907 Road Work 6/15/13 ($1,000) 0.02 5123 0.01
  19. 19. #inspire14 Test and Control Process Meet with Business Owner Performed in Alteryx Identify Stores to be Excluded Match Test and Control Stores Perform the Test Meet with Business Owner
  20. 20. #inspire14 Match similar test and control stores Test and Control Stores Match • A good match controls any store fluctuations in the match variables • Match on a number of metrics • Sales, unit movement, margin, number of customers • We matched on total store sales and unit movement • Match on up to 6 metrics per test (recommended 2 – 3) • Match within specific clusters: • Region, urbanicity, strategy initiatives
  21. 21. #inspire14 Test and Control Match Module
  22. 22. #inspire14 The goal is to have test and control stores close to each other Evaluate Test and Control Match Poor Match Remove These Stores Good Match
  23. 23. Test and Control Process Meet with Business Owner Performed in Alteryx Identify Stores to be Excluded Match Test and Control Stores Perform the Test Meet with Business Owner
  24. 24. #inspire14 Test and Control Module
  25. 25. #inspire14 Test and Control Analysis Module You better know what you are doing !!!
  26. 26. #inspire14 Perform the Test and Evaluate the Results 5921 5122 5993 5984 5097 5789 5121 5432 5162 5427 Statistic Treatment Control Average 2.4% 3.9% Minimum -1.4% -13.2% Maximum 6.5% 16.2% Standard Deviation 2.2% 3.2% Some Control Stores May Need Additional Inspection Alteryx generates individual reports for each metric analyzed Sometimes, additional data investigation may be required
  27. 27. #inspire14 Test and Control Results Lift Expected Impact Significance Level -1.5% -$2.7K 99.9% Test t-Statistic Degrees of Freedom p-Value Control vs. Treatment 3.55 110.6 0.001 -25% 0% 25% 04/24/12 05/22/12 06/19/12 07/17/12 08/28/12 Performance Store Performance Lift: -1.5%, Impact: -$2.7K Test Control Pre Period Test Period
  28. 28. #inspire14 Test and Control Process Meet with Business Owner Performed in Alteryx Identify Stores to be Excluded Match Test and Control Stores Perform the Test Meet with Business Owner
  29. 29. #inspire14 Success Criteria: An impact of at least $1.4K per store per week Recommendation: Keep the Sunday flyer in rural and suburban areas; discontinue in urban areas Meeting with Business Owners Stores Metric Lift Expected Impact (Per Store Per Week) Significance Level Decision All Stores Total Store Sales -1.5% -$2.7K 99.9% Success Rural Total Store Sales -2.8% -$5.2K 100.0% Success Suburban Total Store Sales -0.9% -$1.4K 99.0% Success Urban Total Store Sales -0.3% -$0.5K 97.7% Fail
  30. 30. #inspire14 Questions
  31. 31. #inspire14 THANK YOU!

×