Moving Forward with Big Data: The Future of Retail Analytics


Published on

Out new report Moving Forward with Big Data: The Future of Retail Analytics goes deeper into new territory that's relevant to changes taking place across retailing.
It calls out significant progress in the past 9 months.
• The definition of big data has grown beyond technical, i.e. “what it is,” to include “what it does.”
• A lot more companies are executing big data projects (an increase from < 20% to now 65% of sample respondents).
• Most of the focus is on driving top line growth.

Published in: Business, Technology

Moving Forward with Big Data: The Future of Retail Analytics

  1. 1. Moving Forward with Big DataThe Future of Retail Analytics By Bill Bishop, Chief Architect, Brick Meets ClickOverview 1. Where’s the value?Where big data creates value for retailersThe most valuable data sources2. What’s the focus?Demand and supply side perspectives3. What’s happening today?Who’s doing whatThe status of big data projects4. What’s ahead for big data?Clarifying the definitionOvercoming barriers to progressA process for moving forwardBRICK MEETS CLICK is a strategic resource for retailers, suppliers, and technology providers who want to make insightful decisions about meetingshoppers needs. Visit us at BRICK MEETS CLICK ORIGINAL PAPER April 2013Readers of this paper are invited to comment usingthis link. Or, go to
  2. 2. © 2013 Brick Meets Click 2Moving Forward with Big Data: The Future of Retail AnalyticsOverviewWhen we surveyed retailing professionals on big data in 2012, fewer than 20% indicated that retailers were actively working on big data and “betterunderstanding of customers” was the most widely perceived benefit. By March 2013, 64% report they are participating in a project that involves Big Data,and the biggest focus is on how to use it to create competitive advantage.Clearly the relationship between big data and retail is evolving rapidly, but it’s also clear that many retailers still need to answer questions about how muchto invest in big data and where to focus efforts initially to produce the best returns.This survey explored:• Where and how big data is adding value for retailers on both the demand and supply sides of the business.• The type of big data projects that are underway, the stage of development or implementation they’ve reached, and whether a business case hasbeen established.• The definition of big data and how clarifying this can accelerate progress in using it.• Barriers that are limiting its application in retail.So, how can retailers use these findings to more quickly and confidently decide if big data can help them create competitive advantage? The answerdepends on where your particular organization is with big data. For retailers who haven’t yet made a decision in favor of big data, use these results to build awareness within your organization about howcompetitors are already working to measure the benefits and validate the business case for the strategic use of big data. For retailers who are committed but looking for a starting place, use these findings to provide guidance on which business problems hold thegreatest potential. For retailers who are already moving forward with big data projects, find a “second opinion” on the direction you’ve taken by learning moreabout what others in the business are doing.
  3. 3. © 2013 Brick Meets Click 3Moving Forward with Big Data: The Future of Retail AnalyticsABOUT THE SURVEY AND COMMENTSBrick Meets Click’s second big data survey was conducted during March 2013.115 retailing professionals responded. 50% identified themselves as consultants or market researchers, 20% as technology/information suppliers, 13% asretailers and wholesalers, 9% as product supplier/manufacturer, and 8% as other.Participants had the option to answer questions related to the demand side of the business or the supply side of the business, or both. 89% chose toanswer demand side questions; 60% chose to answer supply side questions; and 44% answered both sets of questions.Survey respondents had an opportunity to comment on the findings. In the text, these are set off by a blue background.Readers of this paper are invited to comment using this link. Or, go to
  4. 4. © 2013 Brick Meets Click 4Moving Forward with Big Data: The Future of Retail Analytics1. Where’s the value?The big data experts and leading retailers we’ve talked with in the past year frequently cite the following ways they expect big data to create value forretailers:• Analytical results will become more accurate because big data will make it possible to study the entire information set instead of sampling data. Dealingwith sampling errors will no longer be necessary.• Large-scale controlled experiments will become both possible and practical. This is the only way to establish with certainty what really drives aparticular improvement.• By creating a single source of data that the entire enterprise could access for analyses, big data could eliminate much of the friction, effort, andcoordination required to bring together data from different parts of the business.• Big data analytics will support faster, fact-based decision-making by shortening the time that elapses between the identification of a problem oropportunity and when management is able to take effective action.• It will strengthen analysis by incorporating sources of data that werepreviously impractical or impossible to integrate.“Supporting faster decisions” ranked #1While big data offers many ways for retailers to create value, the ability toexecute on fact-based decisions more quickly was ranked highest. The ability toinclude new sources of data in analyses ranked second. Using big data to createa single source of data for the enterprise ranked third. All other choices scored inthe single digits. (See Figure 1.)Figure 1 * multiple responses allowed
  5. 5. © 2013 Brick Meets Click 5Moving Forward with Big Data: The Future of Retail AnalyticsThe familiar is still favorite: Shopper transaction dataFive sources of data were identified as having high potential for improving retail performance (Figure 2).Shopper identified transaction data, selected by 52%, still gets the biggest vote of confidence from retail professionals. These transactions can be tieddirectly to a shopper or household via loyalty programs, credit or debit purchases, andother methods.Mobile device data that could be used for geo-tracking and geo-targeting is alsoseen as having big potential.In-store tracking has been around for some time, but its potential to create value forretailers is increased by new data collection methods and more powerful analytics.Shopper feedback can now be collected on a more continuous basis. Much oftoday’s focus is on translating it into information that can be used quickly andeffectively by management.The ranking of social media reflects the challenge many retailers still face inconnecting these digital breadcrumbs with existing marketing and merchandisingtactics.Figure 2 * multiple responses allowed
  6. 6. © 2013 Brick Meets Click 6Moving Forward with Big Data: The Future of Retail AnalyticsBMC PERSPECTIVE: Where’s the value?The recent advances in data storage and analysis will allow retailers to extract value from big data but, at the same time, this has created an almostoverwhelming number of never-before-available possibilities. It’s no wonder that there’s confusion about how to begin to capture this value.At the start of this study, we had a long list of ways big data can create value for retailers; we were interested to find out how others viewed this landscape.Here are the opportunities we see:1. DOING THE SAME THING BETTER AND FASTER. While the results show no consensus, the emphasis on supporting faster decisions is clearly themost popular option. This comes as no surprise since it’s easier and probably safer to try to do something better than to take an entirely newapproach.2. EXPLORING NEW DATA SETS. Conversations with big data practitioners (outside of retailing) reveal that they find significant value in incorporatingnew external sets of data that provide a richer context for the analysis. The survey shows that some retailing professionals see this value as well. Wehope this report will encourage exploration of which external data sets can be of most value to retailers.3. A SINGLE DATA LIBRARY. The idea of creating a single source of data was endorsed by some in the survey, but challenged by others as not alegitimate way for big data to create value. Conversations with those deeply involved in analysis, however, opened our eyes to some trulytransformational possibilities. Just think of the time and labor that would be saved if everything from the weekly sales reports to the revenuecomponent of quarterly financials could be extracted from the same database.From survey participantsCollecting and managing big data in real-time acts as the catalyst to reach shoppers at the moment of truth, and not just tool for planningcampaigns. A mobile platform will generate relevant data which can be used to further engage shoppers in a precise and personal way anddrive predictable purchases." -- Aaron Roberts, CEO, QThruAs a believer in “CRM Big Data”—the shopper name, address and SKU linked, made into a database and then analyzed to findapplications across the retail enterprise, it is heartening to find that the “big data” concept continues to become embedded as a decisiontool and not just the idea du jour. My takeaway: “big results” from “big data”. -- Francey Smith, Francey Smith & AssociatesIt’s noteworthy shopper feedback appears in the top five. It isn’t enough to just know what a shopper purchased (which might have been asubstitute for the out-of-stock item they really wanted), how they moved through the store or if they purchased a geo-targeted ad item.Shopper feedback connects all of these elements with the “experience” behind the purchase, which may be most revealing!-- Brian Numainville, Principal, The Retail Feedback Group
  7. 7. © 2013 Brick Meets Click 7Moving Forward with Big Data: The Future of Retail Analytics2. What’s the focus?Demand and supply side thinkingPeople working on the demand side of retail tend to think about big data opportunities differently than those who work on the supply side. For demand-siders, winning market share is the major goal. Supply-siders are responsible for retail operations, distribution, and supply chain relationships and oftenfocus on managing costs, productivity, and optimization.For retailers, the decision about which “side” of the business to start with will reflect the retailer’s strategy for growing the business. If the focus is on: Demand side, these survey findings are likely to encourage that retailer to concentrate on marketing more effectively to individual shoppers. Supply-side, then the results will direct them to strengthen their management of inventory and look for possible workforce applications.To capture both types of thinking, the survey offered supply side and demand side “tracks.” Participants had the option to answer either (or both) sets of questions.Eighty-nine percent of the survey participants chose to answer demand-side questions, 60% chose to answer supply side questions, and 44% answered both.On the demand side . . .One of the first questions retailers need to answer as they move deeper into big data on thedemand side of the business is what they want to achieve – what problem do they want tosolve? Once that’s been decided, the job of identifying data and analytics becomes mucheasier.. . . marketing to individuals more effectively is the focus.The idea of one-to-one marketing has been around for some time, but it’s difficult to implement.With 69% currently focusing on marketing to individuals, it would appear that retailers areseeing big data as the light at the end of the tunnel that will enable them to achievecustomization on a mass scale. (See Figure 3.)Figure 3
  8. 8. © 2013 Brick Meets Click 8Moving Forward with Big Data: The Future of Retail AnalyticsTurning to the specific opportunities to drive demand, we asked: Where will big data create the most value in the next two years?There are some strong contenders. (See Figure 4.)Given that strengthening shopper engagement is the highest rated opportunity at76%, expect retailers to interact even more actively with shoppers along the entire“path to purchase.” This will change marketing and merchandising practices.High interest in personalized promotions (70%) reflects continuing focus on movingaggressively toward one-to-one marketing tactics.The focus on shopper solutions (43%) opens additional opportunities forcollaboration with suppliers who can bring shopper insights to power development ofhighly relevant solutions.The power of implementing store specific assortments (38%) comes from betterensuring that shoppers will find the products they’re looking for and making sure thatthey’re in stock.Figure 4 *multiple responses allowed
  9. 9. © 2013 Brick Meets Click 9Moving Forward with Big Data: The Future of Retail AnalyticsBMC PERSPECTIVE: On the demand sideWith continuing slow economic growth, retailers’ top priority is understandably on driving demand.SHORT TERM. The clear direction from the survey is towards strengthening relationships with existing customers. There’s strong evidence that this isprobably the best way to drive incremental sales, but this is a short-term justification.LONG TERM. The more strategic reason for this focus is to put in place a clearly defined and manageable base for driving sustainable growth. Enhancedcustomer relationships are the basis for customer retention and managing the lifecycle value of each shopper. This is where retailers will capture long-lasting value from personalized promotions and stronger shopper engagement.From survey participantsPutting the emphasis on business value (what problem(s) does the retailer need to solve?) is the bestand most practical way to go about conquering big data. It ensures value, investment, and outcomesare aligned. If executed well, the results should be easy to quantify. Taking a “build it and they willcome” approach to big data is too risky from a time, opportunity cost, and financial standpoint.-- Jim Butera, FusionPointThe ability to quickly analyze transaction data down to the SKU level will open a gateway tounderstanding how consumers are shopping each product in the store and how those specific itemsare inter-related. Retailers that adapt this level of basket analytics will certainly have a competitiveadvantange in the marketplace by partnering with vendors/CPG to develop and create very uniquecustomer-centric promotions that will resonate with their targeted shopper segments.-- Victor Andedo, Principal, Linque Marketing Inc.
  10. 10. © 2013 Brick Meets Click 10Moving Forward with Big Data: The Future of Retail AnalyticsOn the supply side . . .. . . once again, POS is MVP.By far, POS transaction data was identified as the most valuable source of big data forthe supply side of retailing; 51% of respondents selected it. Three other sources wereidentified as potentially most valuable – sensors in the retail environment, shopperfeedback, and automated product recognition – though none even came close torivaling the confidence shown POS (Figure 5).POS has been a key source of data since the inception of checkout scanning and it’sclear that the growth of big data will not change this any time soon. POS data drivesreplenishment, maps labor requirements, and provides a unique operational signaturefor each store.The growing importance of sensors has been evident at recent trade shows wherethey were used for cold chain and energy management, as well as, workforcemanagement.Shopper feedback is used mainly to identify opportunities for improved employee engagement with customers.Automated product recognition, the technology at the core of tools for monitoring plan-o-gram compliance on the shelf, was identified by 8% eventhough it was unfamiliar to many.Figure 5
  11. 11. © 2013 Brick Meets Click 11Moving Forward with Big Data: The Future of Retail AnalyticsOptimizing inventory is a big opportunityAs retailers move to apply big data analytics to the supply side of the business,the question is, where will they find the greatest opportunities?Inventory is by far where respondents see big data has the most positive impacttoday. Two other opportunity areas identified – customer service managementand labor management – are closely related. (See Figure 6.)Attention on inventory management focused mainly on optimization rather thanreduction. Customer service management is becoming a more importantcompetitive dimension. Labor management -- both scheduling and salesconversion are both related to shopper engagement.Workforce management will changeLabor represents an important element in most brick and mortar retail businessmodels, so the response to this question showed a remarkable degree ofconfidence (Figure 7).Figure 6Figure 7
  12. 12. © 2013 Brick Meets Click 12Moving Forward with Big Data: The Future of Retail AnalyticsBMC PERSPECTIVE: On the supply side.Expect strong focus on supply side applications because these have the potential to both reduce costs and drive incremental sales. The good news is thatthese results show a range of ways to do that.SHORT TERM. While inventory management was identified as the most important in these results, it’s hard to imagine that there are not equallyimpressive ROI opportunities related to labor and customer service management. Solid decisions will leverage accurate measurements in both areas.LONG TERM. The strategic opportunity is to use big data to reconfigure the supply side of the business to deliver products and services cost-effectively ina multi-channel shopping environment. Already, retailers show signs of recognizing this need (online retailers are opening stores and brick and mortarretailers are leveraging digital) but the details of how to optimize inventory in this complex “mixed” environment are still being worked out. Retailers whosolve this problem will have a real competitive advantage.
  13. 13. © 2013 Brick Meets Click 13Moving Forward with Big Data: The Future of Retail Analytics3. What’s happening today?Who’s doing what?Almost two-thirds of those who answered the survey were actively involved in a projectusing big data (Figure 8). The projects addressed a broad range of problems, but somepatterns did appear.About half targeted a better understanding of shopper behavior and buying habits. Thesefell mainly in two areas: Developing more relevant and effective promotional offers. Creating more sharply focused behavior-based shopper segmentation.About 20% related to improving product availability: assortment optimization and “gettingthe right product to the right place,” and reducing out-of-stock.Four unique business problems being addressed stood out from the rest: Leveraging insight from social media to drive product display with specific markets in a way that intersects with natural events (like flu outbreaks). Maximizing customer retention while minimizing incentive spending. Monitoring risk in the environment. Analyzing the shopping behavior of millions of insurance customers.Figure 8
  14. 14. © 2013 Brick Meets Click 14Moving Forward with Big Data: The Future of Retail AnalyticsThe status of big data projectsMarketing is driving the bus.Marketing sponsored almost half of the projects reported in the survey (Figure 8).This was followed by merchandising, operations, and IT. In the “other” category,projects were sponsored by: Research and development Executive management Supply chain Sales/category managementThe horses are spreading out on the track.The status of big data projects spanned the project lifecycle, suggesting that somecompanies are pulling ahead of the pack.A little less than a quarter of the projects are in startup, while nearly half, 49%, areeither in pilot or in production (Figure 10). This indicates that while the practice ofbig data is still young, some retailers are already beginning to reap the benefits.The fact that 58% of those responding report that they have already been able tomeasure the benefits indicates that confidence is building in the value of thisapproach.Figure 9Figure 10
  15. 15. © 2013 Brick Meets Click 15Moving Forward with Big Data: The Future of Retail AnalyticsWhat’s Happening Today?Having business case for big data projects is no longer the exception, 65% of the projectshad established one (Figure 11). This shows that nearly two-thirds of those working withbig data now see how it will solve important business problems. These people are nolonger groping to figure out how to use a new tool.BMC PERSPECTIVE: The status of big data projects.The use of big data in retailing has gained significant traction in just a matter of months. While the practices are still in an early stage of development, it’sclear that many more retailers are making a real commitment to figure this out. This is encouraging, because in the larger scheme of things, this evolutionis not only sensible, but inevitable.STILL EARLY DAYS. Even this progress doesn’t obscure the fact that big data is still in the early stages of the adoption curve for business innovation. Alot more evidence will need to be generated about its value before a significant percentage of retailers adopt the practice – but we see this happening fast,probably a lot faster than most can imagine now.SOME RETAILERS ARE PULLING AHEAD. The key insight? Now is the time for retailers to begin figuring out how big data can benefit them over thelong term and to take the first steps in that direction, so they don’t fall too far behind.Figure 11
  16. 16. © 2013 Brick Meets Click 16Moving Forward with Big Data: The Future of Retail Analytics4. What’s ahead for Big Data?Clarifying the definitionWhen McKinsey published their big data report1in 2011, the definition they offered focused mainly on scale: “Data sets whose size is beyond the ability oftypical database tools to capture, store, manage, and analyze.” The emphasis on size has proved a distraction at times as retailing professionals grappledwith how to make practical use of big data. In BMC’s 2012 survey, participants quickly went beyond “what it is” to focus on “what it does” and how it couldbe used to improve sales performance.Don’t get distracted by the BIG in Big DataIn this 2013 survey, we wanted to explore whether extending and clarifying thedefinition could add value and make the term more useful; 94% of agreed that doingso would help to speed up the adoption and effective use of big data. (See Figure 12.)Almost 9 out of 10 indicated that a stronger working definition would make it easier forsenior management to see the connection between big data analysis and changesthat would drive improved performance. The focus on the large-scale nature of bigdata doesn’t do much to help bridge that gap.1Big Data: The Frontier for Innovation, Competition, and Productivity – McKinsey Global Institute 2011Figure 12 *Multiple responses so total exceeds 100%
  17. 17. © 2013 Brick Meets Click 17Moving Forward with Big Data: The Future of Retail AnalyticsNearly half said a better definition would make it easier to gain agreement on the scope of projects and help to build justification for projects. Betteragreement on scope would help in minimizing costs and coordinating project support across business functions.In “other” comments, participants said a stronger definition would: Help focus on the real value – speeding up innovation, fine-tuning customer focused offers, providing competitive context for analysis, andemphasizing important business questions Clarify ambiguity. The term big data feels too ambitious for many people. Said one participant: We need to replace it with more focused definitions . . . since often the data we are talking about is closer to “mid-sized” or even“small sized.”Favorite addition: New data combinationsWe asked those surveyed to start with a core definition – an approach to making thehandling of data at large scale “effective and affordable” – and then indicate how helpful itwould be to add each of three features to the definition.Nearly 70% chose “Analyzing data not previously combined.” Almost 40% wanted to includeopportunities to extract value from unstructured data, and about a quarter wanted to includethe opportunity to use data not previously collected or previously considered of low value.(See Figure 13.)These additions make it easier to talk about what big data does. When a retailer combinesshort-term temperature forecasts with their beverage replenishment system, they areanalyzing data not previously combined in ways that improve their results. The grocery retailer who is analyzing comments on social media to tailor menuand recipe recommendations is extracting value from unstructured data. And the major retailer who is using sensors to track shoppers in order to makeshort-term adjustments in labor at the front end is making use of data not previously collected to reduce customer wait time and increase labor productivity.Figure 13
  18. 18. © 2013 Brick Meets Click 18Moving Forward with Big Data: The Future of Retail AnalyticsBMC PERSPECTIVE: Clarifying the definition.With appreciation to the McKinsey Global Institute and those who developed the initial working definitions of big data, it’s now clear that more meaningfulprogress will require a next-generation definition – or set of definitions. More descriptive definitions will be easier to use in business settings, and benefit allstakeholders by encouraging more rapid adoption of big data practices.WHAT’S NEXT? The findings in this survey make it clear that it is time for the retail community to work together to hammer out this new generation ofdefinitions. The only caution would be to not leave this work to those whose self-interest would influence the results.From survey participantsI believe we are all just talking about input, because thats what data (big, medium, or small) is. It may bemore productive from a leadership perspective to talk of Testing (big, medium, or small - although I likethe idea of Big Testing), and focus on continually learning through tests that can now be fully analyzed andacted upon because of the masses of data that we can now gain access to and the speed at which it can becrunched.-- Craig Elston, SVP Insight and Strategy, The Integer Group
  19. 19. © 2013 Brick Meets Click 19Moving Forward with Big Data: The Future of Retail AnalyticsOvercoming barriers to progressSurvey participants were concerned that organizations weren’t making full use ofalready available data (Figure 14.). This comes either from not having enoughstaff to analyze the data or to make full use of the insights from existing data.They were also concerned that organizations don’t have the staff or infrastructureneeded to handle all the insights generated by big data and translate them toaction.The low concern for privacy and security was perhaps the only surprise among theother barriers identified (Figure 15). While shoppers are showing less concernabout privacy, the incidence of security breaches indicates that this is an area ofrisk warranting careful management.Figure 14 *multiple responses allowedFigure 15 *multiple responses allowed
  20. 20. © 2013 Brick Meets Click 20Moving Forward with Big Data: The Future of Retail AnalyticsBMC PERSPECTIVE: Overcoming barriers to progress.The main barriers to successfully using big data are not technical; they are related to organizational alignment and capability. This may sound like jargon,but it’s the truth and those participating in the survey know it.This raises a key question: What it will take for senior retail leaders to make big data and the organizational readiness to exploit it a key strategic priority?The increase in big data projects is encouraging, but by itself it doesn’t reveal much about the vision for and the magnitude of commitment to taking fulladvantage of this important new resource. Effective use of big data by itself won’t guarantee success in 21stcentury retailing, but it will be a key part ofthat success formula.From survey respondentsThe issue companies face is not the technology to amass the data but the analytical skill to understandthe insights from the data to take the proper marketing action. We must invest in people with thoseskill sets to be competitive. As an industry we have never really invested in staff to do the analyticalwork necessary to build the right targeted programs. Kroger clearly figured it out when they beganworking with Dunnhumby. Technology has moved quickly and at a lower cost than before. We need toharness the information to make businesses successful.-- Ann Raider, President and CEO, inStreamThe report shows a glimpse of the future. We will forget today’s clunky user-intensive big data andmobility platform interaction model one day. Background-processed real-time prescriptive analyticswill ultimately “get big data out of the way.” Retailers and consumers alike with engage trulyseamless Brick Meets Click shopping experiences. Supply chains, markets, and buyers will integratewithout regard to location. I can’t wait for this new future.-- Andrew John Stein, Pervasive Strategy Group
  21. 21. © 2013 Brick Meets Click 21Moving Forward with Big Data: The Future of Retail AnalyticsBMC PERSPECTIVE: A process for moving forward.As we reviewed the findings from our big data survey at Brick Meets Click, we also thought about the many conversations we’ve had with colleaguesduring the past year on this topic. It’s clearly going to take large-scale solutions to address some of the big issues identified – like developing theorganizational alignment and skill sets needed to take full advantage of big data’s potential. But, the fact that some big issues remain to be resolved hasn’tstopped people from moving forward to apply big data practices to solve retail problems and create value.5 SIMPLE STEPS. Here’s an approachable, grounded process that any retailer can use to move forward with big data. It unlocks the power of big data byfocusing on specific business challenges.1. Identify a key business challenge worth overcoming through the analysis and use of big data.2. Find a tool – an application, analysis, method, etc. – with the potential to address that business problem in a new and better way.3. Put in place a baseline measurement you can use to evaluate progress in overcoming the challenge.4. Focus first on incremental improvements to ensure you’re making real progress.5. Expand to bring in new external sets of data that will improve results.From survey participantsIt seems clear that senior management needs to adjust as failure to could lead to disaster.Just ask Blockbuster, Borders, and on the ropes retailers: Best Buy, JC Penney, RadioShack, Sears and Abercrombie & Fitch. All have in common a failure to adjust tochanging conditions. Perhaps this is the real promise of Big Data. Used correctly itoffers the ability to not only adjust but to lead the way to realizing the holy grail ofcreating a truly multi-channel shopping environment.-- Tom Van Aman, Marketing Strategy, Allstate