Your SlideShare is downloading. ×
The Case for Big Data and Analytics in Retail: Unlocking Hidden Opportunities, by IDC Retail Insights
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

The Case for Big Data and Analytics in Retail: Unlocking Hidden Opportunities, by IDC Retail Insights


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Big Data and Analytics in Retail: Unlocking Hidden Opportunities WHITE PAPER Sponsored by: HP G re g G ir ar d S ep tem b er 2 01 IDC RETAIL INSIGHTS OPINION The new information economy — increasingly digital and intelligent — defines the terms of reference for retail today, and within this milieu, mastering Big Data has become a cornerstone for success.F.508.988.7881 Mastering the four dimensions of Big Data — social, channel/customer, market, and supplier/product — for actionable and monetized insight requires newfound technical, organizational, process, and decision management frameworks to handle three challenging Vs of Big Data and analytics — volume, velocity, andP.508.935.4400 variety — in order to deliver the other V — value. Value from Big Data and analytics can come from three sources — gaining insight to improve processes and resource allocation, personalizing and localizing offers, and creating community as the nexusGlobal Headquarters: 5 Speen Street Framingham, MA 01701 USA for branding and customer engagement. Extracting value economically from Big Data requires a new generation of technologies and architectures — shared information ecosystems designed to enable the high-velocity capture, discovery, analysis, and application of insights. Several analytical underpinnings of value creation are clearly evident, including context-aware and pattern-based analytics, quantitative natural language processing and social network analysis, and the extension of retail data models beyond conventional enterprise entities and ontologies. Applying these insights to control and optimize operations, tactics, and strategies may require new business processes, decision management frameworks, and roles and organizational constructs. Opportunities to apply Big Data analytical insight reach deep into digital or social marketing or customer insight — the organizations where these efforts often originate — but just as important and likely more important, these opportunities extend into brand management and marketing, merchandising, product management, localization, pricing, fulfillment, and all channels of commerce. What do the dimensions of "big," "fast," and "varied" mean with respect to the opportunities to create value from Big Data and analytics in your business? Accurately evaluating competitive and market scenarios that impact your Big Data and analytics business case calculus requires the strategic and tactical points of view of line-of-business leadership. Dont September 2012, IDC Retail Insights #GRI237012
  • 2. let Big Data and analytics become the bright shiny new object of yourattention. Forgetting that value from Big Data and analytics doesntmaterialize just because you have invested in these capabilities will leadto too many "dry wells" filled with misspent time, talent, and treasure.SITUATION OVERVIEWThe New Information Environment of RetailAs the digital world continues to emerge, information has become thesingular environment within which retailers earn customer loyalty,bring successful new products to market, collaborate through supplychains with business partners, enable associates, reduce risk, ensurecompliance, and above all, burnish their brand. At least thats thepotential — if only retailers master Big Data.As more Big Data and potential insight amass within and beyond theenterprise, traditional approaches to acquiring and analyzing relevant dataand applying insight gained from interrogating the data no longer suffice.Barriers to capitalizing on this potential include the technical capabilitiesof conventional information technologies as well as the processes,decision management frameworks, and organizational constructs bywhich organizations govern data, develop information from the data, andapply insight to operational, tactical, and strategic decisions.By some estimates, up to 90% of the worlds information lies hidden informs and formats beyond the competencies of the informationmanagement technologies commonly used in retail. Retailers ownsome of this hidden information within their four walls or a cloud —some of it unstructured (e.g., in product descriptions, customer emails,contact center notes, promotional agreements) and some of itstructured (Excel spreadsheets being the nearly ubiquitous example).More and more information hidden in Big Data, however, lies beyonda retailers information management assets — residing in social data,customer data, market data, and supplier data.Big Data in the Social Context of RetailThe impact of Big Data — its promise amid peril — is easiest to see inhow it plays out in the inherently social context of retail. A growingand diverse number of consumers now interconnect and influenceone another with their use of social networking and media —YouTube, Facebook, Twitter, LinkedIn, Google+, blogs, wikis, andratings and reviews. The speed, scale, and viral aspects of social mediaand networking influences, whether they are facts, opinions, ormisinformation, dwarf old modalities of "word of mouth" and "powerof the pen" pressures on shaping consumers attitudes and opinionsabout brands as well as consumers buying patterns and propensities.Page 2 #GRI237012 ©2012 IDC Retail Insights
  • 3. Conventional ways by which retailers seek to understand consumersentiments and the sources of influence on those sentiments fall shortof the mark when their customers engage one another through socialmedia and networks. Traditional information management approacheswill no longer suffice to address these emerging demands and theresulting complexity.The disparity between the speed and strength of consumer sentimentsspreading across social networks and the limits of what retailers haveto track these sentiments puts retailers at risk of losing control of theirbrand, the loyalty of their best customers, and market share, revenue,and margins.The imperative retailers face is deciding how they should readythemselves to understand, react, and ideally engage their consumers inthis brave new world. The key lies in mastering new competencies inBig Data and analytics to successfully and automatically integrate theirinsight into the various retail business processes.Latent Value: The First V in Big DataSocial data is one of four dimensions of Big Data in retail. Customerdata sources, market data sources, and supply data sources round outthe IDC Retail Insights typology of Big Data in retail (see Figure 1).Insights hidden in the Big Data of each dimension can improve decisionsinherent in that dimension; for example, insight from loyalty and shopperprofile data can improve customer segmentation and promotionaltargeting, and in another example, real-time stock (distribution center[DC], in-transit, store) data can improve next-day or same-dayreplenishment. Increasingly, however, new insight can be drawn bymashing up data across domains. For example, contextual analysis ofcontact center call logs, tweets, ratings and reviews, lot tracking ofshipment records, product design information, and loyalty data takentogether could spot product defects from customer feedback, identifyroot causes from product design information, identify stores holding anydefective merchandise, and find loyal customers who purchased the item.In more general terms, Big Data and analytics hold the potential todramatically increase three imperatives of omnichannel retail:● Insight applied in commerce (online and in-store); marketing (brand management, advertising, and promotions); fulfillment (from private label management to customer orders); and merchandising (from curating core assortments to localizing prices and promotions)● Personalization applied in private promotions and offers, dynamic configuration of online assortments, and digital content presentation based on knowing the customer at any touch point and anticipating his or her needs©2012 IDC Retail Insights #GRI237012 Page 3
  • 4. ● Community applied in physical or digital social venues to harness the influence of social networks and media to create personal engagement with the brandFIGURE 1Retail Big Data Typology • Twitter • Facebook • Games • Shutterfly • Pinterest • Google+ • YouTube • LinkedIn • Blogs • Wikis • Foursquare • Shopkick • Yelp • Citysearch Social Data Sources • Market baskets • Shopping carts • Loyalty and profiles • Offer/response • Purchase orders • Shopping list Customer Data Sources Enterprise Systems of Record • Supply Data Sources Shipments • Click stream • Returns • SMS • RFID and sensors Billions of Interactions • Associates black books • DC stock • Search terms and patterns • Store stock • Ratings and reviews • Receipts Millions of • Downloads • DC issues Transactions • Geospatial • Trade promotions • QR scans • Product information • App usage • Design specifications • Web chat • Market intelligence • Contact center • Compliance • Sensor Market Data Sources • Survey and focus group • Trade • Email • Syndicated • Demographic • Geospatial • Events • Venues • Business • Economic • WeatherSource: IDC Retail Insights, 2012Volume, Velocity, Variety: The Other Vs ofBig Data OpportunityRealizing the opportunities from analyzing Big Data in individualdomains and mashing up Big Data across domains, in other words,harvesting its latent value, is a function of an organizations ability toPage 4 #GRI237012 ©2012 IDC Retail Insights
  • 5. master the inherent volume, variety, and velocity of the types ofBig Data shown in Figure 1:● Data volumes growing from terabytes to petabytes, exabytes, or more. IDC Retail Insights estimates a base of 9.6PB held in social media today with 10.2PB being added in the year ahead. We also estimate — conservatively — that 19.8PB of created content drives 14,000PB to 18,000PB of content consumed.● Data varieties extending in formats from structured to semistructured and unstructured and in sources from core transactional enterprise systems — ones own or those of trading partners — through which retailers operate and control their businesses; to collaborative and communications systems (e.g., email, enterprise wikis, and contact center logs); and to external informational systems — some structured (e.g., demographic data) but the vast volume unstructured (e.g., tweets or posts to a companys Facebook page).● Data velocities and their business process and decision management frameworks changing from those amenably serviced by batch processes to those stubbornly dependent on the streaming of data acquisition, analysis, and application of insight.FUTURE OUTLOOKMarket and financial imperatives present the strategic context of anurgent and critical need to master the "supply" and "apply" sides ofBig Data and analytics in retail. Merchandising and marketing presentan excellent example of this need in the context of the ongoing shift ofpower to the consumer. The forefront of insights in these two domainshas long since marched past the private knowledge domains of themerchant prince who controlled product risk and is now marching pastthe enterprise knowledge domain of spreadsheet merchants as theycontrol market risk to the Big Data knowledge domain of the socialmerchant controlling customer risk. Todays retailer is dependent onmastering all three knowledge domains.Supply Side and Apply Side of Big Data andAnalyticsMastery of Big Datas four Vs is a function of two capabilities —supply and apply. Both dimensions require agility in the solutions thatsupply Big Data and the processes that apply its insights. Fullrealization of the value hidden in Big Data in the strategic context ofreal-time omnichannel retail wherein the consumer holds the balanceof power requires new consumption models — from real-timecontextual decision management frameworks to mobile presentation ofvisually rendered analytics in an enterprise context and mobilepersonalized content in the context of instrumented, informed, andinterconnected customers.©2012 IDC Retail Insights #GRI237012 Page 5
  • 6. Supply Side of Extracting Value from the Volume, Variety,and Velocity of Big DataExtracting value economically from Big Data requires a newgeneration of technologies and architectures designed to enable thehigh-velocity capture, discovery, analysis, and application of insights.These IT assets encompass the hardware and the software thatintegrate, organize, manage, analyze, and present data that ischaracterized by the four Vs of Big Data opportunity. This set ofcapabilities is the "supply side" of Big Data competence.Apply Side of Extracting Value from the Volume, Variety,and Velocity of Big DataA second set of capabilities — of equal importance — addresses the"apply side" of Big Data competence — the business processes,decision management frameworks, and roles and organizationalconstructs by which an organization consumes Big Data insight tocontrol and optimize its operations, tactics, and strategies to achieve itsbusiness objectives.Shared Information EcosystemsThe value of Big Data and analytics cannot be fully realized unless itsinsights are applied in the retail enterprise ecosystem context of otherbusiness intelligence systems, in particular, in their service ofmarketing and positioning, merchandising and product development,supply and fulfillment, and omnichannel customer engagementthrough store and ecommerce channels. The contextualization of BigData insight in the ecosystem of enterprise systems of processes andsystems of record includes:● Aligning analytics with business concerns (e.g., product quality, brand position, customer relationships, product development, marketing campaigns)● Managing the life cycles of Big Data — defining, harvesting, organizing, analyzing, storing, reporting, and disposing of the data at the clock speed of the business processes in which it is consumed● Reporting and applying insights in process decision management frameworks (e.g., roles; decision rights; operational, tactical, and strategic time frames and spans of control; and plan, do, analyze cycle times)Page 6 #GRI237012 ©2012 IDC Retail Insights
  • 7. Retail Big Data and Analytics MaturityFigure 2 presents the five dimensions of Big Data and analytics in thecontext of retail:● Intent — context of strategic, tactical, and operational decision management● Process — relevancy of Big Data and analytics to roles, responsibilities, and outcomes● People — alignment of organization and cultural norms with the use of Big Data and analytics● Data — governance and life-cycle management of voluminous, varied, and high-velocity data● Technology — priority focus on hardware, software, and services for economical supply of relevant Big Data and analyticsFIGURE 2Retail Big Data and Analytics Maturity Model Clear view of business and market trends relating to the power of Big Data analytics f or insight, personalization, and community f ormation; the competitive risks and rewards related to the use of Big Data analytics; and commitment to inf using the omnichannel enterprise with the capabilities of Big Data analytics Intent Organizational alignment to collaborate Big Data insight made relevant and applied in with Big Data analytics, def ine roles and processes that span departments and f unctions training f or skills to apply the insights of People Process with clarity of roles, responsibilities, decision Big Data analytics, and align Big Data management f rameworks, and dependencies f or compensation to create a Big Data continuous plan, do, analyze cycles analytics retail culture Maturity Data Technology IT priority f ocus on technologies and Governance and MDM of structured and architectures designed to extract value unstructured Big Data f or consistent enterprise economically f rom very large volumes of a and external data f rameworks f or analytical wide variety of data by enabling high-velocity support of business processes capture, discovery, analysis, and application of insights to decision makingSource: IDC Retail Insights, 2012©2012 IDC Retail Insights #GRI237012 Page 7
  • 8. Value Creation Opportunities fromBig Data and Analytics in RetailUltimately, in the business context of retail, value creation from BigData and analytics should result in the monetization of hidden insightrevealed and applied. Commercial paths to monetization can lead toproduct and service innovation and with it differentiation, premiumpricing, and customer loyalty; process control — compliance, productquality, security, and fraud prevention; and optimization of processesand decisions — cost control, margin growth, and efficiencies.Analytical Underpinnings of Value CreationThe range of new analytical underpinnings for monetizing Big Datainsight includes the following:● Context-aware analytics: Integrating Big Data insight with enterprise transactional data and business intelligence (e.g., predictive analytics, forecasting, and root cause analysis) such that the situational awareness of Big Data analytics complements and supports human judgment● Broad-scope pattern-based analytics: Proactively seeking episodic, recurrent, and systemic patterns or systems of factors within process and organizational domains ("silos") and supersystems of factors spanning silos from conventional and unconventional sources that can positively or negatively impact outcomes● Quantitative natural language processing analytics: Discovery and disambiguation of meaning primarily in digital communications (e.g., emails, tweets, blogs, and Facebook posts) and content repositories (e.g., product descriptions in product data management system or customer ratings and reviews)● Extension of retail data models beyond conventional enterprise entities and ontologies (e.g., foundational notions of products, customers, regions, stores, promotions, channels) to include friends, followers, sentiments, and interests as extensions of customer attributes; "likes," "wants," and reviews as extensions of product attributes; and forums and topics at the nexus of people, places, and productsOpportunities for Applying Big Data and Analytics in theLine of BusinessOpportunities to monetize Big Data and analytics span core retailprocesses including brand management and marketing; merchandising;fulfillment; and commerce. Examples and use cases of Big Data andanalytics for each of these processes are shown in Table 1.Page 8 #GRI237012 ©2012 IDC Retail Insights
  • 9. TABLE 1 Opportunities to Monetize Big Data and Analytics in Retail Process Process Responsibilities Sample Applications and Use Cases Brand  Brand differentiation  Identification and characterization of threats to brand values management (e.g., boycotts or campaigns reacting to unmet corporate social and marketing  Support and execution of brand responsibility commitments) management differentiation strategy via development and  Intelligence about strengths and weaknesses of competitors by execution of media campaigns customer segments, merchandise categories, product quality, pricing, locations, and other categories of strategic importance  Customer analytics, insights, and segmentation  Intelligence about customer sentiment trends by customer segments, merchandise categories, product quality, pricing,  Creation of promotional locations, and other categories of strategic importance campaigns to shape customer demand in collaboration with  Identification of networks, nodes, and personalities of viral marketing influence  Creation of promotional  Assessment of the impact of marketing campaigns and events campaigns to shape customer on consumer awareness, sentiments, behaviors, and intentions demand in collaboration with merchandising Merchandising  Selection, curation, localization,  Intelligence about new product development, design, and allocation, distribution, and introduction — aspects that delight customers, meet their pricing of merchandise in all expectations, or are of low value commerce channels  Intelligence about local demand for products and services not  Creation of promotional carried in local assortment campaigns to shape customer demand in collaboration with marketing Fulfillment  Sourcing, developing,  Early warning of product defects and shortfalls of performance designing, and delivering of and attributes against customer expectations merchandise into commerce channels in collaboration with  Intelligence about new product development, design, and merchandising introduction — aspects that delight customers, meet their expectations, or are of low value  Fulfilling customer orders from own network, suppliers, and marketplace Commerce  Presenting and selling  Store- and market-specific customer concerns and delights merchandise and services regarding store operations, associates engagement practices across all channels and product knowledge, customer service, crowds, wait times, and so forth  Operating all channels of trade, including stores, catalog, call  Real-time in-store, near-store customer tweets for customer centers, and digital — mobile, service, product information, and product location ecommerce, social, and third- party channels  Customer concerns and delights about performance and characteristics of ecommerce, mobile, social, and other digital channels Source: IDC Retail Insights, 2012©2012 IDC Retail Insights #GRI237012 Page 9
  • 10. CHALLENGESWhat do the dimensions of "big," "fast," and "varied" mean withrespect to the opportunities to create value from Big Data and analyticsin your business? The answers will define the terms of reference foryour Big Data and analytics value creation journey. Gaining clarityand consensus on these definitions will require you to addresstechnical, cultural, and organizational challenges. Left unattended,these challenges will delay and distort identification and definition ofgaps in the five dimensions of your Big Data and analytics maturityprofile — intent, process, people, data, and technology.Ignoring the dictum that "better can be the enemy of good enough" canderail your Big Data business case calculus. Accurately evaluatingcompetitive and market scenarios that impact your Big Data andanalytics business case calculus requires the strategic and tacticalpoints of view of line-of-business leadership. Against that input, theability to define investment tripwires — investment priorities,dependencies, and time lines for scenarios most likely to come soonerand create more risk — and using them to sequence and scheduleinvestments under likely scenarios requires the strategic and tacticalpoints of view of information technology leadership.Becoming mesmerized by the allure of Big Data and analytics as abright shiny new object of your attention presents another challenge tobe managed. Opportunities to create value from Big Data and analyticsprojects dont materialize just because you have invested in theseprojects. Forgetting that will lead to too many "dry wells" filled withlost time, talent, and treasure.OVERVIEW OF HP BIG DATA ANDANALYTICS OFFERINGSHP offers a wide range of information management and analytics(IM&A) capabilities drawn from its enterprise services, softwareproducts, and cloud and security platforms to address Big Datachallenges and opportunities in retail.HP IM&A services include information strategy and organization,information management and architecture, business analytics andinformation delivery, and social intelligence. HP provides theseconsulting services around its own Big Data software assets,Autonomy and Vertica, and third-party software assets, in particularSAP HANA and Microsoft SharePoint/BI platform. HP supports thiswith a variety of software and solution delivery models including on-premise installation, hosted, software as a service (SaaS), cloudcomputing, and multitenant SaaS (cloud deployment).Page 10 #GRI237012 ©2012 IDC Retail Insights
  • 11. HP focuses its capabilities to enable retailers to proactively manageinformation-related business risk, enhance customer experiences, andoptimize business performance to create competitive advantage.HPs Autonomy software asset helps retailers develop connectedintelligence from structured and unstructured data for actionabledecisions that improve business performance.HPs Vertica analytics database delivers scalable performance on BigData queries enabling real-time decision making to be embedded inretailer processes in order to optimize business performance.Vertica and Autonomy deliver a powerful combination for real-timeanalytics and decision making using structured and unstructured dataacross the enterprise.Strengths and ChallengesHP offers a product and services portfolio that is consistent with whatIDC Retail Insights expects from a market-leading technologyprovider to the retail industry. In addition to the software and servicescapabilities noted previously, HP provides the following infrastructurecomponent to enable Big Data application deployments:● Cloud computing. HP can offer clients a variety of enterprise- grade cloud computing solutions, including public, hybrid, and private. The ability to offer private cloud to retailers is considered a strength for a number of reasons, not the least of which is a tighter security model.● Security. HP offers clients a security strategy through the HP Security Framework, designed to offer end-to-end information security plans and execution road maps. Because of the sensitive nature of customer data and the requirements and penalties imposed by the regulators, security is top of mind for industry IT executives.● Mobility. HPs approach to enabling enterprise mobility is suited for organizations that wish to reach their constituents across multiple networks and devices by delivering applications, content, and services in a scalable, secure, and reliable way. This approach leverages HPs global applications services capabilities to provide the architecture, systems engineering, development, and support services. Combined, they help an organization simplify its applications and extend them where necessary as well as build mobile business-to-business, business-to-consumer, and business- to-employee applications. This approach also leverages HPs service-oriented architecture–based integration architecture and is enabled by development and security frameworks that help create componentized building blocks from monolithic legacy applications to develop and deploy mobile applications.©2012 IDC Retail Insights #GRI237012 Page 11
  • 12. HP faces several unique market challenges as well as many of thesame market challenges as other enterprise vendors servicing the BigData marketplace:● Demonstrating leadership in the transition from information management to Big Data. HPs legacy information management solutions, including TRIM, Data Protect, and others, are not broadly implemented across the retail marketplace. HP must more effectively demonstrate leading capabilities at the component level with high-profile Autonomy and Vertica wins as well as highlight its ability to deliver end-to-end Big Data software, services, and hardware with lighthouse and marquee clients.● Offering competitive, differentiated solutions portfolios. The retail market, as a traditional early adopter of emerging technologies, senses the value of the Big Data trend and is budgeting accordingly. HPs competitors are also focused on expanding their solution portfolios on the Big Data trend in terms of breadth and depth of product capabilities, IT infrastructure, and professional services. HP must leverage its industry position and demonstrate the value of a Big Data relationship with HP to gain Big Data market share.● Channel network management. HPs channel strength is also a weakness. Successful execution of HPs Big Data strategy to broaden its portfolio will require that HP reinforce its position in the enterprise space without alienating the channel that has been so beneficial to the organization.ESSENTIAL GUIDANCE● Adopt an enterprisewide approach to Big Data analytics. The design, development, and deployment of Big Data analytical capabilities should be seen from the outset as an enterprisewide undertaking even as nascent initiatives of the sort incubate within marketing, in particular digital marketing, where Big Data analytics are applied to social Big Data, or in ecommerce and personalization where Big Data analytics are applied to customer Big Data. The design objectives of an enterprisewide approach to Big Data analytics should include: ○ Program governance led by a leadership team drawn from across four core omnichannel processes: marketing, commerce, merchandising, and fulfillment ○ Alignment of insights from the four quadrants of Big Data sources — social, customer, market, and supply — with the four dimensions of retail intelligence — customer intelligence, offer intelligence, channel intelligence, and merchandise intelligencePage 12 #GRI237012 ©2012 IDC Retail Insights
  • 13. ○ Delivering insights within the decision management context of the roles they inform — that is, task aligned and just in time — and the style in which and the speed at which the decisions are made● Look for monetization opportunities across the board. Big Data analytics can inform decisions beyond marketing in merchandising, commerce, and fulfillment. Actionable insights include marketing campaigns; product design and quality; new product development and introduction; assortment localization; pricing, promotions, and personalization; crisis management, selling techniques, and store associate management; store operations and design; and competitive intelligence and differentiation.● Maintain a readiness for rapid evolution of Big Data analytics. Big Data analytics is a new area; new capabilities for and opportunities to apply Big Data analytics are emerging quickly. Managing rapid change on the supply and apply sides of Big Data analytics requires agility in a host of technical competencies (e.g., data management and governance, commodity hardware configurations); organizational dimensions of skills, roles, and decision rights; consumption models and use cases; and more. Retailers will increasingly need to find sources of innovation and should look to Big Data information partners. Retailers should screen for partners that provide best-in-class solutions consisting of open source technologies/capabilities and proprietary technologies/capabilities, as needed, that are best suited for their unique requirements.Copyright NoticeCopyright 2012 IDC Retail Insights. Reproduction without writtenpermission is completely forbidden. External Publication of IDCRetail Insights Information and Data: Any IDC Retail Insightsinformation that is to be used in advertising, press releases, orpromotional materials requires prior written approval from theappropriate IDC Retail Insights Vice President. A draft of theproposed document should accompany any such request. IDC RetailInsights reserves the right to deny approval of external usage for anyreason.This document was reprinted by HP with permission from IDC RetailInsights.©2012 IDC Retail Insights #GRI237012 Page 13