Big Data Business Wins: Real-time Inventory Tracking with Hadoop


Published on

Published in: Technology, Business

Big Data Business Wins: Real-time Inventory Tracking with Hadoop

  1. 1. MetaScale is a subsidiary of Sears Holdings Corporation MetaScale is a subsidiary of Sears Holdings Corporation Ankur Gupta General Manager Big Data Business Wins: Real-Time Inventory Tracking with Hadoop
  2. 2. MetaScale A big data technology solution provider and subsidiary of Sears Holdings that delivers a full spectrum of services focused on big data and Hadoop MetaScale is a ‘big data accelerator’. We provide the technology, talent, and solutions to accelerate the value from big data Offices in Chicago, San Jose, and Pune, India A Fortune 100 company, nearly $40 billion in annual revenue The nation’s fourth largest broad line retailer with almost 2,500 full- line and specialty retail stores in the US and Canada A front runner in big data efforts including driving personalized marketing and generating savings from legacy migration Running one of the biggest rewards programs that captures and analyzes very large number of customer transactions quickly Our Parent Company Sears Holdings 2
  3. 3.  What tools can be used to migrate Point-of-Sales (POS) data from different legacy systems to Hadoop  Establishing an Enterprise Data Hub with Hadoop in order to create a single version of truth  What is a reference architecture for near real-time inventory tracking 3 Objectives
  4. 4. From a recent Wikibon survey:  Enterprise practitioners believe the potential value of Big Data is significant  However, many are struggling to derive maximum value from their big data investments • 46% of Big Data practitioners report that they have only realized partial value from their Big Data deployments • 2% declared their Big Data deployments total failures, with no value achieved Challenge of Achieving Big Data ROI Source: Enterprises Struggling to Derive Maximum Value from Big Data, Wikibon, Sep 2013 4
  5. 5. According to Wikibon, three compelling reasons for this struggle to achieve maximum business value from big data… 1. A lack of skilled Big Data practitioners 2. "Raw" and relatively immature technology 3. A lack of compelling business use case Challenge of Achieving Big Data ROI Source: Enterprises Struggling to Derive Maximum Value from Big Data, Wikibon, Sep 2013 5
  6. 6. Making Business Decisions Quickly 6  The Hadoop ecosystem gives business the ability to create value from its data by being able to process and store vast amounts of data from disparate sources.  Hadoop enables faster processing on larger data sets for analytics and deep analytics.  Storm, Kafka and Cassandra provide the technology for real-time analytics to make business agile.
  7. 7. Keys for Achieving Big Data Success 7  Bring IT and Business together  Define realistic success criteria  Ask “what are you really trying to accomplish?”  Understand how Hadoop will fit into your environment  See the end results first before you start your journey  Discover your big data use case!
  8. 8. Real-Time Inventory Management 8
  9. 9. Real-Time Analytics with Cassandra By implementing Hadoop and Cassandra into a traditional environment, Business Intelligence teams are able to provide more accurate and real-time inventory, pricing, sales and return data as well as predicting ideal floor plans. Managing inventory with up-to-the-second data... 9 In-Store Purchases Online Purchases Real-time inventory data ensures that items ordered are in-stock.
  10. 10.  POS data was stored in different formats in different legacy systems (Mainframe and Teradata)  No single version of truth  No real-time capability Inventory Batch File Sent ONCE A DAY CHALLENGE This latency resulted in potential loss of sales and customer dissatisfaction when items are ordered that are no longer in stock. 10 Real-Time Analytics with Cassandra POS Volume  Average 100,000 message per day  Peak 77,000 messages in 1 hour at 4:00am the day after Thanksgiving
  11. 11. SOLUTION – Phase 1  Condense all POS data from different legacy systems and applications into Hadoop Enterprise Data Hub  Create a Single Version of Truth 11 Real-Time Analytics with Cassandra Hadoop enables a single version of truth for deep analytics, but there is still no real-time capability…
  12. 12. SOLUTION – Phase 2 12 Real-Time Analytics with Cassandra  Use Kafka to extract messages from POS queue  Kafka sends messages to Cassandra for real-time processing
  13. 13. SOLUTION – End-to-End Messages are sent from Cassandra to Hadoop for back-end, deep analytics. 13 Real-Time Analytics with Cassandra 4 Node 4 Node 11 Node
  14. 14. Faster decision making… Business Intelligence Teams are able to provide more accurate and real-time inventory, pricing, sales and return data. BEFORE Cassandra Real-Time Solution: Inventory Batch File Sent Once a Day Real-Time Analytics with Cassandra AFTER Cassandra Real-Time Solution: Inventory Data Sent in Sub-Milliseconds 14 RESULT
  15. 15. Increased sales by improving item availability. Real-Time Analytics with Cassandra 15 Value for the Organization Increased customer satisfaction because customer is able to get what was ordered.
  16. 16. Real-Time Analytics with Cassandra 16 Value for the Organization Cost savings from reduced customer service center calls. Aha Moments Cost savings from reduced truck load times.
  17. 17. Additional Components 17
  18. 18. Hadoop Enterprise Data Hub gives business users access to more data from more sources for deep analytics. Hadoop Enterprise Data Hub 18 Single Version of Truth
  19. 19. Firewall Issues Normally, Storm or Kafka can be used to send POS messages to Cassandra. In certain situations where a firewall exists between data source and processing cluster - such as created by mergers or spin-outs – both Storm and Kafka can be used to send messages over the firewall. 19 Unique Challenge for a Complex Enterprise
  20. 20. Real-Time Over Firewall 20 Unique Challenge for a Complex Enterprise 3 Node Storm Cluster
  21. 21. Advanced Analytics 21 Inventory forecasting with Machine Learning on data from Weather Reports Data-Driven Decision Making Once the Hadoop / Cassandra framework is in place, data from virtually any source can be consumed in the Enterprise Data Hub for Advanced Analytics. New ways to use Social, Geo, Sensor data to develop predictive models…
  22. 22. KEY TAKEAWAYS 22
  23. 23.  Enterprise Data Hub and single version of truth for all data  Hadoop can help you answer questions that were difficult or cost prohibitive to answer before  Hadoop can transform your organization’s approach to how you use data and ask questions you never even thought of  Must have a clear strategy and long-term plan  Leverage the right partnerships to achieve your goals 23 Big Data Business Wins
  24. 24. Q & A Questions? 24
  25. 25. Your One-Stop Big Data Helpline phone: email: visit: 1-800-234-8769