SAP Big Data Forum 2013 T2 Chris Harris - Hortonworks


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SAP Big Data Forum 2013 T2 Chris Harris - Hortonworks

  1. 1. Apache Hadoop's Role in Your Big Data Architecture Chris Harris EMEA, Hortonworks Twitter : cj_harris5 © Hortonworks Inc. 2012 Page 1
  2. 2. Agenda • The Growth of Enterprise Data • Hadoop Market Drivers • Hortonworks – an Overview • The Future of Hadoop and Big Data © Hortonworks Inc. 2013 Page 2
  3. 3. The Growth of Data in the Enterprise 15x Data Explosion 1 Zettabyte (ZB) = 1 Billion TBs growth rate of machine generated data by 2020 Source: IDC By 2015, organizations that build a modern information management system will outperform their peers financially by 20 percent. – Gartner, Mark Beyer, “Information Management in the 21st Century” © Hortonworks Inc. 2013 Page 3
  4. 4. Next Generation Data Architecture Drivers Business Drivers • From reactive analytics to proactive customer interaction • Find insights for competitive advantage & optimal returns Technical Drivers • Data continues to grow exponentially • Data is increasingly everywhere and in many formats Financial Drivers • Cost of data systems, as % of IT spend, continues to grow • Cost advantages of commodity hardware & open source © Hortonworks Inc. 2013
  5. 5. Innovators, technology enthusiasts Early adopters, visionaries The CHASM relative % customers Market Transitioning into Early Majority Early majority, pragmatists Late majority, conservatives Laggards, Skeptics time Customers want technology & performance Customers want solutions & convenience Source: Geoffrey Moore - Crossing the Chasm © Hortonworks Inc. 2013 Page 5
  6. 6. 6 Key Hadoop DATA TYPES 1. Sentiment Understand how your customers feel about your brand and products – right now 2. Clickstream Capture and analyze website visitors’ data trails and optimize your website 3. Sensor/Machine Discover patterns in data streaming automatically from remote sensors and machines 4. Geographic Value Analyze location-based data to manage operations where they occur 5. Server Logs Research logs to diagnose process failures and prevent security breaches 6. Text Understand patterns in text across millions of web pages, emails, and documents © Hortonworks Inc. 2013 Page 6
  7. 7. Apache Hadoop Enterprise Use Cases (1 of 2) Vertical Trading Risk Server Logs Maximize Deposit Spread Text, Server Logs Geographic, Sensor, Text Accelerate Loan Processing Text Call Detail Records (CDRs) Machine, Geographic Infrastructure Investment Machine, Server Logs Next Product to Buy (NPTB) Clickstream Real-time Bandwidth Allocation Server Logs, Text, Sentiment New Product Development Machine, Geographic 360° View of the Customer Clickstream, Text Analyze Brand Sentiment Sentiment Localized, Personalized Promotions Geographic Website Optimization Clickstream Optimal Store Layout © Hortonworks Inc. 2013 Server Logs Insurance Underwriting Retail Text, Server Logs Fraud Prevention Telecom Data Type New Account Risk Screens Financial Services Use Case Sensor Page 7
  8. 8. Apache Hadoop Enterprise Use Cases (2 of 2) Vertical Assembly Line Quality Assurance Sensor Proactive Maintenance Machine Sentiment Genomic Sequencing Structured Real-time Data for Blood Sampling Sensor, Server Logs Rapid, Mobile Detection of Autism Unstructured Reducing Cost of Cancer Treatment Sensor, Unstructured Perpetual Storage of Research Data © Hortonworks Inc. 2013 Sensor Crowdsourced Quality Assurance Healthcare Data Type Supply Chain and Logistics Manufacturing Use Case Sensor, Unstructured Page 8
  9. 9. New Account Risk Screens Financial Services Data: Text, Server Logs Business Problem • Banks take thousands of new account applications daily • Text-based 3rd party risk reports displayed to banker • Bankers can (and do) override risk recs to open account • Account charge-offs and fraud costs banks millions Solution • HDP helps senior managers control new account risk • Match banker decisions with multiple sources of information they use to make those decisions • Correct risky behavior by sanctioning individuals, updating policies, improving training or identifying fraud. © Hortonworks Inc. 2012 Page 9
  10. 10. Fraud Prevention Financial Services Data: Server Logs Business Problem • Financial institutions are always at risk of fraud • Fraudsters test bank systems for vulnerabilities • This testing leaves subtle patterns often undetected by bank employees or law enforcement • Fraud losses costs banks millions Solution • HDP reduces the cost to detect fraudulent activity • HDP stores more types of data for longer • Analysis of data in the “data lake” exposes fraudulent patterns that would have gone undetected © Hortonworks Inc. 2012 Page 10
  11. 11. Call Detail Records (CDRs) Telecom Data: Machine, Geo Business Problem • Telcos perform forensics on dropped calls and sound quality • Call detail records flow in at a rate of millions per second • High volume makes pattern recognition and root cause analysis difficult, which need to happen in real-time • Delay causes attrition and harms servicing margins Solution • HDP can ingest millions of CDRs per second • HDP facilitates data retention and root cause analysis • Continuously improve call quality, customer satisfaction and servicing margins © Hortonworks Inc. 2012 Page 11
  12. 12. Infrastructure Investment Telecom Data: Machine, Logs Business Problem • Telecom marketing and capacity planning are coordinated • Consumption of bandwidth and services can be out of sync with plans for new towers and transmission lines • Mismatch between infrastructure investments and the actual return on investment puts revenue at risk Solution • HDP helps telcos understand service consumption in a particular state, county or neighborhood • Analyze Call Detail Records (CDRs) and network loads, more intelligently, over longer periods of time • Plan infrastructure with more precision and less variability © Hortonworks Inc. 2012 Page 12
  13. 13. 360° View of the Customer Retail Data: Clickstream, Text Business Problem • Retailers interact with customers across multiple channels • Customer interaction and purchase data is often siloed • Few retailers can correlate customer purchases with marketing campaigns and online browsing behavior • Merging data in relational databases is expensive Solution • HDP gives retailers a 360° view of customer behavior • Store data longer & track phases of the customer lifecycle • Gain competitive advantage: increase sales, reduce supply chain expenses and retain the best customers © Hortonworks Inc. 2012 Page 13
  14. 14. Analyze Brand Sentiment Retail Data: Sentiment Business Problem • Enterprises lack a reliable way to track their brand health • It is difficult to analyze how advertising, competitor moves, product launches or news stories affect the brand • Internal brand studies can be slow, expensive and flawed Solution • HDP allows quick, unbiased brand sentiment snapshots • Analyze sentiment from Twitter, Facebook, LinkedIn or industryspecific social media streams • Retailers better understand customer perceptions, to align their communications, products and promotions with those perceptions and expectations © Hortonworks Inc. 2012 Page 14
  15. 15. Supply Chain and Logistics Manufacturing Data: Sensor Business Problem • Manufacturers need just-in-time availability of components • Stock-outs cause harmful production delays • Sensors and RFID tags reduce the cost of capturing more supply chain data, which needs storage and processing Solution • HDP stores unstructured, streaming, “dirty” sensor data • Manufacturers get lead time to make alternative arrangements for supply chain disruptions • Prevent stock-outs, reduce supply chain costs and improve margins for the finished product © Hortonworks Inc. 2012 Page 15
  16. 16. Assembly Line Quality Assurance Manufacturing Data: Sensor Business Problem • High-tech manufacturing uses sensors to capture data at critical steps in the manufacturing process • Sensor data helps diagnose errors with returned products • Much data is discarded, because of high storage costs • Lean margins mean small budgets for data analysis Solution • HDP stores unstructured, streaming, “dirty” sensor data • Manufacturers can proactively analyze more data, over a longer time, to detect subtle issues otherwise undetected • Sensor data managed with HDP can help a manufacturer reduce warranty costs and earn a reputation for quality © Hortonworks Inc. 2012 Page 16
  17. 17. APPLICATIONS Growth Pressures Existing Data Architectures Packaged Analytic App Custom Analytic App DEV & DATA TOOLS DATA SOURCES DATA SYSTEMS BUILD & TEST OPERATIONAL TOOLS RDBMS EDW TRADITIONAL REPOS MP P Traditional Sources OLTP,(RDBMS, POS SYSTEMS OLTP, OLAP) © Hortonworks Inc. 2013 MANAGE & MONITOR Data growth 8% annually Page 17
  18. 18. APPLICATIONS An Emerging Data Architecture Packaged Analytic App Custom Analytic App DEV & DATA TOOLS DATA SOURCES DATA SYSTEMS BUILD & TEST RDBMS EDW TRADITIONAL REPOS MP P Traditional Sources OLTP,(RDBMS, POS SYSTEMS OLTP, OLAP) © Hortonworks Inc. 2013 ENTERPRISE HADOOP PLATFORM New Sources (web logs, email, sensors, social media) OPERATIONAL TOOLS MANAGE & MONITOR Data growth 85% annually Page 18
  19. 19. An Emerging Data Architecture © Hortonworks Inc. 2013 Page 19
  20. 20. Agenda • The Growth of Enterprise Data • Hadoop Market Drivers • An Overview • The Future of Hadoop and Big Data © Hortonworks Inc. 2013 Page 20
  21. 21. A Brief History of Apache Hadoop Apache Project Established Yahoo! begins to Operate at scale Hortonworks Data Platform 2013 2004 2006 2008 2010 2012 Enterprise Hadoop 2005: Yahoo! creates team under E14 to work on Hadoop 2011: Hortonworks created to focus on “Enterprise Hadoop“ © Hortonworks Inc. 2013 Page 21
  22. 22. Leadership Starts at the Core • Driving next generation Hadoop – YARN, MapReduce2, HDFS2, High Availability, Disaster Recovery • 420k+ lines authored since 2006 – More than twice nearest contributor • Deeply integrating w/ecosystem – Enabling new deployment platforms – (ex. Windows & Azure, Linux & VMware HA) – Creating deeply engineered solutions – (ex. Teradata big data appliance) • All Apache, NO holdbacks – 100% of code contributed to Apache © Hortonworks Inc. 2013 Page 22
  23. 23. Agenda • The Growth of Enterprise Data • Hadoop Market Drivers • Hortonworks – an Overview • The Future of Hadoop and Big Data © Hortonworks Inc. 2013 Page 23
  24. 24. The 1st Generation of Hadoop: Batch HADOOP 1.0 Built for Web-Scale Batch Apps Single App Single App INTERACTIVE ONLINE Single App Single App Single App BATCH BATCH BATCH HDFS HDFS HDFS © Hortonworks Inc. 2013 • All other usage patterns must leverage that same infrastructure • Forces the creation of silos for managing mixed workloads
  25. 25. The Enterprise Requirement: Beyond Batch To become an enterprise viable data platform, customers have told us they want to store ALL DATA in one place and interact with it in MULTIPLE WAYS – Simultaneously & with predictable levels of service BATCH INTERACTIVE ONLINE STREAMING GRAPH IN-MEMORY HPC MPI SEARCH HDFS (Redundant, Reliable Storage) © Hortonworks Inc. 2013 Page 25
  26. 26. YARN: Taking Hadoop Beyond Batch • Created to manage resource needs across all uses • Ensures predictable performance & QoS for all apps • Enables apps to run “IN” Hadoop rather than “ON” – Key to leveraging all other common services of the Hadoop platform: security, data lifecycle management, etc. Applications Run Natively IN Hadoop BATCH INTERACTIVE (MapReduce) (Tez) ONLINE (HBase) STREAMING (Storm, S4,…) GRAPH (Giraph) IN-MEMORY (Spark) HPC MPI (OpenMPI) OTHER (Search) (Weave…) YARN (Cluster Resource Management) HDFS2 (Redundant, Reliable Storage) © Hortonworks Inc. 2013 Page 26
  27. 27. The Future of the Hadoop and Big Data • The next generation data architecture evolving rapidly – Store ALL data in a Hadoop data reservoir – Push subsets of data to a final platform for processing • Hadoop 2.0 takes Hadoop beyond “Batch” – 2.0 YARN based architecture enabling mixed use workloads with enterprise resource management • Enabling a new generation of applications at scale – Based on new data types (sensor, sentiment, clickstream, etc.) or keeping existing types for much longer © Hortonworks Inc. 2013
  28. 28. Hortonworks Sandbox Hands on tutorials integrated into Sandbox © Hortonworks Inc. 2013 HDP environment for evaluation Page 28
  29. 29. THANK YOU! Chris Harris Download Sandbox © Hortonworks Inc. 2013 Page 29