6 enriching your data warehouse with big data and hadoop

2,442 views

Published on

Published in: Technology, Business

6 enriching your data warehouse with big data and hadoop

  1. 1. Enriching Your Data Warehouse With Hadoop Presenting with Peter Yu, Senior Director iTech Asia-Pacific & Japan
  2. 2. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.2 Agenda  Opportunity  Challenges  Strategy  Examples  Best Practices
  3. 3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3 Connecting With Your Customer
  4. 4. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4 Big Data Improves Operational Performance Source: Economist Intelligence Unit, .”The Deciding Factor: Big Data and Decision Making“ Big data benefits seen growing substantially
  5. 5. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5 Use Data 12% Executives who feel they understand the impact data will have on their organizations Produce Data The Problem
  6. 6. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6
  7. 7. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7 DON’T BELIEVE EVERYTHING YOU READ
  8. 8. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.8 Electricity: AC or DC?
  9. 9. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9 RDBMSRDBMS Today Discovery & Analytics Business Intelligence
  10. 10. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10 RDBMSRDBMS Today Discovery & Analytics Business Intelligence External ETL Cluster
  11. 11. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.11 RDBMSRDBMS What about Archived Data? Discovery & Analytics Business Intelligence Archive External ETL Cluster
  12. 12. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.12 RDBMSRDBMS What about New Data? Discovery & Analytics Business Intelligence ? Archive External ETL Cluster
  13. 13. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.13 RDBMSRDBMS Expand Your Data Warehouse Discovery & Analytics Business Intelligence ? External ETL Cluster Archive
  14. 14. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.14 RDBMS Hadoop RDBMS Hadoop Integrate Hadoop and RDBMS Discovery & Analytics Business Intelligence
  15. 15. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.15 “[Facebook] started in the Hadoop world. We are now bringing in relational to enhance that. We're kind of going [in] the other direction ... We've been there, and [we] realized that using the wrong technology for certain kinds of problems can be difficult.” Ken Rubin Director of Analytics Facebook http://tdwi.org/Articles/2013/05/06/Facebooks-Relational-Platform.aspx?Page=1
  16. 16. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.16 Get Fast Answers to New Questions Create a Data Reservoir Predict More, More Accurately Accelerate Data-Driven Action Key Hadoop Use Cases Complementing An Existing Data Warehouse ETL
  17. 17. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.17 • Transactional data • Customer information • Web log and session data • Machine/Sensor data • Historical data Data Reservoir Keep All Potentially Valuable Data in One Place
  18. 18. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.18 RDBMSRDBMS Today Discovery & Analytics Business Intelligence ? Archive External ETL Cluster
  19. 19. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.19 RDBMS Hadoop RDBMS Hadoop Create an Active Archive with Hadoop Discovery & Analytics Business Intelligence Σ
  20. 20. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.20  80 years of historical data in Hadoop  Structured and unstructured data includes customer data, economic trends, telematic sensors, weather, public data  Integrated with mainframes and EDWs  Before Hadoop, could analyze only one state, took 24 hours  With Cloudera, can analyze risk across all 50 states, in 16 hours (500x improvement)  First 3 use cases: Data hub, ETL offload, advanced analytics Comprehensive risk analysis Customer Example: Insurer Real-Time Data Hub Cloudera Hadoop EDW and Mainframe Customer Data
  21. 21. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.21  Optimize offers  Increase revenue  Lower costs (offer bytes?)  Reduce complexity  Faster time to value Maximize offer effectiveness Customer Example: Travel Industry Big Data Appliance Legacy Data Warehouse Customer Data
  22. 22. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.22  Fast access to 85% more data  Lowered costs  Simplified architecture  Faster time to value Compliance, cost reduction Customer Example: Regional Bank Big Data Appliance Oracle Exadata Mainframe, RDBMS Oracle Data Integrator
  23. 23. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.23 • Batch window constraints • Adding value vs. adding cost • Analysis vs. Transformation • Analysis vs. Data movement and replication • Uncertain value of new data sources ETL Challenges Today
  24. 24. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.24 Discovery & Analytics Business Intelligence RDBMSRDBMS Typical ETL Today External ETL Cluster
  25. 25. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.25 RDBMS Hadoop RDBMS Hadoop ETL Offload with Hadoop Discovery & Analytics Business Intelligence Σ
  26. 26. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.26  Exponential growth in data, generated by new consumer devices  ETL and storage constraints limited analytics to 1% sample  Now combined Oracle Exadata and Cloudera Hadoop delivers analytics on 100% of data (half a PB per day!)  Query times reduced dramatically (i.e. from 4 days to 53 minutes)  90% reduction of ETL code base From 1% sampling to 100% analysis Customer Example: Communications Services Archive Storage Data Warehouse Complex Correlation Alerting Filter & Split Event Monitoring Streaming ETL Streaming ETL Teleco m Services Before
  27. 27. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.27  Exponential growth in data, generated by new consumer devices  ETL and storage constraints limited analytics to 1% sample  Now combined Oracle Exadata and Cloudera Hadoop delivers analytics on 100% of data (half a PB per day!)  Query times reduced dramatically (i.e. from 4 days to 53 minutes)  90% reduction of ETL code base From 1% sampling to 100% analysis Customer Example: Communications Services Archive Storage Data Warehouse Complex Correlation Alerting Filter & Split Event Monitoring Streaming ETL Streaming ETL Teleco m Services Before Data Warehouse Alerting Filter & Split Event Monitoring Hadoop Archive Storage ETL Correlation Stage 1 DWH Teleco m Services After
  28. 28. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.28 Identifying Potential Opportunities Do you have a problem with ETL performance? Do you have potentially valuable data that you aren’t using, but might deliver new insight? Should you focus on analyzing structured data, unstructured data, or a combination? Are Big Data solutions already being built as silos?
  29. 29. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.29 Challenges in the Way & Oracle Strategies CHALLENGES ORACLE STRATEGY • Fragmented Solutions • Specialized but integrated data stores and tools • Difficulty of Self-Service BI • Flexible, guided, automated BI & data discovery • Data Not Current • Solutions for Just-in-Time well defined data • Time to ROI / Development Time • Horizontal & industry pre-built solutions, engineered systems • Growing Diversity of Data & Users • Enterprise solutions for 1000s diverse users, petabytes data • Manageability, Security, Cost • Centrally managed with advanced security / governance
  30. 30. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.30 Big Data Readiness The R&D Prototype Stage  Skills needed – Distributed data deployment (e.g. Hadoop) – Python or Java programming with MapReduce – Statistical analysis (e.g. R) – Data integration – Ability to formulate business hypotheses – Ability to convey business value of Big Data
  31. 31. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.31 Are You Ready for Big Data? Contact Your Account Team  Have you delivered mature analytics solutions for structured data?  Can Big Data make a difference to the business?  Have you built a Big Data prototype, built skills, and proved value?  Do you have an enterprise integration & deployment strategy for Big Data?
  32. 32. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.32 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.32 Continuous Innovation Big Data at Work
  33. 33. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.33
  34. 34. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.34
  35. 35. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.35 RDBMS Hadoop RDBMS Hadoop Master diagram Discovery & Analytics Business IntelligenceExternal ETL Cluster Data Mart Data Mart Archive ΣΣEvent Processing
  36. 36. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.36 Discovery & Analytics Business Intelligence RDBMSRDBMS Master diagram without Hadoop External ETL Cluster Data Mart Data Mart Archive ΣEvent Processing

×