Breakout: Data Discovery with Hadoop
•
3 likes•1,512 views
Siloed data is difficult to access and causes data consumers to only have partial views of the problem at hand. By limiting access to large volumes of disparate data, analysts and business users alike don’t have the ability to included important data in their reports and models leading to suboptimal analytic outputs. Even when this data is available to countless users, traditional systems limit them to querying small volumes of data in order to return the results in a timely matter.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Breakout: Data Discovery with Hadoop
Similar to Breakout: Data Discovery with Hadoop (20)
More from Cloudera, Inc.
More from Cloudera, Inc. (20)
Recently uploaded
Recently uploaded (20)
Breakout: Data Discovery with Hadoop
- 1. 1© Cloudera, Inc. All rights reserved. Better Analytics in Less Time Enabling Data Discovery & Analytics with Cloudera
- 2. 2© Cloudera, Inc. All rights reserved. Trends in the Market Only 12% of enterprise data is leveraged for analytics. Source: Forrester Research Analytics returns $13.01 for every dollar invested. Source: Nucleus Research 80% of CEOs said analytics is a strategic objective for 2015. Source: PWC CEO Survey Too Much Data Early Returns Strategic Direction
- 3. 3© Cloudera, Inc. All rights reserved. Trends in the Market Data Discovery is the #1 fastest growing workload for enterprise analytics.
- 4. 4© Cloudera, Inc. All rights reserved. Data Discovery & Analytics (DD&A) Overview The ability to find enterprise data and quickly uncover new insights and optimize existing analytics. (AKA: Self-service BI, BI, Data Discovery, Advance Analytics, Machine Learning)
- 5. 5© Cloudera, Inc. All rights reserved. Data Discovery & Analytics (DD&A) Overview Report and dashboard building for business user consumption (Tableau, Oracle) Statistical and machine learning analyses using advanced analytic tools (SAS, Matlab) Command line access for custom analytic development (Spark, Python) Self-Service BI Advanced Analytics Data Science
- 6. 6© Cloudera, Inc. All rights reserved. Objectives of Data Discovery & Analytics Report Model Rule Marketing analysis Log analysis Churn analysis Product recommendation Predictive support Trade recommendation Ad targeting Transaction classification Lead scoring
- 7. 7© Cloudera, Inc. All rights reserved. It’s an Iterative Process Report, Model, or Rules Ingest Transformation 80% of Time Diverse Ingest Search and lineage Agile Transforms 20% of Time SQL Statistical Machine Learning Implement Point Solution Custom App Analysis Technique Access Data Generation Data Discovery & Analytics Flow
- 8. 8© Cloudera, Inc. All rights reserved. A Traditional Architecture Access Data Experiment FastAnalyze Data Enterprise Data Warehouse ImplementData Sources ETLStructured Unstructured Ingest Storage #1, 2, N ELT Store & Process Traditional Architecture EDW Archive ETL Access Data Analyze Data Search Serve Serve Serve Optimize Implement Custom Application Point Solution ELT ELT Statistical Machine Learning SQL
- 9. 9© Cloudera, Inc. All rights reserved. Enterprise Data Warehouse ImplementData Sources ETLStructured Unstructured Ingest Storage #1, 2, N ELT Store & Process Traditional Architecture EDW Archive ETL Access Data Analyze Data Search Serve Serve Serve Optimize Implement Custom Application Point Solution ELT ELT Statistical Machine Learning SQL Challenges with Traditional Architectures 1) Limited Data 2) Long Time to Value 1 2 2 3) Compliance & Privacy Concerns 3
- 10. 10© Cloudera, Inc. All rights reserved. The New Way Forward 1) Unlimited Data Access 2) Reduce Time to Value 3) Secure and Compliant Enterprise Data Warehouse ImplementData Sources ELT Store & Process Modern Architecture Access Data Analyze Data Optimize Implement Custom Application Point Solution Statistical Machine Learning SQL Structured Unstructured EDW ETL Serve ETL Active Ingest Ingest EDH Archive Load Cloudera ELT 3 2 2 3 1 Search
- 11. 11© Cloudera, Inc. All rights reserved. Concur Customer Story
- 12. 12© Cloudera, Inc. All rights reserved. About Concur What do we do - Global customer base of 20,000 clients and 25 million users Processing more than $50 Billion in Travel & Expense (T&E) spend each year Who is Concur - Leading provider of spend management solutions and (Travel, Invoice, TripIt, etc.) services in the world
- 13. 13© Cloudera, Inc. All rights reserved. Our Objectives Deeper BI Reports Starbucks Store #3313 601 108th Ave NE Bellevue, WA (425) 646-9602 ------------------------------- Chk 713452 05/14/2014 11:04 AM 1961558 Drawer: 1 Reg: 1 ------------------------------- Bacon Art Brkfst 3.45 Warmed T1 Latte 2.70 Triple 1.50 Soy 0.60 Gr Vanilla Mac 4.15 Reload Card 50.00 AMEX $50.00 XXXXXXXXXXXXXXXXXX1004 SBUX Card $13.56 SUBTOTAL $62.40 New Caffe Espresso Frappuccino(R) Blended beverage Our Signature Frappuccino(R) roast coffee and fresh milk, blended with ice. Topped with our new espresso whipped cream and new Italian roast drizzle Better Classifications Smarter Predictions
- 14. 14© Cloudera, Inc. All rights reserved. The Pre-Hadoop Environment Travel Invoice Expense 1 Difficult to connect to multiple sources 1 BI Challenges
- 15. 15© Cloudera, Inc. All rights reserved. The Pre-Hadoop Environment Travel Invoice Expense 1 2 Difficult to connect to multiple sources Couldn’t analyze new data sources BTS Google Analytics Weather 1 2 BI Challenges
- 16. 16© Cloudera, Inc. All rights reserved. The Pre-Hadoop Environment Travel Invoice Expense 1 2 3 Difficult to connect to multiple sources Couldn’t analyze new data sources Challenges BTS Google Analytics Weather Advanced analytics was a dream 1 2 BI Advanced Analytics 3
- 17. 17© Cloudera, Inc. All rights reserved. Concur’s Modern Hadoop Architecture Complete data access (structured and unstructured) 1 Improvements New Data Pig Process 1 1 Store HDFS
- 18. 18© Cloudera, Inc. All rights reserved. Concur’s Modern Hadoop Architecture Complete data access (structured and unstructured) 1 Large scale ad-hoc queries2 Pig Process 1 1 Store HDFS HUE Discover Impala Solr Access BI 2 HiveImprovements New Data
- 19. 19© Cloudera, Inc. All rights reserved. Concur’s Modern Hadoop Architecture Complete data access (structured and unstructured) Quick Iterations for advanced analytics 3 1 Large scale ad-hoc queries2 Pig Process HUE Model R MLlib 1 1 Discover Impala Solr Spark Access BI Store HDFS 2 3 HiveImprovements New Data
- 20. 20© Cloudera, Inc. All rights reserved. The Modern Analyst Flow Find Data Prepare Data Analyze Data
- 21. 21© Cloudera, Inc. All rights reserved. Business and Technical ROI Technology ROI Business ROI Customer 360 view leading to better experience Insight into customer travel, expenses, and T&E Single platform for emerging advanced use cases Merge together storage systems for simpler management Unified access to disparate data Scale affordably
- 22. 22© Cloudera, Inc. All rights reserved. Key Leanings Crawl, walk, run It takes time, start now Lean on experts in the community It is a cultural shift as much as a technology shift
- 23. 23© Cloudera, Inc. All rights reserved. Thank you
Editor's Notes
- Key takeaway: It is not just a BI or analytics challenge, it is the way that data is managed. Keeping 3 main high level objectives of an architecture built for Data Discovery in mind- accessing data, analyzing data, and experimenting and iterating fast- we can examine a traditional architecture and see where organizations might run into issues. Questions for customer: Does this look like your architecture?
- Limited Data Access Data siloes Archived or deleted data No unstructured data Only SQL Long Time to Value Resource intensive ad-hoc ELT Adding dimensions takes months Slow large scale queries Compliance & Privacy Concerns Data copied Managing users across systems Protecting data without limiting users
- Key takeaway: An EDH provides the foundation to change the way you collect and manage data in order to provide your analyst what they need in less time. ETL on the fly: Talk to schema-on-write vs schema-on-read (http://www.slideshare.net/awadallah/schemaonread-vs-schemaonwrite). 1) Unlimited Data Access (Active archive, Scalable storage, Unstructured data) 2) Reduce Time to Value (ETL on the fly, Parallel processing, Complete data access) 3) Secure and Compliant (Compliant-ready, Tool Integration, Data lineage)
- Pulling from the “Insights Section”
- Why Hadoop slide content: Even with primarily relational systems, it involved hundreds of sources Getting a BI tool to connect to so many sources is … not fun More times than not, we needed to understand a subset or aggregate of this data - not all of the data! Can use Pig to process, extract, filter the data Can use Hive - a SQL like query language - to query my data
- Why Hadoop slide content: Even with primarily relational systems, it involved hundreds of sources Getting any BI tool to connect to so many sources is … not fun More times than not, we needed to understand a subset or aggregate of this data - not all of the data! Can use Pig to process, extract, filter the data Can use Hive - a SQL like query language - to query my data
- Why Hadoop slide content: Even with primarily relational systems, it involved hundreds of sources Getting any BI tool to connect to so many sources is … not fun More times than not, we needed to understand a subset or aggregate of this data - not all of the data! Can use Pig to process, extract, filter the data Can use Hive - a SQL like query language - to query my data