James Serra, profile picture
Uploaded byJames Serra
PPTX, PDF55,627 views

Big data architectures and the data lake

The document provides an overview of big data architectures and the data lake concept. It discusses why organizations are adopting data lakes to handle increasing data volumes and varieties. The key aspects covered include: - Defining top-down and bottom-up approaches to data management - Explaining what a data lake is and how Hadoop can function as the data lake - Describing how a modern data warehouse combines features of a traditional data warehouse and data lake - Discussing how federated querying allows data to be accessed across multiple sources - Highlighting benefits of implementing big data solutions in the cloud - Comparing shared-nothing, massively parallel processing (MPP) architectures to symmetric multi-processing (

Embed presentation

Downloaded 5,466 times
Big data architectures and the data lake James Serra Big Data Evangelist Microsoft JamesSerra3@gmail.com
About Me  Microsoft, Big Data Evangelist  In IT for 30 years, worked on many BI and DW projects  Worked as desktop/web/database developer, DBA, BI and DW architect and developer, MDM architect, PDW/APS developer  Been perm employee, contractor, consultant, business owner  Presenter at PASS Business Analytics Conference, PASS Summit, Enterprise Data World conference  Certifications: MCSE: Data Platform, Business Intelligence; MS: Architecting Microsoft Azure Solutions, Design and Implement Big Data Analytics Solutions, Design and Implement Cloud Data Platform Solutions  Blog at JamesSerra.com  Former SQL Server MVP  Author of book “Reporting with Microsoft SQL Server 2012”
Agenda  Big Data Architectures  Why data lakes?  Top-down vs Bottom-up  Data lake defined  Hadoop as the data lake  Modern Data Warehouse  Federated Querying  Solution in the cloud  SMP vs MPP
? ? ? ? Big Data Architectures
Enterprise data warehouse augmentation • Seen when EDW has been in existence a while and EDW can’t handle new data • Cons: not offloading EDW work, can’t use existing tools, data hub difficulty understanding data
Data hub plus EDW (temporary) • Data hub is used as temporary staging and refining, no reporting • Cons: data hub is temporary, no reporting/analyzing done with the data hub
All-in-one • Data hub is total solution, no EDW • Cons: queries are slower, new training for reporting tools, difficulty understanding data, security limitations
Modern Data Warehouse • Evolution of three previous scenarios • Ultimate goal • Supports future data needs • Data harmonized and analyzed in the data lake or moved to EDW for more quality and performance
? ? ? ? Why data lakes?
ETL pipeline Dedicated ETL tools (e.g. SSIS) Defined schema Queries Results Relational LOB Applications Traditional business analytics process 1. Start with end-user requirements to identify desired reports and analysis 2. Define corresponding database schema and queries 3. Identify the required data sources 4. Create a Extract-Transform-Load (ETL) pipeline to extract required data (curation) and transform it to target schema (‘schema-on-write’) 5. Create reports. Analyze data All data not immediately required is discarded or archived 10
Harness the growing and changing nature of data Need to collect any data StreamingStructured Challenge is combining transactional data stored in relational databases with less structured data Big Data = All Data Get the right information to the right people at the right time in the right format Unstructured “ ”
The three V’s
Store indefinitely Analyze See results Gather data from all sources Iterate New big data thinking: All data has value All data has potential value Data hoarding No defined schema—stored in native format Schema is imposed and transformations are done at query time (schema-on-read). Apps and users interpret the data as they see fit 13
? ? ? ? Top-down vs Bottom-up
Observation Pattern Theory Hypothesis What will happen? How can we make it happen? Predictive Analytics Prescriptive Analytics What happened? Why did it happen? Descriptive Analytics Diagnostic Analytics Confirmation Theory Hypothesis Observation Two Approaches to Information Management for Analytics: Top-Down + Bottoms-Up
Implement Data Warehouse Physical Design ETL Development Reporting & Analytics Development Install and Tune Reporting & Analytics Design Dimension Modelling ETL Design Setup Infrastructure Understand Corporate Strategy Data Warehousing Uses A Top-Down Approach Data sources Gather Requirements Business Requirements Technical Requirements
The “data lake” Uses A Bottoms-Up Approach Ingest all data regardless of requirements Store all data in native format without schema definition Do analysis Using analytic engines like Hadoop Interactive queries Batch queries Machine Learning Data warehouse Real-time analytics Devices
Data Lake + Data Warehouse Better Together Data sources What happened? Descriptive Analytics Diagnostic Analytics Why did it happen? What will happen? Predictive Analytics Prescriptive Analytics How can we make it happen?
? ? ? ? Data lake defined
What is a data lake? A storage repository, usually Hadoop, that holds a vast amount of raw data in its native format until it is needed. • A place to store unlimited amounts of data in any format inexpensively, especially for archive purposes • Allows collection of data that you may or may not use later: “just in case” • A way to describe any large data pool in which the schema and data requirements are not defined until the data is queried: “just in time” or “schema on read” • Complements EDW and can be seen as a data source for the EDW – capturing all data but only passing relevant data to the EDW • Frees up expensive EDW resources (storage and processing), especially for data refinement • Allows for data exploration to be performed without waiting for the EDW team to model and load the data (quick user access) • Some processing in better done with Hadoop tools than ETL tools like SSIS • Easily scalable
Current state of a data warehouse Traditional Approaches CRMERPOLTP LOB DATA SOURCES ETL DATA WAREHOUSE Star schemas, views other read- optimized structures BI AND ANALYTCIS Emailed, centrally stored Excel reports and dashboards Well manicured, often relational sources Known and expected data volume and formats Little to no change Complex, rigid transformations Required extensive monitoring Transformed historical into read structures Flat, canned or multi-dimensional access to historical data Many reports, multiple versions of the truth 24 to 48h delay MONITORING AND TELEMETRY
Current state of a data warehouse Traditional Approaches CRMERPOLTP LOB DATA SOURCES ETL DATA WAREHOUSE Star schemas, views other read- optimized structures BI AND ANALYTCIS Emailed, centrally stored Excel reports and dashboards Increase in variety of data sources Increase in data volume Increase in types of data Pressure on the ingestion engine Complex, rigid transformations can’t longer keep pace Monitoring is abandoned Delay in data, inability to transform volumes, or react to new sources Repair, adjust and redesign ETL Reports become invalid or unusable Delay in preserved reports increases Users begin to “innovate” to relieve starvation MONITORING AND TELEMETRY INCREASING DATA VOLUME NON-RELATIONAL DATA INCREASE IN TIME STALE REPORTING
Data Lake Transformation (ELT not ETL) New Approaches All data sources are considered Leverages the power of on-prem technologies and the cloud for storage and capture Native formats, streaming data, big data Extract and load, no/minimal transform Storage of data in near-native format Orchestration becomes possible Streaming data accommodation becomes possible Refineries transform data on read Produce curated data sets to integrate with traditional warehouses Users discover published data sets/services using familiar tools CRMERPOLTP LOB DATA SOURCES FUTURE DATA SOURCESNON-RELATIONAL DATA EXTRACT AND LOAD DATA LAKE DATA REFINERY PROCESS (TRANSFORM ON READ) Transform relevant data into data sets BI AND ANALYTCIS Discover and consume predictive analytics, data sets and other reports DATA WAREHOUSE Star schemas, views other read- optimized structures
Data Analysis Paradigm Shift OLD WAY: Structure -> Ingest -> Analyze NEW WAY: Ingest -> Analyze -> Structure
Data Lake layers • Raw data layer– Raw events are stored for historical reference. Also called staging layer or landing area • Cleansed data layer – Raw events are transformed (cleaned and mastered) into directly consumable data sets. Aim is to uniform the way files are stored in terms of encoding, format, data types and content (i.e. strings). Also called conformed layer • Application data layer – Business logic is applied to the cleansed data to produce data ready to be consumed by applications (i.e. DW application, advanced analysis process, etc). Also called workspace layer or trusted layer Still need data governance so your data lake does not turn into a data swamp!
Should I use Hadoop or NoSQL for the data lake? Most implementations use Hadoop as the data lake because of these benefits: • Open-source software ecosystem that allows for massively parallel computing • No inherent structure (no conversion to JSON needed) • Good for batch processing, large files, volume writes, parallel scans, sequential access (NoSQL designed for large-scale OLTP) • Large ecosystem of products • Low cost • Con: performance
? ? ? ? Hadoop as the data lake
What is Hadoop? Microsoft Confidential  Distributed, scalable system on commodity HW  Composed of a few parts:  HDFS – Distributed file system  MapReduce – Programming model  Other tools: Hive, Pig, SQOOP, HCatalog, HBase, Flume, Mahout, YARN, Tez, Spark, Stinger, Oozie, ZooKeeper, Flume, Storm  Main players are Hortonworks, Cloudera, MapR  WARNING: Hadoop, while ideal for processing huge volumes of data, is inadequate for analyzing that data in real time (companies do batch analytics instead) Core Services OPERATIONAL SERVICES DATA SERVICES HDFS SQOOP FLUME NFS LOAD & EXTRACT WebHDFS OOZIE AMBARI YARN MAP REDUCE HIVE & HCATALOG PIG HBASEFALCON Hadoop Cluster compute & storage . . . . . . . . compute & storage . . Hadoop clusters provide scale-out storage and distributed data processing on commodity hardware
Hortonworks Data Platform 2.5 Simply put, Hortonworks ties all the open source products together (22)
The real cost of Hadoop http://www.wintercorp.com/tcod-report/
Use cases using Hadoop and a DW in combination Bringing islands of Hadoop data together Archiving data warehouse data to Hadoop (move) (Hadoop as cold storage) Exporting relational data to Hadoop (copy) (Hadoop as backup/DR, analysis, cloud use) Importing Hadoop data into data warehouse (copy) (Hadoop as staging area, sandbox, Data Lake)
? ? ? ? Modern Data Warehouse
Modern Data Warehouse Think about future needs: • Increasing data volumes • Real-time performance • New data sources and types • Cloud-born data • Multi-platform solution • Hybrid architecture
Modern Data WarehouseThe Dream
The Reality
Near Realtime Data Analytics Pipeline using Azure Steam Analytics Big Data Analytics Pipeline using Azure Data Lake Interactive Analytics and Predictive Pipeline using Azure Data Factory Base Architecture : Big Data Advanced Analytics Pipeline Data Sources Ingest Prepare (normalize, clean, etc.) Analyze (stat analysis, ML, etc.) Publish (for programmatic consumption, BI/visualization) Consume (Alerts, Operational Stats, Insights) Machine Learning Telemetry Azure SQL (Predictions) HDI Custom ETL Aggregate /Partition Azure Storage Blob dashboard of predictions / alerts Live / real-time data stats, Anomalies and aggregates Customer MIS Event Hub PowerBI dashboard Stream Analytics (real-time analytics) Azure Data Lake Analytics (Big Data Processing) Azure Data Lake Storage Azure SQL Data in Motion Data at Rest dashboard of operational stats 41 Scheduledhourly transferusingAzure DataFactory Machine Learning (Anomaly Detection)
Roles when using both Data Lake and DW Data Lake/Hadoop (staging and processing environment) • Batch reporting • Data refinement/cleaning • ETL workloads • Store historical data • Sandbox for data exploration • One-time reports • Data scientist workloads • Quick results Data Warehouse/RDBMS (serving and compliance environment) • Low latency • High number of users • Additional security • Large support for tools • Easily create reports (Self-service BI) • A data lake is just a glorified file folder with data files in it – how many end-users can accurately create reports from it?
Microsoft data platform solutions Product Category Description More Info SQL Server 2016 RDBMS Earned top spot in Gartner’s Operational Database magic quadrant. JSON support https://www.microsoft.com/en-us/server- cloud/products/sql-server-2016/ SQL Database RDBMS/DBaaS Cloud-based service that is provisioned and scaled quickly. Has built-in high availability and disaster recovery. JSON support https://azure.microsoft.com/en- us/services/sql-database/ SQL Data Warehouse MPP RDBMS/DBaaS Cloud-based service that handles relational big data. Provision and scale quickly. Can pause service to reduce cost https://azure.microsoft.com/en- us/services/sql-data-warehouse/ Analytics Platform System (APS) MPP RDBMS Big data analytics appliance for high performance and seamless integration of all your data https://www.microsoft.com/en-us/server- cloud/products/analytics-platform- system/ Azure Data Lake Store Hadoop storage Removes the complexities of ingesting and storing all of your data while making it faster to get up and running with batch, streaming, and interactive analytics https://azure.microsoft.com/en- us/services/data-lake-store/ Azure Data Lake Analytics On-demand analytics job service/Big Data-as-a- service Cloud-based service that dynamically provisions resources so you can run queries on exabytes of data. Includes U- SQL, a new big data query language https://azure.microsoft.com/en- us/services/data-lake-analytics/ HDInsight PaaS Hadoop compute A managed Apache Hadoop, Spark, R, HBase, and Storm cloud service made easy https://azure.microsoft.com/en- us/services/hdinsight/ DocumentDB PaaS NoSQL: Document Store Get your apps up and running in hours with a fully managed NoSQL database service that indexes, stores, and queries data using familiar SQL syntax https://azure.microsoft.com/en- us/services/documentdb/ Azure Table Storage PaaS NoSQL: Key-value Store Store large amount of semi-structured data in the cloud https://azure.microsoft.com/en- us/services/storage/tables/
Cortana Intelligence Suite Integrated as part of an end-to-end suite Action People Automated Systems Apps Web Mobile Bots Intelligence Dashboards & Visualizations Cortana Bot Framework Cognitive Services Power BI Information Management Event Hubs Data Catalog Data Factory Machine Learning and Analytics HDInsight (Hadoop and Spark) Stream Analytics Intelligence Data Lake Analytics Machine Learning Big Data Stores SQL Data Warehouse Data Lake Store Data Sources Apps Sensors and devices Data
? ? ? ? Federated Querying
Federated Querying Other names: Data virtualization, logical data warehouse, data federation, virtual database, and decentralized data warehouse. A model that allows a single query to retrieve and combine data as it sits from multiple data sources, so as to not need to use ETL or learn more than one retrieval technology
PolyBase Query relational and non-relational data with T-SQL By preview early this year PolyBase will add support for Teradata, Oracle, SQL Server, MongoDB, and generic ODBC (Spark, Hive, Impala, DB2) Vs U-SQL: PolyBase is interactive while U-SQL is batch. U-SQL more code to query data but more formats (JSON) and libraries/UDOs and supports writes to blob/ADLS
? ? ? ? Solution in the cloud
Benefits of the cloud Agility • Unlimited elastic scale • Pay for what you need Innovation • Quick “Time to market” • Fail fast Risk • Availability • Reliability • Security Total cost of ownership calculator: https://www.tco.microsoft.com/
Constraints of on-premise data • Scale constrained to on-premise procurement • Capex up-front costs, most companies instead prefer a yearly operating expense (OpEx) • A staff of employees or consultants must be retained to administer and support the hardware and software in place • Expertise needed for tuning and deployment
Talking points when using the cloud for DW • Public and private cloud • Cloud-born data vs on-prem born data • Transfer cost from/to cloud and on-prem • Sensitive data on-prem, non-sensitive in cloud • Look at hybrid solutions
? ? ? ? SMP vs MPP
SMP vs MPP • Uses many separate CPUs running in parallel to execute a single program • Shared Nothing: Each CPU has its own memory and disk (scale-out) • Segments communicate using high-speed network between nodes MPP - Massively Parallel Processing • Multiple CPUs used to complete individual processes simultaneously • All CPUs share the same memory, disks, and network controllers (scale-up) • All SQL Server implementations up until now have been SMP • Mostly, the solution is housed on a shared SAN SMP - Symmetric Multiprocessing
50 TB 100 TB 500 TB 10 TB 5 PB 1.000 100 10.000 3-5 Way Joins  Joins +  OLAP operations +  Aggregation +  Complex “Where” constraints +  Views  Parallelism 5-10 Way Joins Normalized Multiple, Integrated Stars and Normalized Simple Star Multiple, Integrated Stars TB’s MB’s GB’s Batch Reporting, Repetitive Queries Ad Hoc Queries Data Analysis/Mining Near Real Time Data Feeds Daily Load Weekly Load Strategic, Tactical Strategic Strategic, Tactical Loads Strategic, Tactical Loads, SLA “Query Freedom“ “Query complexity“ “Data Freshness” “Query Data Volume“ “Query Concurrency“ “Mixed Workload” “Schema Sophistication“ “Data Volume” DW SCALABILITY SPIDER CHART MPP – Multidimensional Scalability SMP – Tunable in one dimension on cost of other dimensions The spiderweb depicts important attributes to consider when evaluating Data Warehousing options. Big Data support is newest dimension.
Summary • We live in an increasingly data-intensive world • Much of the data stored online and analyzed today is more varied than the data stored in recent years • More of our data arrives in near-real time • “Data is the new currency!” This present a large business opportunity. Are you ready for it?
Other Related Presentations  Building a Big Data Solution  Choosing technologies for a big data solution in the cloud  How does Microsoft solve Big Data?  Benefits of the Azure cloud  Should I move my database to the cloud?  Implement SQL Server on a Azure VM  Relational databases vs Non-relational databases  Introduction to Microsoft’s Hadoop solution (HDInsight)  Introducing Azure SQL Database  Introducing Azure SQL Data Warehouse Visit my blog at: JamesSerra.com (where these slide decks are posted under the “Presentation” tab)
Resources  Why use a data lake? http://bit.ly/1WDy848  Big Data Architectures http://bit.ly/1RBbAbS  The Modern Data Warehouse: http://bit.ly/1xuX4Py  Hadoop and Data Warehouses: http://bit.ly/1xuXfu9
Q & A ? James Serra, Big Data Evangelist Email me at: JamesSerra3@gmail.com Follow me at: @JamesSerra Link to me at: www.linkedin.com/in/JamesSerra Visit my blog at: JamesSerra.com (where this slide deck is posted under the “Presentations” tab)

More Related Content

PPTX
Is the traditional data warehouse dead?
byJames Serra
 
PPTX
Building a modern data warehouse
byJames Serra
 
PDF
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
byDatabricks
 
PPTX
Building the Data Lake with Azure Data Factory and Data Lake Analytics
byKhalid Salama
 
PDF
Building End-to-End Delta Pipelines on GCP
byDatabricks
 
PPTX
Data platform modernization with Databricks.pptx
byCalvinSim10
 
PDF
Building Data Lakehouse.pdf
byLuis Jimenez
 
PDF
Build Real-Time Applications with Databricks Streaming
byDatabricks
 
Is the traditional data warehouse dead?
byJames Serra
 
Building a modern data warehouse
byJames Serra
 
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
byDatabricks
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
byKhalid Salama
 
Building End-to-End Delta Pipelines on GCP
byDatabricks
 
Data platform modernization with Databricks.pptx
byCalvinSim10
 
Building Data Lakehouse.pdf
byLuis Jimenez
 
Build Real-Time Applications with Databricks Streaming
byDatabricks
 

What's hot

PPTX
Data Lake Overview
byJames Serra
 
PPTX
Data Lakehouse, Data Mesh, and Data Fabric (r1)
byJames Serra
 
PPTX
Databricks Platform.pptx
byAlex Ivy
 
PPTX
DW Migration Webinar-March 2022.pptx
byDatabricks
 
PDF
Modernizing to a Cloud Data Architecture
byDatabricks
 
PDF
Architect’s Open-Source Guide for a Data Mesh Architecture
byDatabricks
 
PPTX
Azure data platform overview
byJames Serra
 
PPTX
Azure Synapse Analytics Overview (r2)
byJames Serra
 
PDF
Data Warehouse or Data Lake, Which Do I Choose?
byDATAVERSITY
 
PPTX
Introduction to Data Engineering
byDurga Gadiraju
 
PDF
Data Mesh for Dinner
byKent Graziano
 
PPTX
Big Data Tutorial | What Is Big Data | Big Data Hadoop Tutorial For Beginners...
bySimplilearn
 
PDF
Data Engineering Basics
byCatherine Kimani
 
PPTX
Databricks Fundamentals
byDalibor Wijas
 
PDF
Introduction SQL Analytics on Lakehouse Architecture
byDatabricks
 
PPTX
Free Training: How to Build a Lakehouse
byDatabricks
 
PDF
Future of Data Engineering
byC4Media
 
PDF
Data Catalog for Better Data Discovery and Governance
byDenodo
 
PDF
Data Catalog as the Platform for Data Intelligence
byAlation
 
PPTX
Introduction to Data Engineering
byVivek Aanand Ganesan
 
Data Lake Overview
byJames Serra
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
byJames Serra
 
Databricks Platform.pptx
byAlex Ivy
 
DW Migration Webinar-March 2022.pptx
byDatabricks
 
Modernizing to a Cloud Data Architecture
byDatabricks
 
Architect’s Open-Source Guide for a Data Mesh Architecture
byDatabricks
 
Azure data platform overview
byJames Serra
 
Azure Synapse Analytics Overview (r2)
byJames Serra
 
Data Warehouse or Data Lake, Which Do I Choose?
byDATAVERSITY
 
Introduction to Data Engineering
byDurga Gadiraju
 
Data Mesh for Dinner
byKent Graziano
 
Big Data Tutorial | What Is Big Data | Big Data Hadoop Tutorial For Beginners...
bySimplilearn
 
Data Engineering Basics
byCatherine Kimani
 
Databricks Fundamentals
byDalibor Wijas
 
Introduction SQL Analytics on Lakehouse Architecture
byDatabricks
 
Free Training: How to Build a Lakehouse
byDatabricks
 
Future of Data Engineering
byC4Media
 
Data Catalog for Better Data Discovery and Governance
byDenodo
 
Data Catalog as the Platform for Data Intelligence
byAlation
 
Introduction to Data Engineering
byVivek Aanand Ganesan
 

Viewers also liked

PDF
Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...
byHortonworks
 
PPTX
How to build a successful Data Lake
byDataWorks Summit/Hadoop Summit
 
PPTX
Azure SQL Database Managed Instance
byJames Serra
 
PDF
Big Data Tech Stack
byAbdullah Çetin ÇAVDAR
 
PPTX
Using Hadoop as a platform for Master Data Management
byDataWorks Summit
 
PDF
A beginners guide to Cloudera Hadoop
byDavid Yahalom
 
PPTX
Ai big dataconference_eugene_polonichko_azure data lake
byOlga Zinkevych
 
PDF
Organising the Data Lake - Information Management in a Big Data World
byDataWorks Summit/Hadoop Summit
 
PPTX
Top Trends in Building Data Lakes for Machine Learning and AI
byHolden Ackerman
 
PPTX
10 Amazing Things To Do With a Hadoop-Based Data Lake
byVMware Tanzu
 
PPTX
50 Shades of Data - how, when and why Big,Relational,NoSQL,Elastic,Event,CQRS...
byLucas Jellema
 
PPTX
CWIN17 India / Insights platform architecture v1 0 virtual - subhadeep dutta
byCapgemini
 
PDF
The Future of Data
byblynnbuckley
 
Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...
byHortonworks
 
How to build a successful Data Lake
byDataWorks Summit/Hadoop Summit
 
Azure SQL Database Managed Instance
byJames Serra
 
Big Data Tech Stack
byAbdullah Çetin ÇAVDAR
 
Using Hadoop as a platform for Master Data Management
byDataWorks Summit
 
A beginners guide to Cloudera Hadoop
byDavid Yahalom
 
Ai big dataconference_eugene_polonichko_azure data lake
byOlga Zinkevych
 
Organising the Data Lake - Information Management in a Big Data World
byDataWorks Summit/Hadoop Summit
 
Top Trends in Building Data Lakes for Machine Learning and AI
byHolden Ackerman
 
10 Amazing Things To Do With a Hadoop-Based Data Lake
byVMware Tanzu
 
50 Shades of Data - how, when and why Big,Relational,NoSQL,Elastic,Event,CQRS...
byLucas Jellema
 
CWIN17 India / Insights platform architecture v1 0 virtual - subhadeep dutta
byCapgemini
 
The Future of Data
byblynnbuckley
 

Similar to Big data architectures and the data lake

PPTX
Data Lakehouse, Data Mesh, and Data Fabric (r2)
byJames Serra
 
PDF
Data Virtualization: An Essential Component of a Cloud Data Lake
byDenodo
 
PDF
The Marriage of the Data Lake and the Data Warehouse and Why You Need Both
byAdaryl "Bob" Wakefield, MBA
 
PDF
Building the Modern Data Hub
byDatavail
 
PPTX
Data lake-itweekend-sharif university-vahid amiry
bydatastack
 
PDF
Hadoop data-lake-white-paper
bySupratim Ray
 
PDF
Data Lakes: A Logical Approach for Faster Unified Insights
byDenodo
 
PDF
Data lakes
byŞaban Dalaman
 
PPTX
Chap3-Data Warehousing and OLAP operations..pptx
bystuti8985
 
PDF
Building the Modern Data Hub: Beyond the Traditional Enterprise Data Warehouse
byFormant
 
PDF
Whitepaper-The-Data-Lake-3_0
byJane Roberts
 
PDF
Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...
byData Con LA
 
PPTX
Designing modern dw and data lake
bypunedevscom
 
PDF
Modern data warehouse
byStephen Alex
 
PDF
Modern data warehouse
byStephen Alex
 
PDF
Architecting Agile Data Applications for Scale
byDatabricks
 
PDF
New World Hadoop Architectures (& What Problems They Really Solve) for Oracle...
byRittman Analytics
 
PDF
Data Integration and Data Warehousing for Cloud, Big Data and IoT: 
What’s Ne...
byRittman Analytics
 
PDF
Harness the power of Data in a Big Data Lake
bySaurabh K. Gupta
 
PPTX
Big Data 2.0: Hadoop as part of a Near-Real-Time Integrated Data Era
byDataWorks Summit
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
byJames Serra
 
Data Virtualization: An Essential Component of a Cloud Data Lake
byDenodo
 
The Marriage of the Data Lake and the Data Warehouse and Why You Need Both
byAdaryl "Bob" Wakefield, MBA
 
Building the Modern Data Hub
byDatavail
 
Data lake-itweekend-sharif university-vahid amiry
bydatastack
 
Hadoop data-lake-white-paper
bySupratim Ray
 
Data Lakes: A Logical Approach for Faster Unified Insights
byDenodo
 
Data lakes
byŞaban Dalaman
 
Chap3-Data Warehousing and OLAP operations..pptx
bystuti8985
 
Building the Modern Data Hub: Beyond the Traditional Enterprise Data Warehouse
byFormant
 
Whitepaper-The-Data-Lake-3_0
byJane Roberts
 
Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...
byData Con LA
 
Designing modern dw and data lake
bypunedevscom
 
Modern data warehouse
byStephen Alex
 
Modern data warehouse
byStephen Alex
 
Architecting Agile Data Applications for Scale
byDatabricks
 
New World Hadoop Architectures (& What Problems They Really Solve) for Oracle...
byRittman Analytics
 
Data Integration and Data Warehousing for Cloud, Big Data and IoT: 
What’s Ne...
byRittman Analytics
 
Harness the power of Data in a Big Data Lake
bySaurabh K. Gupta
 
Big Data 2.0: Hadoop as part of a Near-Real-Time Integrated Data Era
byDataWorks Summit
 

More from James Serra

PPTX
Introduction to Azure Databricks
byJames Serra
 
PPTX
Microsoft Data Platform - What's included
byJames Serra
 
PPTX
Azure Synapse Analytics Overview (r1)
byJames Serra
 
PPTX
Data Warehousing Trends, Best Practices, and Future Outlook
byJames Serra
 
PPTX
Power BI Overview, Deployment and Governance
byJames Serra
 
PPTX
Microsoft Fabric Introduction
byJames Serra
 
PPTX
Differentiate Big Data vs Data Warehouse use cases for a cloud solution
byJames Serra
 
PPTX
Power BI Overview
byJames Serra
 
PPTX
Power BI for Big Data and the New Look of Big Data Solutions
byJames Serra
 
PPTX
Machine Learning and AI
byJames Serra
 
PPTX
Microsoft cloud big data strategy
byJames Serra
 
PPTX
Choosing technologies for a big data solution in the cloud
byJames Serra
 
PPTX
Overview on Azure Machine Learning
byJames Serra
 
PPTX
Introduction to PolyBase
byJames Serra
 
PPTX
Introducing DocumentDB
byJames Serra
 
PPTX
AI for an intelligent cloud and intelligent edge: Discover, deploy, and manag...
byJames Serra
 
PPTX
What’s new in SQL Server 2017
byJames Serra
 
PPTX
Learning to present and becoming good at it
byJames Serra
 
PPTX
What's new in SQL Server 2016
byJames Serra
 
PPTX
How to build your career
byJames Serra
 
Introduction to Azure Databricks
byJames Serra
 
Microsoft Data Platform - What's included
byJames Serra
 
Azure Synapse Analytics Overview (r1)
byJames Serra
 
Data Warehousing Trends, Best Practices, and Future Outlook
byJames Serra
 
Power BI Overview, Deployment and Governance
byJames Serra
 
Microsoft Fabric Introduction
byJames Serra
 
Differentiate Big Data vs Data Warehouse use cases for a cloud solution
byJames Serra
 
Power BI Overview
byJames Serra
 
Power BI for Big Data and the New Look of Big Data Solutions
byJames Serra
 
Machine Learning and AI
byJames Serra
 
Microsoft cloud big data strategy
byJames Serra
 
Choosing technologies for a big data solution in the cloud
byJames Serra
 
Overview on Azure Machine Learning
byJames Serra
 
Introduction to PolyBase
byJames Serra
 
Introducing DocumentDB
byJames Serra
 
AI for an intelligent cloud and intelligent edge: Discover, deploy, and manag...
byJames Serra
 
What’s new in SQL Server 2017
byJames Serra
 
Learning to present and becoming good at it
byJames Serra
 
What's new in SQL Server 2016
byJames Serra
 
How to build your career
byJames Serra
 

Recently uploaded

PDF
Implementing A Data Lakehouse Using Iceberg & Spark
byAnass Nabil
 
PDF
How Automation Powers Data Quality and Governance at Scale
bySafe Software
 
PDF
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
PPTX
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
PDF
Unlocking Gen AI Capabilities Within UiPath Document Understanding
byDianaGray10
 
PPTX
Webinar: EVerest for Production: Accelerating EV Charger Development w/ Indus...
byDanBrown980551
 
PDF
Why Enterprises Are Moving to Dedicated Servers in the Netherlands
byAtal Networks
 
PDF
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
PDF
Profiting from Technological Innovation: Managing Intellectual Property to Bu...
byDavid Teece
 
PDF
6 Insights from the Patron Saint of AI Failure
byScott M. Graffius
 
PPTX
Robot Vacuums are Cleaning Up. The global robot vacuum segment has high marke...
byRobert March
 
PDF
Are AI Hallucinations Getting Better or Worse? We Analyzed the Data.
byScott M. Graffius
 
PDF
Salesforce Sales Cloud and Generative AI Present a Fresh Approach to Personal...
byMinuscule Technologies
 
PPTX
The Abomination of AI – keynote at ICoSCI 2026
byAlan Dix
 
PDF
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
PPTX
WIFI Enabled 8 by 32 Matrix.pptx for final year project
bymdsabbirhossain524856
 
PDF
Just Eat Data Scraping For Restaurant Reviews And Pricing.pdf
byfusiondata2026
 
PPTX
About Computer Hardware ................
bymaratiakshaya07
 
PDF
Zniber Partners. Audience Foresight for Confident Decisions
bySamuel Zniber
 
PPTX
congo red stain nd PAS STAIN histopathology
bybushrariaz1240
 
Implementing A Data Lakehouse Using Iceberg & Spark
byAnass Nabil
 
How Automation Powers Data Quality and Governance at Scale
bySafe Software
 
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
Unlocking Gen AI Capabilities Within UiPath Document Understanding
byDianaGray10
 
Webinar: EVerest for Production: Accelerating EV Charger Development w/ Indus...
byDanBrown980551
 
Why Enterprises Are Moving to Dedicated Servers in the Netherlands
byAtal Networks
 
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
Profiting from Technological Innovation: Managing Intellectual Property to Bu...
byDavid Teece
 
6 Insights from the Patron Saint of AI Failure
byScott M. Graffius
 
Robot Vacuums are Cleaning Up. The global robot vacuum segment has high marke...
byRobert March
 
Are AI Hallucinations Getting Better or Worse? We Analyzed the Data.
byScott M. Graffius
 
Salesforce Sales Cloud and Generative AI Present a Fresh Approach to Personal...
byMinuscule Technologies
 
The Abomination of AI – keynote at ICoSCI 2026
byAlan Dix
 
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
WIFI Enabled 8 by 32 Matrix.pptx for final year project
bymdsabbirhossain524856
 
Just Eat Data Scraping For Restaurant Reviews And Pricing.pdf
byfusiondata2026
 
About Computer Hardware ................
bymaratiakshaya07
 
Zniber Partners. Audience Foresight for Confident Decisions
bySamuel Zniber
 
congo red stain nd PAS STAIN histopathology
bybushrariaz1240
 

Big data architectures and the data lake

  • 1.
    Big data architecturesand the data lake James Serra Big Data Evangelist Microsoft JamesSerra3@gmail.com
  • 2.
    About Me  Microsoft,Big Data Evangelist  In IT for 30 years, worked on many BI and DW projects  Worked as desktop/web/database developer, DBA, BI and DW architect and developer, MDM architect, PDW/APS developer  Been perm employee, contractor, consultant, business owner  Presenter at PASS Business Analytics Conference, PASS Summit, Enterprise Data World conference  Certifications: MCSE: Data Platform, Business Intelligence; MS: Architecting Microsoft Azure Solutions, Design and Implement Big Data Analytics Solutions, Design and Implement Cloud Data Platform Solutions  Blog at JamesSerra.com  Former SQL Server MVP  Author of book “Reporting with Microsoft SQL Server 2012”
  • 3.
    Agenda  Big DataArchitectures  Why data lakes?  Top-down vs Bottom-up  Data lake defined  Hadoop as the data lake  Modern Data Warehouse  Federated Querying  Solution in the cloud  SMP vs MPP
  • 4.
  • 5.
    Enterprise data warehouseaugmentation • Seen when EDW has been in existence a while and EDW can’t handle new data • Cons: not offloading EDW work, can’t use existing tools, data hub difficulty understanding data
  • 6.
    Data hub plusEDW (temporary) • Data hub is used as temporary staging and refining, no reporting • Cons: data hub is temporary, no reporting/analyzing done with the data hub
  • 7.
    All-in-one • Data hubis total solution, no EDW • Cons: queries are slower, new training for reporting tools, difficulty understanding data, security limitations
  • 8.
    Modern Data Warehouse •Evolution of three previous scenarios • Ultimate goal • Supports future data needs • Data harmonized and analyzed in the data lake or moved to EDW for more quality and performance
  • 9.
  • 10.
    ETL pipeline Dedicated ETLtools (e.g. SSIS) Defined schema Queries Results Relational LOB Applications Traditional business analytics process 1. Start with end-user requirements to identify desired reports and analysis 2. Define corresponding database schema and queries 3. Identify the required data sources 4. Create a Extract-Transform-Load (ETL) pipeline to extract required data (curation) and transform it to target schema (‘schema-on-write’) 5. Create reports. Analyze data All data not immediately required is discarded or archived 10
  • 11.
    Harness the growingand changing nature of data Need to collect any data StreamingStructured Challenge is combining transactional data stored in relational databases with less structured data Big Data = All Data Get the right information to the right people at the right time in the right format Unstructured “ ”
  • 12.
  • 13.
    Store indefinitely AnalyzeSee results Gather data from all sources Iterate New big data thinking: All data has value All data has potential value Data hoarding No defined schema—stored in native format Schema is imposed and transformations are done at query time (schema-on-read). Apps and users interpret the data as they see fit 13
  • 14.
  • 15.
    Observation Pattern Theory Hypothesis What will happen? How canwe make it happen? Predictive Analytics Prescriptive Analytics What happened? Why did it happen? Descriptive Analytics Diagnostic Analytics Confirmation Theory Hypothesis Observation Two Approaches to Information Management for Analytics: Top-Down + Bottoms-Up
  • 16.
    Implement Data Warehouse PhysicalDesign ETL Development Reporting & Analytics Development Install and Tune Reporting & Analytics Design Dimension Modelling ETL Design Setup Infrastructure Understand Corporate Strategy Data Warehousing Uses A Top-Down Approach Data sources Gather Requirements Business Requirements Technical Requirements
  • 17.
    The “data lake”Uses A Bottoms-Up Approach Ingest all data regardless of requirements Store all data in native format without schema definition Do analysis Using analytic engines like Hadoop Interactive queries Batch queries Machine Learning Data warehouse Real-time analytics Devices
  • 18.
    Data Lake +Data Warehouse Better Together Data sources What happened? Descriptive Analytics Diagnostic Analytics Why did it happen? What will happen? Predictive Analytics Prescriptive Analytics How can we make it happen?
  • 19.
  • 20.
    What is adata lake? A storage repository, usually Hadoop, that holds a vast amount of raw data in its native format until it is needed. • A place to store unlimited amounts of data in any format inexpensively, especially for archive purposes • Allows collection of data that you may or may not use later: “just in case” • A way to describe any large data pool in which the schema and data requirements are not defined until the data is queried: “just in time” or “schema on read” • Complements EDW and can be seen as a data source for the EDW – capturing all data but only passing relevant data to the EDW • Frees up expensive EDW resources (storage and processing), especially for data refinement • Allows for data exploration to be performed without waiting for the EDW team to model and load the data (quick user access) • Some processing in better done with Hadoop tools than ETL tools like SSIS • Easily scalable
  • 21.
    Current state ofa data warehouse Traditional Approaches CRMERPOLTP LOB DATA SOURCES ETL DATA WAREHOUSE Star schemas, views other read- optimized structures BI AND ANALYTCIS Emailed, centrally stored Excel reports and dashboards Well manicured, often relational sources Known and expected data volume and formats Little to no change Complex, rigid transformations Required extensive monitoring Transformed historical into read structures Flat, canned or multi-dimensional access to historical data Many reports, multiple versions of the truth 24 to 48h delay MONITORING AND TELEMETRY
  • 22.
    Current state ofa data warehouse Traditional Approaches CRMERPOLTP LOB DATA SOURCES ETL DATA WAREHOUSE Star schemas, views other read- optimized structures BI AND ANALYTCIS Emailed, centrally stored Excel reports and dashboards Increase in variety of data sources Increase in data volume Increase in types of data Pressure on the ingestion engine Complex, rigid transformations can’t longer keep pace Monitoring is abandoned Delay in data, inability to transform volumes, or react to new sources Repair, adjust and redesign ETL Reports become invalid or unusable Delay in preserved reports increases Users begin to “innovate” to relieve starvation MONITORING AND TELEMETRY INCREASING DATA VOLUME NON-RELATIONAL DATA INCREASE IN TIME STALE REPORTING
  • 23.
    Data Lake Transformation(ELT not ETL) New Approaches All data sources are considered Leverages the power of on-prem technologies and the cloud for storage and capture Native formats, streaming data, big data Extract and load, no/minimal transform Storage of data in near-native format Orchestration becomes possible Streaming data accommodation becomes possible Refineries transform data on read Produce curated data sets to integrate with traditional warehouses Users discover published data sets/services using familiar tools CRMERPOLTP LOB DATA SOURCES FUTURE DATA SOURCESNON-RELATIONAL DATA EXTRACT AND LOAD DATA LAKE DATA REFINERY PROCESS (TRANSFORM ON READ) Transform relevant data into data sets BI AND ANALYTCIS Discover and consume predictive analytics, data sets and other reports DATA WAREHOUSE Star schemas, views other read- optimized structures
  • 24.
    Data Analysis ParadigmShift OLD WAY: Structure -> Ingest -> Analyze NEW WAY: Ingest -> Analyze -> Structure
  • 25.
    Data Lake layers •Raw data layer– Raw events are stored for historical reference. Also called staging layer or landing area • Cleansed data layer – Raw events are transformed (cleaned and mastered) into directly consumable data sets. Aim is to uniform the way files are stored in terms of encoding, format, data types and content (i.e. strings). Also called conformed layer • Application data layer – Business logic is applied to the cleansed data to produce data ready to be consumed by applications (i.e. DW application, advanced analysis process, etc). Also called workspace layer or trusted layer Still need data governance so your data lake does not turn into a data swamp!
  • 26.
    Should I useHadoop or NoSQL for the data lake? Most implementations use Hadoop as the data lake because of these benefits: • Open-source software ecosystem that allows for massively parallel computing • No inherent structure (no conversion to JSON needed) • Good for batch processing, large files, volume writes, parallel scans, sequential access (NoSQL designed for large-scale OLTP) • Large ecosystem of products • Low cost • Con: performance
  • 27.
  • 28.
    What is Hadoop? MicrosoftConfidential  Distributed, scalable system on commodity HW  Composed of a few parts:  HDFS – Distributed file system  MapReduce – Programming model  Other tools: Hive, Pig, SQOOP, HCatalog, HBase, Flume, Mahout, YARN, Tez, Spark, Stinger, Oozie, ZooKeeper, Flume, Storm  Main players are Hortonworks, Cloudera, MapR  WARNING: Hadoop, while ideal for processing huge volumes of data, is inadequate for analyzing that data in real time (companies do batch analytics instead) Core Services OPERATIONAL SERVICES DATA SERVICES HDFS SQOOP FLUME NFS LOAD & EXTRACT WebHDFS OOZIE AMBARI YARN MAP REDUCE HIVE & HCATALOG PIG HBASEFALCON Hadoop Cluster compute & storage . . . . . . . . compute & storage . . Hadoop clusters provide scale-out storage and distributed data processing on commodity hardware
  • 29.
    Hortonworks Data Platform2.5 Simply put, Hortonworks ties all the open source products together (22)
  • 30.
    The real costof Hadoop http://www.wintercorp.com/tcod-report/
  • 31.
    Use cases usingHadoop and a DW in combination Bringing islands of Hadoop data together Archiving data warehouse data to Hadoop (move) (Hadoop as cold storage) Exporting relational data to Hadoop (copy) (Hadoop as backup/DR, analysis, cloud use) Importing Hadoop data into data warehouse (copy) (Hadoop as staging area, sandbox, Data Lake)
  • 32.
  • 33.
    Modern Data Warehouse Thinkabout future needs: • Increasing data volumes • Real-time performance • New data sources and types • Cloud-born data • Multi-platform solution • Hybrid architecture
  • 34.
  • 35.
  • 36.
    Near Realtime DataAnalytics Pipeline using Azure Steam Analytics Big Data Analytics Pipeline using Azure Data Lake Interactive Analytics and Predictive Pipeline using Azure Data Factory Base Architecture : Big Data Advanced Analytics Pipeline Data Sources Ingest Prepare (normalize, clean, etc.) Analyze (stat analysis, ML, etc.) Publish (for programmatic consumption, BI/visualization) Consume (Alerts, Operational Stats, Insights) Machine Learning Telemetry Azure SQL (Predictions) HDI Custom ETL Aggregate /Partition Azure Storage Blob dashboard of predictions / alerts Live / real-time data stats, Anomalies and aggregates Customer MIS Event Hub PowerBI dashboard Stream Analytics (real-time analytics) Azure Data Lake Analytics (Big Data Processing) Azure Data Lake Storage Azure SQL Data in Motion Data at Rest dashboard of operational stats 41 Scheduledhourly transferusingAzure DataFactory Machine Learning (Anomaly Detection)
  • 37.
    Roles when usingboth Data Lake and DW Data Lake/Hadoop (staging and processing environment) • Batch reporting • Data refinement/cleaning • ETL workloads • Store historical data • Sandbox for data exploration • One-time reports • Data scientist workloads • Quick results Data Warehouse/RDBMS (serving and compliance environment) • Low latency • High number of users • Additional security • Large support for tools • Easily create reports (Self-service BI) • A data lake is just a glorified file folder with data files in it – how many end-users can accurately create reports from it?
  • 38.
    Microsoft data platformsolutions Product Category Description More Info SQL Server 2016 RDBMS Earned top spot in Gartner’s Operational Database magic quadrant. JSON support https://www.microsoft.com/en-us/server- cloud/products/sql-server-2016/ SQL Database RDBMS/DBaaS Cloud-based service that is provisioned and scaled quickly. Has built-in high availability and disaster recovery. JSON support https://azure.microsoft.com/en- us/services/sql-database/ SQL Data Warehouse MPP RDBMS/DBaaS Cloud-based service that handles relational big data. Provision and scale quickly. Can pause service to reduce cost https://azure.microsoft.com/en- us/services/sql-data-warehouse/ Analytics Platform System (APS) MPP RDBMS Big data analytics appliance for high performance and seamless integration of all your data https://www.microsoft.com/en-us/server- cloud/products/analytics-platform- system/ Azure Data Lake Store Hadoop storage Removes the complexities of ingesting and storing all of your data while making it faster to get up and running with batch, streaming, and interactive analytics https://azure.microsoft.com/en- us/services/data-lake-store/ Azure Data Lake Analytics On-demand analytics job service/Big Data-as-a- service Cloud-based service that dynamically provisions resources so you can run queries on exabytes of data. Includes U- SQL, a new big data query language https://azure.microsoft.com/en- us/services/data-lake-analytics/ HDInsight PaaS Hadoop compute A managed Apache Hadoop, Spark, R, HBase, and Storm cloud service made easy https://azure.microsoft.com/en- us/services/hdinsight/ DocumentDB PaaS NoSQL: Document Store Get your apps up and running in hours with a fully managed NoSQL database service that indexes, stores, and queries data using familiar SQL syntax https://azure.microsoft.com/en- us/services/documentdb/ Azure Table Storage PaaS NoSQL: Key-value Store Store large amount of semi-structured data in the cloud https://azure.microsoft.com/en- us/services/storage/tables/
  • 39.
    Cortana Intelligence Suite Integratedas part of an end-to-end suite Action People Automated Systems Apps Web Mobile Bots Intelligence Dashboards & Visualizations Cortana Bot Framework Cognitive Services Power BI Information Management Event Hubs Data Catalog Data Factory Machine Learning and Analytics HDInsight (Hadoop and Spark) Stream Analytics Intelligence Data Lake Analytics Machine Learning Big Data Stores SQL Data Warehouse Data Lake Store Data Sources Apps Sensors and devices Data
  • 40.
  • 41.
    Federated Querying Other names:Data virtualization, logical data warehouse, data federation, virtual database, and decentralized data warehouse. A model that allows a single query to retrieve and combine data as it sits from multiple data sources, so as to not need to use ETL or learn more than one retrieval technology
  • 42.
    PolyBase Query relational andnon-relational data with T-SQL By preview early this year PolyBase will add support for Teradata, Oracle, SQL Server, MongoDB, and generic ODBC (Spark, Hive, Impala, DB2) Vs U-SQL: PolyBase is interactive while U-SQL is batch. U-SQL more code to query data but more formats (JSON) and libraries/UDOs and supports writes to blob/ADLS
  • 43.
  • 44.
    Benefits of thecloud Agility • Unlimited elastic scale • Pay for what you need Innovation • Quick “Time to market” • Fail fast Risk • Availability • Reliability • Security Total cost of ownership calculator: https://www.tco.microsoft.com/
  • 45.
    Constraints of on-premisedata • Scale constrained to on-premise procurement • Capex up-front costs, most companies instead prefer a yearly operating expense (OpEx) • A staff of employees or consultants must be retained to administer and support the hardware and software in place • Expertise needed for tuning and deployment
  • 46.
    Talking points whenusing the cloud for DW • Public and private cloud • Cloud-born data vs on-prem born data • Transfer cost from/to cloud and on-prem • Sensitive data on-prem, non-sensitive in cloud • Look at hybrid solutions
  • 47.
  • 48.
    SMP vs MPP •Uses many separate CPUs running in parallel to execute a single program • Shared Nothing: Each CPU has its own memory and disk (scale-out) • Segments communicate using high-speed network between nodes MPP - Massively Parallel Processing • Multiple CPUs used to complete individual processes simultaneously • All CPUs share the same memory, disks, and network controllers (scale-up) • All SQL Server implementations up until now have been SMP • Mostly, the solution is housed on a shared SAN SMP - Symmetric Multiprocessing
  • 49.
    50 TB 100 TB 500TB 10 TB 5 PB 1.000 100 10.000 3-5 Way Joins  Joins +  OLAP operations +  Aggregation +  Complex “Where” constraints +  Views  Parallelism 5-10 Way Joins Normalized Multiple, Integrated Stars and Normalized Simple Star Multiple, Integrated Stars TB’s MB’s GB’s Batch Reporting, Repetitive Queries Ad Hoc Queries Data Analysis/Mining Near Real Time Data Feeds Daily Load Weekly Load Strategic, Tactical Strategic Strategic, Tactical Loads Strategic, Tactical Loads, SLA “Query Freedom“ “Query complexity“ “Data Freshness” “Query Data Volume“ “Query Concurrency“ “Mixed Workload” “Schema Sophistication“ “Data Volume” DW SCALABILITY SPIDER CHART MPP – Multidimensional Scalability SMP – Tunable in one dimension on cost of other dimensions The spiderweb depicts important attributes to consider when evaluating Data Warehousing options. Big Data support is newest dimension.
  • 50.
    Summary • We livein an increasingly data-intensive world • Much of the data stored online and analyzed today is more varied than the data stored in recent years • More of our data arrives in near-real time • “Data is the new currency!” This present a large business opportunity. Are you ready for it?
  • 51.
    Other Related Presentations Building a Big Data Solution  Choosing technologies for a big data solution in the cloud  How does Microsoft solve Big Data?  Benefits of the Azure cloud  Should I move my database to the cloud?  Implement SQL Server on a Azure VM  Relational databases vs Non-relational databases  Introduction to Microsoft’s Hadoop solution (HDInsight)  Introducing Azure SQL Database  Introducing Azure SQL Data Warehouse Visit my blog at: JamesSerra.com (where these slide decks are posted under the “Presentation” tab)
  • 52.
    Resources  Why usea data lake? http://bit.ly/1WDy848  Big Data Architectures http://bit.ly/1RBbAbS  The Modern Data Warehouse: http://bit.ly/1xuX4Py  Hadoop and Data Warehouses: http://bit.ly/1xuXfu9
  • 53.
    Q & A? James Serra, Big Data Evangelist Email me at: JamesSerra3@gmail.com Follow me at: @JamesSerra Link to me at: www.linkedin.com/in/JamesSerra Visit my blog at: JamesSerra.com (where this slide deck is posted under the “Presentations” tab)