Lynn Langit, profile picture
Uploaded byLynn Langit
4,697 views

Beyond Relational

The document discusses building data pipelines in the cloud. It covers serverless data pipeline patterns using services like BigQuery, Cloud Storage, Cloud Dataflow, and Cloud Pub/Sub. It also compares Cloud Dataflow and Cloud Dataproc for ETL workflows. Key questions around ingestion and ETL are discussed, focusing on volume, variety, velocity and veracity of data. Cloud vendor offerings for streaming and ETL are also compared.

Embed presentation

Downloaded 94 times
1 / 144
2 / 144
3 / 144
4 / 144
5 / 144
6 / 144
7 / 144
8 / 144
9 / 144
10 / 144
11 / 144
12 / 144
13 / 144
14 / 144
15 / 144
16 / 144
17 / 144
18 / 144
19 / 144
20 / 144
21 / 144
22 / 144
23 / 144
24 / 144
25 / 144
26 / 144
27 / 144
28 / 144
29 / 144
30 / 144
31 / 144
32 / 144
33 / 144
34 / 144
35 / 144
36 / 144
37 / 144
38 / 144
39 / 144
40 / 144
41 / 144
42 / 144
43 / 144
44 / 144
45 / 144
46 / 144
47 / 144
48 / 144
49 / 144
50 / 144
51 / 144
52 / 144
53 / 144
54 / 144
55 / 144
56 / 144
57 / 144
58 / 144
59 / 144
60 / 144
61 / 144
62 / 144
63 / 144
64 / 144
65 / 144
66 / 144
67 / 144
68 / 144
69 / 144
70 / 144
71 / 144
72 / 144
73 / 144
74 / 144
75 / 144
76 / 144
77 / 144
78 / 144
79 / 144
80 / 144
81 / 144
82 / 144
83 / 144
84 / 144
85 / 144
86 / 144
87 / 144
88 / 144
89 / 144
90 / 144
91 / 144
92 / 144
93 / 144
94 / 144
95 / 144
96 / 144
97 / 144
98 / 144
99 / 144
100 / 144
101 / 144
102 / 144
103 / 144
104 / 144
105 / 144
106 / 144
107 / 144
108 / 144
109 / 144
110 / 144
111 / 144
112 / 144
113 / 144
114 / 144
115 / 144
116 / 144
117 / 144
118 / 144
119 / 144
120 / 144
121 / 144
122 / 144
123 / 144
124 / 144
125 / 144
126 / 144
127 / 144
128 / 144
129 / 144
130 / 144
131 / 144
132 / 144
133 / 144
134 / 144
135 / 144
136 / 144
137 / 144
138 / 144
139 / 144
140 / 144
141 / 144
142 / 144
143 / 144
144 / 144
Beyond Relational: Applying Big Data Cloud Patterns Lynn Langit QCon London – March 2017
About this Workshop Learning from Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
1. Understand the Landscape
History
‘Old’ Big Data
Save ALL of your Data (in a Data Lake)
But NOT in HDFS…sorry Cloudera What is a Cloud-based Data Lake Architecture?
AWS Data Lake Architecture
Disruptor: Data Lake on GCP ✘Well-priced,Highly Available,Auto-scaled ✘Simple, One API, NoOps ✘SQL ‘over’ it via Query Services
Practice Applying Concepts – Data Lake Features (AWS or GCP)
“Finding a path -- the ACTUAL Cost of… ✘ Saving all Data into a Data Lake ✘ Using newer database technologies ✘ Going beyond Relational
About this Workshop Learning from Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
2. Evaluate Cloud Data Severs & Services
Choice… is good, right?
Past -- Workloads and Terms Example Products Other Names Operational Database MySQL, SQL Server, Oracle OTLP Analytic Database SSAS, Cognos, SAS, OLAP NoSQL Database MongoDB, Redis, Cassandra, Neo4J Document, K/V, Graph, Column-store Hadoop Cloudera, Hortonworks Big Data
??? The fastest growing cloud-basedBig Data products are…
Relational The fastest growing cloud-basedBig Data products are…
“When do I use…? ✘Hadoop / Spark ✘NoSQL ✘Big Relational
“Existing relational databases to the cloud? Operational, Data Warehouse
Prerequisite – move / store files in the cloud – prefer Data Lake to HDFS
Cloud Vendors – Files as Data Lake AWS ✘ S3/Glacier ✘ Most complete offering ✘ Most mature offering ✘ Notable: S3 management GCP ✘ GCS with 4 types ✘ Lean, mean and cheap ✘ Unified API ✘ Notable: Simple pricing
Serverless Do you need to manage servers?
Place your screenshot here AWS Console 20 Data services
Place your screenshot here GCP Console 11 Data Services
GCP Serverless Data Warehouse Reference table Query / Compute BigQuery Customer Lists / Reference Data Export Ad Data Cloud Storage Id matching Cloud Dataflow Marketing List DoubleClick Campaign Manager Google Analytics Relevant Users Cloud Storage Analysts DataStudio 360 Dashboards
BigQuery ✘ Mature– based on Dremel (Google) ✘ Import (batch or stream) from GCS++ ✘ ANSI SQL queries ✘ Interactive or batch pricing ✘ NoOps / Serverless Head-to-Head Data Warehouse -- GCP BigQuery vs. AWS Redshift Redshift ✘ Mature – big market share ✘ Big MySQL – ANSI SQL query ✘ PostgreSQL dialect ✘ Cost compare? ✘ Manual partition/scale up ✘ Huge partner ecosystem (ETL, viz)
Disruptor: AWS Athena ✘Well-priced ✘Highly Available ✘Auto-scaled ✘Simple ✘NoOps ✘Analytics ✘NoSQL
Practice Applying Concepts – Serverless SQL Queries (AWS Aurora or GCP Big Query)
BigQuery ✘ Mature – based on Dremel (Google) ✘ Import (batch or stream) from GCS++ ✘ ANSI SQL queries ✘ Interactive or batch pricing (streaming) ✘ 3rd party vendor integration (ETL, viz) Head-to-Head SQL Query -- GCP BigQuery vs. AWS Athena Athena ✘ New – based on Apache Presto (from FB) ✘ SQL query against S3 data ✘ Cost compare - $ 5/TB scanned
36 What is Spanner?
Disruptor: GCP Cloud Spanner ✘Well-priced ✘Highly Available ✘Auto-scaled ✘Transactional ✘NoOps ✘Operational ✘RDBMS
Aurora ✘ Mature ✘ ANSI SQL queries ✘ MySQL compliant ✘ 3rd party vendor integration (ETL, viz) Head-to-Head Relational Database – AWS Aurora vs. GCP Spanner Spanner ✘ New (beta) ✘ ANSI SQL queries ✘ CAP – auto scales, transactional consistent, auto partitions ✘ Cost compare?
Cloud Vendors RDBMS Database (Operational or Analytics) AWS ✘ 14X market share of next competitor ✘ Most complete offering ✘ Most mature offering - partners ✘ Notable: Aurora, Big Relational RDBMS GCP ✘ Lean, mean and cheap ✘ Fastest player ✘ Requires top developers ✘ Notable: BigQuery, MPP SQL Query as a Service
Cloud Offerings – RDBMS and Files AWS Google Managed RDBMS RDS Aurora RDS <Other> Spanner Cloud SQL Data Warehouse Redshift Athena* BigQuery* NoSQL buckets S3* Glacier* Cloud Storage (Multi-regional, Regional)* Nearline, Coldline* *Serverless services
Practice Applying Concepts – Big Relational
Reasons to use Big Relational Cloud Services Developers DevOps Cloud Vendors – AWS Developers DevOps Cloud Vendors – GCP
Reasons to use Big Relational Cloud Services Developers ✘ Most know RDBMS query patterns ✘ Many know basic administration DevOps ✘ Most know RDBMS administration ✘ Many know basic RDBMS queries ✘ Many know query optimization Cloud Vendors - AWS ✘ Aurora – RDBMS up to 64 TB ✘ Redshift - $ 1k USD / 1 TB / year ✘ Rich partner ecosystem – ETL ✘ Integration with AWS products Developers ✘ Most know coding language patterns to interact with RDBMS systems DevOps ✘ Familiar RDBMS security patterns ✘ Familiar auditing ✘ Partner tooling integration Cloud Vendors - GCP ✘ Spanner / Big Query – familiar SQL queries ✘ No hassle streaming ingest ✘ No hassle pay-as-you-go ✘ NoOps administration
My past top Big Data Cloud Services
“When do I use…? ✘Hadoop ✘NoSQL ✘Big Relational
225 NoSQL Database Types to Choose From
Let’s review some NoSQL concepts Key-Value Redis, Riak, Aerospike Graph Neo4j Document MongoDB Wide-Column Cassandra, HBase
“
Key Questions - Storage ✘Volume – how much now, what growth rate? ✘Variety – what type(s) of data? ‘rectangular’,‘graph’,‘k-v’, etc… ✘Velocity – batches, streams,both, what ingest rate? ✘Veracity – current state (quality) of data, amount of duplicationof data stores, existence of authoritative (master) data management?
Actual Questions - NoSQL ✘When do you use Redis for caching? ✘When do you use MongoDB or AWS DynamoDB for JSON data? ✘When do you use Neo4j for graph-type data? ✘Why are you using somethingelse, i.e. Cassandra, Riak, etc..?
51 ✘Open Source is Free ✘Not Free § Rapid iteration, innovation § Can start up for free (on premise) § Can ‘rent’ for cheap or free on the cloud § Can use with the command line for free § Some vendors offer free online training § Ex. www.neo4j.org § Constant releases § Can be deceptively hard to set up (time is money) § Don’t forget to turn it off if on the cloud! § GUI tools, support, training cost $$$ § Ex. www.neo4j.com NoSQL Example
Practice Applying Concepts - NoSQL
NoSQL Applied Log Files • ??? Product Catalogs • ??? Social Games • ??? Social aggregators • ??? Line-of- Business • ???
NoSQL Applied Log Files • Columnstore • HBase Product Catalogs • Key/Value • Redis Social Games • Document • MongoDB Social aggregators • Graph • Neo4j Line-of- Business • RDBMS • SQL Server
More than NoSQL NoSQL ✘ Non-relational ✘ Can be optimized in-memory ✘ Eventually consistent ✘ Schema on Read ✘ Example: Aerospike NewSQL ✘ Relational plus more ✘ Often in-memory ✘ Some kind of SQL-layer ✘ Schema on Write ✘ Example: MemSQL U-SQL ✘ What??? ✘ Microsoft’s universal SQL language ✘ Example: Azure Data Lake
Focus
How Best to Store your Data? Complexity Scalability Developer Cost RDBMS easy medium* low NoSQL medium big high *GCP Cloud Spanner is the exception here
Practice Applying Concepts – Real Cost of Storage Types
Cloud NoSQL Applied – AWS Log Files Product Catalogs Social Games Social aggregators Line-of- Business
Cloud NoSQL Applied – AWS Log Files • Stream or Hadoop • Kinesis or EMR Product Catalogs • Key/Value • DynamoDB Social Games • Document • MongoDB Social aggregators • Graph • Neo4j Line-of- Business • RDBMS • RDS
Cloud NoSQL Applied – GCP Log Files Product Catalogs Social Games Social aggregators Line-of- Business
Cloud NoSQL Applied – GCP Log Files • Pub/Sub • BigTable Product Catalogs • Key/Value • DataStore Social Games • Document • Spanner Social aggregators • Graph • Neo4j Line-of- Business • RDBMS • Spanner
Place your screenshot here AWS Console 20 Data services
Place your screenshot here GCP Console 11 Data Services
Cloud Offerings – Big Relational and NoSQL AWS Google Managed RDBMS RDS Aurora Spanner Cloud SQL Data Warehouse Redshift or Athena BigQuery NoSQL buckets S3 Glacier Cloud Storage Nearline NoSQL Key-Value NoSQL Wide Column DynamoDB Big Table Cloud Datastore NoSQL Document NoSQL Graph MongoDB on EC2 Neo4j on EC2 MongoDB on GCE Neo4j on GCE
“When do I use…? ✘Hadoop / Spark ✘NoSQL ✘Big Relational
Size Matters
One Vendor’s View I don’t Want Text here
Where is Hadoop / Spark Used?
“What is Hadoop Spark? ✘ In-memory ✘ Distributed ✘ Integrated with other libraries
Spark is Fast, Unified and Usable
Disruptor: DataBricks ✘Managed Apache Spark on AWS
Practice Applying Concepts – Apache Spark via Databricks
Hadoop is becoming Spark… Volume ✘10 TB or greater to start ✘Growth of 25% YOY ✘Where FROM ✘Where TO Velocity and Variety ✘Spark over Hive ✘Kafka and Samsa Veracity ✘Pay, train and hire team ✘Top $$$ for talent, IF you can find it ✘WATCH OUT for Cloud Vendors who promise ‘easy access’ ✘Complexity of ecosystem ✘DataBricks knows best for Spark ✘Cloudera knows best for on premise
How Best to Store your Data? Complexity Scalability Developer Cost RDBMS easy medium* low NoSQL medium big high Hadoop hard huge very high
Cloud Offerings – Big Data AWS Google Managed RDBMS RDS Aurora Spanner Cloud SQL Data Warehouse Redshift or Athena BigQuery NoSQL buckets S3 Glacier Cloud Storage Nearline NoSQL Key-Value NoSQL Wide Column DynamoDB Big Table Cloud Datastore NoSQL Document NoSQL Graph MongoDB on EC2 Neo4j on EC2 MongoDB on GCE Neo4j on GCE Hadoop / Spark Elastic MapReduce DataProc
Real World Big Data -- When do I use what? RDBMS 65% NoSQL 30% Hadoop 5%
ETL is 75% of all Big Data Projects Surveying, cleaning and loading data is the majorityof the billable time for new Big Data projects.
About this Workshop Learning from Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
3. Construct Data Pipelines Build vs. Buy
Kappa Architecture on the Cloud
Pattern ✘How to build optimized cloud-based data pipelines? -- Cloud-based ETL tools and processes -- includes load-testingpatterns and securitypractices -- including connectingbetween different vendor clouds
Key Questions – Ingestion and ETL ✘Volume – how much and how fast,now and future? ✘Variety – what type(s) or data, any pre-processingneeded? ✘Velocity – batches or steaming? ✘Veracity – verificationon ingest needed? new data needed?
Actual Questions – Data Ingest and ETL ✘When (at what data volume) do you need AWS Snowball and/or AWS Direct Connect for ingest? ✘What does ‘near-real time streaming’ exactlymean? ✘When do you need to use a product vs. a libraryor service for ETL? ✘Which 3rd party partner products are available for the Data services you are using? ✘What are your actual data qualityissues?
Serverless Is this a cloud data pipeline patternnow?
Serverless Time Series Analysis Batch Storage BigQuery Storage Cloud Storage Time Series Processing Cloud Dataflow Analysis Cloud Datalab Storage Cloud Bigtable* Processing Cloud Dataproc Time Series Files Cloud Storage ML Cloud ML Streaming Time Series Streaming Cloud Pub/Sub *Note: Use Bigtable with NoSQL workloads of 1 TB or more
DataFlow ✘ Managed Apache Beam ✘ Elegant pipeline API ✘ Beam as a product -> GCP ✘ Scala? Java? Head-to-Head ETL – GCP DataFlow vs. GCP DataProc DataProc ✘ Managed Apache Spark ✘ Large user base ✘ Spark as a product -> DataBricks ✘ Cost compare?
“Considering… ✘Initial Load/Transform ✘Data Quality ✘Batch vs. Stream
Pipeline Phases Phase 0 Eval Current Data - Quality & Quantity Phase 1 Get New Data - Free or Premium Phase 2 Build MVP & Forecast volume and growth Phase 3 Load test at scale Phase 4 Deploy – secure, audit and monitor
Cloud Big Data Vendors - ETL AWS ✘ 14X market share of next competitor ✘ Notable: Many, strong ETL Partners GCP ✘ Lean, mean and cheap ✘ Fastest player / Serverless ✘ Notable: DataFlow requires Java or Python developers
How Best to Ingest and ETL your Data? Complexity Scalability Developer Cost RDBMS medium medium low NoSQL medium big high Hadoop hard huge very high
“Considering… ✘Initial Load/Transform ✘Data Quality ✘Batch vs. Stream
Building a Streaming Pipeline Stream Interval Window
Key Questions - Streaming ✘Volume – how much data now and predictedover next 12 months? ✘Variety – what types of data now and future? ✘Velocity – volume of input data / time now and near future? ✘Veracity – volume of EXISTING data now
Cloud Big Data Vendors - Streaming AWS ✘ 14X market share of next competitor ✘ Most complete offering ✘ Most mature offering ✘ Notable: Kinesis/Firehose GCP ✘ Lean, mean and cheap ✘ Fastest player ✘ Requires top developers ✘ Notable: DataFlow flexible
Cloud Offerings – Data and Pipelines AWS Google Managed RDBMS RDS Aurora Spanner / Cloud SQL Data Warehouse Redshift/ Athena BigQuery NoSQL buckets S3 Glacier Cloud Storage Nearline NoSQL Key-Value NoSQL Wide Column DynamoDB Big Table Cloud Datastore Streaming or ML Kinesis AWS Machine Learning Pub/Sub or DataFlow Google Machine Learning NoSQL Document NoSQL Graph MongoDB on EC2 Neo4j on EC2 MongoDB on GCE Neo4j on GCE Hadoop / Spark Elastic MapReduce DataProc Cloud ETL Data Pipelines DataFlow
How Best to Stream your Data? Complexity Scalability Developer Cost Micro-Batches easy medium low Stream Windows difficult big high Near Real-time very difficult huge high
Practice Applying Concepts
Designing Streaming Cloud Data Pipelines Log Files Product Catalogs Social Games Social aggregators Line-of- Business
Data Processing Financial Services > Monte Carlo Simulations Bespoke Apps Compute Engine Storage/Analysis BigQuery Storage Cloud Storage Dataflow/Beam Cloud Dataflow Visualization Cloud Datalab Storage Cloud Bigtable Hadoop/Spark Cloud Dataproc
Batch Financial Services > Time Series Analysis Storage BigQuery Storage Cloud Storage Time Series Processing Cloud Dataflow Analysis Cloud Datalab Storage Cloud Bigtable Processing Cloud Dataproc Time Series Files Cloud Storage ML Cloud ML Streaming Time Series Streaming Cloud Pub/Sub
Private Datasets Public Datasets Life Sciences > Variant Analysis MSSNG Autism Cloud Storage Scientist High Throughput Genome Sequencers 1000 Genomes Cloud Storage Patient Data Cloud Storage Illumina Platform Cloud Storage Ref Genomes Cloud Storage TCGA Cloud Storage Analytics Online Analytics BigQuery Batch Analytics Cloud Dataflow Lab Notebooks Cloud Datalab Data Ingest Genomics BAM FASTQ
Streaming Big Data > Complex Event Processing Cloud Apps Compute Engine Streaming Batch Push to Devices App Engine Rules Engine Cloud Dataflow Data Analysis Cloud Datalab Mobile Devices Push Notifications Report & Share Business Analysis Cloud Apps Compute Engine On-Premises Databases On-Premises Applications Processed Events Cloud Bigtable Events Time Series Data Warehouse BigQuery Execution Results Streaming Cloud Pub/Sub Transactions Processing Cloud Dataflow Transaction Streams Messaging Cloud Pub/Sub Rules Actions ETL Cloud Dataflow Transform Data Cloud Data Cloud Storage Rules Engine Cloud Dataproc
About this Workshop Learning from Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
4. Select ETL and Visualization Analytics and Presentation
Pattern ✘How best to Query and Visualize -- When to use business analytics vs. predictive analytics (machine learning) -- how best to present data to clients - partner visualizationproducts or roll your own
Making Sense of Data Machine Learning Reports Presentation
Actual Questions - Query ✘What query skills do the Dev / DevOps teams have? ✘Do we use an ORM? ✘Are transactions needed? ✘Which RDBMs features are we using now? i.e. PL-SQL, indexing… ✘Which language(s) do our analysts use? Excel, R, Python?
Graphs What is nature of your questions?
Practice Applying Concepts – Trying out Cypher on Neo4j
Cloud Big Data Vendors - Query AWS ✘ 5X market share of next competitor ✘ Most complete offering ✘ Most mature offering ✘ Notable: Big Relational GCP ✘ Lean, mean and cheap ✘ Fastest player ✘ Notable: Flexible, powerful machine learning
Query Languages SQL Everyone knows it But how well do they know it? NoSQL Vendor Language Too many to list How will you learn it? Cypher Query language for graph databases The future? ORM Good, bad or horrible? Again, how well do they know it? Spark Scala or Python (functional-style) Machine Learning Queries SciPy, NumPy or Python R Language Julie Language Many more…
Practice Applying Concepts – Understanding Query Language Costs
How Best to Query your Data? Business Analytics Predictive Analytics Developer Cost RDBMS NoSQL Hadoop
How Best to Query your Data? Business Analytics Predictive Analytics Developer Cost RDBMS easy medium low NoSQL hard very hard very high Hadoop hard hard very high
Machine Learning aka Predictive Analytics AWS ✘ ML for developers ✘ GUI-based – regression/classification only ✘ MxNet for deep neural networks, works best with GPUs GCP ✘ ML for ML developers ✘ Scaled or specialized ML models ✘ Many model types – vision, text, translate, speech, jobs, video ✘ TensorFlow for neural networks, can use GPUs
Presentation If you can’t see it, it’s not worth it.
Dashboards ✘ More than KPIs ✘ Mobile ✘ Alerts ✘ Data Stories Innovation in Data Visualization Reports ✘ Level of Detail ✘ Meaningful Taxonomies ✘ Fast enough ✘ Drill for Data
D3 The language of Data Visualization
Cloud Big Data Vendors - Visualization AWS ✘ Most complete offering ✘ Notable: Partners & QuickSight GCP ✘ Big Query Partners ✘ Notable: 360 product - New Dashboards, iPython-style notebooks
About this Workshop Learning from Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
5. Review End-to-end Patterns For major cloud vendors
Big Data > Time Series Analysis Batch Storage BigQuery Storage Cloud Storage Time Series Processing Cloud Dataflow Analysis Cloud Datalab Storage Cloud Bigtable* Processing Cloud Dataproc Time Series Files Cloud Storage ML Cloud ML Streaming Time Series Streaming Cloud Pub/Sub *Note: Use Bigtable with NoSQL workloads of 1 TB or more
Big Data > Bioinformatics *Note: Use Bigtable with NoSQL workloads of 1 TB or more
Azure Patterns
About this Workshop Learning from Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
6. Cloud IoT Patterns Internet of Things
Place your screenshot here Data Generation Device
IoT is Big Data Realized
235,000,000,000 $The IoT Market 2017By the year 20 Billion devicesAnd a lot of users
IoT all the Things
139 Ingest Pipelines Storage Analytics Application & Presentation Standard Devices HTTPS Constrained Devices Non-TCP e.g. BLE Gateway Internet of Things > Sensor stream ingest and processing App Engine Container Engine Cloud Storage Cloud Pub/Sub Cloud Dataflow Monitoring Logging Cloud Dataflow Cloud Datastore Cloud Bigtable BigQuery Cloud Dataproc Cloud Datalab Compute Engine
IoT
Cloud Big Data Vendors - IoT AWS ✘ First to market ✘ Most complete offering ✘ Most mature offering ✘ Notable: MQTT support, AWS IoT Rules GCP ✘ Still in Beta / Android Things ✘ Fastest player ✘ Requires top developers ✘ Notable: Android Things, Weave, Nest
“The Future is Functional” @LynnLangit
The Next Generation…

More Related Content

PPTX
New AWS Services for Bioinformatics
byLynn Langit
 
PPTX
AWS for the Data Professional
byLynn Langit
 
PPTX
Scaling Galaxy on Google Cloud Platform
byLynn Langit
 
PPTX
Serverless Reality
byLynn Langit
 
PPTX
SQL Server on Google Cloud Platform
byLynn Langit
 
PDF
Logging infrastructure for Microservices using StreamSets Data Collector
byCask Data
 
PPTX
Benchmarking Aerospike on the Google Cloud - NoSQL Speed with Ease
byLynn Langit
 
PPT
Building a data warehouse with AWS Redshift, Matillion and Yellowfin
byLynn Langit
 
New AWS Services for Bioinformatics
byLynn Langit
 
AWS for the Data Professional
byLynn Langit
 
Scaling Galaxy on Google Cloud Platform
byLynn Langit
 
Serverless Reality
byLynn Langit
 
SQL Server on Google Cloud Platform
byLynn Langit
 
Logging infrastructure for Microservices using StreamSets Data Collector
byCask Data
 
Benchmarking Aerospike on the Google Cloud - NoSQL Speed with Ease
byLynn Langit
 
Building a data warehouse with AWS Redshift, Matillion and Yellowfin
byLynn Langit
 

What's hot

PDF
AWS re:Invent 2016 Recap: What Happened, What It Means
byRightScale
 
PPTX
Cloud computing by Google Cloud Platform - Presentation
byTinarivosoaAbaniaina
 
PDF
AWS Glue - let's get stuck in!
byChris Taylor
 
PDF
Next Generation Cloud Computing With Google - RightScale Compute 2013
byRightScale
 
PPTX
Bleeding Edge Databases
byLynn Langit
 
PDF
Getting Started on Google Cloud Platform
byAaron Taylor
 
PPTX
Cloud native data platform
byLi Gao
 
PPTX
Scaling Traffic from 0 to 139 Million Unique Visitors
byYelp Engineering
 
AWS re:Invent 2016 Recap: What Happened, What It Means
byRightScale
 
Cloud computing by Google Cloud Platform - Presentation
byTinarivosoaAbaniaina
 
AWS Glue - let's get stuck in!
byChris Taylor
 
Next Generation Cloud Computing With Google - RightScale Compute 2013
byRightScale
 
Bleeding Edge Databases
byLynn Langit
 
Getting Started on Google Cloud Platform
byAaron Taylor
 
Cloud native data platform
byLi Gao
 
Scaling Traffic from 0 to 139 Million Unique Visitors
byYelp Engineering
 

Viewers also liked

PPTX
Kafka & Couchbase Integration Patterns
byManuel Hurtado
 
PPTX
A Passionate Teacher is a Great Teacher
byJames Kiger
 
PDF
Klohn Crippen Berger 2017
byHartley Facultad
 
PPTX
El rol del docente
byEvelin Guadalupe
 
PDF
Living, Learning, & Working Smart @KKU
byDenpong Soodphakdee
 
PPTX
России
byLilit Sahakyan
 
PDF
Продвижение в Instagram для регионального бизнеса
byGusarov Group
 
PDF
St nar-40-ebook
byHarrizul Rivai
 
PDF
Mindful mvp 1_preso
byWilson Mullen
 
DOCX
7820303 jorge parra
by2334sdf
 
PPS
2° Domingo de Cuaresma 2017.
byPadre Enrique Camargo
 
PDF
Participantes fecicam
bymiciudadreal
 
PDF
mindful_preso2
byWilson Mullen
 
PDF
Propuesta Técnica de Diálogo
byCarlos Oblitas
 
PPTX
Diagnostico organizacional
byMilton Villalba
 
PPS
2° Domingo de Cuaresma 2017.
byPadre Enrique Camargo
 
PDF
Book reviews (sixteen) from AJL Reviews 2011 to 2017
byNathan Rosen
 
PPT
slideshare
byyulia-pow
 
PPTX
Survey Monkey demographic analysis
byRubio Luis
 
ODP
Chiara giacomini marco plantamura
bysecondag dicambio
 
Kafka & Couchbase Integration Patterns
byManuel Hurtado
 
A Passionate Teacher is a Great Teacher
byJames Kiger
 
Klohn Crippen Berger 2017
byHartley Facultad
 
El rol del docente
byEvelin Guadalupe
 
Living, Learning, & Working Smart @KKU
byDenpong Soodphakdee
 
России
byLilit Sahakyan
 
Продвижение в Instagram для регионального бизнеса
byGusarov Group
 
St nar-40-ebook
byHarrizul Rivai
 
Mindful mvp 1_preso
byWilson Mullen
 
7820303 jorge parra
by2334sdf
 
2° Domingo de Cuaresma 2017.
byPadre Enrique Camargo
 
Participantes fecicam
bymiciudadreal
 
mindful_preso2
byWilson Mullen
 
Propuesta Técnica de Diálogo
byCarlos Oblitas
 
Diagnostico organizacional
byMilton Villalba
 
2° Domingo de Cuaresma 2017.
byPadre Enrique Camargo
 
Book reviews (sixteen) from AJL Reviews 2011 to 2017
byNathan Rosen
 
slideshare
byyulia-pow
 
Survey Monkey demographic analysis
byRubio Luis
 
Chiara giacomini marco plantamura
bysecondag dicambio
 

Similar to Beyond Relational

PDF
Data Engineering Interview Questions & Answers Cloud Data Stacks (AWS, Azure,...
byMindbox Trainings
 
PPTX
Database Choices
byLynn Langit
 
PDF
GCP DataServices Consolidated information
bySrinivasulu Paranduru
 
PPTX
Choosing technologies for a big data solution in the cloud
byJames Serra
 
PDF
Data Platform on GCP
byPatrick Alexander
 
PPT
SQL, NoSQL, BigData in Data Architecture
byVenu Anuganti
 
PDF
GCP Data Engineer cheatsheet
byGuang Xu
 
PPTX
Big Data Infrastructure and Hadoop components.pptx
byGEZWARDGERALD
 
PPTX
Introduction to Big Data
byVipin Batra
 
PDF
Big Data Ecosystem
byLucian Neghina
 
PPTX
Big data presentation
byChinh Vo Wili
 
PDF
Cloud Big Data Architectures
byLynn Langit
 
PPTX
Not only SQL - Database Choices
byLynn Langit
 
PPTX
AWS for Big Data Experts
byLynn Langit
 
PDF
Design Choices for Cloud Data Platforms
byAshish Mrig
 
PPTX
NoSQL for the SQL Server Pro
byLynn Langit
 
PPTX
Essential Data Engineering for Data Scientist
bySoftServe
 
PDF
SQL, NoSQL, Distributed SQL: Choose your DataStore carefully
byMd Kamaruzzaman
 
PDF
Big data Intro by Kaushik Dutta
byKaushik Dutta
 
PPTX
Column Stores and Google BigQuery
byCsaba Toth
 
Data Engineering Interview Questions & Answers Cloud Data Stacks (AWS, Azure,...
byMindbox Trainings
 
Database Choices
byLynn Langit
 
GCP DataServices Consolidated information
bySrinivasulu Paranduru
 
Choosing technologies for a big data solution in the cloud
byJames Serra
 
Data Platform on GCP
byPatrick Alexander
 
SQL, NoSQL, BigData in Data Architecture
byVenu Anuganti
 
GCP Data Engineer cheatsheet
byGuang Xu
 
Big Data Infrastructure and Hadoop components.pptx
byGEZWARDGERALD
 
Introduction to Big Data
byVipin Batra
 
Big Data Ecosystem
byLucian Neghina
 
Big data presentation
byChinh Vo Wili
 
Cloud Big Data Architectures
byLynn Langit
 
Not only SQL - Database Choices
byLynn Langit
 
AWS for Big Data Experts
byLynn Langit
 
Design Choices for Cloud Data Platforms
byAshish Mrig
 
NoSQL for the SQL Server Pro
byLynn Langit
 
Essential Data Engineering for Data Scientist
bySoftServe
 
SQL, NoSQL, Distributed SQL: Choose your DataStore carefully
byMd Kamaruzzaman
 
Big data Intro by Kaushik Dutta
byKaushik Dutta
 
Column Stores and Google BigQuery
byCsaba Toth
 

More from Lynn Langit

PPTX
Google Cloud and Data Pipeline Patterns
byLynn Langit
 
PPTX
Redis Labs and SQL Server
byLynn Langit
 
PPTX
Blastn plus jupyter on Docker
byLynn Langit
 
PPTX
Serverless Architectures
byLynn Langit
 
PPTX
Cloud-centric Internet of Things
byLynn Langit
 
PPTX
Understanding Jupyter notebooks using bioinformatics examples
byLynn Langit
 
PPTX
Bioinformatics Data Pipelines built by CSIRO on AWS
byLynn Langit
 
PPTX
Genomic Scale Big Data Pipelines
byLynn Langit
 
PDF
Testing in Ballerina Language
byLynn Langit
 
PPTX
VariantSpark - a Spark library for genomics
byLynn Langit
 
PPTX
Teaching Kids Programming
byLynn Langit
 
PPTX
Practical Cloud
byLynn Langit
 
PPTX
Practical cloud
byLynn Langit
 
PPTX
Serverless Reality
byLynn Langit
 
PPTX
VariantSpark on AWS
byLynn Langit
 
PPTX
Genome-scale Big Data Pipelines
byLynn Langit
 
PPTX
Teaching Kids to create Alexa Skills
byLynn Langit
 
PPTX
10+ Years of Teaching Kids Programming
byLynn Langit
 
PPTX
Teaching Kids Programming for Developers
byLynn Langit
 
PPTX
What is 'Teaching Kids Programming'
byLynn Langit
 
Google Cloud and Data Pipeline Patterns
byLynn Langit
 
Redis Labs and SQL Server
byLynn Langit
 
Blastn plus jupyter on Docker
byLynn Langit
 
Serverless Architectures
byLynn Langit
 
Cloud-centric Internet of Things
byLynn Langit
 
Understanding Jupyter notebooks using bioinformatics examples
byLynn Langit
 
Bioinformatics Data Pipelines built by CSIRO on AWS
byLynn Langit
 
Genomic Scale Big Data Pipelines
byLynn Langit
 
Testing in Ballerina Language
byLynn Langit
 
VariantSpark - a Spark library for genomics
byLynn Langit
 
Teaching Kids Programming
byLynn Langit
 
Practical Cloud
byLynn Langit
 
Practical cloud
byLynn Langit
 
Serverless Reality
byLynn Langit
 
VariantSpark on AWS
byLynn Langit
 
Genome-scale Big Data Pipelines
byLynn Langit
 
Teaching Kids to create Alexa Skills
byLynn Langit
 
10+ Years of Teaching Kids Programming
byLynn Langit
 
Teaching Kids Programming for Developers
byLynn Langit
 
What is 'Teaching Kids Programming'
byLynn Langit
 

Recently uploaded

PPTX
Cloud-and-AI-Platform-FY26-Partner-Playbook.pptx
byiuiuiuiu1
 
PDF
Day 1 - Cloud Security Strategy and Planning ~ 2nd Sight Lab ~ Cloud Security...
by2nd Sight Lab
 
PDF
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
PDF
Day 3 - Data and Application Security - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
PPTX
Kanban India 2025 | Daksh Gupta | Modeling the Models, Generative AI & Kanban
byLeanKanbanIndia
 
PDF
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
PDF
ElyriaSoftware — Powering the Future with Blockchain Innovation
byElyria Software
 
PDF
Day 5 - Red Team + Blue Team in the Cloud - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
PDF
Making Sense of Raster: From Bit Depth to Better Workflows
bySafe Software
 
PDF
Session 1 - Solving Semi-Structured Documents with Document Understanding
byDianaGray10
 
PDF
Digit Expo 2025 - EICC Edinburgh 27th November
byRay Bugg
 
PDF
DevFest El Jadida 2025 - Product Thinking
byElmehdi AMLOU
 
PDF
API-First Architecture in Financial Systems
bycyy111112
 
PPTX
Software Analysis &Design ethiopia chap-2.pptx
byAbbas484771
 
PDF
Usage Control for Process Discovery through a Trusted Execution Environment
byValerioGoretti
 
PDF
Decoding the DNA: The Digital Networks Act, the Open Internet, and IP interco...
byCSUC - Consorci de Serveis Universitaris de Catalunya
 
PPTX
Building Cyber Resilience for 2026: Best Practices for a Secure, AI-Driven Bu...
byYasir Naveed Riaz
 
PPTX
Cybersecurity Best Practices - Step by Step guidelines
byYasir Naveed Riaz
 
PPTX
Protecting Data in an AI Driven World - Cybersecurity in 2026
byYasir Naveed Riaz
 
PPTX
cybercrime in Information security .pptx
byismaeelsahib032
 
Cloud-and-AI-Platform-FY26-Partner-Playbook.pptx
byiuiuiuiu1
 
Day 1 - Cloud Security Strategy and Planning ~ 2nd Sight Lab ~ Cloud Security...
by2nd Sight Lab
 
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
Day 3 - Data and Application Security - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
Kanban India 2025 | Daksh Gupta | Modeling the Models, Generative AI & Kanban
byLeanKanbanIndia
 
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
ElyriaSoftware — Powering the Future with Blockchain Innovation
byElyria Software
 
Day 5 - Red Team + Blue Team in the Cloud - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
Making Sense of Raster: From Bit Depth to Better Workflows
bySafe Software
 
Session 1 - Solving Semi-Structured Documents with Document Understanding
byDianaGray10
 
Digit Expo 2025 - EICC Edinburgh 27th November
byRay Bugg
 
DevFest El Jadida 2025 - Product Thinking
byElmehdi AMLOU
 
API-First Architecture in Financial Systems
bycyy111112
 
Software Analysis &Design ethiopia chap-2.pptx
byAbbas484771
 
Usage Control for Process Discovery through a Trusted Execution Environment
byValerioGoretti
 
Decoding the DNA: The Digital Networks Act, the Open Internet, and IP interco...
byCSUC - Consorci de Serveis Universitaris de Catalunya
 
Building Cyber Resilience for 2026: Best Practices for a Secure, AI-Driven Bu...
byYasir Naveed Riaz
 
Cybersecurity Best Practices - Step by Step guidelines
byYasir Naveed Riaz
 
Protecting Data in an AI Driven World - Cybersecurity in 2026
byYasir Naveed Riaz
 
cybercrime in Information security .pptx
byismaeelsahib032
 

Beyond Relational

  • 1.
    Beyond Relational: Applying BigData Cloud Patterns Lynn Langit QCon London – March 2017
  • 2.
    About this Workshop Learningfrom Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
  • 3.
  • 4.
  • 5.
  • 8.
  • 9.
    But NOT inHDFS…sorry Cloudera What is a Cloud-based Data Lake Architecture?
  • 10.
    AWS Data LakeArchitecture
  • 12.
    Disruptor: Data Lakeon GCP ✘Well-priced,Highly Available,Auto-scaled ✘Simple, One API, NoOps ✘SQL ‘over’ it via Query Services
  • 14.
    Practice Applying Concepts –Data Lake Features (AWS or GCP)
  • 15.
    “Finding a path-- the ACTUAL Cost of… ✘ Saving all Data into a Data Lake ✘ Using newer database technologies ✘ Going beyond Relational
  • 16.
    About this Workshop Learningfrom Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
  • 17.
    2. Evaluate Cloud DataSevers & Services
  • 19.
  • 20.
    Past -- Workloadsand Terms Example Products Other Names Operational Database MySQL, SQL Server, Oracle OTLP Analytic Database SSAS, Cognos, SAS, OLAP NoSQL Database MongoDB, Redis, Cassandra, Neo4J Document, K/V, Graph, Column-store Hadoop Cloudera, Hortonworks Big Data
  • 22.
    ??? The fastest growingcloud-basedBig Data products are…
  • 23.
    Relational The fastest growingcloud-basedBig Data products are…
  • 24.
    “When do Iuse…? ✘Hadoop / Spark ✘NoSQL ✘Big Relational
  • 25.
    “Existing relational databases tothe cloud? Operational, Data Warehouse
  • 26.
    Prerequisite – move/ store files in the cloud – prefer Data Lake to HDFS
  • 27.
    Cloud Vendors –Files as Data Lake AWS ✘ S3/Glacier ✘ Most complete offering ✘ Most mature offering ✘ Notable: S3 management GCP ✘ GCS with 4 types ✘ Lean, mean and cheap ✘ Unified API ✘ Notable: Simple pricing
  • 28.
    Serverless Do you needto manage servers?
  • 29.
    Place your screenshothere AWS Console 20 Data services
  • 30.
    Place your screenshothere GCP Console 11 Data Services
  • 31.
    GCP Serverless DataWarehouse Reference table Query / Compute BigQuery Customer Lists / Reference Data Export Ad Data Cloud Storage Id matching Cloud Dataflow Marketing List DoubleClick Campaign Manager Google Analytics Relevant Users Cloud Storage Analysts DataStudio 360 Dashboards
  • 32.
    BigQuery ✘ Mature– basedon Dremel (Google) ✘ Import (batch or stream) from GCS++ ✘ ANSI SQL queries ✘ Interactive or batch pricing ✘ NoOps / Serverless Head-to-Head Data Warehouse -- GCP BigQuery vs. AWS Redshift Redshift ✘ Mature – big market share ✘ Big MySQL – ANSI SQL query ✘ PostgreSQL dialect ✘ Cost compare? ✘ Manual partition/scale up ✘ Huge partner ecosystem (ETL, viz)
  • 33.
  • 34.
    Practice Applying Concepts –Serverless SQL Queries (AWS Aurora or GCP Big Query)
  • 35.
    BigQuery ✘ Mature –based on Dremel (Google) ✘ Import (batch or stream) from GCS++ ✘ ANSI SQL queries ✘ Interactive or batch pricing (streaming) ✘ 3rd party vendor integration (ETL, viz) Head-to-Head SQL Query -- GCP BigQuery vs. AWS Athena Athena ✘ New – based on Apache Presto (from FB) ✘ SQL query against S3 data ✘ Cost compare - $ 5/TB scanned
  • 36.
  • 37.
    Disruptor: GCP Cloud Spanner ✘Well-priced ✘HighlyAvailable ✘Auto-scaled ✘Transactional ✘NoOps ✘Operational ✘RDBMS
  • 38.
    Aurora ✘ Mature ✘ ANSISQL queries ✘ MySQL compliant ✘ 3rd party vendor integration (ETL, viz) Head-to-Head Relational Database – AWS Aurora vs. GCP Spanner Spanner ✘ New (beta) ✘ ANSI SQL queries ✘ CAP – auto scales, transactional consistent, auto partitions ✘ Cost compare?
  • 39.
    Cloud Vendors RDBMS Database(Operational or Analytics) AWS ✘ 14X market share of next competitor ✘ Most complete offering ✘ Most mature offering - partners ✘ Notable: Aurora, Big Relational RDBMS GCP ✘ Lean, mean and cheap ✘ Fastest player ✘ Requires top developers ✘ Notable: BigQuery, MPP SQL Query as a Service
  • 40.
    Cloud Offerings –RDBMS and Files AWS Google Managed RDBMS RDS Aurora RDS <Other> Spanner Cloud SQL Data Warehouse Redshift Athena* BigQuery* NoSQL buckets S3* Glacier* Cloud Storage (Multi-regional, Regional)* Nearline, Coldline* *Serverless services
  • 41.
  • 42.
    Reasons to useBig Relational Cloud Services Developers DevOps Cloud Vendors – AWS Developers DevOps Cloud Vendors – GCP
  • 43.
    Reasons to useBig Relational Cloud Services Developers ✘ Most know RDBMS query patterns ✘ Many know basic administration DevOps ✘ Most know RDBMS administration ✘ Many know basic RDBMS queries ✘ Many know query optimization Cloud Vendors - AWS ✘ Aurora – RDBMS up to 64 TB ✘ Redshift - $ 1k USD / 1 TB / year ✘ Rich partner ecosystem – ETL ✘ Integration with AWS products Developers ✘ Most know coding language patterns to interact with RDBMS systems DevOps ✘ Familiar RDBMS security patterns ✘ Familiar auditing ✘ Partner tooling integration Cloud Vendors - GCP ✘ Spanner / Big Query – familiar SQL queries ✘ No hassle streaming ingest ✘ No hassle pay-as-you-go ✘ NoOps administration
  • 44.
    My past topBig Data Cloud Services
  • 45.
    “When do Iuse…? ✘Hadoop ✘NoSQL ✘Big Relational
  • 46.
  • 47.
    Let’s review someNoSQL concepts Key-Value Redis, Riak, Aerospike Graph Neo4j Document MongoDB Wide-Column Cassandra, HBase
  • 48.
  • 49.
    Key Questions -Storage ✘Volume – how much now, what growth rate? ✘Variety – what type(s) of data? ‘rectangular’,‘graph’,‘k-v’, etc… ✘Velocity – batches, streams,both, what ingest rate? ✘Veracity – current state (quality) of data, amount of duplicationof data stores, existence of authoritative (master) data management?
  • 50.
    Actual Questions -NoSQL ✘When do you use Redis for caching? ✘When do you use MongoDB or AWS DynamoDB for JSON data? ✘When do you use Neo4j for graph-type data? ✘Why are you using somethingelse, i.e. Cassandra, Riak, etc..?
  • 51.
    51 ✘Open Source isFree ✘Not Free § Rapid iteration, innovation § Can start up for free (on premise) § Can ‘rent’ for cheap or free on the cloud § Can use with the command line for free § Some vendors offer free online training § Ex. www.neo4j.org § Constant releases § Can be deceptively hard to set up (time is money) § Don’t forget to turn it off if on the cloud! § GUI tools, support, training cost $$$ § Ex. www.neo4j.com NoSQL Example
  • 52.
  • 53.
    NoSQL Applied Log Files •??? Product Catalogs • ??? Social Games • ??? Social aggregators • ??? Line-of- Business • ???
  • 54.
    NoSQL Applied Log Files •Columnstore • HBase Product Catalogs • Key/Value • Redis Social Games • Document • MongoDB Social aggregators • Graph • Neo4j Line-of- Business • RDBMS • SQL Server
  • 55.
    More than NoSQL NoSQL ✘Non-relational ✘ Can be optimized in-memory ✘ Eventually consistent ✘ Schema on Read ✘ Example: Aerospike NewSQL ✘ Relational plus more ✘ Often in-memory ✘ Some kind of SQL-layer ✘ Schema on Write ✘ Example: MemSQL U-SQL ✘ What??? ✘ Microsoft’s universal SQL language ✘ Example: Azure Data Lake
  • 56.
  • 57.
    How Best toStore your Data? Complexity Scalability Developer Cost RDBMS easy medium* low NoSQL medium big high *GCP Cloud Spanner is the exception here
  • 58.
    Practice Applying Concepts –Real Cost of Storage Types
  • 59.
    Cloud NoSQL Applied– AWS Log Files Product Catalogs Social Games Social aggregators Line-of- Business
  • 60.
    Cloud NoSQL Applied– AWS Log Files • Stream or Hadoop • Kinesis or EMR Product Catalogs • Key/Value • DynamoDB Social Games • Document • MongoDB Social aggregators • Graph • Neo4j Line-of- Business • RDBMS • RDS
  • 61.
    Cloud NoSQL Applied– GCP Log Files Product Catalogs Social Games Social aggregators Line-of- Business
  • 62.
    Cloud NoSQL Applied– GCP Log Files • Pub/Sub • BigTable Product Catalogs • Key/Value • DataStore Social Games • Document • Spanner Social aggregators • Graph • Neo4j Line-of- Business • RDBMS • Spanner
  • 63.
    Place your screenshothere AWS Console 20 Data services
  • 64.
    Place your screenshothere GCP Console 11 Data Services
  • 65.
    Cloud Offerings –Big Relational and NoSQL AWS Google Managed RDBMS RDS Aurora Spanner Cloud SQL Data Warehouse Redshift or Athena BigQuery NoSQL buckets S3 Glacier Cloud Storage Nearline NoSQL Key-Value NoSQL Wide Column DynamoDB Big Table Cloud Datastore NoSQL Document NoSQL Graph MongoDB on EC2 Neo4j on EC2 MongoDB on GCE Neo4j on GCE
  • 66.
    “When do Iuse…? ✘Hadoop / Spark ✘NoSQL ✘Big Relational
  • 67.
  • 69.
    One Vendor’s View Idon’t Want Text here
  • 71.
    Where is Hadoop/ Spark Used?
  • 72.
    “What is HadoopSpark? ✘ In-memory ✘ Distributed ✘ Integrated with other libraries
  • 74.
    Spark is Fast,Unified and Usable
  • 75.
  • 76.
    Practice Applying Concepts –Apache Spark via Databricks
  • 77.
    Hadoop is becomingSpark… Volume ✘10 TB or greater to start ✘Growth of 25% YOY ✘Where FROM ✘Where TO Velocity and Variety ✘Spark over Hive ✘Kafka and Samsa Veracity ✘Pay, train and hire team ✘Top $$$ for talent, IF you can find it ✘WATCH OUT for Cloud Vendors who promise ‘easy access’ ✘Complexity of ecosystem ✘DataBricks knows best for Spark ✘Cloudera knows best for on premise
  • 78.
    How Best toStore your Data? Complexity Scalability Developer Cost RDBMS easy medium* low NoSQL medium big high Hadoop hard huge very high
  • 79.
    Cloud Offerings –Big Data AWS Google Managed RDBMS RDS Aurora Spanner Cloud SQL Data Warehouse Redshift or Athena BigQuery NoSQL buckets S3 Glacier Cloud Storage Nearline NoSQL Key-Value NoSQL Wide Column DynamoDB Big Table Cloud Datastore NoSQL Document NoSQL Graph MongoDB on EC2 Neo4j on EC2 MongoDB on GCE Neo4j on GCE Hadoop / Spark Elastic MapReduce DataProc
  • 80.
    Real World BigData -- When do I use what? RDBMS 65% NoSQL 30% Hadoop 5%
  • 81.
    ETL is 75%of all Big Data Projects Surveying, cleaning and loading data is the majorityof the billable time for new Big Data projects.
  • 82.
    About this Workshop Learningfrom Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
  • 83.
  • 86.
  • 87.
    Pattern ✘How to buildoptimized cloud-based data pipelines? -- Cloud-based ETL tools and processes -- includes load-testingpatterns and securitypractices -- including connectingbetween different vendor clouds
  • 88.
    Key Questions –Ingestion and ETL ✘Volume – how much and how fast,now and future? ✘Variety – what type(s) or data, any pre-processingneeded? ✘Velocity – batches or steaming? ✘Veracity – verificationon ingest needed? new data needed?
  • 89.
    Actual Questions –Data Ingest and ETL ✘When (at what data volume) do you need AWS Snowball and/or AWS Direct Connect for ingest? ✘What does ‘near-real time streaming’ exactlymean? ✘When do you need to use a product vs. a libraryor service for ETL? ✘Which 3rd party partner products are available for the Data services you are using? ✘What are your actual data qualityissues?
  • 90.
    Serverless Is this acloud data pipeline patternnow?
  • 91.
    Serverless Time SeriesAnalysis Batch Storage BigQuery Storage Cloud Storage Time Series Processing Cloud Dataflow Analysis Cloud Datalab Storage Cloud Bigtable* Processing Cloud Dataproc Time Series Files Cloud Storage ML Cloud ML Streaming Time Series Streaming Cloud Pub/Sub *Note: Use Bigtable with NoSQL workloads of 1 TB or more
  • 92.
    DataFlow ✘ Managed ApacheBeam ✘ Elegant pipeline API ✘ Beam as a product -> GCP ✘ Scala? Java? Head-to-Head ETL – GCP DataFlow vs. GCP DataProc DataProc ✘ Managed Apache Spark ✘ Large user base ✘ Spark as a product -> DataBricks ✘ Cost compare?
  • 93.
  • 94.
    Pipeline Phases Phase 0 EvalCurrent Data - Quality & Quantity Phase 1 Get New Data - Free or Premium Phase 2 Build MVP & Forecast volume and growth Phase 3 Load test at scale Phase 4 Deploy – secure, audit and monitor
  • 95.
    Cloud Big DataVendors - ETL AWS ✘ 14X market share of next competitor ✘ Notable: Many, strong ETL Partners GCP ✘ Lean, mean and cheap ✘ Fastest player / Serverless ✘ Notable: DataFlow requires Java or Python developers
  • 96.
    How Best toIngest and ETL your Data? Complexity Scalability Developer Cost RDBMS medium medium low NoSQL medium big high Hadoop hard huge very high
  • 97.
  • 98.
    Building a StreamingPipeline Stream Interval Window
  • 99.
    Key Questions -Streaming ✘Volume – how much data now and predictedover next 12 months? ✘Variety – what types of data now and future? ✘Velocity – volume of input data / time now and near future? ✘Veracity – volume of EXISTING data now
  • 100.
    Cloud Big DataVendors - Streaming AWS ✘ 14X market share of next competitor ✘ Most complete offering ✘ Most mature offering ✘ Notable: Kinesis/Firehose GCP ✘ Lean, mean and cheap ✘ Fastest player ✘ Requires top developers ✘ Notable: DataFlow flexible
  • 101.
    Cloud Offerings –Data and Pipelines AWS Google Managed RDBMS RDS Aurora Spanner / Cloud SQL Data Warehouse Redshift/ Athena BigQuery NoSQL buckets S3 Glacier Cloud Storage Nearline NoSQL Key-Value NoSQL Wide Column DynamoDB Big Table Cloud Datastore Streaming or ML Kinesis AWS Machine Learning Pub/Sub or DataFlow Google Machine Learning NoSQL Document NoSQL Graph MongoDB on EC2 Neo4j on EC2 MongoDB on GCE Neo4j on GCE Hadoop / Spark Elastic MapReduce DataProc Cloud ETL Data Pipelines DataFlow
  • 102.
    How Best toStream your Data? Complexity Scalability Developer Cost Micro-Batches easy medium low Stream Windows difficult big high Near Real-time very difficult huge high
  • 103.
  • 104.
    Designing Streaming CloudData Pipelines Log Files Product Catalogs Social Games Social aggregators Line-of- Business
  • 105.
    Data Processing Financial Services> Monte Carlo Simulations Bespoke Apps Compute Engine Storage/Analysis BigQuery Storage Cloud Storage Dataflow/Beam Cloud Dataflow Visualization Cloud Datalab Storage Cloud Bigtable Hadoop/Spark Cloud Dataproc
  • 106.
    Batch Financial Services >Time Series Analysis Storage BigQuery Storage Cloud Storage Time Series Processing Cloud Dataflow Analysis Cloud Datalab Storage Cloud Bigtable Processing Cloud Dataproc Time Series Files Cloud Storage ML Cloud ML Streaming Time Series Streaming Cloud Pub/Sub
  • 107.
    Private Datasets PublicDatasets Life Sciences > Variant Analysis MSSNG Autism Cloud Storage Scientist High Throughput Genome Sequencers 1000 Genomes Cloud Storage Patient Data Cloud Storage Illumina Platform Cloud Storage Ref Genomes Cloud Storage TCGA Cloud Storage Analytics Online Analytics BigQuery Batch Analytics Cloud Dataflow Lab Notebooks Cloud Datalab Data Ingest Genomics BAM FASTQ
  • 108.
    Streaming Big Data >Complex Event Processing Cloud Apps Compute Engine Streaming Batch Push to Devices App Engine Rules Engine Cloud Dataflow Data Analysis Cloud Datalab Mobile Devices Push Notifications Report & Share Business Analysis Cloud Apps Compute Engine On-Premises Databases On-Premises Applications Processed Events Cloud Bigtable Events Time Series Data Warehouse BigQuery Execution Results Streaming Cloud Pub/Sub Transactions Processing Cloud Dataflow Transaction Streams Messaging Cloud Pub/Sub Rules Actions ETL Cloud Dataflow Transform Data Cloud Data Cloud Storage Rules Engine Cloud Dataproc
  • 109.
    About this Workshop Learningfrom Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
  • 110.
    4. Select ETL andVisualization Analytics and Presentation
  • 111.
    Pattern ✘How best toQuery and Visualize -- When to use business analytics vs. predictive analytics (machine learning) -- how best to present data to clients - partner visualizationproducts or roll your own
  • 112.
    Making Sense ofData Machine Learning Reports Presentation
  • 113.
    Actual Questions -Query ✘What query skills do the Dev / DevOps teams have? ✘Do we use an ORM? ✘Are transactions needed? ✘Which RDBMs features are we using now? i.e. PL-SQL, indexing… ✘Which language(s) do our analysts use? Excel, R, Python?
  • 114.
    Graphs What is natureof your questions?
  • 115.
    Practice Applying Concepts –Trying out Cypher on Neo4j
  • 117.
    Cloud Big DataVendors - Query AWS ✘ 5X market share of next competitor ✘ Most complete offering ✘ Most mature offering ✘ Notable: Big Relational GCP ✘ Lean, mean and cheap ✘ Fastest player ✘ Notable: Flexible, powerful machine learning
  • 118.
    Query Languages SQL Everyone knowsit But how well do they know it? NoSQL Vendor Language Too many to list How will you learn it? Cypher Query language for graph databases The future? ORM Good, bad or horrible? Again, how well do they know it? Spark Scala or Python (functional-style) Machine Learning Queries SciPy, NumPy or Python R Language Julie Language Many more…
  • 119.
    Practice Applying Concepts –Understanding Query Language Costs
  • 120.
    How Best toQuery your Data? Business Analytics Predictive Analytics Developer Cost RDBMS NoSQL Hadoop
  • 121.
    How Best toQuery your Data? Business Analytics Predictive Analytics Developer Cost RDBMS easy medium low NoSQL hard very hard very high Hadoop hard hard very high
  • 122.
    Machine Learning akaPredictive Analytics AWS ✘ ML for developers ✘ GUI-based – regression/classification only ✘ MxNet for deep neural networks, works best with GPUs GCP ✘ ML for ML developers ✘ Scaled or specialized ML models ✘ Many model types – vision, text, translate, speech, jobs, video ✘ TensorFlow for neural networks, can use GPUs
  • 123.
    Presentation If you can’tsee it, it’s not worth it.
  • 124.
    Dashboards ✘ More thanKPIs ✘ Mobile ✘ Alerts ✘ Data Stories Innovation in Data Visualization Reports ✘ Level of Detail ✘ Meaningful Taxonomies ✘ Fast enough ✘ Drill for Data
  • 125.
    D3 The language ofData Visualization
  • 127.
    Cloud Big DataVendors - Visualization AWS ✘ Most complete offering ✘ Notable: Partners & QuickSight GCP ✘ Big Query Partners ✘ Notable: 360 product - New Dashboards, iPython-style notebooks
  • 128.
    About this Workshop Learningfrom Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
  • 129.
  • 130.
    Big Data >Time Series Analysis Batch Storage BigQuery Storage Cloud Storage Time Series Processing Cloud Dataflow Analysis Cloud Datalab Storage Cloud Bigtable* Processing Cloud Dataproc Time Series Files Cloud Storage ML Cloud ML Streaming Time Series Streaming Cloud Pub/Sub *Note: Use Bigtable with NoSQL workloads of 1 TB or more
  • 131.
    Big Data >Bioinformatics *Note: Use Bigtable with NoSQL workloads of 1 TB or more
  • 132.
  • 133.
    About this Workshop Learningfrom Cloud (AWS / GCP) Architectures 1. Landscape 2. Data Servers and Services 3. Data Pipelines 4. ETL and VisualizationServices 5. End-to-End Patterns 6. Bonus: IoT
  • 134.
  • 135.
    Place your screenshothere Data Generation Device
  • 136.
  • 137.
    235,000,000,000 $The IoTMarket 2017By the year 20 Billion devicesAnd a lot of users
  • 138.
  • 139.
    139 Ingest Pipelines Storage Analytics Application & Presentation Standard Devices HTTPS Constrained Devices Non-TCP e.g.BLE Gateway Internet of Things > Sensor stream ingest and processing App Engine Container Engine Cloud Storage Cloud Pub/Sub Cloud Dataflow Monitoring Logging Cloud Dataflow Cloud Datastore Cloud Bigtable BigQuery Cloud Dataproc Cloud Datalab Compute Engine
  • 140.
  • 142.
    Cloud Big DataVendors - IoT AWS ✘ First to market ✘ Most complete offering ✘ Most mature offering ✘ Notable: MQTT support, AWS IoT Rules GCP ✘ Still in Beta / Android Things ✘ Fastest player ✘ Requires top developers ✘ Notable: Android Things, Weave, Nest
  • 143.
    “The Future isFunctional” @LynnLangit
  • 144.