Data Lake, Business Intelligence, Enterprise Data Warehouse, Big Data Pipeline, Online Machine Learning, Lambda Architecture, Streaming, Spark, Kafka, Storm, Flink, Hadoop, Mesos and SMACK stack are some of the things you hear about when you want to dive into building a data pipeline. The Big Data Landscape cannot fit on a single screen as seen in the presentation. This is in addition to all the Big Data & Machine Learning offerings AWS has been introducing over the past few years which address many pain points highlighted by the various communities and help you get up and running faster. The objective of this talk is to provide the audience with a framework which helps them define their pipeline problems, isolate components and pick the right tools for the right job. We will talk about: 1. A consistent definition of BIG in big data 2. The lineage of fundamental tools in the ecosystem 3. First principles of a big data pipeline based on the lambda (not lambda functions) and kappa architectures 4. Distinguishing between big data and online machine learning pipelines 5. Technology choices based on first principles, open source solutions and AWS offerings 6. Demo: Serverless, Managed Big Data Pipeline and real-time dashboard on AWS (orchestrated via Terraform) Presented at: https://www.meetup.com/Vancouver-Amazon-Web-Services-User-Group/events/245946651/

1. A Tale of Lambdas, Kappas & Pancakes
@osamakhn

2. Who am I?
Osama Khan
Big Data Engineer @ACLServices
Grad Student @GTComputing
AWS Big Data Specialist+
! Vancouver, BC
" : Java " : C# (met thru J#) # : Python
$ : Golang, NodeJS % : Scala
Previously: Robot Soccer, BI, Credit Rating, AML,
O&G Portfolio, NLP/Governance, Doctor Triage,
Energy Monitoring, Consulting, Private Equity
Recently: Data/ML Pipeline, Tools & Platforms

3. What am I going to talk
about today?
The goal of this talk is to provide a high level overview
of the big data landscape to help software engineers
distinguish signal from noise
1) How big is BIG: Lets get our scales recalibrated to
understand what we mean by BIG Data
2) Lineage: Evolution of the Big Data ecosystem; from
EDW to Data Lakes
3) Lambda & Kappa Architectures: The foundation
of data pipelines and machine learning systems
4) Technology Choices: SMACK that PANCAKE
BUT I ❤ Serverless
5) Demos: Athena, EMR, Redshift, Quicksight,
Sagemaker, ModelDB

4. How big is BIG?
Big is BIG when Bieber breaks the Google Cloud (wat!?!)
Lets get our scales recalibrated to understand what we
mean by BIG Data

5. This is BIG …
§ 390 Hyperscale Datacenters ( < 300, 2016)
§ Hyperscale == (5k servers, 10k sq.ft space)
§ > 400, 100M+ total servers
§ 56% web content in English
§ 8,000 languages spoken globally
§ Hello, friend.
§ 100M+ active users, 40M+ subscribers
§ 30M+ songs, 20K new per day, 2B+ playlists, 1B+ plays per day
§ 2,500 node Hadoop cluster, 100 PB+ Disk, 100TB+ RAM
§ 60TB+ per day log ingestion, 20k+ jobs per day
§ Listening History Query
§ user x track x [day/week/month/all time]
§ 300B elements
§ 800 workers, 32 core, 208 GB ram
§ 240TB in, 90TB out
§ Top Tracks in Vancouver (June 2017)
§ 30 date partitioned tables, 60TB data
§ 1 metadata table, 418GB
§ 94.2s, 4.82TB processed
§ Despacito – Remix (Luis Fonsi)
§ (2017)
§ 2.8 B+ US Tweets
§ Donald Trump (901.8 M)
§ Hillary Clinton (123.2 M)
§ Mike Pence (31.4 M)
§ 30x more than VP, 7x more than opponent
(2013)
§ 170M individual metrics (timeseries) per minute
§ 200M queries served/day, 47 charts/user

6. Lineage
Evolution of the Big Data ecosystem; from EDW to
Data Lakes

7. A journey from ETL to Distributed Transactions via the ELT alley…
2007 20172003 20142009 2011201020042000
FaunaDB,
Aurora
Lambda,
Kappa
Architectures
UC
Berkeley
Spark
Google
File
System
LinkedIn
Kafka
FB
Cassandra
AWS
DynamoDB
Google
Dremel
2012: Google
Spanner
IBM,
Oracle,
MSFT,
Terradata,
SAP
CAP
Theorem
Google
Map
Reduce
AWS, GCP,
Azure,
Hana
2006:
Yahoo! Hadoop
Occupy the Cloud:
Distributed
Computing for the
99%

8. Big Data Landscape
Distributed systems rule the !

9. Yet Another Big Data Framework (YABDF)
Doesn’t fit on a slide or two … and you
thought you had library fatigue in the JS
world !
http://mattturck.com/wp-content/uploads/2017/05/Matt-Turck-FirstMark-2017-Big-Data-Landscape.png

10. Lambda & Kappa Architectures
The foundation of data pipelines for enterprise insights

11. Lambda Architecture: First Principles & Desired Properties
Data special information from which everything else is derived Information processed data
Data System query = function(ALL_DATA)
1. Robustness & Fault Tolerance
2. Low Latency Read & Update
3. Scalability
4. Generalization
5. Ad-hoc
6. Minimal Maintenance
7. Debuggability

12. The Lambda Architecture
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
∑ ALL_DATA
Δ NEW_DATA

13. The Lambda Architecture (in the enterprise)
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR

14. The Lambda Architecture (in the enterprise)
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR

15. The Lambda Architecture (in the enterprise)
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR

16. The Lambda Architecture (deep dive)
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
1
2 3
4
5
> home of ‘master data’
> precomputed_batch_view = fn(ALL_DATA)
> user_query = fn(precomputed_batch_view)

17. The Lambda Architecture (deep dive)
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
1
2 3
4
5
> (distributed) db to store batch view data
> produce fast results for known queries
> allow (random) reads by users/systems

18. The Lambda Architecture (deep dive)
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
1
2 3
4
5
> compensate for high latency of batch updates
> run fast, incremental algorithms (probabilistic data structures, for the win)
> realtime_view = fn(realtime_view, new_data)
> user_query = fn(realtime_view) > user_query =
fn(precomputed_batch_view)

19. The Lambda Architecture (deep dive)
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
1
2 3
4
5
> user_query = fn(
precomputed_batch_view,
realtime_view)

20. The Lambda Architecture (ingest? speedlayer?)
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
1
2 3
4
5

21. The Lambda Architecture (ingest? speedlayer?)
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
1
2 3
4
5

22. The Kappa Architecture
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
1
2 3
4
5

23. The Lambda Architecture (SMACK)
Data
Catalog
Impact
Analysis
Data
Lineage
Access
Control
Secure
Data Rest
GDPR
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
1
2 3
4
5

24. Big Picture of Metadata Management for Data Governance
Access
Control
Secure
Data Rest
GDPR
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
lineage
impact
analysis
semantic
lineage
Enterprise
Vocabulary
Semantic
Mapping
(metadata harvesting)
(metadata stitching)

25. Big Picture of Metadata Management for Data Governance
Access
Control
Secure
Data Rest
GDPR
query
query
new
data
master dataset
batch view
batch view
real-time view real-time view
batch layer serving layer
speed layer
lineage
impact
analysis
semantic
lineage
Enterprise
Vocabulary
Semantic
Mapping

26. Machine Learning Pipelines
BAML PIT

27. Machine Learning Pipelines
BAML PITBAML PIT == $$100MM
Blockchain based Adversarial Machine Learning
Platform for IoT Testing

28. classic model

29. classic model

30. Pancake Stack
[Presto Arrow Nifi Cassandra Airflow Kafka ElasticSearch Spark Tensorflow AlgeBird CoreNLP Kibana]

31. data science silo
Data Source Data & Feature
Engineering
Adaptation of slide by Ben Lorica
Model
Building
Deploy
Monitor

32. maturity spectrum

33. what’s changing(-ed)?
1. Cloud (faas, serverless data pipelines, ml-as-a-service)
2. Consumer demand for ML features/products/applications
3. Targeted Models (we need to manage 20MM models for 10MM users maybe)
4. Localization (ASEAN facial recognition)
5. Security (Adv. ML, Side-channel attacks)
6. Transparency (Bias is a BUG)
7. Many toy sophisticated solutions but conventional, simpler techniques (regression)
still deliver more business value!
8. Monitoring to ensure deployed models are making high quality predictions
9. Need practices to maintain (update or rebuild) models over time
10. and ….

34. feature engineering, wat?
By @MLpuppy

35. rise of machine learning engineers

36. rise of machine learning engineers

37. Online Machine Learning Pipeline
Model Inventory
Model Output Monitoring
Take Action
ML Serving Layer
Hyper-parameter Tuning

38. www.productionml.org