From the Amazon Web Services Singapore & Malaysia Summits 2015 Track 2 Breakout, 'Big Data and Analytics' Presented by Russell Nash – AWS Solutions Architect

1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Big Data and Analytics
End to End on AWS
Russell Nash – AWS Solutions Architect

2. Agenda
End to End Framework
Use Cases
Demonstration
Religion
Greater
Good
FOMO

3. Greater
Good

6. Big Data
End to End Framework

7. Amazon
Redshift
Amazon
Kinesis
Amazon
S3
Amazon
RDS
Impala
Apache
Storm
PIG
Amazon
Machine
Learning
Amazon
EMR
Amazon
Glacier
Amazon
DynamoDB

8. ”I
got
kicked
out
of
the
bookshop
last
week,
because
I
moved
all
of
the
Big
Data
books
into
the
Religion
sec>on”

9. Ingest Store Process Analyse
Data Answers
Simplify Big Data Processing

10. Databases
Database
Flat Files Database
Data
File Data
Event
Producer
Android
iOS
Streaming
Data
Sales Data
Customer Data
Web Logs
Server Logs
Clickstream data
Sensor data
Database
INGEST STORE

11. Databases
Database
Flat Files Database
Data
File Data
Event
Producer
Android
iOS
Streaming
Data
Sales Data
Customer Data
Web Logs
Server Logs
Clickstream data
Sensor data
Database
INGEST
Amazon
Redshift
Amazon
RDS
STORE

12. Data Tier
Search Cache Object Store
RDBMSNoSQL
Data
Warehouse
logging
analyAcs
webscale
transacAons
rich search
hot reads
complex queries
and transacAons
Data Tier
Amazon
DynamoDB
Amazon
RDS
Amazon
ElastiCache
Amazon
S3
Amazon
Redshift
Amazon
CloudSearch
Traditional Relational Database

13. Amazon
Redshift
Amazon
RDS
ScalingVertical Horizontal
StorageRow Column
WorkloadTransactional Analytical
ArchitectureSMP MPP
TypeSQL Relational SQL Relational

14. ”Some
of
the
queries
we’re
running
are
98
percent
faster,
and
most
things
are
running
90
percent
faster”
-­‐
FT
CTO
John
O’Donovan

15. Databases
Database
Flat Files Database
Data
File Data
Event
Producer
Android
iOS
Streaming
Data
Sales Data
Customer Data
Web Logs
Server Logs
Clickstream data
Sensor data
Storage
INGEST
Amazon
Redshift
Amazon
RDS
Application
Amazon
S3
STORE

16. Impala PIG
Amazon
EMR

17. Amazon
S3
Amazon
Redshift
Amazon
EMR
Glacier
Amazon
DynamoDB
Amazon
Machine
Learning
Applications

18. Amazon
Redshift
ScalingAdd nodes Automatic
SpeedFastest Fast
CostHigher Lower
DurabilityConfigurable Built-in
Amazon
S3

19. ”Avoid
vendor
lock-­‐in”
-­‐
Saman
Michael
Far
-­‐SVP

20. Databases
Database
Flat Files Database
Data
File Data
Event
Producer
Android
iOS
Streaming
Data
Sales Data
Customer Data
Web Logs
Server Logs
Clickstream data
Sensor data
Stream
Processor
INGEST
Amazon
Redshift
Amazon
RDS
Amazon
S3
Amazon
Kinesis
STORE

21. Why Stream Storage?
Sensors
Amazon
Kinesis
Apache
Kafka

22. Availability
Zone
Availability
Zone
Availability
Zone
Data
Sources
Data
Sources
Data
Sources
Data
Sources
Data
Sources
Logging
Metrics
Analysis
Complex
Processing
S3
DynamoDB
Redshift
Apache
Storm
Amazon Kinesis
Stream

23. Amazon
Redshift
OrderingYes Yes
Persistence24 Hours Configurable
Size50 KB Configurable
ScalingHigh High
LatencyLow Low
ManagedYes No
Amazon
Kinesis

24. ”The
world
of
gaming
never
sleeps.
We
owe
every
player
a
great
experience,
and
AWS
is
our
main
tool
to
make
that
happen.”
-­‐
Sami
Yliharju,
Services
Lead

25. INGEST STORE PROCESS
Event
Producer
Android
iOS
Databases
Amazon
Redshift
Amazon
Kinesis
Amazon
S3
Amazon
RDS
Impala
Amazon
Redshift
Amazon
EMR
Flat Files
Database
Data
Event
Data
Streaming
Data
InteractiveBatchStreaming
Hadoop

26. Amazon
Redshift
Scaling2 PB+ Nodes
StorageNative HDFS/S3
BI ToolsHigh Medium
DurabilityHigh High
LatencyLow Low
ManagedFully Semi (EMR)
Amazon
Redshift
Nodes
HDFS
Medium
High
Low
Semi (EMR)
Amazon
Redshift Impala

27. INGEST STORE PROCESS
Event
Producer
Android
iOS
Databases
Amazon
Redshift
Amazon
Kinesis
Amazon
S3
Amazon
RDS
Impala
Amazon
Redshift
Flat Files
Database
Data
Event
Data
Streaming
Data
InteractiveBatch
PIG
Streaming
Amazon
EMR
Hadoop

28. PIG
SQL on Hadoop
Eats anything
New Processing Engine

29. Amplab Big Data Benchmark
https://amplab.cs.berkeley.edu/benchmark/

30. INGEST STORE PROCESS
Event
Producer
Android
iOS
Databases
Amazon
Redshift
Amazon
Kinesis
Amazon
S3
Amazon
RDS
Impala
Amazon
Redshift
Apache
Storm
Kinesis
Consumers
Flat Files
Database
Data
Event
Data
Streaming
Data
InteractiveBatchStreaming
PIG
Amazon
EMR
Hadoop

31. INGEST STORE PROCESS
Event
Producer
Android
iOS
Databases
Amazon
Redshift
Amazon
Kinesis
Amazon
S3
Amazon
RDS
Impala
Amazon
Redshift
Apache
Storm
Kinesis
Consumers
Flat Files
Database
Data
Event
Data
Streaming
Data
InteractiveBatchStreaming
PIG
ANALYSE
Amazon
Machine
Learning
L
Amazon
EMR
Hadoop

32. Use Cases

33. FOMO

34. Amazon
EMR
Hadoop
Amazon
Machine
Learning
Kinesis
Producer
Android
iOS
Databases
Amazon
Redshift
Amazon
Kinesis
Amazon
S3
Amazon
RDS
Impala
Amazon
Redshift
Apache
Storm
Kinesis
Consumer
Flat Files
Database
Data
Event
Data
Streaming
Data
Databases
Amazon
Redshift
Amazon
Redshift
Database
Data
SQL Analytics

35. Amazon
Machine
Learning
Kinesis
Producer
Android
iOS
Databases
Amazon
Redshift
Amazon
Kinesis
Amazon
S3
Amazon
RDS
Impala
Amazon
Redshift
Apache
Storm
Kinesis
Consumer
AmazonElasticMapReduce
Flat Files
Database
Data
Event
Data
Streaming
Data
Clickstream Analysis - Batch
AmazonElasticMapReduce
Event
Data
Amazon
EMR
Hadoop

36. Amazon
Machine
Learning
Kinesis
Producer
Android
iOS
Databases
Amazon
Redshift
Amazon
Kinesis
Amazon
S3
Amazon
RDS
Impala
Amazon
Redshift
Apache
Storm
Kinesis
Consumer
AmazonElasticMapReduce
Flat Files
Database
Data
Event
Data
Streaming
Data
Clickstream Analysis – Near Real Time
Event
Producer
Amazon
Kinesis
Amazon
S3
Amazon
Redshift
Kinesis
Consumers
Streaming
Data

37. Amazon
Machine
Learning
Kinesis
Producer
Android
iOS
Databases
Amazon
Redshift
Amazon
Kinesis
Amazon
S3
Amazon
RDS
Impala
Amazon
Redshift
Apache
Storm
Kinesis
Consumer
AmazonElasticMapReduce
Flat Files
Database
Data
Event
Data
Streaming
Data
Data Lake – Self Service Analysis
Databases
Amazon
S3
Database
Data
Event
Data
Streaming
DataAndroid
iOS
Impala
Amazon
Redshift
Amazon
Machine
Learning
Amazon
EMR
Hadoop