1. Managing Big Data in the AWS Cloud
Siva Raghupathy
Principal Solutions Architect
Amazon Web Services

2. Agenda
• Big data challenges
• AWS big data portfolio
• Architectural considerations
• Customer success stories
• Resources to help you get started
• Q&A

3. Data Volume, Velocity, & Variety
• 4.4 zettabytes (ZB) of data exists
in the digital universe today
– 1 ZB = 1 billion terabytes
• 450 billion transaction per day by
2020
• More unstructured data than
structured data GB
TB
PB
ZB
EB
1990 2000 2010 2020

4. Big Data
• Hourly server logs: how your systems were
misbehaving an hour ago
• Weekly / Monthly Bill: What you spent this
past billing cycle?
• Daily customer-preferences report from your
web-site’s click stream: tells you what deal
or ad to try next time
• Daily fraud reports: tells you if there was
fraud yesterday
Real-time Big Data
• Real-time metrics: what just went wrong
now
• Real-time spending alerts/caps:
guaranteeing you can’t overspend
• Real-time analysis: tells you what to offer
the current customer now
• Real-time detection: blocks fraudulent use
now
Big Data : Best Served Fresh

5. Data Analysis Gap
Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011
IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares
Generated data
Available for analysis
Data volume - Gap
1990 2000 2010 2020

6. Big Data
Potentially massive datasets
Iterative, experimental style of data
manipulation and analysis
Frequently not a steady-state workload;
peaks and valleys
Time to results is key
Hard to configure/manage
AWS Cloud
Massive, virtually unlimited capacity
Iterative, experimental style of
infrastructure deployment/usage
At its most efficient with highly
variable workloads
Parallel compute clusters from singe
data source
Managed services

7. AWS Big Data Portfolio
Collect / Ingest
Kinesis
Store Process / Analyze
Visualize / Report
EMR EC2
Redshift Data Pipeline
S3
DynamoDB
Glacier
RDS
Import Export
Direct Connect
Amazon SQS

8. Ingest: The act of collecting and storing data

9. Why Data Ingest Tools?
• Data ingest tools convert
random streams of data into
fewer set of sequential streams
– Sequential streams are easier to
process
– Easier to scale
– Easier to persist
Processing
Processing
Processing
Processing
Processing
Kafka
Or
Kinesis
Processing

10. Data Ingest Tools
• Facebook Scribe  Data collectors
• Apache Kafka  Data collectors
• Apache Flume  Data Movement and Transformation
• Amazon Kinesis  Data collectors

11. Real-time processing of streaming data
High throughput
Elastic
Easy to use
Connectors for EMR, S3, Redshift, DynamoDB
Amazon
Kinesis

12. AmAamzaozno Kn iKneinseiss iAs rAchrcitheictetucrtuere
AZ AZ AZ
Durable, highly consistent storage replicates data
across three data centers (availability zones)
Amazon Web Services
Aggregate and
archive to S3
Millions of
sources producing
100s of terabytes
per hour
Front
End
Authentication
Authorization
Ordered stream
of events supports
multiple readers
Real-time
dashboards
and alarms
Machine learning
algorithms or
sliding window
analytics
Aggregate analysis
in Hadoop or a
data warehouse
Inexpensive: $0.028 per million puts

13. Kinesis Stream:
Managed ability to capture and store data
• Streams are made of Shards
• Each Shard ingests data up to
1MB/sec, and up to 1000 TPS
• Each Shard emits up to 2 MB/sec
• All data is stored for 24 hours
• Scale Kinesis streams by adding or
removing Shards
• Replay data inside of 24Hr. Window

14. Simple Put interface to store data in Kinesis
• Producers use a PUT call to store data in a Stream
• PutRecord {Data, PartitionKey, StreamName}
• A Partition Key is supplied by producer and used to
distribute the PUTs across Shards
• Kinesis MD5 hashes supplied partition key over the hash
key range of a Shard
• A unique Sequence # is returned to the Producer upon a
successful PUT call

15. Building Kinesis Processing Apps: Kinesis Client Library
Client library for fault-tolerant, at least-once, Continuous Processing
o Java client library, source available on Github
o Build & Deploy app with KCL on your EC2 instance(s)
o KCL is intermediary b/w your application & stream
 Automatically starts a Kinesis Worker for each shard
 Simplifies reading by abstracting individual shards
 Increase / Decrease Workers as # of shards changes
 Checkpoints to keep track of a Worker’s location in the
stream, Restarts Workers if they fail
o Integrates with AutoScaling groups to redistribute workers
to new instances

16. Sending & Reading Data from Kinesis Streams
Sending Reading
HTTP Post
AWS SDK
LOG4J
Flume
Fluentd
Get* APIs
Kinesis Client Library
+
Connector Library
Apache Storm
Amazon Elastic
MapReduce
Write Read

17. AWS Partners for Data Load and Transformation
Hparser, Big Data Edition
Flume, Sqoop

18. Storage

19. Storage
Structured – Simple Query
NoSQL
Amazon DynamoDB
Cache
Amazon ElastiCache
(Memcached, Redis)
Structured – Complex Query
SQL
Amazon RDS
Data Warehouse
Amazon Redshift
Search
Amazon CloudSearch
Unstructured – No Query
Cloud Storage
Amazon S3
Amazon Glacier
Unstructured – Custom Query
Hadoop/HDFS
Amazon Elastic Map Reduce
Data Structure Complexity
Query Structure Complexity

20. Store anything
Object storage
Scalable
Designed for 99.999999999% durability
Amazon
S3

21. Why is Amazon S3 good for Big Data?
• No limit on the number of Objects
• Object size up to 5TB
• Central data storage for all systems
• High bandwidth
• 99.999999999% durability
• Versioning, Lifecycle Policies
• Glacier Integration

22. Amazon S3 Best Practices
• Use random hash prefix for keys
• Ensure a random access pattern
• Use Amazon CloudFront for high throughput GETs and PUTs
• Leverage the high durability, high throughput design of Amazon S3 for
backup and as a common storage sink
• Durable sink between data services
• Supports de-coupling and asynchronous delivery
• Consider RRS for lower cost, lower durability storage of derivatives or copies
• Consider parallel threads and multipart upload for faster writes
• Consider parallel threads and range get for faster reads

23. Aggregate All Data in S3 Surrounded by a collection of the right tools
EMR Kinesis
Data Pipeline
Redshift DynamoDB RDS
Cassandra Storm Spark Streaming
Amazon
S3
Amazon S3

24. Fully-managed NoSQL database service
Built on solid-state drives (SSDs)
Consistent low latency performance
Any throughput rate
No storage limits
Amazon
DynamoDB

25. DynamoDB Concepts
table
items
attributes
schema-less
schema is defined per attribute

26. DynamoDB: Access and Query Model
• Two primary key options
• Hash key: Key lookups: “Give me the status for user abc”
• Composite key (Hash with Range): “Give me all the status updates for user ‘abc’
that occurred within the past 24 hours”
• Support for multiple data types
– String, number, binary… or sets of strings, numbers, or binaries
• Supports both strong and eventual consistency
– Choose your consistency level when you make the API call
– Different parts of your app can make different choices
• Global Secondary Indexes

27. DynamoDB: High Availability and Durability

28. What does DynamoDB handle for me?
• Scaling without down-time
• Automatic sharding
• Security inspections, patches, upgrades
• Automatic hardware failover
• Multi-AZ replication
• Hardware configuration designed specifically for DynamoDB
• Performance tuning
…and a lot more

29. Amazon DynamoDB Best Practices
• Keep item size small
• Store metadata in Amazon DynamoDB and blobs in Amazon S3
• Use a table with a hash key for extremely high s