Big Data Analytics

BIG DATA ANALYTICS WITH AWS Carlos Conde │Solutions Architect

BIG DATAWhen innovation is required to collect, store, analyze, and share your data

YOU DON’T HAVE THE CHOICE…

27 TB per dayLarge Hadron Collider – CERN

BIGGERIS BETTER

The more data you collect The more VALUE you can derive from it

Big Data Verticals Media Social Network Oil & Gas Retail Life Sciences Financial Services SecurityAdvertising Gaming User Anti-virus Demographics Targeted Recommendations Monte CarloAdvertising Simulations Seismic Genome Fraud Usage Analysis Analysis Detection analysisImage and Transaction Risk Video Analysis Analysis Image In-gameProcessing Recognition metrics

 2.7 Zetabytes in 2012 Over 90% will be unstructured

VOLUMEVELOCITYVARIETY

COLLECT │ STORE │ ANALYZE │ SHARE

AWSIMPORT / EXPORT

AMAZONS3

AMAZONDYNAMODB

HBaseonAMAZON EMR

GPUGRAPHICS PROCESSING UNIT

CLUSTER GPUQUADRUPLE EXTRA LARGE Intel Xeon X5570, quad-core 2x Nehalem architecture NVIDIA Tesla Fermi 2x M2050 GPUs 22 GB of memory – 1.7 TB of storage $ 2.1 PER HOUR

PARALELIZATION

ON A SINGLE INSTANCECOST: 4h x $2.1 = $8.4RENDERING TIME: 4h

ON MULTIPLE INSTANCESCOST: 2 x 2h x $2.1 = $8.4RENDERING TIME:

"Hadoop is a reliable storage and data analysis system" HDFS MapReduce

Deploying a Hadoop cluster is hard

AMAZON EMR HADOOP + AWS

Doing analytics in Eclipse iswrong…

A real Pig script(used at Twitter)

USE THE RIGHT TOOL FOR THE RIGHT JOB RDBMS Hadoop Interactive Reporting (<1sec)  Affordable Storage/Compute Multistep Transactions  Structured or Not (Agility) Lots of Updates/Deletes  Resilient Auto Scalability

Data Warehouse (Batch Processing)Data Warehouse Data Warehouse (Steady State) (Steady State) Shrink to Expand to 9 instances 25 instances

OPERATIONAL HADOOP EXPERIENCE Operated over a million Hadoop clusters last year

PUBLIC DATA SETShttp://aws.amazon.com/publicdatasets

INNOVATE

« Want to increase innovation? Lower the cost of failure » Joi Ito

AWS LOWERSTHE COST OF INNOVATION Testing a new idea is cheap

Big Data Analytics

by Amazon Web Services on Jun 26, 2012

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics