Analytics in the Cloud

Analytics in the Cloud Deepak Singh, Ph.D. Sr. Business Development Manager

Via butteryflysha under a CC-BY license

Image: Simon Cockell under CC-BY

our reality

lots and lots and lots andlots and lots of data

Astronomy Genome SequencingLogs ClickstreamsFraud SensorsGeolocation

harness data

business insights

better decisions

new business/product opportunities

People drive innovationCredit: Pieter Musterd a CC-BY-NC-ND license

Image: Drew Conway

challenges

SuccessfulYour Idea Product

Data VolumeData StructuresData Dimensionality

Resource constraintsTight BudgetsUndifferentiated Heavy Lifting

enter the cloud

Amazon Web Services

infrastructure services

building blocks

Abstract Resources On Demand Programmable Pay As You Go Secure Elastic

Predicting Infrastructure Needs Actual Usage CustomerCompute Power Dissatisfaction Predicted Usage Waste Time

3000 - 3000 CPU’s for one firm’s risk management processesNumber of EC2 Instances 300 CPU’s on weekends300 - Wednesday Thursday Friday Saturday Sunday Monday Tuesday 4/22/2009 4/23/2009 4/24/2009 4/25/2009 4/26/2009 4/27/2009 4/28/2009

requirements for analytics

scalable storage

scalable compute

on-demand resources

99.999999999 Amazon S3 Elastic Block Store Amazon EC2 Persistent Attached Storage Elastic Computing

database

Amazon RDS

what database do you want?

Oracle Certification, Support and Licensing All products certified on the Oracle Virtual Machine are now Certified & Certified on Amazon EC2 supported managed OVM Full Support from Oracle and AWS Standard Licensing Policies Apply AMIs exist today for many Oracle products

Biomarker Warehousepre-clinical, clinical, 3rd party data and publications Estimated cost: 10 TB warehouse over 3 years

reporting & business intelligence

SAP BusinessObjects Recovery.gov Architecture

new paradigms for data analysis

unconstrained tools for unconstrained growth

Amazon Elastic MapReduce

Amazon EC2 Instances EndDeploy Application Hadoop Hadoop Hadoop Elastic Elastic MapReduce MapReduce Hadoop Hadoop Hadoop NotifyWeb Console, Command line tools Input output dataset results Input S3 Output S3 Get Results Input Data bucket bucket Amazon S3

mapreduce

Amazon ElasticMapReduce Goals

handle undifferentiated heavy lifting

integrated with other cloud services

It is Hadoop

Hive PigCascading

Customers are using for …Targeted advertising / Clickstream analysisData warehousing applicationsBioinformaticsFinancial modelingFile processingWeb indexingData mining and BI

CLICKSTREAM ANALYSIS – RAZORFISH AND BESTBUY Best Buy came to Razorfish  3.5 billion records, 71 million unique cookies, 1.7 million targeted ads required per day User recently purchased a home theater Targeted Ad system and is searching for (1.7 Million per day) video games• Leveraged AWS and Elastic MapReduce – 100 node cluster on demand – Processing time dropped from 2+ days to 8 hours – Increased ROAS by 500%

Clickstream Analysis - Architecture

another example: Etsy

Etsy by the numbers October 2010 $29.9 million in sales 842 million page views 434 GB web logs 97 million total favorite listings

Data collection & EMR integration

Cascading.Jruby source raw_log_data’ assembly raw_log_data do # Helper method to parse fields from raw web logs: session_id, url parameters, etc. parse_log :query_fields => [affiliate_code’,sale_confirmation_id’] branch affiliate_events do where affiliate_code:string != null’ rename created_at =>; affiliate_timestamp’ end branch sales_events do where sale_confirmation_id:string != null’ rename created_at => sale_timestamp’ end end

data warehousing

WEB-SCALE DATA WAREHOUSING

RESIZE RUNNING JOB FLOWS Use Case: Increase speed of running job flows  Speed up job flow execution in response to changing requirements  Dynamically balance cost versus performance without restarting a job Job Flow Job Flow Job Flow Allocate Expand to Expand to4 instances 9 instances 25 instances Time remaining: Time remaining: 14 Hours 7 Hours Time remaining: 3 Hours

Dynamically Resize Job Flows Use Case: Agile Data Warehouse Cluster  Customize cluster size to support varying resource needs (e.g., query support during the day versus batch processing overnight)  Leverage flexibility to reduce costs and increase cluster utilization Data Warehouse (Batch Processing) Data Warehouse Data Warehouse (Steady State) (Steady State) Allocate Expand to Shrink to9 instances 25 instances 9 instances

Ecosystem And Tools Business Intelligence  MicroStrategy, Pentaho Analytics  Datameer, Karmasphere Open source  Beeswax …

some practicalconsiderations

• Launch and monitor job flows • AWS Management Console • Command line interface • REST API • 3rd party libraries, e.g. MrJob

Hardware requirements for Use Cases Data/IO Intensive (m1/m2 instances)  Data Warehouse  Data Mining • Click stream, logs, events, etc. Compute/IO Intensive (c1, cc1 instances)  Credit Ratings  Fraud Models  Portfolio analysis  VaR calculation

additional resources

http://aws.amazon.com/elasticmapreduce

http://aws.amazon.com/articles/Elastic-MapReduce

http://www.youtube.com/user/AmazonWebServices

in summary

Amazon Web Services

Abstract Resources On Demand Programmable Pay As You Go Secure Elastic

many database options

Elastic MapReduce

powerful, flexible, elastic analytics and data warehousing

SuccessfulYour Idea Product

Image: Jeff Hester under a Creative Commons License

AWS will be at Strata 2011

deesingh@amazon.com Twitter:@mndoci Inspiration and material from Matt Wood, Peter Sirota & Larry LessigCredit” Oberazzi under a CC-BY-NC-SA license

http://twitter.com	3
http://www.onlydoo.com	2
http://us-w1.rockmelt.com	1
https://twitter.com	1
http://trunk.ly	1
http://paper.li	1

Analytics in the Cloud

by Amazon Web Services

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 9

Statistics

Analytics in the Cloud Presentation Transcript