Large-scale data processing with Hadoop in the Cloud

by Patrick Salami on Feb 10, 2011

A lightning talk that I gave at CloudCamp San Diego on how Hadoop can be used in the cloud to process more than 100,000,000 records

A lightning talk that I gave at CloudCamp San Diego on how Hadoop can be used in the cloud to process more than 100,000,000 records

http://twitter.com

Large-scale data processing with Hadoop in the Cloud — Presentation Transcript

Large-scale data processing with Hadoop in the Cloud by Patrick Salami Senior Software Engineer

100,000,000+ records

< 1h processing time

HadoopDistributed computing frameworkRuns on commodity hardwareSplits up large data setsEasy to deploy in AWS with ElasticMapReduce

Java Code Raw Product Normalize Data Filter Categorize S3 Shipping Coupons Hadoop Cluster Group Post-Process Solr S3 WWW Index

x32 nodescompute time $0.17/h $5.44/hEMR $0.03/h $0.96/hTotal $0.20/h $6.40/h Source: http://aws.amazon.com/ec2/#pricing

twitter.com/psalamilinkedin.com/in/psalami