Hw09 Matchmaking In The Cloud

Matchmaking in the Cloud: Amazon Web Services and Apache Hadoop at eHarmony ,[object Object],CONFIDENTIAL CONFIDENTIAL

[object Object],[object Object],[object Object],[object Object],[object Object],Why You’re Here CONFIDENTIAL

[object Object],[object Object],[object Object],[object Object],[object Object],About eHarmony CONFIDENTIAL * based on survey conducted by Harris Interactive in 2007.

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],The Science of Matching CONFIDENTIAL

Computational Requirements ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Scaling Challenges CONFIDENTIAL

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Hadoop Addresses Scaling Needs CONFIDENTIAL

Computing on AWS ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

AWS Pricing Model ,[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Architecture CONFIDENTIAL Data Warehouse Amazon Cloud Hadoop Jobs User data dump S3 upload download Result keystore input output update Elastic MapReduce Data Warehouse

MapReduce Overview ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Model Validation with MapReduce ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Data Flow CONFIDENTIAL Match Info Users (Left Side) Users (Right Side) Join Join & Score Group by User Results Temp Files 3 MapReduce Jobs:

AWS Elastic MapReduce ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Elastic MapReduce for eHarmony ,[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Simple Job Control ,[object Object],[object Object],[object Object],CONFIDENTIAL elastic_mapreduce.rb --create --name #{JOB_NAME} --num-instances #{NODES} --instance-type #{INST_TYPE} --key_pair #{KEY} --log-uri #{LOGDIR} --jar #{JAR} --main-class #{JOIN_CLASS} --arg -xconf --arg #{CONF}/join-config.xml --jar #{JAR} --main-class #{SCORER_CLASS} --arg -xconf --arg #{CONF}/scorer-config.xml --jar #{JAR} --main-class #{COMBINER_CLASS} --arg -xconf --arg #{CONF}/combiner-config.xml

Development & Test Environments ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Performance by Instance Type CONFIDENTIAL Minutes

Total Execution Time CONFIDENTIAL

Administration Tools ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

AWS Management Console ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

AWS Management Console CONFIDENTIAL EC2 Console Dashboard

Hadoop DFS – Monitor Disk Usage CONFIDENTIAL

Challenges ,[object Object],[object Object],[object Object],CONFIDENTIAL

Challenges – Elastic MapReduce ,[object Object],[object Object],[object Object],CONFIDENTIAL

Challenges – S3 (Simple Storage Service) ,[object Object],[object Object],[object Object],CONFIDENTIAL

Challenges – Data Shuffling ,[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Future Directions: Hadoop Streaming ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Future Directions: Data Analysis in the Cloud ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Data Analysis with Hive ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Data Analysis with Pig ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Lessons Learned ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CONFIDENTIAL

Any questions? ,[object Object],CONFIDENTIAL

Thank you ,[object Object],[object Object],CONFIDENTIAL CONFIDENTIAL

Hw09 Matchmaking In The Cloud

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to Hw09 Matchmaking In The Cloud

Similar to Hw09 Matchmaking In The Cloud (20)

More from Cloudera, Inc.

More from Cloudera, Inc. (20)

Recently uploaded

Recently uploaded (20)

Hw09 Matchmaking In The Cloud