COSCUP - Open Source Engines Providing Big Data in the Cloud, Markku Lepisto
•
7 likes•3,519 views
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to COSCUP - Open Source Engines Providing Big Data in the Cloud, Markku Lepisto
Similar to COSCUP - Open Source Engines Providing Big Data in the Cloud, Markku Lepisto (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (11)
COSCUP - Open Source Engines Providing Big Data in the Cloud, Markku Lepisto
- 1. Markku Lepistö Technology Evangelist Amazon Web Services @markkulepisto
- 3. #1 ●○○○○
- 4. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance. We are constantly producing more data
- 5. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance. From all types of industries
- 8. The Role of Data is Changing
- 9. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance. Un#l%now,%Ques#ons%you%ask%drove%Data%model% New%model%is%collect%as%much%data%as%possible% –%“Data>First%Philosophy”%
- 10. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance. Data is the new raw material for any business on par with capital, people, labor Datais the new raw material for business on par with capital & labor
- 12. Generated data Available for analysis Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011 IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares
- 14. lunch hours last year?
- 15. ! select productId, count(*) from page_hits where hour in (12,13) group by productId order by count(*) desc! ! cat *-(12|13) | cut –f3 | sort | uniq -c > out! Hit <enter>?
- 16. 1PB = 10^15 (1,000,000,000,000,000) bytes 1 PB = 231 days at 50MB/s
- 17. Solution: Massively Parallel Processing
- 18. #2 ○●○○○
- 23. Very%large% log% (e.g%TBs)% Split into small pieces Lots of actions by John
- 24. Very%large% log% (e.g%TBs)% Process in a hadoop cluster Split into small pieces Lots of actions by John
- 25. Very%large% log% (e.g%TBs)% John’s% history% Process in a hadoop cluster Aggregate the results Split into small pieces Lots of actions by John
- 27. map Input file reduce Output file map Input file reduce Output file map Input file reduce Output file Worker node Worker node Worker node
- 30. #3 ♥ ○○●○○
- 31. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance.
- 32. Elastic On Demand Pay as you go Focus on YOUR business
- 33. Elastic On Demand Pay as you go Focus on YOUR business
- 34. November
- 37. November
- 41. #4 ○○○●○
- 42. EMR is Hadoop in the Cloud!
- 45. Media/ Advertising Targeted Advertising Image and Video Processing Oil & Gas Seismic Analysis Retail Recommendations Transactions Analysis Life Sciences Genome Analysis Financial Services Monte Carlo Simulations Risk Analysis Security Anti-virus Fraud Detection Image Recognition Social Network/ Gaming User Demographics Usage analysis In-game metrics
- 48. Job Flows Custom JAR Cascading Streaming Ruby, Perl, Python, PHP, R, Bash, C++
- 49. Data Warehouse for Hadoop SQL-like query language Hive
- 50. High-level programming Ideal for data flow / ETL Pig
- 51. Near real time key/value store for structured data HBase
- 52. Distributed monitoring of cluster and nodes Ganglia
- 53. Statistical computing and graphics Machine learning library discover Value in Data
- 54. Data Strategist
- 55. Unknown Unknowns
- 56. Elastic On Demand Pay as you go Focus on YOUR business
- 60. elastic-mapreduce --create --key-pair micro --region eu-west-1 --name MyJobFlow --num-instances 5 --instance-type m2.4xlarge –-alive --log-uri s3n://mybucket/EMR/log Instance type/count
- 61. elastic-mapreduce --create --key-pair micro --region eu-west-1 --name MyJobFlow --num-instances 5 --instance-type m2.4xlarge –-alive --pig-interactive --pig-versions latest --hive-interactive –-hive-versions latest --hbase --log-uri s3n://mybucket/EMR/log Adding Hive, Pig and Hbase to the job flow
- 62. Elastic On Demand Pay as you go Focus on YOUR business
- 63. 1 instance for 1000 hours = 1000 instances for 1 hour
- 65. …to Thousands
- 68. Turn Off the Resources and Stop Paying
- 69. Elastic On Demand Pay as you go Focus on YOUR business
- 70. Source: IDC Whitepaper, sponsored by Amazon, The Business Value of Amazon Web Services Accelerates Over Time. July 2012 70% lower 5 year TCO per app AWS On- premises $3.01M $0.90M 50% reduction in analytics costs
- 71. Save more money by using Spot Instances
- 72. 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20% PercentageoftheDistribution Bid Price as Percentage of the On-Demand Price Bid Distribution Typical Spot Bidding Strategies
- 74. 14%hrs% Without#Spot# 4%instances%*%14%hrs%*%$0.50%=%$28% EMR with Spot Instances 14%hrs%
- 76. With#Spot# 4%instances%*%7%hrs%*%$0.50%=%$14%+%%% 14%hrs% Without#Spot# 4%instances%*%14%hrs%*%$0.50%=%$28% EMR with Spot Instances 7%hrs%
- 77. With#Spot# 4%instances%*%7%hrs%*%$0.50%=%$14%+% 5%instances%*%7%hrs%*%$0.25%=%$8.75% Total%=%$22.75% 14%hrs% Without#Spot# 4%instances%*%14%hrs%*%$0.50%=%$28% EMR with Spot Instances 7%hrs%
- 80. #5 ○○○○●
- 81. What kind of movies do people like ?
- 82. More than 25 Million Streaming Members 50 Billion Events Per Day 30 Million plays every day 2 billion hours of video in 3 months 4 million ratings per day 3 million searches Device location , time , day, week etc. Social data
- 83. 10 TB of streaming data per day
- 84. Da ta $C enter S3 Netflix(Data(Center Legacy data from on-premise data center Legacy Data
- 85. Customer dimension data stored in Cassandra
- 86. ~1 PB of data stored in Amazon S3 S3
- 87. Wide range of processing languages used EMR Prod%Cluster% (EMR) S3
- 88. Data consumed in multiple ways S3 EMR Prod%Cluster% (EMR) Recommendation Engine Ad-hoc Analysis Personalization
- 91. Durability
- 92. Versioning
- 95. Foursquare… 33 million users 1.3 million businesses …generates a lot of Data 3.5 billion check-ins 15M+ venues, Terabytes of log data
- 96. Uses EMR for Evaluation of new features Machine learning Exploratory analysis Daily customer usage reporting Long-term trend analysis
- 97. Benefits of EMR Ease-of-Use “We have decreased the processing time for urgent data-analysis” Flexibility To deal with changing requirements & dynamically expand reporting clusters Costs “We have reduced our analytics costs by over 50%”
- 98. ApplicationStack Scala/Liftweb API Machines WWW Machines Batch Jobs Scala Application code Mongo/Postgres/ Flat Files Databases LogsDataStack Amazon S3 Database Dumps Log Files Hadoop Elastic Map Reduce Hive/Ruby/ Mahout Analytics Dashboard Map Reduce Jobs mongoexport postgres dump Flume
- 99. ApplicationStack Scala/Liftweb API Machines WWW Machines Batch Jobs Scala Application code Mongo/Postgres/ Flat Files Databases LogsDataStack Amazon S3 Database Dumps Log Files Hadoop Elastic Map Reduce Hive/Ruby/ Mahout Analytics Dashboard Map Reduce Jobs mongoexport postgres dump Flume
- 100. ApplicationStack Scala/Liftweb API Machines WWW Machines Batch Jobs Scala Application code Mongo/Postgres/ Flat Files Databases LogsDataStack Amazon S3 Database Dumps Log Files Hadoop Elastic Map Reduce Hive/Ruby/ Mahout Analytics Dashboard Map Reduce Jobs mongoexport postgres dump Flume
- 101. ApplicationStack Scala/Liftweb API Machines WWW Machines Batch Jobs Scala Application code Mongo/Postgres/ Flat Files Databases LogsDataStack Amazon S3 Database Dumps Log Files Hadoop Elastic Map Reduce Hive/Ruby/ Mahout Analytics Dashboard Map Reduce Jobs mongoexport postgres dump Flume
- 102. 0 0.1 0.2 0.3 0.4 0.5 0.6 Female Male Gender 0 10 20 30 40 50 60 70 80 Age
- 103. Gorilla Coffee Gray's Papaya Amorino Thursday% Friday% Saturday% Sunday%
- 104. Who is using our service?
- 105. Finding signal in the noise of logs
- 110. Log files 250 EMR clusters spun up and down every week
- 111. Common Crawl 1000 Genomes Project Census Data 54 other datasets http://aws.amazon.com/publicdatasets/
- 114. 150B Soil Observations 3M Daily Weather Measurements 850K Precision Rainfall Grids Tracked 200 TB in Amazon S3
- 115. Training Videos Basic Overview Documentation Getting Started Guide Developer Guide API Reference FAQs Think Big Training (3-day Dev Course) EMR Bootcamp (on-site consulting)
- 117. Elastic and scalable No upfront CapEx Pay per use + + On demand + = Remove constraints
- 118. Remove constraints = More experimentation
- 119. More experimentation = More innovation
- 120. Focus on your business Leave undifferentiated heavy lifting to us