COSCUP - Open Source Engines Providing Big Data in the Cloud, Markku Lepisto
Report
Share
•7 likes•3,519 views
COSCUP - Open Source Engines Providing Big Data in the Cloud, Markku Lepisto
•7 likes•3,519 views
Report
Share
Recommended
More Related Content
What's hot
What's hot(20)
Viewers also liked
Viewers also liked(20)
Similar to COSCUP - Open Source Engines Providing Big Data in the Cloud, Markku Lepisto
Similar to COSCUP - Open Source Engines Providing Big Data in the Cloud, Markku Lepisto(20)
More from Amazon Web Services
More from Amazon Web Services(20)
Recently uploaded
Recently uploaded(7)
COSCUP - Open Source Engines Providing Big Data in the Cloud, Markku Lepisto
- 1. Markku Lepistö Technology Evangelist Amazon Web Services @markkulepisto
- 3. #1 ●○○○○
- 4. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance. We are constantly producing more data
- 5. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance. From all types of industries
- 8. The Role of Data is Changing
- 9. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance. Un#l%now,%Ques#ons%you%ask%drove%Data%model% New%model%is%collect%as%much%data%as%possible% –%“Data>First%Philosophy”%
- 10. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance. Data is the new raw material for any business on par with capital, people, labor Datais the new raw material for business on par with capital & labor
- 12. Generated data Available for analysis Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011 IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares
- 14. lunch hours last year?
- 15. ! select productId, count(*) from page_hits where hour in (12,13) group by productId order by count(*) desc! ! cat *-(12|13) | cut –f3 | sort | uniq -c > out! Hit <enter>?
- 16. 1PB = 10^15 (1,000,000,000,000,000) bytes 1 PB = 231 days at 50MB/s
- 17. Solution: Massively Parallel Processing
- 18. #2 ○●○○○
- 23. Very%large% log% (e.g%TBs)% Split into small pieces Lots of actions by John
- 24. Very%large% log% (e.g%TBs)% Process in a hadoop cluster Split into small pieces Lots of actions by John
- 25. Very%large% log% (e.g%TBs)% John’s% history% Process in a hadoop cluster Aggregate the results Split into small pieces Lots of actions by John
- 27. map Input file reduce Output file map Input file reduce Output file map Input file reduce Output file Worker node Worker node Worker node
- 30. #3 ♥ ○○●○○
- 31. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance.
- 32. Elastic On Demand Pay as you go Focus on YOUR business
- 33. Elastic On Demand Pay as you go Focus on YOUR business
- 34. November
- 37. November
- 41. #4 ○○○●○
- 42. EMR is Hadoop in the Cloud!
- 45. Media/ Advertising Targeted Advertising Image and Video Processing Oil & Gas Seismic Analysis Retail Recommendations Transactions Analysis Life Sciences Genome Analysis Financial Services Monte Carlo Simulations Risk Analysis Security Anti-virus Fraud Detection Image Recognition Social Network/ Gaming User Demographics Usage analysis In-game metrics
- 48. Job Flows Custom JAR Cascading Streaming Ruby, Perl, Python, PHP, R, Bash, C++
- 49. Data Warehouse for Hadoop SQL-like query language Hive
- 50. High-level programming Ideal for data flow / ETL Pig
- 51. Near real time key/value store for structured data HBase
- 52. Distributed monitoring of cluster and nodes Ganglia
- 53. Statistical computing and graphics Machine learning library discover Value in Data
- 54. Data Strategist
- 55. Unknown Unknowns
- 56. Elastic On Demand Pay as you go Focus on YOUR business
- 60. elastic-mapreduce --create --key-pair micro --region eu-west-1 --name MyJobFlow --num-instances 5 --instance-type m2.4xlarge –-alive --log-uri s3n://mybucket/EMR/log Instance type/count
- 61. elastic-mapreduce --create --key-pair micro --region eu-west-1 --name MyJobFlow --num-instances 5 --instance-type m2.4xlarge –-alive --pig-interactive --pig-versions latest --hive-interactive –-hive-versions latest --hbase --log-uri s3n://mybucket/EMR/log Adding Hive, Pig and Hbase to the job flow
- 62. Elastic On Demand Pay as you go Focus on YOUR business
- 63. 1 instance for 1000 hours = 1000 instances for 1 hour
- 65. …to Thousands
- 68. Turn Off the Resources and Stop Paying
- 69. Elastic On Demand Pay as you go Focus on YOUR business
- 70. Source: IDC Whitepaper, sponsored by Amazon, The Business Value of Amazon Web Services Accelerates Over Time. July 2012 70% lower 5 year TCO per app AWS On- premises $3.01M $0.90M 50% reduction in analytics costs
- 71. Save more money by using Spot Instances
- 72. 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20% PercentageoftheDistribution Bid Price as Percentage of the On-Demand Price Bid Distribution Typical Spot Bidding Strategies
- 74. 14%hrs% Without#Spot# 4%instances%*%14%hrs%*%$0.50%=%$28% EMR with Spot Instances 14%hrs%
- 76. With#Spot# 4%instances%*%7%hrs%*%$0.50%=%$14%+%%% 14%hrs% Without#Spot# 4%instances%*%14%hrs%*%$0.50%=%$28% EMR with Spot Instances 7%hrs%
- 77. With#Spot# 4%instances%*%7%hrs%*%$0.50%=%$14%+% 5%instances%*%7%hrs%*%$0.25%=%$8.75% Total%=%$22.75% 14%hrs% Without#Spot# 4%instances%*%14%hrs%*%$0.50%=%$28% EMR with Spot Instances 7%hrs%
- 80. #5 ○○○○●
- 81. What kind of movies do people like ?
- 82. More than 25 Million Streaming Members 50 Billion Events Per Day 30 Million plays every day 2 billion hours of video in 3 months 4 million ratings per day 3 million searches Device location , time , day, week etc. Social data
- 83. 10 TB of streaming data per day
- 84. Da ta $C enter S3 Netflix(Data(Center Legacy data from on-premise data center Legacy Data
- 85. Customer dimension data stored in Cassandra
- 86. ~1 PB of data stored in Amazon S3 S3
- 87. Wide range of processing languages used EMR Prod%Cluster% (EMR) S3
- 88. Data consumed in multiple ways S3 EMR Prod%Cluster% (EMR) Recommendation Engine Ad-hoc Analysis Personalization
- 91. Durability
- 92. Versioning
- 95. Foursquare… 33 million users 1.3 million businesses …generates a lot of Data 3.5 billion check-ins 15M+ venues, Terabytes of log data
- 96. Uses EMR for Evaluation of new features Machine learning Exploratory analysis Daily customer usage reporting Long-term trend analysis
- 97. Benefits of EMR Ease-of-Use “We have decreased the processing time for urgent data-analysis” Flexibility To deal with changing requirements & dynamically expand reporting clusters Costs “We have reduced our analytics costs by over 50%”
- 98. ApplicationStack Scala/Liftweb API Machines WWW Machines Batch Jobs Scala Application code Mongo/Postgres/ Flat Files Databases LogsDataStack Amazon S3 Database Dumps Log Files Hadoop Elastic Map Reduce Hive/Ruby/ Mahout Analytics Dashboard Map Reduce Jobs mongoexport postgres dump Flume
- 99. ApplicationStack Scala/Liftweb API Machines WWW Machines Batch Jobs Scala Application code Mongo/Postgres/ Flat Files Databases LogsDataStack Amazon S3 Database Dumps Log Files Hadoop Elastic Map Reduce Hive/Ruby/ Mahout Analytics Dashboard Map Reduce Jobs mongoexport postgres dump Flume
- 100. ApplicationStack Scala/Liftweb API Machines WWW Machines Batch Jobs Scala Application code Mongo/Postgres/ Flat Files Databases LogsDataStack Amazon S3 Database Dumps Log Files Hadoop Elastic Map Reduce Hive/Ruby/ Mahout Analytics Dashboard Map Reduce Jobs mongoexport postgres dump Flume
- 101. ApplicationStack Scala/Liftweb API Machines WWW Machines Batch Jobs Scala Application code Mongo/Postgres/ Flat Files Databases LogsDataStack Amazon S3 Database Dumps Log Files Hadoop Elastic Map Reduce Hive/Ruby/ Mahout Analytics Dashboard Map Reduce Jobs mongoexport postgres dump Flume
- 102. 0 0.1 0.2 0.3 0.4 0.5 0.6 Female Male Gender 0 10 20 30 40 50 60 70 80 Age
- 103. Gorilla Coffee Gray's Papaya Amorino Thursday% Friday% Saturday% Sunday%
- 104. Who is using our service?
- 105. Finding signal in the noise of logs
- 110. Log files 250 EMR clusters spun up and down every week
- 111. Common Crawl 1000 Genomes Project Census Data 54 other datasets http://aws.amazon.com/publicdatasets/
- 114. 150B Soil Observations 3M Daily Weather Measurements 850K Precision Rainfall Grids Tracked 200 TB in Amazon S3
- 115. Training Videos Basic Overview Documentation Getting Started Guide Developer Guide API Reference FAQs Think Big Training (3-day Dev Course) EMR Bootcamp (on-site consulting)
- 117. Elastic and scalable No upfront CapEx Pay per use + + On demand + = Remove constraints
- 118. Remove constraints = More experimentation
- 119. More experimentation = More innovation
- 120. Focus on your business Leave undifferentiated heavy lifting to us