Beyond the Fridge

The world of Connected Data !
Dr. Werner Vogels!
CTO, Amazon.com!
The amount of information generated during the first day of
a baby’s life today is equivalent to 70 times the information
c...
I. Science"
Observations – Theory – Models – Facts"
Human Genome Project"
Collaborative project to sequence every single letter!
of the human genetic code.!
13 years and $bil...
Beyond the Human Genome"
45+ species sequenced: mouse, rat, gorilla, rabbit, !
platypus, nematode, zebra fish...!
Compare g...
The Next Generation"
New sequencing instruments lead to a dramatic!
drop in cost and time required to sequence a genome.!
...
The 1000 Genomes Projects"
Public/private consortium to build world’s largest!
collection of human genetic variation.!
Hug...
1000 Genomes in the Cloud"
The 1000 Genomes data made available to all on AWS.!
Stored for free as part of the Public Data...
II. Consumer"
Dropcam	
  is	
  the	
  biggest	
  inbound	
  video	
  
service	
  on	
  the	
  Web	
  	
  
•  More	
  data	
  uploaded	
 ...
III. Retail"
UNCERTAINTY"
UNDERSTAND"
YOUR CUSTOMER"
Who	
  is	
  my	
  customer	
  really?	
  	
  
	
  
What	
  do	
  people	
  really	
  like?	
  	
  
What	
  is	
  happenin...
PERSONALIZE"
75% of users select"
movies based on"
recommendations"
More than 27 million users!
~ 30 million plays per day!
More than 40 billion events per day !
~ 4 million ratings per day!...
BIGGER IS BETTER"
IV. Industrial"
V. Sports"
VI. Location"
VII. The Pipeline"
COLLECT	
  |	
  STORE	
  |	
  ORGANIZE	
  |	
  ANALYZE	
  |	
  SHARE	
  
COLLECT	
  |	
  STORE	
  |	
  ORGANIZE	
  |	
  ANALYZE	
  |	
  SHARE	
  
COLLECT	
  |	
  STORE	
  |	
  ORGANIZE	
  |	
  ANALYZE	
  |	
  SHARE	
  
COLLECT	
  |	
  STORE	
  |	
  ORGANIZE	
  |	
  ANALYZE	
  |	
  SHARE	
  
COLLECT	
  |	
  STORE	
  |	
  ORGANIZE	
  |	
  ANALYZE	
  |	
  SHARE	
  
COLLECT	
  |	
  STORE	
  |	
  ORGANIZE	
  |	
  ANALYZE	
  |	
  SHARE	
  
VIII. Real-time"
What was happening 

yesterday?!
What ! right now?!
trades are executing!
is the exception rate!
is the ad click-through!
topics are trending"
inventory re...
Kinesis!
Kinesis	
  architecture	
  
Amazon Web Services
AZ AZ AZ
Durable, highly consistent storage replicates data
across three d...
AWS	
  Internal	
  Metering	
  Service	
  
Capture
Submissions
Process in
Realtime
Store in
Redshift
Clients
Submitting
Da...
Workload	
  
•  Daily	
  load	
  of	
  billions	
  records	
  from	
  millions	
  of	
  files	
  from	
  
hundreds	
  of	
 ...
IX. Beyond the Display"
CONNECTED DATA
REQUIRES

NO LIMITS"
Cloud enables
connected data
collection!
Cloud enables
connected data
processing!
Cloud enables
connected data
collaboration!
werner@amazon.com	
  
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge
Upcoming SlideShare
Loading in …5
×

AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge

1,723 views
1,514 views

Published on

AWS Summit Paris closing keynote by Werner Vogels

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,723
On SlideShare
0
From Embeds
0
Number of Embeds
736
Actions
Shares
0
Downloads
69
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

AWS Paris Summit 2014 - Closing Keynote Werner Vogels - Beyond the fridge

  1. 1. Beyond the Fridge
 The world of Connected Data ! Dr. Werner Vogels! CTO, Amazon.com!
  2. 2. The amount of information generated during the first day of a baby’s life today is equivalent to 70 times the information contained in the Library of Congress"
  3. 3. I. Science"
  4. 4. Observations – Theory – Models – Facts"
  5. 5. Human Genome Project" Collaborative project to sequence every single letter! of the human genetic code.! 13 years and $billions to complete.! Gigabyte scale datasets (transferred between sites on! iPods!)!
  6. 6. Beyond the Human Genome" 45+ species sequenced: mouse, rat, gorilla, rabbit, ! platypus, nematode, zebra fish...! Compare genomes between species to identify! biologically interesting areas of the genome.! 100Gb scale datasets. Increased computational requirements.!
  7. 7. The Next Generation" New sequencing instruments lead to a dramatic! drop in cost and time required to sequence a genome.! Sequence and compare genetic code of individuals to! find areas of variation. Much more interesting.! Terabyte scale datasets. Significant computational requirements.!
  8. 8. The 1000 Genomes Projects" Public/private consortium to build world’s largest! collection of human genetic variation.! Hugely important dataset to drive new insight into! known genetic traits, and the identification of new ones.! Vast, complex data and computational resources required, beyond reach of most research groups and hospitals.!
  9. 9. 1000 Genomes in the Cloud" The 1000 Genomes data made available to all on AWS.! Stored for free as part of the Public Datasets program.! Updated regularly.! 200Tb. 1700 individual genomes. As much compute and storage as required available to all.!
  10. 10. II. Consumer"
  11. 11. Dropcam  is  the  biggest  inbound  video   service  on  the  Web     •  More  data  uploaded  per   minute  than  YouTube     •  Petabytes  of  data   processed  every  month   •  Billions  of  mo=on  events   detected  
  12. 12. III. Retail"
  13. 13. UNCERTAINTY"
  14. 14. UNDERSTAND" YOUR CUSTOMER"
  15. 15. Who  is  my  customer  really?       What  do  people  really  like?     What  is  happening  socially  with  my  products?     Where  do  people  consume  my  product?   How  do  people  really  use  your  product?    
  16. 16. PERSONALIZE"
  17. 17. 75% of users select" movies based on" recommendations"
  18. 18. More than 27 million users! ~ 30 million plays per day! More than 40 billion events per day ! ~ 4 million ratings per day! ~ 3 million searches per day! Geo-location data! Device information! Time of day and week (it now can verify that users watch more TV shows during the week and more movies during the weekend)! Metadata from third parties such as Nielsen! Social media data from Facebook and Twitter!
  19. 19. BIGGER IS BETTER"
  20. 20. IV. Industrial"
  21. 21. V. Sports"
  22. 22. VI. Location"
  23. 23. VII. The Pipeline"
  24. 24. COLLECT  |  STORE  |  ORGANIZE  |  ANALYZE  |  SHARE  
  25. 25. COLLECT  |  STORE  |  ORGANIZE  |  ANALYZE  |  SHARE  
  26. 26. COLLECT  |  STORE  |  ORGANIZE  |  ANALYZE  |  SHARE  
  27. 27. COLLECT  |  STORE  |  ORGANIZE  |  ANALYZE  |  SHARE  
  28. 28. COLLECT  |  STORE  |  ORGANIZE  |  ANALYZE  |  SHARE  
  29. 29. COLLECT  |  STORE  |  ORGANIZE  |  ANALYZE  |  SHARE  
  30. 30. VIII. Real-time"
  31. 31. What was happening 
 yesterday?!
  32. 32. What ! right now?! trades are executing! is the exception rate! is the ad click-through! topics are trending" inventory remains! queries are slow! are the high scores! ! !
  33. 33. Kinesis!
  34. 34. Kinesis  architecture   Amazon Web Services AZ AZ AZ Durable, highly consistent storage replicates data across three data centers (availability zones) Aggregate and archive to S3 Millions of sources producing 100s of terabytes per hour Front End Authentication Authorization Ordered stream of events supports multiple readers Real-time dashboards and alarms Machine learning algorithms or sliding window analytics Aggregate analysis in Hadoop or a data warehouse Inexpensive: $0.028 per million puts
  35. 35. AWS  Internal  Metering  Service   Capture Submissions Process in Realtime Store in Redshift Clients Submitting Data Workload •  Tens of millions records/sec •  Multiple TB per hour •  100,000s of sources New features •  Scale with the business •  Provide real-time alerting •  Inexpensive •  Improved auditing
  36. 36. Workload   •  Daily  load  of  billions  records  from  millions  of  files  from   hundreds  of  sources   •  3  hour  SLA  to  load  and  audit  data   •  Hundreds  of  customers   •  Hundreds  of  queries  per  hour     New  features   •  Our  data  is  fresh,  we  ingest  every  6  hours   •  Now  processing  triple  the  volume  in  less  than  25%  of   the  =me   •  “Hammerstone”  ETL  solu=on     –  Built  on  AWS  Data  Pipeline   –  Build  business  specific  marts   –  Build  workload  specific  clusters   •  Supports  a  variety  of  analy=cs  tools:  Tableau,  R,  Toad,   SQL  Developer,  etc.   Internal  AWS  Data  Warehouse   Over 200 internal data sources Data staged in Amazon S3 "Hammerstone:" Custom ETL using AWS Data Pipeline Data processing Redshift cluster Batch reporting Redshift cluster Ad hoc query Redshift cluster
  37. 37. IX. Beyond the Display"
  38. 38. CONNECTED DATA REQUIRES
 NO LIMITS"
  39. 39. Cloud enables connected data collection!
  40. 40. Cloud enables connected data processing!
  41. 41. Cloud enables connected data collaboration!
  42. 42. werner@amazon.com  

×