How NYTimes.com uses Amazon Web Services - AWS Summit 2011
•Download as KEY, PDF•
11 likes•9,690 views
The deck I presented at AWS Summit 2011 in Hilton New York
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (20)
Similar to How NYTimes.com uses Amazon Web Services - AWS Summit 2011
Similar to How NYTimes.com uses Amazon Web Services - AWS Summit 2011 (20)
Recently uploaded
Recently uploaded (20)
How NYTimes.com uses Amazon Web Services - AWS Summit 2011
- 1. NYTimes.com Vadim Jelezniakov AWS Summit 2011 1
- 2. Next 10 min The Company Why Cloud The App Examples Progress So Far What’s Next 2
- 3. The New York Times Company The New York Times The International Herald Tribune The Boston Globe 15 daily newspapers 50 Web sites NYTimes.com Boston.com About.com 3
- 4. NYTimes.com’s Stats In the US: 38.1 mil unique users / month 810 mil page views / month Worldwide: 62 mil unique users / month 984 mil page views / month #1 individual newspaper site #2 media site bloggers link to the most* #4 news and information Web site comScore Report, April 2011 *Technorati's index 4
- 5. Why Cloud? Why AWS? On Demand Availability Cost-Effective Scalability IaaS = Flexible Rapid Deployment Comprehensive 5
- 6. Why Cloud? Why AWS? On Demand Availability Cost-Effective Scalability IaaS = Flexible Rapid Deployment Comprehensive 6
- 7. Why Cloud? Why AWS? On Demand Availability Cost-Effective Scalability IaaS = Flexible Rapid Deployment Comprehensive 7
- 8. Examples On Demand Capacity - TimesMachine Flexibility - Times Skimmer Scalability - User Generated Content (Comments) 8
- 9. TimesMachine (2007, 2008) 130 years of archives in 36 hours 400K objects, 4T of data S3, EC2, Hadoop http://timesmachine.nytimes.com 9
- 10. Skimmer Private Beta (Feb 14th, 2009) http://www.nytimes.com/skimmer/ 10
- 11. “SkimmerGate” 11:00 News leaks to TechCrunch, Google Finance 11:10 Average load on the instance is 40 11:15 Snapshotting the instance, configuring the LB, testing 12:00 Four Skimmer instances load balanced, quality is restored 1 hour - from prototype to production 11
- 12. Comments on Articles and Blogs 12
- 13. Rate and Review for Movies, Theater, Dining and Travel 13
- 14. We like To be able to scale quickly on demand We don’t like calls from Systems team at 6PM on Fridays 14
- 15. Traffic Spikes - Add Capacity 15
- 16. Traffic Spikes - Add Capacity 15
- 17. Load Balancing / Security View 16
- 19. Gap Analysis • Load Balancing • Communication between instances • Talking back to existing Datacenter • Scaling up and back • Securing the instances 18
- 20. 19
- 21. 20
- 22. 21
- 23. 22
- 24. 23
- 25. 24
- 26. 25
- 27. Tools Developed for the Cloud Nimbul Light Cloud Manager http://github.com/nimbul/nimbul Emissary Fast AMQP Messaging http://github.com/nimbul/emissary CloudSource Simple SVN Deployment http://github.com/nimbul/cloudsource based on ServerMattic developed by WordPress 26
- 28. Progress So Far 27
- 29. NYTimes.com EC2 Expansion Clusters: from 2 to 100. Instances: from 30 to 400. 28
- 30. NYTimes.com EC2 Expansion 2009 2011 EC2 DC in comparison with other DCs from 4% to 36% (VMs/pure physical servers/instance count) 29
- 31. NYTimes.com AWS Expansion AWS contributes ~27% of total operational cost across DCs (including space, power, bandwidth and excluding CDN) 30
- 32. What’s Next Multi-AZ and Multi-Region SNS, SNS2SQS - in Dev Multi-AZ MySQL RDS Need 'smarter' ELB Need multi-AZ support in VPC 31
- 33. email: vadimj@nytimes.com twitter: @vadimj http://www.nytimes.com http://www.nytimes.com/skimmer/ http://timesmachine.nytimes.com http://open.nytimes.com http://github.com/nimbul/ images licensed from http://www.dreamstime.com/ charts generated using google docs 32
Editor's Notes
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- comments on articles and blogs - we get about 130K comments per month and 1.5 million reader recommendations.\n
- rate and review for movies, theater, dining and travel destinations\n
- \n
- The key here we thought was not only scaling up for spikes but perhaps scaling down at night when not as many of you were commenting. \n
- The key here we thought was not only scaling up for spikes but perhaps scaling down at night when not as many of you were commenting. \n
- \n
- We had lots of questions that were fun to answer. How would the front ends know which api instance to request? Or where exactly is that database the api instance is supposed to query? Better yet, how are we going to manage all of these instances? How exactly will it scale? How will we request internal api’s that live back in our data center?\n\n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n