Jeremy Edberg

            Text




    Operations Manager
       reddit.com
http://www.revolutioncloud.com
What is reddit?
• reddit is an online community




         http://www.revolutioncloud.com
Conde Nast owns reddit




    http://www.revolutioncloud.com
Conde Nast websites




   http://www.revolutioncloud.com
Timeline
April 2006 -- S3 for logos


                 September 2007 -- S3 for thumbnails


November 2008 -- EC2 for batc...
What led us to AWS
      (part 1)
• Needed an easy way to distribute and
  upload our logo




        http://www.revoluti...
http://www.revolutioncloud.com
What led us to AWS
      (part 2)
• Thumbnails!




        http://www.revolutioncloud.com
What led us to AWS
      (part 3)
• Didn’t want to rent another cabinet




         http://www.revolutioncloud.com
What led us to AWS
      (part 3)
• Didn’t want to rent another cabinet
• Didn’t want to buy more servers




         htt...
What led us to AWS
      (part 3)
• Didn’t want to rent another cabinet
• Didn’t want to buy more servers

               ...
Imaging and Racking Servers Is
  A (Sometimes Fun) Chore




      http://www.revolutioncloud.com
EC2 for Overflow
• Used openvpn to create a secure link to
  our datacenter for batch processing




        http://www.rev...
Moving to EC2
• Started by migrating all data




         http://www.revolutioncloud.com
Moving to EC2
• Started by migrating all data
• Got a complete stack running on EC2




        http://www.revolutioncloud...
Moving to EC2
• Started by migrating all data
• Got a complete stack running on EC2
• Long Saturday night finishing the mig...
Architecture




http://www.revolutioncloud.com
Stats
• 190 Virtual CPUs
• 338GB of RAM
• 9TB of Elastic Block Storage
• 2TB of S3 Storage
• 6.5 TB of Data Out / mo
• 2TB...
Benefits
Data Center (per month)                EC2 (per month)

  Servers:       $6K                Servers:              ...
Benefits
• Don’t have to procure servers anymore




        http://www.revolutioncloud.com
Benefits
• Don’t have to procure servers anymore
• No racking or imaging servers anymore




        http://www.revolutionc...
Benefits
• Don’t have to procure servers anymore
• No racking or imaging servers anymore
• Time to market is faster



    ...
Benefits
• Don’t have to procure servers anymore
• No racking or imaging servers anymore
• Time to market is faster
• Multi...
Benefits
• Don’t have to procure servers anymore
• No racking or imaging servers anymore
• Time to market is faster
• Multi...
Benefits
• Don’t have to procure servers anymore
• No racking or imaging servers anymore
• Time to market is faster
• Multi...
Pain Points




http://www.revolutioncloud.com
Pain Points




http://www.revolutioncloud.com
Pain Points

Higher Latency


Workaround: Fewer network calls, ask for
more data at a time.



       http://www.revolutio...
Pain Points

EBS sometimes slows down a bit


Workaround: Use caching and replication
with read slaves to avoid relying on...
Pain Points

Instances go away sometimes


Workaround: Avoid single points of failure
and make sure your servers have auto...
Pain Points
Fixing these issues made
our app more reliable and
highly available. We are
better off than when we
started.
 ...
Best Practices
• Keep data in multiple Availability Zones




         http://www.revolutioncloud.com
Best Practices
• Keep data in multiple Availability Zones
• EBS for all persistent data




         http://www.revolution...
Best Practices
• Keep data in multiple Availability Zones
• EBS for all persistent data
• Snapshots



         http://www...
Best Practices
• Keep data in multiple Availability Zones
• EBS for all persistent data
• Snapshots
• No secret keys on th...
Best Practices
• Keep data in multiple Availability Zones
• EBS for all persistent data
• Snapshots
• No secret keys on th...
Best Practices

• A full stack in each zone



         http://www.revolutioncloud.com
Best Practices

• A full stack in each zone
• All data stored as key-value pairs


         http://www.revolutioncloud.com
Best Practices

• A full stack in each zone
• All data stored as key-value pairs
• More use of queues

         http://www...
Conclusion
• AWS saves us a lot of money




        http://www.revolutioncloud.com
Conclusion
• AWS saves us a lot of money
• AWS gives us a lot of flexibility




         http://www.revolutioncloud.com
Conclusion
• AWS saves us a lot of money
• AWS gives us a lot of flexibility
• Moving to AWS has forced us to build
  bette...
Conclusion
• AWS saves us a lot of money
• AWS gives us a lot of flexibility
• Moving to AWS has forced us to build
  bette...
Revolution: Cloud


To learn more, visit
http://www.revolutioncloud.com




        http://www.revolutioncloud.com
Upcoming SlideShare
Loading in …5
×

reddit.com and Our Move to the Cloud

6,192 views

Published on

Presentation by Jeremy Edberg at the AWS Event in Los Angeles on
October 15, 2009. The presentation discusses the experiences and
strategies reddit.com used to migrate IT infrastructure to EC2 and S3.
Best Practices and lessons learned are also presented.

Published in: Technology, Business

reddit.com and Our Move to the Cloud

  1. 1. Jeremy Edberg Text Operations Manager reddit.com http://www.revolutioncloud.com
  2. 2. What is reddit? • reddit is an online community http://www.revolutioncloud.com
  3. 3. Conde Nast owns reddit http://www.revolutioncloud.com
  4. 4. Conde Nast websites http://www.revolutioncloud.com
  5. 5. Timeline April 2006 -- S3 for logos September 2007 -- S3 for thumbnails November 2008 -- EC2 for batch processing May 2009 -- EC2 for entire site http://www.revolutioncloud.com
  6. 6. What led us to AWS (part 1) • Needed an easy way to distribute and upload our logo http://www.revolutioncloud.com
  7. 7. http://www.revolutioncloud.com
  8. 8. What led us to AWS (part 2) • Thumbnails! http://www.revolutioncloud.com
  9. 9. What led us to AWS (part 3) • Didn’t want to rent another cabinet http://www.revolutioncloud.com
  10. 10. What led us to AWS (part 3) • Didn’t want to rent another cabinet • Didn’t want to buy more servers http://www.revolutioncloud.com
  11. 11. What led us to AWS (part 3) • Didn’t want to rent another cabinet • Didn’t want to buy more servers New Servers New Servers http://www.revolutioncloud.com
  12. 12. Imaging and Racking Servers Is A (Sometimes Fun) Chore http://www.revolutioncloud.com
  13. 13. EC2 for Overflow • Used openvpn to create a secure link to our datacenter for batch processing http://www.revolutioncloud.com
  14. 14. Moving to EC2 • Started by migrating all data http://www.revolutioncloud.com
  15. 15. Moving to EC2 • Started by migrating all data • Got a complete stack running on EC2 http://www.revolutioncloud.com
  16. 16. Moving to EC2 • Started by migrating all data • Got a complete stack running on EC2 • Long Saturday night finishing the migration and “forklifting” the last bits of data http://www.revolutioncloud.com
  17. 17. Architecture http://www.revolutioncloud.com
  18. 18. Stats • 190 Virtual CPUs • 338GB of RAM • 9TB of Elastic Block Storage • 2TB of S3 Storage • 6.5 TB of Data Out / mo • 2TB of Data In / mo • 150M+ Pageviews and just one sysadmin! http://www.revolutioncloud.com
  19. 19. Benefits Data Center (per month) EC2 (per month) Servers: $6K Servers: $13K Cabinet (x3): $15K Storage: $1.5K Bandwidth: $2.5K Bandwidth: $1.1K Support: N/A Support: $1.2K Total: $23.5K Total: $16.8K 29% Cheaper! Based on Amazon public pricing, reddit open source Estimated Pricing code, and public configuration information http://www.revolutioncloud.com
  20. 20. Benefits • Don’t have to procure servers anymore http://www.revolutioncloud.com
  21. 21. Benefits • Don’t have to procure servers anymore • No racking or imaging servers anymore http://www.revolutioncloud.com
  22. 22. Benefits • Don’t have to procure servers anymore • No racking or imaging servers anymore • Time to market is faster http://www.revolutioncloud.com
  23. 23. Benefits • Don’t have to procure servers anymore • No racking or imaging servers anymore • Time to market is faster • Multiple physical locations with AZs http://www.revolutioncloud.com
  24. 24. Benefits • Don’t have to procure servers anymore • No racking or imaging servers anymore • Time to market is faster • Multiple physical locations with AZs • Get “free” upgrades http://www.revolutioncloud.com
  25. 25. Benefits • Don’t have to procure servers anymore • No racking or imaging servers anymore • Time to market is faster • Multiple physical locations with AZs • Get “free” upgrades • Elasticity! http://www.revolutioncloud.com
  26. 26. Pain Points http://www.revolutioncloud.com
  27. 27. Pain Points http://www.revolutioncloud.com
  28. 28. Pain Points Higher Latency Workaround: Fewer network calls, ask for more data at a time. http://www.revolutioncloud.com
  29. 29. Pain Points EBS sometimes slows down a bit Workaround: Use caching and replication with read slaves to avoid relying on a single disk. http://www.revolutioncloud.com
  30. 30. Pain Points Instances go away sometimes Workaround: Avoid single points of failure and make sure your servers have automated configuration. http://www.revolutioncloud.com
  31. 31. Pain Points Fixing these issues made our app more reliable and highly available. We are better off than when we started. http://www.revolutioncloud.com
  32. 32. Best Practices • Keep data in multiple Availability Zones http://www.revolutioncloud.com
  33. 33. Best Practices • Keep data in multiple Availability Zones • EBS for all persistent data http://www.revolutioncloud.com
  34. 34. Best Practices • Keep data in multiple Availability Zones • EBS for all persistent data • Snapshots http://www.revolutioncloud.com
  35. 35. Best Practices • Keep data in multiple Availability Zones • EBS for all persistent data • Snapshots • No secret keys on the instance http://www.revolutioncloud.com
  36. 36. Best Practices • Keep data in multiple Availability Zones • EBS for all persistent data • Snapshots • No secret keys on the instance • Different functions in different Security Groups http://www.revolutioncloud.com
  37. 37. Best Practices • A full stack in each zone http://www.revolutioncloud.com
  38. 38. Best Practices • A full stack in each zone • All data stored as key-value pairs http://www.revolutioncloud.com
  39. 39. Best Practices • A full stack in each zone • All data stored as key-value pairs • More use of queues http://www.revolutioncloud.com
  40. 40. Conclusion • AWS saves us a lot of money http://www.revolutioncloud.com
  41. 41. Conclusion • AWS saves us a lot of money • AWS gives us a lot of flexibility http://www.revolutioncloud.com
  42. 42. Conclusion • AWS saves us a lot of money • AWS gives us a lot of flexibility • Moving to AWS has forced us to build better applications and follow better IT practices. http://www.revolutioncloud.com
  43. 43. Conclusion • AWS saves us a lot of money • AWS gives us a lot of flexibility • Moving to AWS has forced us to build better applications and follow better IT practices. • http://code.reddit.com http://www.revolutioncloud.com
  44. 44. Revolution: Cloud To learn more, visit http://www.revolutioncloud.com http://www.revolutioncloud.com

×