Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2011 AWS Tour Australia, Closing Keynote: How Amazon.com migrated to AWS, by Jon Jenkins

7,510 views

Published on

Published in: Spiritual, Technology, Business
  • Hey guys! Who wants to chat with me? More photos with me here 👉 http://www.bit.ly/katekoxx
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

2011 AWS Tour Australia, Closing Keynote: How Amazon.com migrated to AWS, by Jon Jenkins

  1. 1. AWS Cloud Tour 2011, Australia<br />Closing Keynote:<br />How Amazon.com<br />Migrated to AWS<br />Jon Jenkins<br />Director, Software Development<br />Amazon.com<br />
  2. 2. amazon.com’sJourney to the Cloud<br />Jon Jenkins<br />jjenkin@amazon.com<br />Twitter - #awstour<br />AWS Cloud Tour<br />
  3. 3. ≠<br />
  4. 4. 1995 - 2011<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  5. 5. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  6. 6. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  7. 7. +<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  8. 8. First real data center<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  9. 9. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  10. 10. Distribution Center Isolation<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  11. 11. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  12. 12. “We have 50 million lines of C++ code. No, it's more than that now. I don't know what it is anymore. It was 50 million last Christmas, nine months ago, and was expanding at 8 million lines a quarter. The expansion rate was increasing as well. Ouch.”<br />Amazon SDE, internal blog post<br />September 2004<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  13. 13. Decouple<br />Service Oriented Architecture<br />Scale Horizontally<br />Increase Speed of Execution<br />Develop Iteratively<br />Seek Simplicity<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  14. 14. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  15. 15. What could we do with just S3?<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  16. 16. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  17. 17. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />IMDB Widget Architecture<br />
  18. 18. The Problem<br /><ul><li>Release process is coupled
  19. 19. Runtime latency & scale requirements
  20. 20. Service integration issues</li></ul>The Solution<br /><ul><li>Use S3 as a service
  21. 21. Store raw HTML for the feature in S3</li></ul>1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  22. 22. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />Before<br />After<br />
  23. 23. Results<br /><ul><li>Reduced page latency
  24. 24. IMDB doesn’t worry about scaling
  25. 25. Reduced web server CPU utilization
  26. 26. Improved availability through reduced dependencies
  27. 27. Simplified release model
  28. 28. AJAX readiness</li></ul>1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  29. 29. What about a more complex case?<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  30. 30. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  31. 31. The Problem<br /><ul><li>The system has lots of moving parts
  32. 32. It must run in an external data center
  33. 33. It must scale up quickly
  34. 34. Development team is two people</li></ul>The Solution<br /><ul><li>Use as many AWS services as possible</li></ul>1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  35. 35. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  36. 36. Results<br /><ul><li>Very few dev resources required
  37. 37. Launched without having to negotiate any new datacenter co-lo presence
  38. 38. True external performance metrics
  39. 39. We can test site features in development that have not yet launched
  40. 40. The system scales horizontally to large amounts of traffic</li></ul>1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  41. 41. What about amazon.com web servers?<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  42. 42. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  43. 43. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  44. 44. 39%<br />61%<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  45. 45. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  46. 46. 76%<br />24%<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  47. 47. The Problem<br /><ul><li>Retail web site hardware is underutilized
  48. 48. Traffic spikes require heroic effort
  49. 49. Scaling is non-linear</li></ul>The Solution<br /><ul><li>Migrate the entire www.amazon.com web server fleet to AWS</li></ul>1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  50. 50. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  51. 51. November 10, 2010<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  52. 52. Results<br /><ul><li>All traffic for www.amazon.com is now served from AWS
  53. 53. We can dynamically scale the fleet in increments as small as a single host
  54. 54. Traffic spikes can be handled with ease</li></ul>1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  55. 55. What about a DB use case?<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  56. 56. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  57. 57. Basic Order Storage Architecture<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  58. 58. Basic Order Storage Architecture<br />Scaling Pattern 1<br />Scaling Pattern 2<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  59. 59. The Problem<br /><ul><li>Cumulative data impacts scale
  60. 60. No database scaling pattern is ideal
  61. 61. Databases infrastructure is expensive</li></ul>The Solution<br /><ul><li>Create a tiered storage system with AWS</li></ul>1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  62. 62. 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20102011<br />
  63. 63. Results<br /><ul><li>670 million (4TB) orders now stored in S3
  64. 64. We are spending way less on DB hosts
  65. 65. Sets us up for migration to RDS / SDB</li></ul>1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20102011<br />
  66. 66. Lessons learned<br />1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  67. 67. Business Lessons<br /><ul><li>Less time spent on capacity planning
  68. 68. Fewer conversations with finance
  69. 69. More innovation
  70. 70. Happier developers
  71. 71. I get credit for AWS price reductions
  72. 72. Be sure to consider compliance issues
  73. 73. No more lease returns!</li></ul>1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  74. 74. Technical Lessons<br /><ul><li>Start with simple applications
  75. 75. Iterate toward your desired end-state
  76. 76. Identify reusable components
  77. 77. Engage security early and treat them as partners
  78. 78. Migrate to the cloud in concert with your other architectural objectives
  79. 79. The cloud can’t cover up sloppy engineering</li></ul>1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br />
  80. 80. amazon.com’sJourney to the Cloud<br />Jon Jenkins<br />jjenkin@amazon.com<br />Twitter - #awstour<br />AWS Cloud Tour<br />

×