Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Walking Through Cloud Serving at Yahoo!

6,132 views

Published on

From Surendra Reddy's presentation "Walking Through Cloud Serving at Yahoo!" at the 2009 Cloud Computing Expo in Santa Clara, CA, USA. Here's the talk description on the Expo's site: http://cloudcomputingexpo.com/event/session/508

Published in: Technology
  • Be the first to comment

Walking Through Cloud Serving at Yahoo!

  1. 1. Yahoo Cloud Serving<br />Surendra Reddy<br />Cloud Computing Conference & ExpoNovember 3, 2009<br /> <br />
  2. 2. Imagining the Future of Cloud!<br />
  3. 3. Cloud is a Promise, But …<br />
  4. 4. Reference:<br /> Tussle in Cyberspace: Defining Tomorrow’s Internet<br /> http://conferences.sigcomm.org/sigcomm/2002/papers/tussle.pdf<br />
  5. 5. Customer Tussles:<br /><ul><li>Outages: 76% outages caused by change management
  6. 6. Unmet Needs: Rapidly changing needs</li></li></ul><li>Management Tussles:<br /><ul><li>Growing Costs and Complexity
  7. 7. Long Deployment Cycles: Inflexible, peak-provisioned infrastructure</li></li></ul><li>Operations Tussles:<br /><ul><li>Entangled in many day-to-day failures & troubleshooting
  8. 8. Crippled Infrastructure
  9. 9. Building on weak foundation</li></li></ul><li>Developers Tussles:<br /><ul><li>Bloated Operating System, App Stacks
  10. 10. Staggering Complexities of Dependencies
  11. 11. No time for QA – Bugs Creeping in all over</li></li></ul><li>Security Tussles:<br /><ul><li>Whose problem is it?
  12. 12. Auditability and Assurance
  13. 13. Governance
  14. 14. Compliance</li></li></ul><li>
  15. 15. As a Result: <br /><ul><li>More Friction, Waste, and Frustration
  16. 16. Less Innovation </li></li></ul><li>So, What is Cloud?<br /><ul><li>C – Customer Driven
  17. 17. L – Lean
  18. 18. O – Optimized
  19. 19. U – Utility
  20. 20. D – Demand-Driven</li></li></ul><li><ul><li>Cloudis NOT about Cost cutting
  21. 21. It is about Value Creation enabling Innovation</li></li></ul><li><ul><li>Cloud is an opportunity to clean up the Infrastructure mess!</li></li></ul><li>
  22. 22. Yahoo! is Perfect for Cloud Computing<br />600M<br />UNIQUE USERS / MONTH<br />300M+<br />YAHOO! MAIL USERS / MONTH<br />HUNDREDS<br />OF PROPERTIES / PRODUCTS<br />BILLIONS<br />OF OBJECTS STORED<br />HUNDREDS<br />OF PETABYTES OF STORAGE<br />PETABYTES<br />OF TRAFFIC DAILY<br />
  23. 23. <ul><li>Declarative language for specifying the full, end-to-end structure of a service
  24. 24. Includes multiple environments: Dev, QA, Staging, and production
  25. 25. Extreme Automation</li></li></ul><li>Yahoo! Cloud Serving - Principles<br /><ul><li>Metastructure: Abstracts concerns of the underlying infrastructure and the network communication
  26. 26. Declarative application structure
  27. 27. Continuous Integration, Continuous Deployment, & Continuous Compliance*
  28. 28. End-to-End Security</li></ul>* Rich Miller @RHM2K first coined “Continuous Compliance” <br />
  29. 29. Yahoo! Cloud Serving - Principles<br /><ul><li>Virtualized hardware
  30. 30. Standardized software stacks and packaging
  31. 31. Integrated service management
  32. 32. Containers vs. Machines</li></li></ul><li>Yahoo! Cloud Serving - Principles<br /><ul><li>Elastic serving of changing workloads
  33. 33. Controlled/Intelligent traffic direction
  34. 34. Controlled execution environment
  35. 35. Managed Communication
  36. 36. Service Associations, Bindings, and Access Controls</li></li></ul><li>Yahoo! Cloud: Tiers without Tears<br />Entrypoint<br />Naming<br />Presentation<br />Tier<br />Application Logic<br />Tier<br />Bindings<br />Connections<br />Database Tier<br />
  37. 37. Yahoo! Cloud: Tiers without Tears<br />Entrypoint<br />Naming<br />Load<br />Balancer<br />Presentation<br />Tier<br />Application Logic<br />Tier<br />node<br />node<br />node<br />Bindings<br />Connections<br />Database Tier<br />
  38. 38. Yahoo! Cloud: Extreme Automation<br />Abstracting Services<br />Component AnswersCore {<br /> Entry root;<br />Tier pres {<br /> app = “answers_web_tier”;<br /> Exit applogic;<br /> nsize = small;<br /> nnodes = 3;<br /> };<br />Tier applogic {<br /> app = “answers_al_tier”;<br /> Exit data;<br /> nsize = medium;<br /> nnodes = 3;<br /> };<br />Tier data;<br /> Connections {<br /> root -&gt; pres;<br /> pres.applogic -&gt; applogic;<br /> applogic.data -&gt; data;<br /> }<br />} <br />Presentation<br />Tier<br />Application Logic<br />Tier<br />Database Tier<br />
  39. 39. Yahoo! Cloud: Extreme Automation<br />QA Environment<br />Environment qa(ne1) {<br /> TrafficDirector root {<br /> name = “qa.answers.yahoo.com”;<br /> access = testing;<br /> };<br /> AnswersCore core;<br /> Connections {<br /> root -&gt; core;<br /> }<br />} <br />Presentation<br />Tier<br />Presentation<br />Tier<br />Application Logic<br />Tier<br />Application Logic<br />Tier<br />Testing Testing<br />Environment production(ac4,sk1) {<br /> TeeTrafficDirector root {<br /> name = “answers.yahoo.com”;<br /> access = public;<br /> split_percent = 10;<br /> split_destination = qa.root;<br /> };<br /> AnswersCore core;<br /> Connections {<br /> root -&gt; core;<br /> }<br />} <br />Database Tier<br />Database Tier<br />
  40. 40. Yahoo! Cloud: Extreme Automation<br />Apps written against rich stacks<br />Includes core infra and platforms<br /><ul><li>Infrastructure: Security, monitoring, metering, abuse prevention
  41. 41. Platforms: data, user info, ad systems, configuration bundles
  42. 42. All “auto configured”</li></li></ul><li>Serving Cloud Serving for Y!’s Cloud<br />
  43. 43. <ul><li>Infrastructure to Metastructure
  44. 44. Machines to Containers
  45. 45. Friction to Innovation
  46. 46. Tears to Tiers
  47. 47. Complexity to Simplexity
  48. 48. Costs to Revenue Streams</li></li></ul><li>Cloud is a Promise, Service is the fulfillment!<br />
  49. 49. OPEN CLOUD ACCESS PROTOCOL<br />
  50. 50. OPEN CLOUD ACCESS PROTOCOL<br /><ul><li>Definition, Deployment, and Life Cycle Management of Cloud Resources
  51. 51. Allocation, Provisioning, and Metering of Cloud Resources
  52. 52. Metadata/Registry for Cloud Resources
  53. 53. Virtual Infrastructure Management</li></li></ul><li>Yahoo! Submitted OpenCAP to IETF<br /><ul><li>Why IETF?
  54. 54. Specification behind Yahoo Implementation
  55. 55. Plan to open source implementation</li></li></ul><li>Open Cloud Access Protocol<br />Resource Model<br /><ul><li>All Infrastructure and Functional Components are modeled as Resources
  56. 56. Nodes, Networks, Storage Devices, Databases, Load Balancers, Identities, Key Value Store, Object Stores, Caching Services</li></ul>Resource Properties<br /><ul><li>Modeled as JSON objects
  57. 57. Standard Catalog of attributes, extensible</li></ul>Resource Operations<br /><ul><li>Operation, Control, and Notification</li></li></ul><li>Open Cloud Access Protocol<br />Management Operation Services<br /><ul><li>CREATE - Create an instance of a managed resource
  58. 58. DELETE - Delete an instance of a managed resource
  59. 59. GET - Request managed resource properties (for one resource or a set of resource)
  60. 60. CANCEL - Cancel an outstanding request
  61. 61. SET - Set managed resource properties
  62. 62. ACTION - Request an action to be performed on a managed resource</li></li></ul><li>Open Cloud Access Protocol<br />Management Notification Services<br /><ul><li>EVENT- Send events occurring on managed resources (VM events, VM stats etc, SNMP events etc)
  63. 63. ALERT- Send alerts for resource failures and exceptions</li></ul>Life Cycle Management Services<br /><ul><li>START- Starts the resource
  64. 64. TERMINATE – Stop the resource
  65. 65. SUSPEND – Suspend the resource
  66. 66. RESUME – Resume the resource
  67. 67. ARCHIVE – Archive the resource</li></li></ul><li>Yahoo! Open Source & Research<br />Open Source<br /><ul><li>Cloud Serving Platform, Yahoo Implementation of OpenCAP
  68. 68. Traffic Server</li></ul>Research Activities<br /><ul><li>Open Cloud Consortium
  69. 69. OpenCirrus & M45</li></li></ul><li>THANK YOU<br />Surendra Reddy<br />@sureddy twitter<br />sureddy@yahoo-inc.com email<br />yCloudBlog.com blog<br />
  70. 70. TUESDAY, 11/3<br />4:50pm – 5:35 pm<br />Hadoop @ Yahoo! – Internet<br />Scale Data Processing<br />Eric Baldeschwieler<br />VP, Hadoop Software Development<br />WEDNESDAY, 11/4<br />9:10 am - 9:55 am<br />Yahoo! Scalable Storage and<br />Delivery Services<br />Chuck Neerdaels<br />VP, Storage and Edge Services<br />VISIT BOOTH #103 <br />TO TALK WITH YAHOO! ENGINEERS AND LEARN MORE ABOUT YAHOO!’S VISION FOR CLOUD COMPUTING.<br />

×