Cloud Serving Engine

733 views

Published on

Yahoo Cloud Serving Platform

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
733
On SlideShare
0
From Embeds
0
Number of Embeds
47
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • _____We believe, Cloud computing is about Driving Innovation.At Yahoo!, we are developing and deploying Cloud Services to help us drive innovationBy providing a set of horizontal Cloud services that are broadly applicable to building many Yahoo! products We are able to reallocate many developers to building products and features from building product-centric infrastructure. We are providing a stable base of Cloud Services that enable product developers to iterate quickly and innovate without risking product quality We are providing the ability to process and analyze enormous amounts of data easily and quickly for purposes of improving consumer experience with our products.
  • Hundreds of properties/productsEnormous Scale 500M+ unique visitors monthly 300M+ Mail users 100’s of petabytes of storage 100’s of billions of objects stored Billions of daily requestsGlobal footprint
  • Cloud Serving Engine

    1. 1. Yahoo Cloud Serving<br />Surendra Reddy<br />Cloud Computing Conference & ExpoNovember 3, 2009<br /> <br />
    2. 2. Imagining the Future of Cloud!<br />
    3. 3. Cloud is a Promise, But …<br />
    4. 4. Reference:<br /> Tussle in Cyberspace: Defining Tomorrow’s Internet<br /> http://conferences.sigcomm.org/sigcomm/2002/papers/tussle.pdf<br />
    5. 5. Customer Tussles:<br /><ul><li>Outages: 76% outages caused by change management
    6. 6. Unmet Needs: Rapidly changing needs</li></li></ul><li>Management Tussles:<br /><ul><li>Growing Costs and Complexity
    7. 7. Long Deployment Cycles: Inflexible, peak-provisioned infrastructure</li></li></ul><li>Operations Tussles:<br /><ul><li>Entangled in many day-to-day failures & troubleshooting
    8. 8. Crippled Infrastructure
    9. 9. Building on weak foundation</li></li></ul><li>Developers Tussles:<br /><ul><li>Bloated Operating System, App Stacks
    10. 10. Staggering Complexities of Dependencies
    11. 11. No time for QA – Bugs Creeping in all over</li></li></ul><li>Security Tussles:<br /><ul><li>Whose problem is it?
    12. 12. Auditability and Assurance
    13. 13. Governance
    14. 14. Compliance</li></li></ul><li>As a Result: <br /><ul><li>More Friction, Waste, and Frustration
    15. 15. Less Innovation </li></li></ul><li>So, What is Cloud?<br /><ul><li>C – Customer Driven
    16. 16. L – Lean
    17. 17. O – Optimized
    18. 18. U – Utility
    19. 19. D – Demand-Driven</li></li></ul><li><ul><li>Cloudis NOT about Cost cutting
    20. 20. It is about Value Creation enabling Innovation</li></li></ul><li><ul><li>Cloud is an opportunity to clean up the Infrastructure mess!</li></li></ul><li>
    21. 21. Yahoo! is Perfect for Cloud Computing<br />HUNDREDS<br />OF PROPERTIES / PRODUCTS<br />600M<br />UNIQUE USERS / MONTH<br />HUNDREDS<br />OF PETABYTES OF STORAGE<br />BILLIONS<br />OF OBJECTS STORED<br />THOUSANDS<br />OF SERVERS<br />300M+<br />YAHOO! MAIL USERS / MONTH<br />
    22. 22. <ul><li>Declarative language for specifying the full, end-to-end structure of a service
    23. 23. Includes multiple environments: Dev, QA, Staging, and production
    24. 24. Extreme Automation</li></li></ul><li>Yahoo! Cloud Serving - Principles<br /><ul><li>Metastructure: Abstracts concerns of the underlying infrastructure and the network communication
    25. 25. Declarative application structure
    26. 26. Continuous Integration, Continuous Deployment, & Continuous Compliance*
    27. 27. End-to-End Security</li></ul>* Rich Miller @RHM2K first coined “Continuous Compliance” <br />
    28. 28. Yahoo! Cloud Serving - Principles<br /><ul><li>Virtualized hardware
    29. 29. Standardized software stacks and packaging
    30. 30. Integrated service management
    31. 31. Containers vs. Machines</li></li></ul><li>Yahoo! Cloud Serving - Principles<br /><ul><li>Elastic serving of changing workloads
    32. 32. Controlled/Intelligent traffic direction
    33. 33. Controlled execution environment
    34. 34. Managed Communication
    35. 35. Service Associations, Bindings, and Access Controls</li></li></ul><li>Yahoo! Cloud: Tiers without Tears<br />Entrypoint<br />Naming<br />Presentation<br />Tier<br />Application Logic<br />Tier<br />Bindings<br />Connections<br />Database Tier<br />
    36. 36. Yahoo! Cloud: Tiers without Tears<br />Entrypoint<br />Naming<br />Load<br />Balancer<br />Presentation<br />Tier<br />Application Logic<br />Tier<br />node<br />node<br />node<br />Bindings<br />Connections<br />Database Tier<br />
    37. 37. Yahoo! Cloud: Extreme Automation<br />Abstracting Services<br />Component AnswersCore {<br /> Entry root;<br />Tier pres {<br /> app = “answers_web_tier”;<br /> Exit applogic;<br /> nsize = small;<br /> nnodes = 3;<br /> };<br />Tier applogic {<br /> app = “answers_al_tier”;<br /> Exit data;<br /> nsize = medium;<br /> nnodes = 3;<br /> };<br />Tier data;<br /> Connections {<br /> root -&gt; pres;<br /> pres.applogic -&gt; applogic;<br /> applogic.data -&gt; data;<br /> }<br />} <br />Presentation<br />Tier<br />Application Logic<br />Tier<br />Database Tier<br />
    38. 38. Yahoo! Cloud: Extreme Automation<br />QA Environment<br />Environment qa(ne1) {<br />TrafficDirector root {<br /> name = “qa.answers.yahoo.com”;<br /> access = testing;<br /> };<br />AnswersCore core;<br /> Connections {<br /> root -&gt; core;<br /> }<br />} <br />Presentation<br />Tier<br />Presentation<br />Tier<br />Application Logic<br />Tier<br />Application Logic<br />Tier<br />Testing Testing<br />Environment production(ac4,sk1) {<br /> TeeTrafficDirector root {<br /> name = “answers.yahoo.com”;<br /> access = public;<br /> split_percent = 10;<br /> split_destination = qa.root;<br /> };<br /> AnswersCore core;<br /> Connections {<br /> root -&gt; core;<br /> }<br />} <br />Database Tier<br />Database Tier<br />
    39. 39. Yahoo! Cloud: Extreme Automation<br />Apps written against rich stacks<br />Includes core infra and platforms<br /><ul><li>Infrastructure: Security, monitoring, metering, abuse prevention
    40. 40. Platforms: data, user info, ad systems, configuration bundles
    41. 41. All “auto configured”</li></li></ul><li>Serving Cloud Serving for Y!’s Cloud<br />
    42. 42. <ul><li>Infrastructure to Metastructure
    43. 43. Machines to Containers
    44. 44. Friction to Innovation
    45. 45. Tears to Tiers
    46. 46. Complexity to Simplexity
    47. 47. Costs to Revenue Streams</li></li></ul><li>Cloud is a Promise, Service is the fulfillment!<br />
    48. 48. OPEN CLOUD ACCESS PROTOCOL<br />
    49. 49. OPEN CLOUD ACCESS PROTOCOL<br /><ul><li>Definition, Deployment, and Life Cycle Management of Cloud Resources
    50. 50. Allocation, Provisioning, and Metering of Cloud Resources
    51. 51. Metadata/Registry for Cloud Resources
    52. 52. Virtual Infrastructure Management</li></li></ul><li>Yahoo! Submitted OpenCAP to IETF<br /><ul><li>Why IETF?
    53. 53. Specification behind Yahoo Implementation
    54. 54. Plan to open source implementation</li></li></ul><li>Open Cloud Access Protocol<br />Resource Model<br /><ul><li>All Infrastructure and Functional Components are modeled as Resources
    55. 55. Nodes, Networks, Storage Devices, Databases, Load Balancers, Identities, Key Value Store, Object Stores, Caching Services</li></ul>Resource Properties<br /><ul><li>Modeled as JSON objects
    56. 56. Standard Catalog of attributes, extensible</li></ul>Resource Operations<br /><ul><li>Operation, Control, and Notification</li></li></ul><li>Open Cloud Access Protocol<br />Management Operation Services<br /><ul><li>CREATE - Create an instance of a managed resource
    57. 57. DELETE - Delete an instance of a managed resource
    58. 58. GET - Request managed resource properties (for one resource or a set of resource)
    59. 59. CANCEL - Cancel an outstanding request
    60. 60. SET - Set managed resource properties
    61. 61. ACTION - Request an action to be performed on a managed resource</li></li></ul><li>Open Cloud Access Protocol<br />Management Notification Services<br /><ul><li>EVENT- Send events occurring on managed resources (VM events, VM stats etc, SNMP events etc)
    62. 62. ALERT- Send alerts for resource failures and exceptions</li></ul>Life Cycle Management Services<br /><ul><li>START- Starts the resource
    63. 63. TERMINATE – Stop the resource
    64. 64. SUSPEND – Suspend the resource
    65. 65. RESUME – Resume the resource
    66. 66. ARCHIVE – Archive the resource</li></li></ul><li>Yahoo! Open Source & Research<br />Open Source<br /><ul><li>Cloud Serving Platform, Yahoo Implementation of OpenCAP
    67. 67. Traffic Server</li></ul>Research Activities<br /><ul><li>Open Cloud Consortium
    68. 68. OpenCirrus & M45</li></li></ul><li>THANK YOU<br />Surendra Reddy<br />@sureddytwitter<br />sureddy@yahoo-inc.comemail<br />yCloudBlog.comblog<br />
    69. 69. TUESDAY, 11/3<br />4:50pm – 5:35 pm<br />Hadoop @ Yahoo! – Internet<br />Scale Data Processing<br />Eric Baldeschwieler<br />VP, Hadoop Software Development<br />WEDNESDAY, 11/4<br />9:10 am - 9:55 am<br />Yahoo! Scalable Storage and<br />Delivery Services<br />Chuck Neerdaels<br />VP, Storage and Edge Services<br />VISIT BOOTH #103 <br />TO TALK WITH YAHOO! ENGINEERS AND LEARN MORE ABOUT YAHOO!’S VISION FOR CLOUD COMPUTING.<br />

    ×