Yahoo Cloud Serving<br />Surendra Reddy<br />Cloud Computing Conference & ExpoNovember 3, 2009<br /> <br />
Imagining the Future of Cloud!<br />
Cloud is a Promise, But …<br />
Reference:<br />	Tussle in Cyberspace: Defining Tomorrow’s Internet<br />	http://conferences.sigcomm.org/sigcomm/2002/paper...
Customer Tussles:<br /><ul><li>Outages: 76% outages caused by change management
Unmet Needs: Rapidly changing needs</li></li></ul><li>Management Tussles:<br /><ul><li>Growing Costs and Complexity
Long Deployment Cycles: Inflexible, peak-provisioned infrastructure</li></li></ul><li>Operations Tussles:<br /><ul><li>Ent...
Crippled Infrastructure
Building on weak foundation</li></li></ul><li>Developers Tussles:<br /><ul><li>Bloated Operating System, App Stacks
Staggering Complexities of Dependencies
No time for QA – Bugs Creeping in all over</li></li></ul><li>Security Tussles:<br /><ul><li>Whose problem is it?
Auditability and Assurance
Governance
Compliance</li></li></ul><li>
As a Result: <br /><ul><li>More Friction, Waste, and Frustration
Less Innovation </li></li></ul><li>So, What is Cloud?<br /><ul><li>C – Customer Driven
L – Lean
O – Optimized
U – Utility
D – Demand-Driven</li></li></ul><li><ul><li>Cloudis NOT about Cost cutting
It is about Value Creation enabling Innovation</li></li></ul><li><ul><li>Cloud is an opportunity to clean up the Infrastru...
Yahoo! is Perfect for Cloud Computing<br />600M<br />UNIQUE USERS / MONTH<br />300M+<br />YAHOO! MAIL USERS / MONTH<br />H...
<ul><li>Declarative language for specifying the full, end-to-end structure of a service
Includes multiple environments: Dev, QA, Staging, and production
Extreme Automation</li></li></ul><li>Yahoo! Cloud Serving - Principles<br /><ul><li>Metastructure: Abstracts concerns of t...
Declarative application structure
Continuous Integration, Continuous Deployment, & Continuous Compliance*
End-to-End Security</li></ul>* Rich Miller @RHM2K first coined “Continuous Compliance”  <br />
Yahoo! Cloud Serving - Principles<br /><ul><li>Virtualized hardware
Standardized software stacks and packaging
Integrated service management
Containers vs. Machines</li></li></ul><li>Yahoo! Cloud Serving - Principles<br /><ul><li>Elastic serving of changing workl...
Controlled/Intelligent traffic direction
Controlled execution environment
Managed Communication
Service Associations, Bindings, and Access Controls</li></li></ul><li>Yahoo! Cloud: Tiers without Tears<br />Entrypoint<br...
Yahoo! Cloud: Tiers without Tears<br />Entrypoint<br />Naming<br />Load<br />Balancer<br />Presentation<br />Tier<br />App...
Upcoming SlideShare
Loading in...5
×

Walking Through Cloud Serving at Yahoo!

4,724

Published on

From Surendra Reddy's presentation "Walking Through Cloud Serving at Yahoo!" at the 2009 Cloud Computing Expo in Santa Clara, CA, USA. Here's the talk description on the Expo's site: http://cloudcomputingexpo.com/event/session/508

Published in: Technology
0 Comments
18 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,724
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
18
Embeds 0
No embeds

No notes for slide
  • _____We believe, Cloud computing is about Driving Innovation.At Yahoo!, we are developing and deploying Cloud Services to help us drive innovationBy providing a set of horizontal Cloud services that are broadly applicable to building many Yahoo! products We are able to reallocate many developers to building products and features from building product-centric infrastructure. We are providing a stable base of Cloud Services that enable product developers to iterate quickly and innovate without risking product quality We are providing the ability to process and analyze enormous amounts of data easily and quickly for purposes of improving consumer experience with our products.
  • Hundreds of properties/productsEnormous Scale 500M+ unique visitors monthly 300M+ Mail users 100’s of petabytes of storage 100’s of billions of objects stored Billions of daily requestsGlobal footprint
  • Transcript of "Walking Through Cloud Serving at Yahoo!"

    1. 1. Yahoo Cloud Serving<br />Surendra Reddy<br />Cloud Computing Conference & ExpoNovember 3, 2009<br /> <br />
    2. 2. Imagining the Future of Cloud!<br />
    3. 3. Cloud is a Promise, But …<br />
    4. 4. Reference:<br /> Tussle in Cyberspace: Defining Tomorrow’s Internet<br /> http://conferences.sigcomm.org/sigcomm/2002/papers/tussle.pdf<br />
    5. 5. Customer Tussles:<br /><ul><li>Outages: 76% outages caused by change management
    6. 6. Unmet Needs: Rapidly changing needs</li></li></ul><li>Management Tussles:<br /><ul><li>Growing Costs and Complexity
    7. 7. Long Deployment Cycles: Inflexible, peak-provisioned infrastructure</li></li></ul><li>Operations Tussles:<br /><ul><li>Entangled in many day-to-day failures & troubleshooting
    8. 8. Crippled Infrastructure
    9. 9. Building on weak foundation</li></li></ul><li>Developers Tussles:<br /><ul><li>Bloated Operating System, App Stacks
    10. 10. Staggering Complexities of Dependencies
    11. 11. No time for QA – Bugs Creeping in all over</li></li></ul><li>Security Tussles:<br /><ul><li>Whose problem is it?
    12. 12. Auditability and Assurance
    13. 13. Governance
    14. 14. Compliance</li></li></ul><li>
    15. 15. As a Result: <br /><ul><li>More Friction, Waste, and Frustration
    16. 16. Less Innovation </li></li></ul><li>So, What is Cloud?<br /><ul><li>C – Customer Driven
    17. 17. L – Lean
    18. 18. O – Optimized
    19. 19. U – Utility
    20. 20. D – Demand-Driven</li></li></ul><li><ul><li>Cloudis NOT about Cost cutting
    21. 21. It is about Value Creation enabling Innovation</li></li></ul><li><ul><li>Cloud is an opportunity to clean up the Infrastructure mess!</li></li></ul><li>
    22. 22. Yahoo! is Perfect for Cloud Computing<br />600M<br />UNIQUE USERS / MONTH<br />300M+<br />YAHOO! MAIL USERS / MONTH<br />HUNDREDS<br />OF PROPERTIES / PRODUCTS<br />BILLIONS<br />OF OBJECTS STORED<br />HUNDREDS<br />OF PETABYTES OF STORAGE<br />PETABYTES<br />OF TRAFFIC DAILY<br />
    23. 23. <ul><li>Declarative language for specifying the full, end-to-end structure of a service
    24. 24. Includes multiple environments: Dev, QA, Staging, and production
    25. 25. Extreme Automation</li></li></ul><li>Yahoo! Cloud Serving - Principles<br /><ul><li>Metastructure: Abstracts concerns of the underlying infrastructure and the network communication
    26. 26. Declarative application structure
    27. 27. Continuous Integration, Continuous Deployment, & Continuous Compliance*
    28. 28. End-to-End Security</li></ul>* Rich Miller @RHM2K first coined “Continuous Compliance” <br />
    29. 29. Yahoo! Cloud Serving - Principles<br /><ul><li>Virtualized hardware
    30. 30. Standardized software stacks and packaging
    31. 31. Integrated service management
    32. 32. Containers vs. Machines</li></li></ul><li>Yahoo! Cloud Serving - Principles<br /><ul><li>Elastic serving of changing workloads
    33. 33. Controlled/Intelligent traffic direction
    34. 34. Controlled execution environment
    35. 35. Managed Communication
    36. 36. Service Associations, Bindings, and Access Controls</li></li></ul><li>Yahoo! Cloud: Tiers without Tears<br />Entrypoint<br />Naming<br />Presentation<br />Tier<br />Application Logic<br />Tier<br />Bindings<br />Connections<br />Database Tier<br />
    37. 37. Yahoo! Cloud: Tiers without Tears<br />Entrypoint<br />Naming<br />Load<br />Balancer<br />Presentation<br />Tier<br />Application Logic<br />Tier<br />node<br />node<br />node<br />Bindings<br />Connections<br />Database Tier<br />
    38. 38. Yahoo! Cloud: Extreme Automation<br />Abstracting Services<br />Component AnswersCore {<br /> Entry root;<br />Tier pres {<br /> app = “answers_web_tier”;<br /> Exit applogic;<br /> nsize = small;<br /> nnodes = 3;<br /> };<br />Tier applogic {<br /> app = “answers_al_tier”;<br /> Exit data;<br /> nsize = medium;<br /> nnodes = 3;<br /> };<br />Tier data;<br /> Connections {<br /> root -&gt; pres;<br /> pres.applogic -&gt; applogic;<br /> applogic.data -&gt; data;<br /> }<br />} <br />Presentation<br />Tier<br />Application Logic<br />Tier<br />Database Tier<br />
    39. 39. Yahoo! Cloud: Extreme Automation<br />QA Environment<br />Environment qa(ne1) {<br /> TrafficDirector root {<br /> name = “qa.answers.yahoo.com”;<br /> access = testing;<br /> };<br /> AnswersCore core;<br /> Connections {<br /> root -&gt; core;<br /> }<br />} <br />Presentation<br />Tier<br />Presentation<br />Tier<br />Application Logic<br />Tier<br />Application Logic<br />Tier<br />Testing Testing<br />Environment production(ac4,sk1) {<br /> TeeTrafficDirector root {<br /> name = “answers.yahoo.com”;<br /> access = public;<br /> split_percent = 10;<br /> split_destination = qa.root;<br /> };<br /> AnswersCore core;<br /> Connections {<br /> root -&gt; core;<br /> }<br />} <br />Database Tier<br />Database Tier<br />
    40. 40. Yahoo! Cloud: Extreme Automation<br />Apps written against rich stacks<br />Includes core infra and platforms<br /><ul><li>Infrastructure: Security, monitoring, metering, abuse prevention
    41. 41. Platforms: data, user info, ad systems, configuration bundles
    42. 42. All “auto configured”</li></li></ul><li>Serving Cloud Serving for Y!’s Cloud<br />
    43. 43. <ul><li>Infrastructure to Metastructure
    44. 44. Machines to Containers
    45. 45. Friction to Innovation
    46. 46. Tears to Tiers
    47. 47. Complexity to Simplexity
    48. 48. Costs to Revenue Streams</li></li></ul><li>Cloud is a Promise, Service is the fulfillment!<br />
    49. 49. OPEN CLOUD ACCESS PROTOCOL<br />
    50. 50. OPEN CLOUD ACCESS PROTOCOL<br /><ul><li>Definition, Deployment, and Life Cycle Management of Cloud Resources
    51. 51. Allocation, Provisioning, and Metering of Cloud Resources
    52. 52. Metadata/Registry for Cloud Resources
    53. 53. Virtual Infrastructure Management</li></li></ul><li>Yahoo! Submitted OpenCAP to IETF<br /><ul><li>Why IETF?
    54. 54. Specification behind Yahoo Implementation
    55. 55. Plan to open source implementation</li></li></ul><li>Open Cloud Access Protocol<br />Resource Model<br /><ul><li>All Infrastructure and Functional Components are modeled as Resources
    56. 56. Nodes, Networks, Storage Devices, Databases, Load Balancers, Identities, Key Value Store, Object Stores, Caching Services</li></ul>Resource Properties<br /><ul><li>Modeled as JSON objects
    57. 57. Standard Catalog of attributes, extensible</li></ul>Resource Operations<br /><ul><li>Operation, Control, and Notification</li></li></ul><li>Open Cloud Access Protocol<br />Management Operation Services<br /><ul><li>CREATE - Create an instance of a managed resource
    58. 58. DELETE - Delete an instance of a managed resource
    59. 59. GET - Request managed resource properties (for one resource or a set of resource)
    60. 60. CANCEL - Cancel an outstanding request
    61. 61. SET - Set managed resource properties
    62. 62. ACTION - Request an action to be performed on a managed resource</li></li></ul><li>Open Cloud Access Protocol<br />Management Notification Services<br /><ul><li>EVENT- Send events occurring on managed resources (VM events, VM stats etc, SNMP events etc)
    63. 63. ALERT- Send alerts for resource failures and exceptions</li></ul>Life Cycle Management Services<br /><ul><li>START- Starts the resource
    64. 64. TERMINATE – Stop the resource
    65. 65. SUSPEND – Suspend the resource
    66. 66. RESUME – Resume the resource
    67. 67. ARCHIVE – Archive the resource</li></li></ul><li>Yahoo! Open Source & Research<br />Open Source<br /><ul><li>Cloud Serving Platform, Yahoo Implementation of OpenCAP
    68. 68. Traffic Server</li></ul>Research Activities<br /><ul><li>Open Cloud Consortium
    69. 69. OpenCirrus & M45</li></li></ul><li>THANK YOU<br />Surendra Reddy<br />@sureddy twitter<br />sureddy@yahoo-inc.com email<br />yCloudBlog.com blog<br />
    70. 70. TUESDAY, 11/3<br />4:50pm – 5:35 pm<br />Hadoop @ Yahoo! – Internet<br />Scale Data Processing<br />Eric Baldeschwieler<br />VP, Hadoop Software Development<br />WEDNESDAY, 11/4<br />9:10 am - 9:55 am<br />Yahoo! Scalable Storage and<br />Delivery Services<br />Chuck Neerdaels<br />VP, Storage and Edge Services<br />VISIT BOOTH #103 <br />TO TALK WITH YAHOO! ENGINEERS AND LEARN MORE ABOUT YAHOO!’S VISION FOR CLOUD COMPUTING.<br />

    ×