Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Cloud Computing @Yahoo! Dekel Tankel Director, Product Management Yahoo!  Cloud Computing [email_address] IGT, June 2009
What we’ll cover today… <ul><li>Why Cloud?  </li></ul><ul><ul><li>Scale and Abstraction; Quality and Agility </li></ul></u...
Why Cloud? Benefits for Yahoo! <ul><li>Higher Agility & Stability while maintaining Scale </li></ul><ul><li>Abstraction  <...
Yahoo!’s Unique Cloud:  Unprecedented Scale <ul><li>Massive user base and engagement </li></ul><ul><ul><li>500M+ unique us...
Yahoo! Cloud Services ROI & Innovation Y!OS, BOSS, YQL, APT, Analytics, … Storage, Batch,  Edge Serving,… Users Applicatio...
Yahoo! Cloud Services:  Focus on PaaS offerings ROI & Innovation IaaS PaaS SaaS Users Applications Functional Cloud Servic...
From Infrastructure to  Shareholders benefit <ul><li>Horizontal Cloud </li></ul><ul><ul><li>Focus on open source and colla...
Yahoo! Cloud Strategy in Action: The Front Page Case Study <ul><li>Horizontal Cloud – Storage & Hadoop </li></ul><ul><ul><...
Yahoo! Cloud Strategy in Action: The Inquisitor Case Study <ul><li>Horizontal Cloud – Hadoop </li></ul><ul><ul><li>Analyze...
Horizontal Cloud Services ROI & Innovation Users Applications Functional Cloud Services Horizontal Cloud  Services  Physic...
Horizontal Cloud Services <ul><li>Optimized for Yahoo!-scale </li></ul><ul><ul><li>Yahoo!-internal focus </li></ul></ul><u...
Horizontal Cloud Services:  Conceptual View Common Approaches to QA, Production Engineering, Performance Engineering, Data...
Horizontal Cloud Services: Use Cases Ads Optimization Content Optimization Search Index Image/Video Storage & Delivery Mac...
Yahoo! Distribution of Hadoop <ul><li>Hadoop in a nutshell </li></ul><ul><ul><li>Open source distributed file system & par...
Yahoo! runs the largest  Hadoop Clusters in the World <ul><li>25,000+ nodes  </li></ul><ul><ul><li>Clusters of up to 4,000...
Case Study  -  Search Assist™ <ul><li>Database for  Search Assist™  is built using Hadoop.  </li></ul><ul><li>3 years of l...
Functional Cloud Services ROI & Innovation Users Applications Functional Cloud Services Horizontal Cloud  Services  Physic...
Functional Cloud Services <ul><li>Provides functional capabilities for applications </li></ul><ul><ul><li>Help developers ...
Functional Cloud Services:  YQL & BOSS <ul><li>  </li></ul>A single endpoint service that enables developers to query, fil...
Build your Own Search Service (BOSS) <ul><li>Yahoo!'s open search web services platform </li></ul><ul><ul><li>Serving hund...
Yahoo! Query Language (YQL) <ul><li>Single endpoint service to query, filter and combine data across Yahoo! and beyond </l...
Y!OS and Cloud
Yahoo! Open Stagey (Y!OS): Goals
Y!OS and Cloud Strategy CLOUD SERVICES
Open Collaborations  around the globe <ul><li>M45  - Yahoo!’s supercomputing cluster </li></ul><ul><ul><li>4,000 cores, 3 ...
Questions? Dekel Tankel Director, Product Management Yahoo! Cloud Computing [email_address]
Upcoming SlideShare
Loading in …5
×

Cloud Computing @Yahoo!

1,636 views

Published on

Published in: Business, Technology
  • Be the first to comment

Cloud Computing @Yahoo!

  1. 1. Cloud Computing @Yahoo! Dekel Tankel Director, Product Management Yahoo! Cloud Computing [email_address] IGT, June 2009
  2. 2. What we’ll cover today… <ul><li>Why Cloud? </li></ul><ul><ul><li>Scale and Abstraction; Quality and Agility </li></ul></ul><ul><ul><li>Yahoo!’s unique footprint </li></ul></ul><ul><li>Yahoo!’s Cloud Strategy </li></ul><ul><ul><li>Overview of the Yahoo! Cloud vision and portfolio </li></ul></ul><ul><ul><li>Deep dive on Horizontal & Functional Cloud Services </li></ul></ul><ul><li>The Yahoo! Open Strategy </li></ul><ul><ul><li>Marrying Yahoo!’s “Open Strategy”, its platforms and ethic with external Cloud services </li></ul></ul>
  3. 3. Why Cloud? Benefits for Yahoo! <ul><li>Higher Agility & Stability while maintaining Scale </li></ul><ul><li>Abstraction </li></ul><ul><ul><li>Enable developers to focus on their applications, not infrastructure </li></ul></ul><ul><li>Accelerating innovation </li></ul><ul><ul><li>Adding new features and products at an ever faster rate </li></ul></ul><ul><li>Increasing Scale & Availability </li></ul><ul><ul><li>More robustly, more globally, more completely, for a given budget </li></ul></ul>Cloud is pushing up the Operation Excellence Curve Agility & Innovation Quality & Stability
  4. 4. Yahoo!’s Unique Cloud: Unprecedented Scale <ul><li>Massive user base and engagement </li></ul><ul><ul><li>500M+ unique users per month </li></ul></ul><ul><ul><li>Hundreds of petabyte of storage </li></ul></ul><ul><ul><li>Hundreds of billions of objects </li></ul></ul><ul><ul><li>Hundred of thousands of requests/sec </li></ul></ul><ul><li>Global </li></ul><ul><ul><li>Tens of globally distributed data centers </li></ul></ul><ul><ul><li>Serving each region at low latencies </li></ul></ul><ul><li>Challenging Users </li></ul><ul><ul><li>Rapidly extracting value from voluminous data </li></ul></ul><ul><ul><li>Downtime is not an option (outages cost $millions) </li></ul></ul><ul><ul><li>Variable usage patterns </li></ul></ul>
  5. 5. Yahoo! Cloud Services ROI & Innovation Y!OS, BOSS, YQL, APT, Analytics, … Storage, Batch, Edge Serving,… Users Applications Functional Cloud Services Horizontal Cloud Services Physical Layer
  6. 6. Yahoo! Cloud Services: Focus on PaaS offerings ROI & Innovation IaaS PaaS SaaS Users Applications Functional Cloud Services Horizontal Cloud Services Physical Layer
  7. 7. From Infrastructure to Shareholders benefit <ul><li>Horizontal Cloud </li></ul><ul><ul><li>Focus on open source and collaborative R&D with industry, academia and government </li></ul></ul><ul><li>Functional Cloud </li></ul><ul><ul><li>Focus on developing &quot;open strategy&quot; frameworks, tools and services for developers (at Yahoo! and beyond) </li></ul></ul><ul><li>Combined Together </li></ul><ul><ul><li>Leverage our unique scale, assets and data to drive disruptive innovations in the market and expand Yahoo!’s competitive differentiation </li></ul></ul>
  8. 8. Yahoo! Cloud Strategy in Action: The Front Page Case Study <ul><li>Horizontal Cloud – Storage & Hadoop </li></ul><ul><ul><li>Analyze extremely large content data sets </li></ul></ul><ul><li>Functional Cloud – Content Optimization </li></ul><ul><ul><li>Rate content items based on various parameters </li></ul></ul><ul><li>Applications – Yahoo’s Front Page </li></ul><ul><ul><li>Display “high rating” items to the right users </li></ul></ul><ul><ul><li>Benefit consumers and advertisers and grow Yahoo!’s revenue </li></ul></ul>
  9. 9. Yahoo! Cloud Strategy in Action: The Inquisitor Case Study <ul><li>Horizontal Cloud – Hadoop </li></ul><ul><ul><li>Analyze large search-index data sets </li></ul></ul><ul><li>Functional Cloud - BOSS </li></ul><ul><ul><li>Expose the data in a structured, open, flexible and “cloud like” way </li></ul></ul><ul><li>Applications - iPhone TM Inquisitor </li></ul><ul><ul><li>Leverage BOSS to provide innovative consumer experience </li></ul></ul><ul><ul><li>Benefit consumers and grow Yahoo!’s revenue </li></ul></ul>
  10. 10. Horizontal Cloud Services ROI & Innovation Users Applications Functional Cloud Services Horizontal Cloud Services Physical Layer
  11. 11. Horizontal Cloud Services <ul><li>Optimized for Yahoo!-scale </li></ul><ul><ul><li>Yahoo!-internal focus </li></ul></ul><ul><ul><li>Data processing and serving environments </li></ul></ul><ul><li>Drive faster innovation and agility </li></ul><ul><ul><li>Shorter product development cycles </li></ul></ul><ul><ul><li>Reduce labor and costs for infrastructure </li></ul></ul><ul><li>Multi-year effort </li></ul><ul><ul><li>Strategic investment across the company </li></ul></ul>
  12. 12. Horizontal Cloud Services: Conceptual View Common Approaches to QA, Production Engineering, Performance Engineering, Datacenter Management, and Optimization Shared Infrastructure Simple API’s ID & Account Management Provisioning & Virtualization (Xen) Operational Storage Structured, unstructured Batch Storage & Processing Hadoop, PIG Edge Content Services Caching, Proxies Online Serving Web, Data Security and Authentication Metering, Billing Monitoring & QoS
  13. 13. Horizontal Cloud Services: Use Cases Ads Optimization Content Optimization Search Index Image/Video Storage & Delivery Machine Learning (e.g. Spam filters) Attachment Storage
  14. 14. Yahoo! Distribution of Hadoop <ul><li>Hadoop in a nutshell </li></ul><ul><ul><li>Open source distributed file system & parallel execution environment to process massive amounts of data </li></ul></ul><ul><ul><li>Started in 2005, became top-level Apache project in 2008 </li></ul></ul><ul><ul><li>Simple Design for Horizontal Scaling on commodity HW </li></ul></ul><ul><li>Yahoo! Distribution of Hadoop </li></ul><ul><ul><li>Source distribution of Yahoo!’s implementation of Hadoop (Based entirely on code found in the Apache Hadoop) </li></ul></ul><ul><ul><li>Tested and deployed at Yahoo!’s massive scale </li></ul></ul><ul><ul><li>Benefit the larger ecosystem , Increase pace of innovation </li></ul></ul><ul><ul><li>http://developer.yahoo.com/hadoop </li></ul></ul>
  15. 15. Yahoo! runs the largest Hadoop Clusters in the World <ul><li>25,000+ nodes </li></ul><ul><ul><li>Clusters of up to 4,000 nodes </li></ul></ul><ul><li>4 Tiers of clusters </li></ul><ul><ul><li>Development & Testing, POCs, Science & Research, Production </li></ul></ul><ul><li>Terasort Benchmarks </li></ul><ul><ul><li>62 seconds to sort One Terabyte (run on 1,500 nodes) </li></ul></ul><ul><ul><li>16.25 hours to sort One Petabyte (run on 3,700 nodes) </li></ul></ul><ul><li>Webmap application </li></ul><ul><ul><li>~490 TB shuffling </li></ul></ul><ul><ul><li>~280 TB output </li></ul></ul>
  16. 16. Case Study - Search Assist™ <ul><li>Database for Search Assist™ is built using Hadoop. </li></ul><ul><li>3 years of log-data, 20-steps of map-reduce </li></ul><ul><ul><li>Leverage Hadoop’s scalability, load balancing and resiliency </li></ul></ul><ul><ul><li>Simplified access, flexibility for rapid innovation (from C++ to Python) </li></ul></ul>Before Hadoop After Hadoop Time 26 days 20 minutes Development Time 2-3 weeks 2-3 days
  17. 17. Functional Cloud Services ROI & Innovation Users Applications Functional Cloud Services Horizontal Cloud Services Physical Layer
  18. 18. Functional Cloud Services <ul><li>Provides functional capabilities for applications </li></ul><ul><ul><li>Help developers to accomplish integrated web experiences in a faster and easier way </li></ul></ul><ul><ul><li>Provides common set of functional “building blocks” </li></ul></ul><ul><li>“ Powered by” the horizontal cloud services </li></ul><ul><ul><li>Abstracts infrastructure services from the Application </li></ul></ul><ul><ul><ul><li>E.g. Storage, Compute, Serving, Robustness and Scalability </li></ul></ul></ul><ul><ul><li>Self-Served, Global, Managed, Elastic and Metered </li></ul></ul>
  19. 19. Functional Cloud Services: YQL & BOSS <ul><li> </li></ul>A single endpoint service that enables developers to query, filter and combine data across Yahoo! and beyond http://developer.yahoo.com/yql/console/ Providing Yahoo! Search infrastructure and technology to developers and companies to help them build their own search experiences Build your Own Search Service http://developer.yahoo.com/search/boss / Yahoo! Query Language
  20. 20. Build your Own Search Service (BOSS) <ul><li>Yahoo!'s open search web services platform </li></ul><ul><ul><li>Serving hundreds of millions of users across the Web. </li></ul></ul><ul><li>Goal: foster innovation in the search industry </li></ul><ul><ul><li>Build and launch web-scale search products that utilize the entire Yahoo! Search index. </li></ul></ul><ul><ul><li>Access to Yahoo!'s investments in crawling and indexing, ranking and relevancy algorithms </li></ul></ul>
  21. 21. Yahoo! Query Language (YQL) <ul><li>Single endpoint service to query, filter and combine data across Yahoo! and beyond </li></ul><ul><ul><li>The “Internet API” </li></ul></ul><ul><li>SQL-like SELECT syntax for getting the right data </li></ul><ul><ul><li>Quickly discover available data sources and structure </li></ul></ul><ul><ul><li>Combined data from a single web browser </li></ul></ul><ul><li>Easy-to-use Consol </li></ul><ul><ul><li>http://developer.yahoo.com/yql/console/ </li></ul></ul>
  22. 22. Y!OS and Cloud
  23. 23. Yahoo! Open Stagey (Y!OS): Goals
  24. 24. Y!OS and Cloud Strategy CLOUD SERVICES
  25. 25. Open Collaborations around the globe <ul><li>M45 - Yahoo!’s supercomputing cluster </li></ul><ul><ul><li>4,000 cores, 3 TB RAM, 1.5 PB disks, 27 teraflops! </li></ul></ul><ul><ul><li>Operational since November 2007, 4 major Universities </li></ul></ul><ul><ul><li>Focus on highly parallel computing </li></ul></ul><ul><li>Open Cirrus™ with HP & Intel </li></ul><ul><ul><li>A global, multi-data center, open source test bed </li></ul></ul><ul><ul><li>Target to advance cloud computing research & education </li></ul></ul><ul><ul><li>Simulates a real-life, Internet-scale environment </li></ul></ul><ul><ul><li>9 Global sites, more than 50 research projects </li></ul></ul>
  26. 26. Questions? Dekel Tankel Director, Product Management Yahoo! Cloud Computing [email_address]

×