• Save
Yahoo! Scalable Storage and Delivery Services
Upcoming SlideShare
Loading in...5
×
 

Yahoo! Scalable Storage and Delivery Services

on

  • 4,459 views

From Chuck Neerdaels' presentation "Yahoo! Scalable Storage and Delivery Services" at the 2009 Cloud Computing Expo in Santa Clara, CA, USA. Here's the talk description on the Expo's site: ...

From Chuck Neerdaels' presentation "Yahoo! Scalable Storage and Delivery Services" at the 2009 Cloud Computing Expo in Santa Clara, CA, USA. Here's the talk description on the Expo's site: http://cloudcomputingexpo.com/event/session/541

Statistics

Views

Total Views
4,459
Slideshare-icon Views on SlideShare
4,429
Embed Views
30

Actions

Likes
9
Downloads
0
Comments
0

2 Embeds 30

http://www.slideshare.net 23
http://blog.sahsu.mobi 7

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Online

Yahoo! Scalable Storage and Delivery Services Yahoo! Scalable Storage and Delivery Services Presentation Transcript

  • Chuck Neerdaels VP of Storage and Edge Services Y! Scalable Storage and Delivery Service
  • Cloud Storage & Edge Services Overview
    • Bulk Storage (Video, Mail attachments, Images, …) :
      • MObStor
      • DORA (Clustered Object Store)
    • Structured Storage (key-value and user data):
      • Sherpa/PNUTs
      • User Database (UDB)
      • UDS/UPS
  • CS&E cont.
    • Messaging and Replication:
      • Many variations
    • Edge:
      • ySquid
      • Traffic Server
      • Core Proxy Infrastructure
      • Multi-vendor CDN
      • Global DNS & local load balancing technology
  • Typical Use Cases User Data and Preferences Image/Video Storage & Delivery RSS Feeds Caching, Load Balancing
  • A Quick Demo…
  • more cute puppy photos
  • Want a real demo? Sports.yahoo.com Travel.yahoo.com Finance.yahoo.com Mail.yahoo.com News.yahoo.com *.yahoo.com
  • Infrastructure and Platform Service Reality
    • Demos are only notable when they fail
    • On-line 7x24x365
        • Through upgrades, attacks, outages, human errors, …
        • Legitimate use cases can be as challenging as abuse
    • It’s all assumed to, “just work”
    • But, what if they didn’t…
  • But what if they didn’t…
    • Globally:
      • Sites offline
      • Load balancing fails
      • Images, CSS, and JavaScript not served
      • No logins and personalization
      • Normalized RSS feeds go stale
      • Site thumbnails fail
      • Server-to-server communication ceases
    • Ads
      • Ads not served
      • Ad pipelines halt
    • Flickr:
      • No more video or images
    • Mail:
      • Webmail slow of offline, attachments missing
    • Mobile & Social:
      • Features go offline
    • Sports
      • Overwhelmed every Sunday
      • Fantasy Sports draft data gets stale
    Yahoo! sites would become stale, boring, slow, non-profitable, or non-existent
  • What Makes an On-line Cloud Service at Y!… Global
  • Internal Expectations
    • Global Solution, Including Emerging Markets
    • Fully Managed
    • Multi-tenant (Finance)
    • Single tenant (Customers)
    • Elastic
    • Predictable SLA’s
    • Simple to provision and manage
    • Providing the illusion of infinite bandwidth, capacity, (and single tenancy)…
  • Common Challenges
    • Speed of light
    • Huge traffic spike potential
    • Cost
    • Partitioned network failures are assumed
    • Consumer/User intuition
  • A Little More Detail: MObStor & Sherpa
  • MObStor: Blowing Through the Cloudiness Global Servers Caching Protocol Termination Authentication Content Routing Local Servers Auto Expiration De-duplication Object Placement Re-replication
  • MObStor: Trying to Increase 9’s DNS GSLB Hardware VIP Scalable Session Manager Internal DNS with Load Feedback Hardware VIP Local Replication Global Replication Separate Metadata Replication
  • Sherpa/PNUTs
  • Sherpa/PNUTs DNS GSLB Hardware VIP Scalable Router & Session Manager Consistency vs. Availability Knobs Local Replication Global Replication Ordered Messaging
  • Physics and Economics for a Global Cloud…
  • What’s your target SLA? 0 Distance between producer/consumer *Illustrative, and not intended to portray reality Latency Speed of light + network degradation
  • What’s your budget? Latency Replication Costs *Illustrative, and not intended to portray reality $$$ $
  • Lessons Learned
    • Intuition is often wrong – let data drive your data
    • Provide hooks, experimental feedback, and mobility
    • N-way global replication is really expensive
    • Customers don’t understand 95/5 billing
    • Customers don’t understand CAP theorem
    • Verify all provisioning
    • Strive for quality, but plan for failure
  • Why would any of you care about a Y! proprietary cloud?
  • Committed to Open Source
    • Component approach – current pipeline:
    • ZooKeeper (already available)
    • Traffic Server (just released!)
    • More to come
    • YAHOO! IS OPEN SOURCING
    • Cloud building block
    • Processes up to 35K req/second per box
    • Extensible Framework
    • Edge Caching, Load Balancing, Proxying
    • Leveraged in Edge, Storage, Serving services
    • Supports more than half of all Yahoo! traffic
    • Available via the Apache Software Foundation
  • Questions? Y! Scalable Storage and Delivery Service