Leveraging the Web
for Services at
       Mark Nottingham
     <mnot@yahoo-inc.com>
Y! Calendar
                                                            Y! Mexico
                                        ...
Y! Auctions                                                 Y! Mexico                                                   Y!...
Between     properties


  With     partners


 With     acquisitions


 Within    over time
“   Scale”
Media Group

                      Y! Music

          Y! TV       Y! Personal Finance

      Y! Kids     Y! Finance      ...
Front-End




   Slave
Database




  Master
Database
Large Datasets Don’t Push Well
Adding Capacity is Expensive




Large Datasets Don’t Push Well
Adding Capacity is Expensive




Large Datasets Don’t Push Well
                             What to do after push failure?
User-Generated Content         Adding Capacity is Expensive




Large Datasets Don’t Push Well
                           ...
Building New Sites
Requirements
Requirements



massive scalability
flexible deployment
highly dynamic
separation of concerns
mashability
i.e.,


Services
Scalability      Simplicity




  Reuse       Interoperability
i.e.,


HTTP
Front-End




 Caching




 Back-End




Database
Single Source of Truth
Single Source of Truth   Cache Replicates Naturally
Adding Capacity Is Easy




Single Source of Truth   Cache Replicates Naturally
UGC Pushes Through         Adding Capacity Is Easy




Single Source of Truth   Cache Replicates Naturally
HTTP Caching Intermediaries
HTTP Caching Intermediaries


Freshness
HTTP Caching Intermediaries


Freshness

Validation
HTTP Caching Intermediaries


Freshness

Validation

  Metrics
                      0101011010010101101010011001010101
HTTP Caching Intermediaries


Freshness                     Load Balancing

Validation

  Metrics
HTTP Caching Intermediaries


Freshness                     Load Balancing

Validation                    Cache Peering

 ...
HTTP Caching Intermediaries


Freshness                     Load Balancing

Validation                    Cache Peering

 ...
HTTP Caching Intermediaries


Freshness                     Load Balancing

Validation                    Cache Peering

 ...
HTTP Caching Intermediaries


Freshness                     Load Balancing

Validation                    Cache Peering

 ...
HTTP Caching Intermediaries


Freshness                     Load Balancing

Validation                    Cache Peering

 ...
HTTP Caching Intermediaries


Freshness                     Load Balancing

Validation                    Cache Peering

 ...
12,000 req/sec/core
Tens of Thousands of Connections
pitfalls
REST vs. WS-* wars
theory vs. practice
human-intuitive, but not programmer-intuitive
different deployment/operati...
what’s needed

tools
web description language
data-oriented schema language
investment in the Atom stack
HTTP test suite
Leveraging the Web for Services at Yahoo!
Leveraging the Web for Services at Yahoo!
Leveraging the Web for Services at Yahoo!
Leveraging the Web for Services at Yahoo!
Leveraging the Web for Services at Yahoo!
Leveraging the Web for Services at Yahoo!
Leveraging the Web for Services at Yahoo!
Leveraging the Web for Services at Yahoo!
Upcoming SlideShare
Loading in...5
×

Leveraging the Web for Services at Yahoo!

1,992

Published on

QCon London 2007: When moving to a Service-Oriented Architecture, SOAP and WSDL aren't the only option. HTTP, URIs and well-defined formats are the basis of the Web, and more architects are looking at them as a serious alternative. Beyond the hype of REST vs. WS-* and Web 2.0, it's possible to do some remarkable things with very simple tools.

In this talk, I relate our experiences so far using HTTP and the Web for services in Yahoo!'s Media Group, covering our reasons for choosing this approach, the places where it's most effective (and where it's not), the pitfalls we've found along the way and what's still needed to make it developers' first choice for connecting their applications and data.

Published in: Technology, Business
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total Views
1,992
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
36
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

Leveraging the Web for Services at Yahoo!

  1. 1. Leveraging the Web for Services at Mark Nottingham <mnot@yahoo-inc.com>
  2. 2. Y! Calendar Y! Mexico Y! Shopping Y! Sports Y! China Y! Local Y! Auctions Y! Personal Finance Y! Travel Y! Messenger Upcoming Y! France Y! Small Business Bix Y! Mail Y! Brazil del.icio.us Y! Germany Y! Spain Y! Tech Y! Korea Y! Health Flickr Y! UK Y! Movies Y! India Y! Food Y! Australia Broadcast.com HotJobs Y! Podcasts Yahooligans! Y! Italy Y! Argentina Inktomi Y! Video Y! Classifieds Y! Entertainment LAUNCH Overture Y! Music Y! Japan Y! Games Y! Photos Y! Maps Kelkoo Y! Go Y! Y! Singapore GeoCities Y! Groups Y! Search Y! 360 992 1994 1996 1998 2000 2002 2004 2006 200 100,000 25 million 200 million 500 million users users users users 49 employees 11,400 employees 65 million 1.3 billion 4 billion daily page views daily page views daily page views
  3. 3. Y! Auctions Y! Mexico Y! Messenger Y! Germany Y! India Y! Entertainment Y! China Bix Y! Travel Inktomi Upcoming Y! Sports Y! Korea Y! Tech Y! Small Business Y! Brazil Y! Mail del.icio.us Y! UK Y! Japan Y! Integration Y! Food Flickr Y! Go Nightmare Y! 360 Y! Spain Overture Y! France Y! Health Y! Music Y! Search Y! Italy Y! Australia Yahooligans! Y! Argentina Y! Chinese Y! Video Broadcast.com Y! Games LAUNCH Y! Photos Y! Singapore Kelkoo Y! Classifieds GeoCities Y! Maps Y! Podcasts HotJobs Y! Shopping Y! Groups
  4. 4. Between properties With partners With acquisitions Within over time
  5. 5. “ Scale”
  6. 6. Media Group Y! Music Y! TV Y! Personal Finance Y! Kids Y! Finance Y! Movies Y! Fantasy Sports Y! Games Y! Health Y! Sports Y! Astrology Y! Wii Y! Tech Y! Food Y! News
  7. 7. Front-End Slave Database Master Database
  8. 8. Large Datasets Don’t Push Well
  9. 9. Adding Capacity is Expensive Large Datasets Don’t Push Well
  10. 10. Adding Capacity is Expensive Large Datasets Don’t Push Well What to do after push failure?
  11. 11. User-Generated Content Adding Capacity is Expensive Large Datasets Don’t Push Well What to do after push failure?
  12. 12. Building New Sites
  13. 13. Requirements
  14. 14. Requirements massive scalability flexible deployment highly dynamic separation of concerns mashability
  15. 15. i.e., Services
  16. 16. Scalability Simplicity Reuse Interoperability
  17. 17. i.e., HTTP
  18. 18. Front-End Caching Back-End Database
  19. 19. Single Source of Truth
  20. 20. Single Source of Truth Cache Replicates Naturally
  21. 21. Adding Capacity Is Easy Single Source of Truth Cache Replicates Naturally
  22. 22. UGC Pushes Through Adding Capacity Is Easy Single Source of Truth Cache Replicates Naturally
  23. 23. HTTP Caching Intermediaries
  24. 24. HTTP Caching Intermediaries Freshness
  25. 25. HTTP Caching Intermediaries Freshness Validation
  26. 26. HTTP Caching Intermediaries Freshness Validation Metrics 0101011010010101101010011001010101
  27. 27. HTTP Caching Intermediaries Freshness Load Balancing Validation Metrics
  28. 28. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics
  29. 29. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics Negative Caching
  30. 30. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics Negative Caching Collapsed Forwarding
  31. 31. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics Negative Caching Collapsed Forwarding stale-while-revalidate
  32. 32. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics Negative Caching Collapsed Forwarding ? stale-while-revalidate stale-if-error
  33. 33. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics Negative Caching Collapsed Forwarding stale-while-revalidate stale-if-error Invalidation Channels
  34. 34. 12,000 req/sec/core
  35. 35. Tens of Thousands of Connections
  36. 36. pitfalls REST vs. WS-* wars theory vs. practice human-intuitive, but not programmer-intuitive different deployment/operational concerns formats are hard format / interface proliferation authentication isn’t there yet tools have a way to go
  37. 37. what’s needed tools web description language data-oriented schema language investment in the Atom stack HTTP test suite
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×