Leveraging the Web for Services at Yahoo!

2,258 views
2,154 views

Published on

QCon London 2007: When moving to a Service-Oriented Architecture, SOAP and WSDL aren't the only option. HTTP, URIs and well-defined formats are the basis of the Web, and more architects are looking at them as a serious alternative. Beyond the hype of REST vs. WS-* and Web 2.0, it's possible to do some remarkable things with very simple tools.

In this talk, I relate our experiences so far using HTTP and the Web for services in Yahoo!'s Media Group, covering our reasons for choosing this approach, the places where it's most effective (and where it's not), the pitfalls we've found along the way and what's still needed to make it developers' first choice for connecting their applications and data.

Published in: Technology, Business
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total views
2,258
On SlideShare
0
From Embeds
0
Number of Embeds
29
Actions
Shares
0
Downloads
37
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

Leveraging the Web for Services at Yahoo!

  1. 1. Leveraging the Web for Services at Mark Nottingham <mnot@yahoo-inc.com>
  2. 2. Y! Calendar Y! Mexico Y! Shopping Y! Sports Y! China Y! Local Y! Auctions Y! Personal Finance Y! Travel Y! Messenger Upcoming Y! France Y! Small Business Bix Y! Mail Y! Brazil del.icio.us Y! Germany Y! Spain Y! Tech Y! Korea Y! Health Flickr Y! UK Y! Movies Y! India Y! Food Y! Australia Broadcast.com HotJobs Y! Podcasts Yahooligans! Y! Italy Y! Argentina Inktomi Y! Video Y! Classifieds Y! Entertainment LAUNCH Overture Y! Music Y! Japan Y! Games Y! Photos Y! Maps Kelkoo Y! Go Y! Y! Singapore GeoCities Y! Groups Y! Search Y! 360 992 1994 1996 1998 2000 2002 2004 2006 200 100,000 25 million 200 million 500 million users users users users 49 employees 11,400 employees 65 million 1.3 billion 4 billion daily page views daily page views daily page views
  3. 3. Y! Auctions Y! Mexico Y! Messenger Y! Germany Y! India Y! Entertainment Y! China Bix Y! Travel Inktomi Upcoming Y! Sports Y! Korea Y! Tech Y! Small Business Y! Brazil Y! Mail del.icio.us Y! UK Y! Japan Y! Integration Y! Food Flickr Y! Go Nightmare Y! 360 Y! Spain Overture Y! France Y! Health Y! Music Y! Search Y! Italy Y! Australia Yahooligans! Y! Argentina Y! Chinese Y! Video Broadcast.com Y! Games LAUNCH Y! Photos Y! Singapore Kelkoo Y! Classifieds GeoCities Y! Maps Y! Podcasts HotJobs Y! Shopping Y! Groups
  4. 4. Between properties With partners With acquisitions Within over time
  5. 5. “ Scale”
  6. 6. Media Group Y! Music Y! TV Y! Personal Finance Y! Kids Y! Finance Y! Movies Y! Fantasy Sports Y! Games Y! Health Y! Sports Y! Astrology Y! Wii Y! Tech Y! Food Y! News
  7. 7. Front-End Slave Database Master Database
  8. 8. Large Datasets Don’t Push Well
  9. 9. Adding Capacity is Expensive Large Datasets Don’t Push Well
  10. 10. Adding Capacity is Expensive Large Datasets Don’t Push Well What to do after push failure?
  11. 11. User-Generated Content Adding Capacity is Expensive Large Datasets Don’t Push Well What to do after push failure?
  12. 12. Building New Sites
  13. 13. Requirements
  14. 14. Requirements massive scalability flexible deployment highly dynamic separation of concerns mashability
  15. 15. i.e., Services
  16. 16. Scalability Simplicity Reuse Interoperability
  17. 17. i.e., HTTP
  18. 18. Front-End Caching Back-End Database
  19. 19. Single Source of Truth
  20. 20. Single Source of Truth Cache Replicates Naturally
  21. 21. Adding Capacity Is Easy Single Source of Truth Cache Replicates Naturally
  22. 22. UGC Pushes Through Adding Capacity Is Easy Single Source of Truth Cache Replicates Naturally
  23. 23. HTTP Caching Intermediaries
  24. 24. HTTP Caching Intermediaries Freshness
  25. 25. HTTP Caching Intermediaries Freshness Validation
  26. 26. HTTP Caching Intermediaries Freshness Validation Metrics 0101011010010101101010011001010101
  27. 27. HTTP Caching Intermediaries Freshness Load Balancing Validation Metrics
  28. 28. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics
  29. 29. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics Negative Caching
  30. 30. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics Negative Caching Collapsed Forwarding
  31. 31. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics Negative Caching Collapsed Forwarding stale-while-revalidate
  32. 32. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics Negative Caching Collapsed Forwarding ? stale-while-revalidate stale-if-error
  33. 33. HTTP Caching Intermediaries Freshness Load Balancing Validation Cache Peering Metrics Negative Caching Collapsed Forwarding stale-while-revalidate stale-if-error Invalidation Channels
  34. 34. 12,000 req/sec/core
  35. 35. Tens of Thousands of Connections
  36. 36. pitfalls REST vs. WS-* wars theory vs. practice human-intuitive, but not programmer-intuitive different deployment/operational concerns formats are hard format / interface proliferation authentication isn’t there yet tools have a way to go
  37. 37. what’s needed tools web description language data-oriented schema language investment in the Atom stack HTTP test suite

×