Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

FinagleCon_2015_Pinterest

916 views

Published on

How we use Finagle at Pinterest. Presented at FinagleCon hosted by Twitter on 08/13/2015.

Published in: Technology
  • Be the first to comment

FinagleCon_2015_Pinterest

  1. 1. Finagle at Pinterest 1 Yongsheng Wu Email: yongsheng@pinterest.com Linkedin: www.linkedin.com/in/yongshengwu Pinterest: www.pinterest.com/yswu Twitter: @yswu 08-13-2015
  2. 2. 2 Introduction of Pinterest SOA Service Proxy Dynamic Service Discovery Python Finagle Thrift Client Genesis
  3. 3. Pinterest is solving the discovery problem. 3
  4. 4. 4
  5. 5. 5
  6. 6. 6 50+ Billion Pins categorized by people into more than 1 Billion Boards
  7. 7. 7 Introduction of Pinterest SOA Service Proxy Dynamic Service Discovery Python Finagle Thrift Client Genesis
  8. 8. Pinterest March, 2010 8
  9. 9. 10
  10. 10. Why Finagle? 11 ๏Reactive Manifesto ✓Responsive ✓Resilient ✓Elastic ✓Message Driven ๏Efficiency ๏Future Programming Model ๏Visibility ๏Community
  11. 11. Best Practices for Service Migration 12 ๏Avoid Changes of Functionalities ๏Test-Driven Development ‣ Unit Test ‣ Integration Test ๏Dark Traffic ‣ Comparison ‣ Capacity ๏Light Traffic
  12. 12. Pin Creation 13
  13. 13. 14 Introduction of Pinterest SOA Service Proxy Dynamic Service Discovery Python Finagle Thrift Client Genesis
  14. 14. Service Proxy 15
  15. 15. Service Proxy 16
  16. 16. Service Proxy Request Logging 17 ๏Request logging automatically handled by proxy ‣ Sampled Access Logging ‣ Sampled Slow Request Logging ‣ Sampled Failure Request Logging ๏Log format
  17. 17. Service Proxy Request Logging 18 ๏Request logging automatically handled by proxy ‣ Sampled Access Logging ‣ Sampled Slow Request Logging ‣ Sampled Failure Request Logging ๏Log format
  18. 18. Service Proxy Request Logging 19 ๏Request logging automatically handled by proxy ‣ Sampled Access Logging ‣ Sampled Slow Request Logging ‣ Sampled Failure Request Logging ๏Log format
  19. 19. Service Proxy Request Logging 20 ๏Request logging automatically handled by proxy ‣ Sampled Access Logging ‣ Sampled Slow Request Logging ‣ Sampled Failure Request Logging ๏Log format
  20. 20. Service Proxy Request Logging 21 ๏Request logging automatically handled by proxy ‣ Sampled Access Logging ‣ Sampled Slow Request Logging ‣ Sampled Failure Request Logging ๏Log format
  21. 21. Service Exception Monitoring 22
  22. 22. Service Proxy Metrics 23
  23. 23. Service Proxy Metrics 24
  24. 24. Service Proxy Metrics 25
  25. 25. Service Proxy Rate Limiting 26 ๏Guava RateLimiter (Token Bucket based) and ServerSet ๏No coordination among servers in the cluster ๏Rate limiting rules ๏Max QPS to service ๏Max QPS to a particular API endpoint ๏Max QPS from a specific client to service ๏Max QPS from a specific client to a particular API endpoint
  26. 26. 27
  27. 27. 28 Introduction of Pinterest SOA Service Proxy Dynamic Service Discovery Python Finagle Thrift Client Genesis
  28. 28. Dynamic Service Discovery 29
  29. 29. Dynamic Service Discovery 30
  30. 30. Dynamic Service Discovery 31
  31. 31. 32 Introduction of Pinterest SOA Service Proxy Dynamic Service Discovery Python Finagle Thrift Client Genesis
  32. 32. Finagle Thrift Protocol 33
  33. 33. Finagle Thrift Protocol 34
  34. 34. 35 Introduction of Pinterest SOA Service Proxy Dynamic Service Discovery Python Finagle Thrift Client Genesis
  35. 35. Genesis 36 ๏Interactive Q&A Shell ๏Finagle Server and Client Generation ‣ Existing thrift IDL ๏Python Thrift Client Generation ๏Statsboard Generation ‣ System Metrics (CPU, Memory, Network, Disk) ‣ Finagle Stats (requests, failures, GC) ‣ Per API Endpoint Stats ‣ QPS ‣ Failure Count ‣ Latency
  36. 36. Genesis 37
  37. 37. Genesis 38
  38. 38. Genesis 39
  39. 39. 40 Introduction of Pinterest SOA Service Proxy Dynamic Service Discovery Python Finagle Thrift Client Genesis
  40. 40. Future Challenges 41 ๏Unified Data Model, Partial Retrieval ๏One-step Service Generation ๏Distributed Tracing ๏Auto-scaling ๏Cell-based Architecture ๏MUX, HTTP/2 ๏… ….

×