Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 15

Waiter: An Open-Source Distributed Auto-Scaler

0

Share

Download to read offline

One of the key challenges in developing a service-oriented architecture (SOA) is anticipating traffic patterns and scaling the number of running instances of services to meet demand. In many situations, it’s hard to know how much traffic a service will receive and when that traffic will come. A service may see no requests for several days in a row and then suddenly see thousands of requests per second. If developers underestimate peak traffic, their service can quickly become overwhelmed and unresponsive, and may even crash, resulting in constant human intervention and poor developer productivity. On the other hand, if they provision sufficient capacity upfront, the resources they allocate will be completely wasted when there’s no traffic. In order to allow for better resource utilization, many cluster management platforms provide auto-scaling features. These features tend to auto-scale at the machine/resource level (as opposed to the request level) or by deferring to logic in the application layer. A more optimal approach would be to run services when–and only when–there is traffic. Waiter is a distributed auto-scaler that delivers this optimal type of request-level auto-scaling. It requires no input or handling from applications and is agnostic to underlying cluster managers; it currently uses Mesos, but can easily run on top of Kubernetes or other solutions. Another challenge with SOAs is enabling the evolution of service implementations without breaking downstream customers. On this front, Waiter supports service-versioning for downstream consumers by running multiple, individually-addressable versions of services. It automatically manages service lifecycles and reaps older versions after a period of inactivity. With a variety of unique features, Waiter is a compelling platform for applications across a broad range of industries. Existing web services can run on Waiter without modification as long as they communicate over HTTP and support the transmission of client requests to arbitrary backends. Two Sigma has employed the platform in a variety of critical production contexts for over two years, with use cases rising to hundreds of millions of requests per day.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Waiter: An Open-Source Distributed Auto-Scaler

  1. 1. Waiter https://github.com/twosigma/waiter Shams Imam Software Engineer, Two Sigma Investments, LP Two Sigma Open Source Meetup November 13th, 2017
  2. 2. • Motivation? • Waiter 2 Agenda
  3. 3. 3 A typical architecture Client(s) Load Balancer Backend Backend Backend Request
  4. 4. 4 Problem 1: Dead Backend Client(s) Load Balancer AWOL Backend Backend Request
  5. 5. 5 Problem 2 : Load Imbalance Client(s) Load Balancer Backend Backend Backend Request
  6. 6. 6 Problem 3a: Too Many Requests Client(s) Load Balancer Backend Backend Backend Request
  7. 7. 7 Problem 3b: Manual Scaling Client(s) Load Balancer Backend Request Backend Backend Backend Backend Backend
  8. 8. 8 Problem 3c: Wasting Resources Client(s) Load Balancer Backend Request Backend Backend Backend Backend Backend
  9. 9. 9 Problem 4: Version Mismatch RequestClient Version f8d2 Client Version c8b5 Backend Version f8d2
  10. 10. • Service platform that – Runs and auto-scales your services • Parameterized services • Usage metrics • Authentication – Automatic management of resources – Provides resiliency when instances fail – Easy version management 10 What is Waiter?
  11. 11. 11 Architecture – MITM HTTP Proxy Client(s) WAITER Backend Backend Backend Request
  12. 12. 12 Architecture – Distributed version Client(s) Service-1 Service-1 Service-1 Request WAITER WAITER WAITER Service-2 Service-2
  13. 13. • We manage both open source and internal versions of the products – Internal version relies on plug-in mechanism • We rely heavily on automated tests to help prevent regression – Travis (open source version) – Jenkins (internal version) 13 Workflow
  14. 14. • Leverage the wisdom of crowds to evolve the platform and integrate the platform with new technologies we cannot foresee yet • By open sourcing, we open the door for many other opportunities – Waiter UI being built by a collaborator 14 Why open source
  15. 15. 15 (loop [] (when-let [comment (or (audience/question) (audience/feedback))] (respond-to comment) (recur))

×