Load balancingtheory and practice
WelcomeMe:• Dave Rosenthal• Co-founder of FoundationDB• Spent last three years building a distributed  transactional NoSQL...
WarningThere is an ugly downside to learning about loadbalancing: TSA checkpoints, grocery store lines,and traffic lights ...
What is load balancing?Wikipedia: “…methodology to distributeworkload across multiple computers … toachieve optimal resour...
The latency curve                                           Overload          10000          1000                         ...
Goal for real-time systems          10000          1000Latency            100                  Low latency at             ...
Goal for batch systems          10000          1000                    High Jobs/sec at aLatency            100     reason...
The latency curve               1000               100Latency (ms)                              Better load balancing stra...
Load balancing tensions• We want to reduce queue lengths in the  system to yield better latency• We want to lengthen queue...
Can we just limit queue sizes?                    40                    35                    30% of dropped jobs         ...
Simple strategiesGlobal job queue: for slow tasksRound robin: for highly uniform situationsRandom: probably won’t screw yo...
Use a global queue if possible                         10Latency under 80% load                          1                ...
Options for information transfer•   None (rare)•   Latency (most common)•   Failure detection•   Explicit    – Load averag...
FoundationDB’s approach1. Request to random of three servers2. Server either answers query or replies “busy” if its   queu...
FDB latency curve before/after          100          10Latency            1          0.1                0   200000   40000...
Tackling load balancing•   Queuing theory: One useful insight•   Simulation: Do this•   Instrumentation: Do this•   Contro...
The one insight: Little’s law                       Q = R*W•   (Q)ueue size = (R)ate * (W)ait-time•   Q is the average num...
Little’s law example 1                     Q = R*W•   We get 1,000,000 request per second (R=1E6)•   We take 100 ms to ser...
Little’s law example 2                   W = Q/R• We have 100 users in the system making  continuous requests (Q=100)• We ...
Little’s law ramifications                     Q = R*W• In distributed system:  – R scales up  – W remains the same, or ge...
The rest of queuing theoryErlang• A language• A man (Agner Krarup Erlang)• And a unit! (Q from little’s law AKA offered lo...
Abandon hope                             Math for queuing theory                     10000Complexity of Math              ...
SimulationThe best way to explore distributed systembehavior
QuizModel: Jobs of random durations. 80% load.Goal: Minimize average job latency.What to work a bit more on?• First task r...
Simulation code snippits
Simulation results at 80% load            First task received            Last task received                 Shortest task ...
Simulation results at 95% load            First task received            Last task received                 Shortest task ...
FoundationDB’s approach• Strategy validated using simulation used for a  single server’s fiber scheduling• High priority: ...
Load spikesLow load system               High load system  Bursts of job requests can destroy latency. The  effect is quad...
Burst-avoiding tip1. Search for any delay/interval in your system2. If system correctness depends on the   delay/interval ...
Overload                                 Overload          10000          1000Latency            100             10       ...
OverloadWhat happens when work comes in too fast?• Somewhere in your system a queue is going to  get huge. Where?• Lowered...
Overload (cont’d): SloshingLoading 10 million rows into popular NoSQL K/Vstore shows sloshing                   12.5 minutes
Overload (cont’d): No sloshingLoading 10 million rows into FDB shows smoothbehavior:
System queuing       Queue    Queue    QueueWork A B C     Node 1   Node 2   Node 3 D E
System queuing       Queue    Queue    Queue         AWork B C     Node 1   Node 2   Node 3 D E
Internal queue buildup          Queue   Queue    Queue           A           B            CWork       D         Node 1   N...
Even queues, external buildup           Queue    Queue    Queue                               A                      B    ...
Our approach“Ratekeeper”• Active management of internal queue sizes  prevents sloshing• Avoids every subcomponent needing ...
Ratekeeper in action                        1400000                        1200000Operations per second                   ...
Ratekeeper internals
What can go wrongWell, we are controlling the queue depths of thesystem, so, basically, everything in controltheory…Namely...
Recognizing oscillation• Something moving up and down :)  – Look for low utilization of parallel resources  – Zoom in!• Th...
What oscillation looks like                70                60                50Utilization %                40          ...
What oscillation looks like                120                100                80Utilization %                60        ...
Avoiding oscillation• This is control theory—avoid if possible!• The major thing to know: control gets harder  at frequenc...
Instrumentation If you can’t measure, you can’t make it betterThings that might be nice to measure:• Latencies• Queue leng...
Measuring latenciesOur approach:• We want information about the distribution, not  just the average• We use a “Distributio...
Measuring queue lengthsOur approach:• Track the % of time that a queue is at zero length• Measure queue length snapshots a...
Measuring latency from blocking• Easy to calculate:   – L = (b0^2 + b1^2 … bN^2) / elapsed   – Total all squared seconds o...
SummaryThanks for listening, and remember:• Everything has a latency curve• Little’s law• Randomize regular intervals• Val...
Prioritization/QOS• Can help in systems under partial load• Vital in systems that handle batch and real-  time loads simul...
Congestion pricing• My favorite topic• Priority isn’t just a function of the benefit of  your job• To be a good citizen, y...
Other FIFO alternatives?• LIFO  – Avoids the reason to line up early  – In situations where there is adequate capacity to ...
Upcoming SlideShare
Loading in...5
×

Load balancing theory and practice

2,456
-1

Published on

Presentation about load balancing by David Rosenthal, one of the founders at FoundationDB.

Published in: Technology, Business
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,456
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
84
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Load balancing theory and practice

  1. 1. Load balancingtheory and practice
  2. 2. WelcomeMe:• Dave Rosenthal• Co-founder of FoundationDB• Spent last three years building a distributed transactional NoSQL database• It’s my birthdayAny time you have multiple computers working on a job, you have a load balancing problem!
  3. 3. WarningThere is an ugly downside to learning about loadbalancing: TSA checkpoints, grocery store lines,and traffic lights may become even morefrustrating.
  4. 4. What is load balancing?Wikipedia: “…methodology to distributeworkload across multiple computers … toachieve optimal resource utilization, maximizethroughput, minimize response time, and avoidoverload” All part of the latency curve
  5. 5. The latency curve Overload 10000 1000 SaturationLatency 100 10 1 Nominal Interesting Jobs/second
  6. 6. Goal for real-time systems 10000 1000Latency 100 Low latency at 10 given load 1 Jobs/second
  7. 7. Goal for batch systems 10000 1000 High Jobs/sec at aLatency 100 reasonable latency 10 1 Jobs/second
  8. 8. The latency curve 1000 100Latency (ms) Better load balancing strategies can dramatically improve both latency and throughput 10 1 0 0.2 0.4 0.6 0.8 1 Load
  9. 9. Load balancing tensions• We want to reduce queue lengths in the system to yield better latency• We want to lengthen queue lengths to keep a “buffer” of work to keep busy during irregular traffic and yield better throughput• For distributed systems, equalizing queue lengths sounds good
  10. 10. Can we just limit queue sizes? 40 35 30% of dropped jobs 25 20 15 10 5 0 0 5 10 15 20 Queued job limit
  11. 11. Simple strategiesGlobal job queue: for slow tasksRound robin: for highly uniform situationsRandom: probably won’t screw youSticky: for cacheable situationsFastest of N tries: tradeoff throughput forlatency. I recommend N = 2 or 3.
  12. 12. Use a global queue if possible 10Latency under 80% load 1 Random assignment Global Job Queue 0.1 1 2 3 4 5 6 7 8 9 10 Cluster Size
  13. 13. Options for information transfer• None (rare)• Latency (most common)• Failure detection• Explicit – Load average – Queue length – Response times
  14. 14. FoundationDB’s approach1. Request to random of three servers2. Server either answers query or replies “busy” if its queue is longer than the queue limit estimate3. Queries that were busy are sent to second random server with “must do” flag set.Queue limit = 25 * 2^(20*P)• A global queue limit is implicitly shared by estimating the fraction of incoming requests (P) that are flagged “must do”• Converges to a P(redirect)/queue-size equilibrium
  15. 15. FDB latency curve before/after 100 10Latency 1 0.1 0 200000 400000 600000 800000 1000000 1200000 Operations per second
  16. 16. Tackling load balancing• Queuing theory: One useful insight• Simulation: Do this• Instrumentation: Do this• Control theory: Know how to avoid this• Operations research: Read about this for fun – Blackett: Shield planes where they are not shot!
  17. 17. The one insight: Little’s law Q = R*W• (Q)ueue size = (R)ate * (W)ait-time• Q is the average number of jobs in the system• R is the average arrival rate (jobs/second)• W is the average wait time (seconds)• For any (!) steady-state systems – Or sub-systems, or joint systems, or…
  18. 18. Little’s law example 1 Q = R*W• We get 1,000,000 request per second (R=1E6)• We take 100 ms to service each request• (Q = 1E6*0.100)• Little’s Law: Average queue depth is 100,000!
  19. 19. Little’s law example 2 W = Q/R• We have 100 users in the system making continuous requests (Q=100)• We get 10,000 requests per second• (W = 100 / 10,000)• Little’s Law: Average wait time is 10 ms
  20. 20. Little’s law ramifications Q = R*W• In distributed system: – R scales up – W remains the same, or gets a bit worse• To maintain performance, you’re going to need a whole lot of jobs in flight
  21. 21. The rest of queuing theoryErlang• A language• A man (Agner Krarup Erlang)• And a unit! (Q from little’s law AKA offered load is measured in dimensionless Erlang units)• Erlang-B formula (for limited-length queues)• Erlang-C formula (P(waiting))
  22. 22. Abandon hope Math for queuing theory 10000Complexity of Math 1000 100 10 1 Little’s law Real-world applicability ?
  23. 23. SimulationThe best way to explore distributed systembehavior
  24. 24. QuizModel: Jobs of random durations. 80% load.Goal: Minimize average job latency.What to work a bit more on?• First task received• Last task received• Shortest task• Longest task• Random task• Task with least work remaining• Task with most work remaining
  25. 25. Simulation code snippits
  26. 26. Simulation results at 80% load First task received Last task received Shortest task Longest task Random taskTask with least work remainingTask with most work remaining 0 10 20 30 40 50 Latency
  27. 27. Simulation results at 95% load First task received Last task received Shortest task Longest task Random taskTask with least work remainingTask with most work remaining 1 10 100 1000 10000 100000 Latency
  28. 28. FoundationDB’s approach• Strategy validated using simulation used for a single server’s fiber scheduling• High priority: Work on the next task to finish• But be careful to enqueue incoming work from the network with highest priority—we want to know about all our jobs to make good decisions• Low priority: Catch up with housekeeping (e.g. non-log writing)
  29. 29. Load spikesLow load system High load system Bursts of job requests can destroy latency. The effect is quadratic: A burst produces a queue of size B that lasts time proportional to B. On highly- loaded systems, the effect is multiplied by 1/(1- load), leading to huge latency impacts.
  30. 30. Burst-avoiding tip1. Search for any delay/interval in your system2. If system correctness depends on the delay/interval being exact, first fix that3. Now change that delay/interval to randomly wait 0.8-1.2 times the nominal time on each executionYMMV, but this tends to diffuse system events more evenly in time and help utilization and latency.
  31. 31. Overload Overload 10000 1000Latency 100 10 1 Jobs/second
  32. 32. OverloadWhat happens when work comes in too fast?• Somewhere in your system a queue is going to get huge. Where?• Lowered efficiency due to: – Sloshing – Poor caching• Unconditional acceptance of new work means no information transfer to previous system!
  33. 33. Overload (cont’d): SloshingLoading 10 million rows into popular NoSQL K/Vstore shows sloshing 12.5 minutes
  34. 34. Overload (cont’d): No sloshingLoading 10 million rows into FDB shows smoothbehavior:
  35. 35. System queuing Queue Queue QueueWork A B C Node 1 Node 2 Node 3 D E
  36. 36. System queuing Queue Queue Queue AWork B C Node 1 Node 2 Node 3 D E
  37. 37. Internal queue buildup Queue Queue Queue A B CWork D Node 1 Node 2 Node 3 E
  38. 38. Even queues, external buildup Queue Queue Queue A B CWork Node 1 Node 2 Node 3 D E …
  39. 39. Our approach“Ratekeeper”• Active management of internal queue sizes prevents sloshing• Avoids every subcomponent needing it’s own well-tuned load balancing strategy• Explicitly send queue information at 10hz back to a centrally-elected control algorithm• When queues get large, slow system input• Pushes latency into an external queue at the front of the system using “tickets”
  40. 40. Ratekeeper in action 1400000 1200000Operations per second 1000000 800000 600000 400000 200000 0 0 100 200 300 400 500 600 Seconds
  41. 41. Ratekeeper internals
  42. 42. What can go wrongWell, we are controlling the queue depths of thesystem, so, basically, everything in controltheory…Namely, oscillation:
  43. 43. Recognizing oscillation• Something moving up and down :) – Look for low utilization of parallel resources – Zoom in!• Think about sources of feedback—is there some way that having a machine getting more job done feeds either less or more work for that machine in the future? (probably yes)
  44. 44. What oscillation looks like 70 60 50Utilization % 40 Node A 30 Node B 20 10 0 1 2 3 4 5
  45. 45. What oscillation looks like 120 100 80Utilization % 60 Node A Node B 40 20 0 2 2.05 2.1 2.15 2.2 2.25 2.3 -20
  46. 46. Avoiding oscillation• This is control theory—avoid if possible!• The major thing to know: control gets harder at frequencies get higher. (e.g. Bose headphones)• Two strategies: – Control on a longer time scale – Introduce a low-pass-filer in the control loop (e.g. exponential moving average)
  47. 47. Instrumentation If you can’t measure, you can’t make it betterThings that might be nice to measure:• Latencies• Queue lengths• Causes of latency?
  48. 48. Measuring latenciesOur approach:• We want information about the distribution, not just the average• We use a “Distribution” class – addSample(X) – Stores 500+ samples – Throws away half of them when it hits 1000 samples, and halves probability of accepting new samples – Also tracks exact min, max, mean, and stddev
  49. 49. Measuring queue lengthsOur approach:• Track the % of time that a queue is at zero length• Measure queue length snapshots at intervals• Watch out for oscillations – Slow ones you can see – Fast ones look like noise (which, unfortunately, is also what noise looks like) – “Zoom in” to exclude the possibility of micro- oscillations
  50. 50. Measuring latency from blocking• Easy to calculate: – L = (b0^2 + b1^2 … bN^2) / elapsed – Total all squared seconds of blocking time over some interval, divide by the duration of the interval.• Measures impact of unavailability on mean latency from random traffic• Example: Is server’s slow latency explained by this lock?• Doesn’t count catch-up time.
  51. 51. SummaryThanks for listening, and remember:• Everything has a latency curve• Little’s law• Randomize regular intervals• Validate designs with simulation• Instrument May your queues be small, but not emptydavid.rosenthal@foundationdb.com
  52. 52. Prioritization/QOS• Can help in systems under partial load• Vital in systems that handle batch and real- time loads simultaneously• Be careful that high priority work doesn’t generate other high priority work plus other jobs in the queue. This can lead to poor utilization analogous to the internal queue buildup case.
  53. 53. Congestion pricing• My favorite topic• Priority isn’t just a function of the benefit of your job• To be a good citizen, you should subtract the costs to others• For example, jumping into the front of a long queue has costs proportional to the queue size
  54. 54. Other FIFO alternatives?• LIFO – Avoids the reason to line up early – In situations where there is adequate capacity to serve everyone, can yield better waiting times for everyone involved
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×