Download as PDF, PPTX
More Related Content
Similar to Resilient Design 101 (JeeConf 2017)
Resilient Design 101 (JeeConf 2017)
- 1. Resilient design 101 AvishaiIsh-Shalom github.com/avishai-ish-shalom@nukembergavishai.is@wix.com
- 2. Wix in numbers ~500 Engineers ~ 1500 employees ~ 100M users ~ 500 micro services Lithuania Ukraine Vilnius Kyiv Dnipro Wix Engineering Locations Israel Tel-Aviv Be’er Sheva
- 3.
- 4. Queues are everywhere! ▪Futures/Executors ▪ Sockets ▪ Locks (DB Connection pools) ▪ Callbacks in node.js/Netty Anything async?!
- 5. Queues ▪ Incoming load(arrival rate) ▪ Service from the queue (service rate) ▪ Service discipline (FIFO/LIFO/Priority) ▪ Latency = Wait time + Service time ▪ Service time independent of queue
- 6. It varies ▪ Arrivalrate fluctuates ▪ Service times fluctuates ▪ Delays accumulate ▪ Idle time wasted Queues are almost always full or near-empty!
- 7. Capacity & Latency ▪ Latency(and queue size) rises to infinity as utilization approaches 1 ▪ For QoS ρ << 0.75 ▪ Decent latency -> over capacity ρ = arrival rate / service rate (utilization)
- 8. Implications Infinite queues: ▪ Memorypressure / OOM ▪ High latency ▪ Stale work Always limit queue size! Work item TTL*
- 9. Latency & Service time λ= wait time σ = service time ρ = utilization
- 10. Utilization fluctuates! ▪ 10%fluctuation at = 0.5 will hardly affects latency (~ 1.1x) ▪ 10% fluctuation at = 0.9 will kill you (~ 10x latency) ▪ Be careful when overloading resources ▪ During peak load we must be extra careful ▪ Highly varied load must be capped
- 11. Practical advice ▪ Usechokepoints (throttling/load shedding) ▪ Plan for low utilization of slow resources Example Resource Latency Planned Utilization RPC thread pool 1ms 0.75 DB connection pool 10ms 0.5
- 12. Backpressure ▪ Internal queuesfill up and cause latency ▪ Front layer will continue sending traffic ▪ We need to inform the client that we’re out of capacity ▪ E.g.: Blocking client, HTTP 503, finite queues for threadpools
- 13. Backpressure ▪ Blocking codehas backpressure by default ▪ Executors, remote calls and async code need explicit backpressure ▪ E.g. producer/consumer through Kafka
- 14. Load shedding ▪ Atradeoff between latency and error rate ▪ Cap the queue size / throttle arrival rate ▪ Reject excess work or send to fallback service Example: Facebook uses LIFO queue and rejects stale work http://queue.acm.org/detail.cfm?id=2839461
- 15.
- 16. Jetty architecture Thread pool(QTP) Socket Acceptor thread
- 17. Too many threads ▪O/S also has a queue ▪ Threads take memory, FDs, etc ▪ What about shared resources? Bad QoS, GC storms, ungraceful degradation Not enough threads wrong ▪ Work will queue up ▪ Not enough RUNNING threads High latency, low resource utilization
- 18. Capacity/Latency tradeoffs When optimizingfor Latency: For low latency, resources must be available when needed Keep the queue empty ▪ Block or apply backpressure ▪ Keep the queue small ▪ Overprovision
- 19. Capacity/Latency tradeoffs When optimizingfor Capacity For max capacity, resources must always have work waiting Keep the queue full ▪ We use a large queue to buffer work ▪ Queueing increases latency ▪ Queue size >> concurrency
- 20. How may threads? ▪Assuming CPU is the limiting resource ▪ Compute by maximal load (opt. latency) ▪ With a Grid: How many cores??? Java Concurrency in Practice (http://jcip.net/)
- 21. How may threads? Howto compute? ▪ Transaction time = W + C ▪ C ~ Total CPU time / throughput ▪ U ~ 0.5 – 0.7 (account for O/S, JVM, GC - and 0.75 utilization target) ▪ Memory and other resource limits
- 22. What about asyncservers?
- 23. Async servers architecture Socket Eventloop epoll Callbacks O/S Syscalls
- 24. Async systems ▪ Eventloop callback/handler queue ▪ The callback queue is unlimited (!!!) ▪ Event loop can block (ouch) ▪ No inherent concurrency limit ▪ No backpressure (*)
- 25. Async systems -overload ▪ No preemption -> no QoS ▪ No backpressure -> overload ▪ Hard to tune ▪ Hard to limit concurrency/queue size ▪ Hard to debug
- 26. So what’s thepoint? ▪ High concurrency ▪ More control ▪ I/O heavy servers Still evolving…. let’s revisit in a few years?
- 27.
- 28. Little’s law ▪ Holdsfor all distributions ▪ For “stable” systems ▪ Holds for systems and their subsystems ▪ “Throughput” is either Arrival rate or Service rate depending on the context. Be careful! L = λ⋅W L = Avg clients in the system λ = Avg Throughput W = Avg Latency
- 29. Using Little’s law ▪How many requests queued inside the system? ▪ Verifying load tests / benchmarks ▪ Calculating latency when no direct measurement is possible Go watch Gil Tene’s "How NOT to Measure Latency" Read Benchmarking Blunders and Things That Go Bump in the Night
- 30.
- 31. How not totimeout People use arbitrary timeout values ▪ DB timeout > Overall transaction timeout ▪ Cache timeout > DB latency ▪ Huge unrealistic timeouts ▪ Refusing to return errors P.S: connection timeout, read timeout & transaction timeout are not the same thing
- 32. Deciding on timeouts Usethe distribution luke! ▪ Resources/Errors tradeoff ▪ Cumulative distribution chart ▪ Watch out for multiple modes ▪ Context, context, context
- 33. Timeouts should bederived from real world constraints!
- 34. UX numbers everydeveloper needs to know ▪ Smooth motion perception threshold: ~ 20ms ▪ Immediate reaction threshold: ~ 100ms ▪ Delay perception threshold: ~ 300ms ▪ Focus threshold: ~ 1sec ▪ Frustration threshold: ~ 10sec Google's RAIL model UX powers of 10
- 35. Hardware latency numbersevery developer needs to know ▪ SSD Disk seek: 0.15ms ▪ Magnetic disk seek: ~ 10ms ▪ Round trip within same datacenter: ~ 0.5ms ▪ Packet roundtrip US->EU->US: ~ 150ms ▪ Send 1M over typical user WAN: ~ 1sec Latency numbers every developer needs to know (updated)
- 36. Timeout Budgets ▪ Decideon global timeouts ▪ Pass context object ▪ Each stage decrements budget ▪ Local timeouts according to budget ▪ If budget too low, terminate preemptively Think microservices Example Global: 500ms Stage Used Budget Timeout Authorization 6ms 494ms 100ms Data fetch (DB) 123ms 371ms 200ms Processing 47ms 324ms 371ms Rendering 89ms 235ms 324ms Audit 2ms - - Filter 10ms 223ms 233ms
- 37. The debt buyer ▪Transactions may return eventually after timeout ▪ Does the client really have to wait? ▪ Timeout and return error/default response to client (50ms) ▪ Keep waiting asynchronously (1 sec) Can’t be used when client is expecting data back
- 38.
- 39.