Артем Оробец «На пути к low-latency»
•Download as PPTX, PDF•
0 likes•1,238 views
Путь реализации сервиса с очень строгими требованиями по latency
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Артем Оробец «На пути к low-latency»
Similar to Артем Оробец «На пути к low-latency» (20)
More from DataArt
More from DataArt (20)
Recently uploaded
Recently uploaded (20)
Артем Оробец «На пути к low-latency»
- 2. On the way to low latency Artem Orobets Smartling Inc
- 3. You mostly care about throughput
- 4. Java for low latency? • Increasingly Java is being used to build applications with low latency requirements • Developers should have a deeper understanding of the JVM
- 5. What is low latency?
- 6. Latency is a time interval between the stimulation and response
- 7. What is latency? total response time = service time + time waiting for service
- 8. Those guys consider 10µs latencies slow We are not a trading company
- 9. Latencies about 50ms is barely noticeable for human
- 10. Requirements • We have latency restriction 100ms • After this time request is considered as failed
- 11. Is that what you call low latency?
- 12. Storage
- 13. * where latency is 99th percentile
- 14. Context switch problem In production we have about 4k connections opened simultaneously
- 15. Context switch problem • Thread per request doesn’t work • Too much overhead on context switching • Too much overhead on memory Usually a Thread takes memory from 256kb to 1mb for the stack space!
- 16. Great architecture in theory But in practice it is not enough!
- 17. We have have fixed a lot of things that we believed were the most problematic parts. But they weren’t.
- 18. Find an evidence that proves your suggestion
- 19. A good tool can give you a clue • Proper logging and log analysis tool • Performance tests • Monitoring
- 20. A good tool can give you a clue
- 21. KPI is necessity
- 22. A big amount of wrappers Significant allocation pressure Intensive usage of lazy initialization. First requests very slow
- 23. Smoke tests • A good practice when you have continuous delivery • It makes all your code initialized by the time real load comes in
- 24. Logging us logging is not appropriate for asynchronous
- 25. Synch logging 82.83% <= 8 milliseconds 99.90% <= 19 milliseconds 99.94% <= 34 milliseconds 99.97% <= 39 milliseconds 99.98% <= 43 milliseconds 99.99% <= 48 milliseconds 100.00% <= 53 milliseconds 251.59 requests per second
- 26. Asynch logging 99.86% <= 5 milliseconds 99.91% <= 6 milliseconds 99.96% <= 7 milliseconds 99.98% <= 11 milliseconds 99.99% <= 13 milliseconds 100.00% <= 14 milliseconds 1657.28 requests per second
- 27. Another prod issue • A long pauses which happened quite often • We couldn’t repeat the issue in local setup
- 28. DNS lookups • After hours of looking through tcp dumps • We have found that DNS lookups sometimes take more than 100ms
- 30. TCP_NODELAY
- 32. GC logging • -Xloggc:path_to_log_file • -XX:+PrintGCDetails • -XX:+PrintGCDateStamps • -XX:+PrintHeapAtGC • -XX:+PrintTenuringDistribution
- 33. -XX:+PrintGCDetails [GC (Allocation Failure) 260526.491: [ParNew … [Times: user=0.02 sys=0.00, real=0.01 secs]
- 34. -XX:+PrintHeapAtGC Heap after GC invocations=43363 (full 3): par new generation total 59008K, used 1335K eden space 52480K, 0% from space 6528K, 20% used to space 6528K, 0% used concurrent mark-sweep generation total 2031616K, used 1830227K
- 35. -XX:+PrintTenuringDistribution Desired survivor size 3342336 bytes, new threshold 2 (max 2) - age 1: 878568 bytes, 878568 total - age 2: 1616 bytes, 880184 total : 53829K->1380K(59008K), 0.0083140 secs] 1884058K->1831609K(2090624K), 0.0084006 secs]
- 36. Too many alive objects during young gen GC • Minimize survivors • Watch the tenuring threshold, might need to tune it to tenure long lived objects faster • Reduce NewSize • Reduce survivor spaces
- 37. Watch your GC *time span is 2h
- 38. Watch your GC