The document discusses the significance of low latency in application performance, comparing different latency benchmarks and highlighting challenges in measuring and optimizing latency. It emphasizes the impact of garbage collection (GC) on latency and advocates for using asynchronous logging to improve performance in low-latency environments. Additionally, it addresses issues related to TCP packet transmission and DNS lookup delays, offering diagnostic tools and suggestions for optimization.

1. On the way to low
latency
Artem Orobets
Smartling Inc

2. Long story short
We realized that latency is important for us
Our fabulous architecture supposed to work, but it didn’t
The issues that we have faced on the way

3. Those guys consider 10µs
latencies slow
We have only 100ms
threshold
We are not a trading company

4. Latencies about 50ms
is barely noticeable for human
Trans-Atlantic Path 91 ms*
Trans-Pacific Path 141 ms*
From Earth to Mars 3-22 min

5. Importance of latency
• SLA
• Negative correlation
to income

6. How to measure it?

7. Duration of a single
test run

8. Average of test run
durations

10. Quantiles of test run
durations
(usually 95th, 99th percentiles)

11. Performance benchmark
98.47% <= 2 ms
99.95% <= 10 ms
99.98% <= 16 ms
99.99% <= 17 ms
100.00% <= 18 ms
750 rps
Throughput
Latency percentiles

12. Benchmarks are hard
Almost all latency
benchmarks are broken
because almost all
benchmarking tools are
broken.

13. 0 100 200
System Stalled
10000 req
Avg = 50 sec
10000 req
Avg = 1 ms
Normal Operation
Real system

14. 0 100 200
System Stalled
Avg = 25 sec
50th perc = 1 ms
75th perc = 50 sec
99.99th perc = 100 sec
Real system

15. 0 100 200
System Stalled
1 req
Avg = 100 sec
10000 req
Avg = 1 ms
Normal Operation
Performance test

16. 0 100 200
System Stalled
Avg = 25 sec
50th perc = 1 ms
75th perc = 1 ms
99.99th perc = 1 ms
Perfomance test

17. Imagine we’ve made some
amazing improvement

18. 0 100 200
Degradation
10000 req
Avg = 5 ms
10000 req
Avg = 1 ms
Normal Operation
After improvement

19. 0 100 200
Degradation
Avg = 25 sec
50th perc = 1 ms
75th perc = 2.5 ms (was 1ms)
99.99th perc = 5 ms (was 1ms)
Perfomance test

20. But what can do?

21. A good tool
can give you a clue

22. KPI

23. Problem that we faced

24. How much time GC
could take?

25. GC logging
• -Xloggc:path_to_log_file
• -XX:+PrintGCDetails
• -XX:+PrintGCDateStamps
• -XX:+PrintHeapAtGC
• -XX:+PrintTenuringDistribution

26. -XX:+PrintGCDetails
[GC (Allocation Failure) 260526.491: [ParNew
…
[Times: user=0.02 sys=0.00, real=0.01 secs]

27. -XX:+PrintHeapAtGC
Heap after GC invocations=43363
(full 3):
par new generation total 59008K, used
1335K
eden space 52480K, 0%
from space 6528K, 20% used
to space 6528K, 0% used
concurrent mark-sweep generation total
2031616K, used 1830227K

28. -XX:+PrintTenuringDistribution
Desired survivor size 3342336 bytes, new
threshold 2 (max 2)
- age 1: 878568 bytes, 878568 total
- age 2: 1616 bytes, 880184 total
: 53829K->1380K(59008K), 0.0083140 secs]
1884058K->1831609K(2090624K), 0.0084006 secs]

29. ~100ms GC pauses
in logs

30. -XX:+UseConcMarkSweepGC

31. Note: CMS collector on young
generation uses the same algorithm
as that of the parallel collector.
Java GC documentation at oracle.com
* http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html

33. Too many alive objects
during young gen GC
• Minimize survivors
• Watch the tenuring threshold, might need
to tune it to tenure long lived objects faster
• Reduce NewSize
• Reduce survivor spaces

34. Watch your GC
*time span is 2h

35. Watch your GC

36. Some requests take
almost a second
And it seems it always happens after deploy

37. is so lazy

38. Smoke tests
• A good practice when you have continuous
delivery
• It makes all your code initialized by the time
real load comes in

39. Logging
Synchronous logging is not appropriate for
asynchronous application

40. log4j2: Asynchronous Loggers
for Low-Latency Logging
http://logging.apache.org/log4j/2.x/manual/async.html

41. Sync Async
98.85% <= 1 ms
99.95% <= 7 ms
99.98% <= 13 ms
99.99% <= 15 ms
100.00% <= 18 ms
1658 rps
98.47% <= 2 ms
99.95% <= 10 ms
99.98% <= 16 ms
99.99% <= 17 ms
100.00% <= 18 ms
769.05 rps
Logging

42. Pauses 50-150ms
A network according
to logs

43. Disappear when I scroll
through logs via SSH

45. Any ideas?

46. TCP_NODELAY

48. Nagle's algorithm
• the "small packet problem”
• TCP packets have a 40 byte header
(20 bytes for TCP, 20 bytes for IPv4)
• combining a number of small outgoing messages,
and sending them all at once

49. • Pauses ~100 ms every couple of hours
• During connection creation
• Doesn’t reproduces on a local setup

50. How to diagnose that?

51. tcpdump -i eth0

52. TCPDUMP
15:47:57.250119 IP (tos 0x0, ttl 64, id 44402, offset
0, flags [DF], proto TCP (6), length 569)
192.168.3.131.58749 > 93.184.216.34.80: Flags
[P.], cksum 0x76b5 (correct), seq
3847355529:3847356046, ack 3021125542, win
4096, options [nop,nop,TS val 848825338 ecr
1053000005], length 517: HTTP, length: 517
GET / HTTP/1.1
Host: example.com
Connection: keep-alive
…

53. TCPDUMP
15:58:32.009884 IP (tos 0x0, ttl 255, id 39809, offset 0,
flags [none], proto UDP (17), length 63)
192.168.3.131.56546 > 192.168.3.1.53: [udp sum ok]
52969+ A? www.google.com.ua. …
15:58:32.012844 IP (tos 0x0, ttl 64, id 0, offset 0, flags
[DF], proto UDP (17), length 127)
192.168.3.1.53 > 192.168.3.131.56546: [udp sum ok]
52969 q: A? www.google.com.ua. …

54. DNS lookups
• After hours of looking through tcp dumps
• We have found that DNS lookups
sometimes take more than 100ms

55. aorobets@smartling.com

56. http://tech.smartling.com/
aorobets@smartling.com