This document provides guidance on optimizing system and application performance. It discusses choosing relevant metrics like transactions per second (TPS) and latency. An experimental setup is needed to properly test performance, which should reproduce the production environment. Testing steps include scripting usage patterns, load testing, and validating results. The document then covers topics like TPS vs latency, resource utilization, Amdahl's law, and taking a top-down or iterative approach to identify bottlenecks like the CPU, network, I/O, operating system, Java Virtual Machine (JVM), garbage collection (GC), and class loading.

1. Code Above Lab LLC
Performance optimization
(balancer optimization)
Vitaly Pronto & Max Hwang

2. Metrics
1. Choose metrics:
a. TPS
b. Latency
c. Memory
d. CPU
e. HA
2. Prove correctness of metric:
a. Relevantness
b. Repeatable

3. Experimental Setup
You can't go any further without the
proper test environment
• Relevant: reproduces the phenomena
• Isolated: leaves out unwanted effects
• Measurable: provides the metrics
• Reliable: produces consistent result

4. Performance Testing Steps
• Script usage patterns into a load test
• Install/configure application to the same specs as production
• Setup monitoring
• Performance requirements
• OS performance counters and garbage collection
• Kill everything on your system
• Spike test to ensure correctness
• Load test
• Validate results
• Repeat as necessary

5. TPS vs Latency

6. Utilization
Utilization = ResourceBusyTime/TotalTime
Idle = 1 - Utilization
Efficiency != Utilization

7. Amdahl's law
Amdahl's Law is a law governing the speedup of using parallel processors on a
problem, versus using only one serial processor. Before we examine Amdahl's Law, we
should gain a better understanding of what is meant by speedup.
Speedup:
The speed of a program is the time it takes the program to excecute. This could be
measured in any increment of time. Speedup is defined as the time it takes a program
to execute in serial (with one processor) divided by the time it takes to execute in
parallel (with many processors).
Performance( A seq B ) ??? Performance( A par B ) No one really knows

8. Amdahl's law

9. Applying Ahmdal's Law
Imagine the application with two distinct phases
Part A takes 70% of time, potential speedup = 2x
Part B takes 30% of time, potential speedup = 6x
Which one to invest in?

10. What Where How
What is the bottleneck
Where is it located
How to fix it

11. Top-Down Approach (classic)
• System Level
• Network
• IO
• OS
• CPU/memory
• JVM
• Heap/GC tuning
• JVM tuning
• JIT
• Application level
• Locks
• Execution threads
• API
• Algorithms
• Microarch level
• Code/data alignment
• Cache optimization
• Processor stalls
• Branch prediction

12. Iterative Approach

13. System Level (CPU)
The entry point is CPU utilization!
• Then, you have multiple things to test for
• Depending on sys% irq% iowait% idle% user%
• Need tools to examine each particular branch

14. Gateway TPS

15. System Level* (Gateway)

16. System Level* Gateway

17. System Level Gateway, Don’t do this
w/o understanding
• echo "2048 64512" > /proc/sys/net/ipv4/ip_local_port_range
• echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle
• echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
• echo "10" > /proc/sys/net/ipv4/tcp_fin_timeout
• echo "65536" > /proc/sys/net/core/somaxconn
• echo "65536" > /proc/sys/net/ipv4/tcp_max_syn_backlog
• echo "262144" > /proc/sys/net/netfilter/nf_conntrack_max

18. System Level* Gateway

19. System Level* Gateway tomcat
settings
factory.addConnectorCustomizers(new TomcatConnectorCustomizer() {
@Override
public void customize(Connector connector) {
Http11NioProtocol protocolHandler = (Http11NioProtocol)
connector.getProtocolHandler();
protocolHandler.setThreadPriority(8);
protocolHandler.setAcceptorThreadPriority(10);
protocolHandler.setPollerThreadCount(10);
protocolHandler.setPollerThreadCount(3);
protocolHandler.setProperty("socket.performanceBandwidth","2");
protocolHandler.setProperty("socket.performanceLatency","1");
protocolHandler.setProperty("socket.unlockTimeout","0");
}
});
https://tomcat.apache.org/tomcat-8.0-doc/config/http.html

20. System Level* Gateway tomcat
settings

21. System Level (CPU, network)
• One of the major contributors to sys%
• In many cases, hardware/OS configuration is enough
• In other cases, application changes might be necessary

22. System Level (CPU, scheduling)
• The symptom of the unbalanced threading
• Lots of voluntary context switches (thread thrashing)
• Lots of involuntary context switches (over-saturation)

23. System Level (CPU, swapping)
• Swapping is the killer for Java performance
• The target is to avoid swapping at all costs
• Swapping out other processes to save the memory is good

24. System Level (CPU, swapping)
• Swapping is the killer for Java performance
• The target is to avoid swapping at all costs
• Swapping out other processes to save the memory is good

25. System Level (irq%, soft%)
• Swapping is the killer for Java performance
• The target is to avoid swapping at all costs
• Swapping out other processes to save the memory is good

26. System Level (irq%, soft%)
• Swapping is the killer for Java performance
• The target is to avoid swapping at all costs
• Swapping out other processes to save the memory is good

27. System Level (irq%, soft%)
• Usual thing when interacting with the devices
• Sometimes IRQ balancing is required
• Sometimes IRQ balancing is expensive

28. System Level (iowait%)
• Expected contributor with disk I/O
• Watch for disk activity
• Watch for disk throughput
• Watch for disk IOPS

29. JVM Level

30. JVM Level (GC)
• Most usual contender in JVM layer
• Lots of things to try fixing (not covered here, see elsewhere)

31. JVM Level (GC)
• Most usual contender in JVM layer
• Lots of things to try fixing (not covered here, see elsewhere)

32. JVM Level (Classload)
• Classload
• Important for startup metrics; not really relevant for others
• Removing the loading obstacles is the road to aw