Performance Testing in Web and Batch Application, Trips and Tricks, Use cases, Tools usage

1. Performance Engineering in
Batch and Web Application
Tips and Tricks
Prakash Sahu

2. Why Performance Testing
• Identifies problems early, before they become costly to resolve
• Reduces development cycles
• Produces better quality, more scalable code
• Prevents revenue and credibility loss due to poor performance
• Enables better planning for future expansion
• To ensure that the system meets performance expectations such as response
time and throughput under given levels of load
• Expose bugs that do not surface in functional testing phase, such as memory
management bugs, memory leaks, buffer overflows, distribution of system
resources utilization

3. Factors that governs Performance testing:
 Response Time
 Throughput
 Tuning
 Benchmarking

4. Throughput
• Capability of a application to handle multiple transactions in a give period
• Throughput represents the number of business transactions processed by
the application in a specified time duration
• The throughput should increase almost linearly with the number of
requests/number of concurrent users. Which can only happen when, there
is very little or no congestion within the Application queues

5. Response Time
10
20
30
40
100 200 300 400
Time(ms)
Concurent Users
Response Time vs Users
• It is important to find the time
duration of the complete
transactions
• Response time is a measure of
delay between the point of request
and the first response from the
application
• The response time increases
proportionally to the user load.
• More resource (CPU, Memory
utilization)

6. Tuning
• Tuning is the process by which application
performance is enhanced by setting
different values to the parameters of the
• Application for testing
• Operating system,
• Database,
• Network and other Components.
• Tuning improves the Application
performance without making any change to
the source code of the application
• The tunning lifecycle is shown pictorially
Test And Measure
CollectData
Analyze Result
Configure
Parameter
Code Baseline

7. Performance Testing Process
Test Planning
Creating vUser Scripts
Creating Scenario
Executing Scenario
Monitoring Scenario
Analyzing Test Results
Creating Scenario
Executing Scenario
Monitoring Scenario
Analyzing Test Results
Test Planning
Web Application Batch /command line Application

8. • Determine the performance test objectives
• N number of TPS from Web App
• N number of throughput for a period in Batch application
• Describe the Hardware environment
• Create a Benchmark to be recorded in subsequent
Phases
A. Define user tasks to be performed
B. Define percentage of users per task.
Test Planning

9. Virtual Users (VUs): Test Goals
Start with: 10 user Max Response Time <= 20 Sec
Incremented by: 10 user
Maximum: 200 user
Think Time: 5 sec
Test Script of typical WEB APP:
One typical user from login through completion
Test Script of typical BATCH JOB:
Execute the batch JOB
EXECUTE

10. • Monitoring the scenario: Monitor scenario execution using the various
runtime tools, NMON, Dynatrace, AppDynamics, Wily Introscope
• Analysing test results: During scenario execution, the tool records the
performance of the application under different loads and use-cases
• Use, graphs, charts and reports to analyse the application’s performance to
take corrective actions
MONITOR
ANALYZE

11. Schematic of WEB APP
Micro-service-1
Load testing in Web applications And micro-services
Optimize Code and configuration in one instance of MS, hence only one
connection is being shown in the picture
Micro-service-N
Micro-service-2
Micro-service-3 Database
Load
Balancer
Load Controller
Collect transaction response time
Collect system metrics (CPU, Memory, Threads)
Generate analysis reports and improvement suggestion
Load Generator
Simulate user activity
Simulate many user on each
generator
Agent

12. Schematic of Batch APP
Load testing in Batch applications
NMON  System metric like CPU, Memory, Disk Usage for VM hosting APP and DB
Agent Base  Code and Database drill down, Slow code and DB query identification
Batch
Application Database
Load Controller [Introscope, AppDynamics,
DynaTrace]
Collect transaction response time
Collect system metrics (CPU, Memory, Threads)
Generate analysis reports and improvement suggestion
Agent
Test File
System Metric
Collector
NMON
VM running Batch App VM hosting Database
System Metric
Collector
NMON

13. Application scalability
 Golden Rule
 Do One change at a time till the change is neutral to change in performance
 Hardware
 Memory
 Monitor JVM memory  Adjust till, change in Memory has no affect, Keep in mind Garbage
Collection Pause time with larger heap size
 CPU
 Monitor and adjust CPU value till it has no affect, a good system should give linear increase in
performance with increase in CPU value, speed and number of core
 Disk
 Monitor the disk I/O
 Read and Write operation per second in terms of Byte/KB/MB
 Implement caching to avoid Read/Write to disk if permitted by design
 Software/Code
 Analyze log to find any duplicate query, some of which could be sometime so heavy that system can
slow down by 10-20 times
 Tune JPA/Hibernate and database parameters
 Monitor the affect of multi-threading to set optimum number of concurrent threads in the application
 Database
 Change default DB parameters to appropriate value
 PostgreSQL  effective_cache_size & shared_buffers
 Increase in cache can reduce SWAP resulting in enhance CPU utilization

14. Application scalability
 Multi Threading
• Too many are not always good
• Non are not suitable
 Multi-Threading Applicability scenario
• Multiple reads of non changing data are ok
• No Write by multiple threads to same resource (file system)
 Independent Instances
• No session in web application
• Small functions  small stack, better memory management
• No synchronization
• No usage of common system
• DB sequence for unique identifiers

15. Batch Insert without Explicit Flush
 Batch inserts, updates property in hibernate
 <property name ="hibernate.jdc.batch_size" value="20"/>
 More than 20-30 does not give better result
 0-20 improves time by 300%
 Must be coupled with the ordered parameters ,Cascading and batching, the batching
is propagated to child
 <property name="hibernate.order_inserts" value="true"/>
 <property name="hibernate.order_updates" value="true"/>
 Multiple insert/update/delete statement are combined together to do in batch
 If our entities use GenerationType.IDENTITY identifier generator, Hibernate will silently
disable batch inserts/updates.
 Use Pagination to avoids OOM on client side and improved performance

16. Using
EntityManager.flush() and
EntityManager.clear() in application code, whenever batch size is reached, removes
OutOfMemory errors and enhances performance, since the persistence context are
regulary cleared
@Transactional
@Test
public void flushingAfterBatch() {
for (int i = 0; i < 1000; i++) {
if (i > 0 && i % BATCH_SIZE == 0) {
entityManager.flush();
entityManager.clear();
}
MyObj obj = createMyObj(i);
entityManager.persist(obj);
}
}
Batch Insert with Explicit Flush

17. Spring Batch-Partition to parallelize the batch without code
Offline Tables
Insert
Online Tables
Rename Offline
to Online Table
No impact to online tables, parallel processing
Step-2- split to
smaller files
Step-3-Insert
in Parallel
Offline Tables
Daily Refresh
Truncate
Step-1-
Truncate
Step-4-Reanme
Spring-Batch Partition for Parallel Processing

18. Spring Batch-Partition Configuration for Parallel execution

19. PostgreSQL Slow Query Report
postgresql.conf:
shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.max = 10000  Maximum number of statements tracked by the module, default=5K
pg_stat_statements.track = all  top, all, none, default =top
Query to check above is applied
SELECT * FROM pg_available_extensions
WHERE
name = 'pg_stat_statements' and
installed_version is not null;
• top  those issued directly by clients
• all  track nested statements (such as
statements invoked within functions)
• none  disable statement statistics
collection

20. SELECT queryid,query AS short_query, round(total_time::numeric, 2) AS total_time, calls, rows,
round(total_time::numeric / calls, 2) AS avg_time, round((100 * total_time / sum(total_time::numeric)
OVER ())::numeric, 2) AS percentage_cpu FROM pg_stat_statements ORDER BY total_time DESC LIMIT
20;
https://gist.github.com/anvk/475c22cbca1edc5ce94546c871460fdd

21. Use non-commercial tool During Development
Stage To figure out performance bottlenecks
Using HPROF
HPROF is a tool to profile Heap and CPU, which is shipped along with Java.
It can be used during development.
The JVM creates a huge file with the name java.hprof.txt after it shuts down.
This contains information about heap profiles, allocated memories to instances, dynamic stack traces, etc.
java agentlib:hprof=heap=sites Hello.java
Other JDK Tools
• JVisualVM
• JConsole
• Java mission control JMC
• JHAT  heap dump analysis
• JSTACK
• Eclipse Memory Analyzer (MAT)
• Heap dump analysis

22. Heap Dump Creation and Analysis
Create Heap Dump
1. Using JMAP (JDK tool)
1. jmap -dump:live,file=/tmp/heapDump.bin PID
1. live objects in the heap are dumped
2. Using JVM Option
1. -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/heapdump
Analyze Heap Dump with Eclipse Memory Analyzer
1. More convenient and easy
Analyze Heap Dump JHAT (JDK tool)
1. jhat -J-Xmx2g -port 7001 /tmp/heapDump.bin
2. WEB ACCESS TO JHAT
1. http://localhost:7001

23. Thread Dump Creation and Analysis
Create Thread Dump
1. Using JSTACK (JDK tool)
1. jstack -l <PID> > <FILE-PATH>
2. Using KILL
1. kill -3 <PID>
3. Take in sample of say few minute (e.g. 5,10,15 ) interval
to check whether the threads present in first thread dump are still present in subsequent dumps
Analyze Thread Dump
1. These are text file which could be manually analyzed
2. Introscope

24. PostgreSQLhttps://www.revsys.com/writings/postgresql-performance.html
shared_buffers  Editing this option is the simplest way to improve the performance of your database server.
The default is pretty low for most modern hardware. Thumb rule  roughly 25% of available RAM on the system
Most people find that setting it larger than a third starts to degrade performance.
effective_cache_size  This value tells PostgreSQL's optimizer how much memory PostgreSQL has available for
caching data and helps in determine whether or not it use an index or not. The larger the value increases the likely hood
of using an index. This should be set to the amount of memory allocated to shared_buffers plus the amount of OS
cache available. Often this is more than 50% of the total system memory.
work_mem  This option is used to control the amount of memory using in sort operations and hash tables.
While you may need to increase the amount of memory if you do a ton of sorting in your application, care needs to be taken.
This isn't a system wide parameter, but a per operation one. So if a complex query has several sort operations in
it it will use multiple work_mem units of memory. Not to mention that multiple backends could be doing this at once.
This query can often lead your database server to swap if the value is too large.

25. Hardware
RAM — The more RAM you have the more disk cache you will have.
This greatly impacts performance considering memory I/O is thousands of times faster than disk I/O.
Disk types — Obviously fast Ultra-320 SCSI ( Solid State) disks are your best option, however high end SATA drives are also
very good. With SATA each disk is substantially cheaper and with that you can afford more spindles than with SCSI on
the same budget.
Disk configuration — The optimum configuration is RAID 1+0 with as many disks as possible.
Separate disk for pg_xlog <transaction log , can not be deleted> ,
and pg_log < error messages, executed query log, dead lock information, can be deleted>
CPUs — The more CPUs the better, however if your database does not use many complex functions your money is best
spent on more RAM or a better disk subsystem

26. Leak Detection and Connection Pool in Hikari
maximumPoolSize=10
leakDetectionThreshold=60000
hikari.properties
Explain check to Check Index scan strategy
Index the join keys in child tables
leakDetectionThreshold 
• This property controls the amount of time that a connection can be out of the pool before a
message is logged indicating a possible connection leak.
• The default value of 0 means leak detection is disabled.
• Lowest acceptable value for enabling leak detection is 2000 (2 secs).
• E.g. A slow DB server can delay the query response time more than this value, and report leak.
• E.g. The query itself is slower than the threshold value
• Useful for transactional system
• Be careful, a reporting application may take much longer to load data

27. Check blocking Query
PostgreSQL 9.6
SELECT * FROM pg_stat_activity WHERE waiting = TRUE;
PostgreSQL 10
SELECT * FROM pg_stat_activity WHERE wait_event IS NOT NULL AND
backend_type = 'client backend';

28. @Entity
@Table(name = “table_parent")
public class ParentEntity {
@OneToMany(mappedBy = "ParentEntity, fetch = FetchType.EAGER,)
private Set<ChildEntity> ChildEntity;
}
@Entity
@Table(name = “table_child")
public class ChildEntity {
}
Common N+1 issue in Eager load situation and solution
Parent Child data required to be loaded in single Query by Eager Loading, In case of Lazy loading the Fetch Join not required

29. Common N+1 issue in Eager load situation and solution
Results into Single Query
CriteriaBuilder builder = entityManager.getCriteriaBuilder();
CriteriaQuery<ParentEntity> criteria = builder.createQuery(ParentEntity.class);
Root<ParentEntity> root = criteria.from(ParentEntity.class);
// changes multiple query  Single Query
Fetch<ParentEntity, ChildEntity> jn = root.fetch("ChildEntity", JoinType.INNER);
return entityManager.createQuery(criteria).getResultList();
Select *
from table_parent as a inner join table_child as b on a.parent_id = b.child_id where a.parent_id=‘xyz;
Results into Multiple Queries:
CriteriaBuilder builder = entityManager.getCriteriaBuilder();
CriteriaQuery<ParentEntity> criteria = builder.createQuery(ParentEntity.class);
Root<ParentEntity> root = criteria.from(ParentEntity.class);
return entityManager.createQuery(criteria).getResultList();

30. Wily Introscope :
Gives below metrics for Performance Monitoring and Analysis
 CPU Utilization
 Java Thread Monitoring
 Java Memory Utilization
 OS memory Utilization
 File System Usage
 SQL Query Monitoring
 Java Code Profiling

31. Wily Introscope: Code Profiling Configuration
• SetFlag: ServiceTracing
•
• IdentifyMatchingClassesAs: "com.application.services.internal.*ServiceImpl" ServiceTracing
• TraceComplexMethodsIfFlagged: ServiceTracing BlamePointTracer "Service|{classname}|{method}"
•
• TraceComplexMethodsIfFlagged: ServiceTracing BlamePointTracer "Service|{classname}"
•
• TraceComplexMethodsIfFlagged: ServiceTracing BlamePointTracer "Service"
•
• TurnOn: ServiceTracing

32. Nmon – Agentless, system data collector
Postmortem Graphing and Analysis
• nmon -s <second> -c <count> -f <filename>
• f Save to File mode
• <filename>  name of file saved to CSV format
• <second>  Time between data capture in seconds
• <count>  Number of Capture

33. nmon  Community version
Performance Monitoring Tool

34. nmon

35. Suddenly the world has come to a crawl
1. Watch out for the disk space in app server which may often run out during
high volume load test due to large application logs, big request, and
response files in application server
2. Watch out for the disk space in database server which may run out due to
large transaction queries, temp file generation, DB log generation.

36. Infrastructure scalability
• Multiple instances of application in legacy system
• Container based independent deployment

37. RAID 1+0

38. Important CPU states
system (sy)
The low-level kernel tasks, like interacting with the hardware, memory allocation, communicating between OS processes,
running device drivers and managing the file system, CPU scheduler.
user (us)
One level up, the “user” CPU state shows CPU time used by user space processes. Like your application, or the database
server running on your machine.
idle (id)
The “idle” CPU state shows the CPU time that’s not actively being used. Internally, idle time is usually calculated by a task
with the lowest possible priority.
iowait (wa)
“iowait” is a sub category of the “idle” state. It marks time spent waiting for input or output operations, like reading or
writing to disk. When the processor waits for a file to be opened, for example, the time spend will be marked as “iowait”.
For example, when an in-memory database needs to flush a lot of data to disk, or when memory is swapped to disk.
Other statistics
The “hardware interrupt” (hi or irq) and “software interrupt” (si, or softirq) categories are time spent servicing interrupts,
and the “steal” (st) subcategory marks time spent waiting for a virtual CPU in a virtual machine.

39. Introscope agent:
https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/wily_java_agent_performance_tuning_recommendations?lang=en
Chose of APM tool
http://karunsubramanian.com/apm/7-critical-things-you-should-look-for-in-an-apm-solution/
Coonfiguration of introscope
https://docplayer.net/11351610-Ca-wily-introscope-webview.html
Over head 2-3 % of CPU/Memory
References

40. THANK YOU