This document discusses performance forensics and optimization techniques. It emphasizes the importance of collecting multi-layered measurements from the user level down to the system level to understand performance problems. Common measurements include response time, memory usage, CPU usage, database queries and latency. Identifying the problem area and isolating it is key before applying optimizations like caching, reducing interactions and data locality. Tuning may be needed at the application, web or database layers. The goal is to make problems reproducible and ensure optimizations address the underlying issues rather than just symptoms.

1. Performance ForensicsAlois ReitbauerdynaTrace Software

2. Forensicsforensisadj - "of or before the forum." In Roman times, a criminal charge meant presenting the case before a group of public individuals in the forum. … The individual with the best argument and delivery would determine the outcome of the case.

3. Collecting Evidence

4. What does this chart tell usResponse TimeTime

5. A- Response time problem?B - CPU problem?C - Sync problem?D - Database problem?

6. We don‘t know

7. We have to collect our evidence first

8. Multi-Layered MeasurementUser-LevelApplication-Level Container/App Server-Level System/OS/Virtualization-Level

9. UnderstandyourmeasurementsResponse Time and GCResponse Time only

10. How can this happen?A: Our response time is 2.3 secondsB: Our response time is 1.5 secondsC: Our response time is 6 seconds

11. The beauty of measures ... Page Load Time HTTP Request Time95 % Servlet TimeNetworkSnifferBrowserWeb ServerFirewallApplicationServerRequest Time (max)... is that there are so many to choose from

12. Types of MeasurementsCyclic MeasurementsAre collected ar regular time intervalsAre time basedJMX, CPU, MemoryEvent-based measurementsAre collected as a request occursAre transactionalResponse Times, CPU consumption

13. Types of StatisticsMin/MaxAverageMedianPercentilesUse percentiles for event-based measures and averages (or max) for cyclical measures

14. How you measure is important

15. Typical Measurements we work withMemoryConsumption, GCCPUUsage, Load AverageResponse TimeTransactionsDatabaseStatements, Pool Sizes CommunicationCalls, Latency, Size, Threads

16. A Clearer Picture

17. What is the problem?We have response times of 6 seconds.We have response times of 6 seconds for 95 percent of our users at a load of 500 users with a CPU utilization of 10 percent.

18. Indentify the suspect

19. Don‘t trust your assumptions

20. Top 10 Optimizations are bad

21. Seperate transaction types

22. Baseline and Delta

23. Understand the difference

24. Make the problem reproducable. Otherwise you cannot check whether you fixed it.

25. When are we done?

26. ... when there are no more why questions

27. Solving the case

28. Ensure you solved the real problem

29. Fight problems not symptoms

30. When a measure supports a problem. Check for all measures affected by the problem.

31. Have you tuned at the right place

32. Watch out for side effects

33. The usual suspects

34. Memory and GCState HandlingData VolumeWeb TierDatabaseNumber of RequestsJavaScriptLatencyData VolumeRenderingComm. BehaviorBusiness TierO/R AccessMemory and GCBrowser

35. Remoting StackApplication Developers ViewClient ApplicationServer ApplicationStubFacadeSerializationDeserialisationClient InfrastructureServer InfrastructureNetworkingNetworking

37. Create Data LocalityReduce InteractionsAvoid Protocol OverheadAdjust Interfaces

38. The DB layerApplicationApplication CodeO/RMapping LayerCaching LayerSQLTCP/IPResult SetStatementConnectionConnectionDatabaseConnectionPoolPreparedStatement...PreparedStatement

39. Caching in the DB layerCross SessionCacheSession CachesQueryCache(s)Execution PlanCachePrepared StatementCacheDatabasePersistenceFrameworkJDBCLayer

40. Tune LoadingBehaviorOptimizeforcachingselect … from a,b,cselect … from b,c,a… join fetch ….Reduce DB CallsDefine Proper Entities

41. The Web Layer

42. HighDatavolumeThread-/ConnectionpoolsHTML RenderingHighLatencyDatabaseAccessNetworkWebService/Backend CallsMany AJAX/HTTPcallsServerBrowserBrowserBrowserJavaScriptPerformance

43. Caching on the WebServerside Data CacheServer providingCaching InformationCache per ClientServerProxy Cachefor Many ClientsClients

44. Alois Reitbaueralois.reitbauer@dynatrace.com@AloisReitbauerblog.dynatrace.com