Andreas Grabner maintains that most performance and scalability problems don’t need a large or long running performance test or the expertise of a performance engineering guru. Don’t let anybody tell you that performance is too hard to practice because it actually is not. You can take the initiative and find these often serious defects. Andreas analyzed and spotted the performance and scalability issues in more than 200 applications last year. He shares his performance testing approaches and explores the top problem patterns that you can learn to spot in your apps. By looking at key metrics found in log files and performance monitoring data, you will learn to identify most problems with a single functional test and a simple five-user load test. The problem patterns Andreas explains are applicable to any type of technology and platform. Try out your new skills in your current testing project and take the first step toward becoming a performance diagnostic hero.

1.
T21
Performance
Testing
10/6/16
15:00
Become
a
Performance
Diagnostics
Hero
Presented
by:
Andreas
Grabner
Dynatrace
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Andreas
Grabner
Performance
enthusiast
Andreas
Grabner
has
been
a
developer,
tester,
architect,
and
product
evangelist
for
the
past
fifteen
years
for
several
testing
and
diagnostics
companies
including
Segue,
Borland,
Compuware,
and
Dynatrace.
Andreas
now
helps
organizations
find
performance,
scalability,
and
architectural
problems
in
their
applications.
Speaking
at
meetups,
user
groups,
and
international
conferences,
he
shares
his
knowledge
and
teaches
others
how
to
recognize
and
avoid
the
formerly-­‐mentioned
problems.

3. Become a Performance
Diagnostics Hero
Without running large scale load tests
Andreas Grabner: @grabnerandi, andreas.grabner@dynatrace.com
http://www.slideshare.net/grabnerandi

4. Why
Performance?
Confidential, Dynatrace, LLC

5. DevOps @ Target
presented at Velocity, DOES and more …
http://apmblog.dynatrace.com/2016/07/07/measure-frequent-successful-software-releases/
“We increased from monthly to 80
deployments per week
… only 10 incidents per month …
… over 96% successful! ….”

6. „We increased from monthly to
250+ deployments per week“
dev.otto.de

7. Not only fast delivered but also delivering fast!
-1000ms +2%
Response Time Conversions
-1000ms +10%
+100ms -1%

8. Performance == Respone Time | UX + Resource Usage
CPU & Mem Impact of updated Dependency Injection Lib

9. Performance --> User Behavior

10. How to
analyze perf?
Confidential, Dynatrace, LLC

11. Time: Wall Clock, CPU, I/O, Wait/Sync, Susp, Page Load
Throughput: # of Requests per Timeinterval
Resources: CPU Cycles, Memory, I/O, Log Messages, ...
Pools and Queues: Sizes, Utilization, Acquisition Time,
# Publishers vs # Subscribers, Process Time
Interactions: # SQLs, # Messages, # Services, # Images, # CSS
Errors: Exceptions, HTTPs, TCP Packet Loss

12. AND MANY MORE

13. 0.02ms
0.01ms

15. your tool of choice
Functional Result + # SQLs Executed by Application
Fail the build early!

16. “We Deliver High
Quality Working Software Faster“
„Shift-Left Quality to Optimize Pipelines“
https://github.com/capitalone/Hygieia & https://www.spreaker.com/user/pureperformance

17. Where do your
Stories come
from?

18. Export & Share
Share Your PurePath - http://bit.ly/sharepurepath

20. 20%
80%

21. How To: Metrics-Based Problem Pattern Detection
http://apmblog.dynatrace.com/2016/06/23/automatic-problem-detection-with-dynatrace/

23. Frontend Performance
We are getting FATer!

24. m.pepsi.com during Super Bowl
434 Resources in total on that page:
230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~
20MB

25. Tip: Make F12 your friend!

26. Tip: Monitor your Real End Users
#1: Which Geo has which
“User Experience”?
#2: Who are
these users?

27. Key Metrics
# and Size of Resources
Total Size of Content
# of HTTP 3xx, 4xx, 5xx
# of Domains
W3C Resource & Nav Timings
JavaScript Errors

30. Backend Performance
“The Usual Suspects”

31. cite the database as the most
common challenge or issue
with application performance
88%

32. Confidential, Dynatrace, LLC
Inefficient Loading of Too Much Data!
SQL Heavy: 2111 SQL
Calls, 6.97s Exec TimeADO.NET: Most of this time is
reading and processing SQL Results
Garbage Collection:
impacting response time
Resource Impact: CPU
& Memory due to
Execution Plan
Generation

33. Confidential, Dynatrace, LLC
N+1 Query Problem Pattern
N+1 Query: Same SQL
called 359! times
N+1 Query: Even with the same
Bind Values called 26! times

34. Confidential, Dynatrace, LLC
The Impact on SQL Servers Execution Plan Cache
CPU intensive operation:
A new execution plan is generated for
each unique SQL statement
The plan cache is used to save all the
execution plans in case they can be reused

35. Confidential, Dynatrace, LLC
The Impact of too many unique SQLs

36. Confidential, Dynatrace, LLC
SQL Server Perf Metrics: Requests vs Compilations
Ratio: More than 50% of the queries
(Batch Requests/s) requires the generation
of a new execution plan (Compilation/s)
High CPU caused by
execution plan generation
High Disk I/O Operations
because the plan cache is
using memory “stolen”
from the buffer cache

37. Confidential, Dynatrace, LLC
Tip: Ad Hoc vs Parameterized Statements

38. Resource Improvements with Parameterized Queries
Throughput improved by half! High
number of queries processed but
low number of compilations
Disk I/O reduced by more
than half as there is more
memory available to cache
the data in memory
Reduced CPU usage

39. Key App Metrics
# of SQL Calls per Request
# of same SQL Execs (1+N)
Rows/Data Transferred
Ratio Batch Requests to
Compilations

40. Pools & Queues
„Proper Sizing“
„Proper Usage“

41. #1: Pick Proper Pool Sizes
Do we have enough DB
CONNECTIONS per pool?

42. #2: Monitoring Pool Usage and Impact
How long to wait for a new
connection?
How long connections are
in use?

43. #3: Correct Load Balancing

44. #4: Excessive Thread Usage per Request
Excessive Remoting:
40! RMI Calls
on 20! parallel threads
Excessive SQL: 2790!
SQL Calls Total
Worker Thread Exhaustion:
Excessive use of threads in App Server also
causes backup of requests in Web Server.
Watch your Worker Thread Pools!

45. Tip: Follow Threads through PurePath
Thread Name:
Count Unique Thread IDs
Async/Sync Nodes:
RMI calls done on
background threads

46. Tip: Watch out for Sync / Wait
1.63s in Object.wait
Means that this thread is put to hold
Waiting on the next
Connection to become
available!

47. Key Metrics
Pool and Queue Sizes
Time in Sync & Wait
# of Threads per Request
Balanced Requests

48. When Logging
„Impacts Performance“
LOG

49. #1: Log Hotspots in Frameworks!
callAppenders clear CPU and I/O Hotspot
Excessive logging through Spring Framework

50. #2: Debug Log and outdated log4j library
#1: Top Problem: log4j.callAppenders
-> 71% Sync Time
#2: Most of logging done from
fillDetail method
#3: Doing “DEBUG” log
output: Is this necessary?

51. #3: Overhead caused by Exceptions
fillInStackTrace is Top 2 in CPU Hotspots
All these Exceptions that never show up in
a log file are consuming all CPU

52. Identify “hidden” Technical Debt
2-5 Log Messages per 5 Min
Looking at the important
(SEVERE, FATAL, …) log messages
written
Up to 20000 Custom Exceptions
That’s about 4000x the number
of Exceptions per Log Message

53. Key Metrics
# of Log Entries
Size of Logs per Use Case
Ratio Exceptions to Logs

54. „(Micro-)Service
Migration Mistakes“

55. Online Sports Club Search Service
2015201420xx
Response Time
2016+
1) Started as a
small project
2) Slowly growing
user base
3) Expanding to
new markets –
1st performance
degradation!
4) Adding more markets
– performance becomes
a business impact Users
4) Potentially start
loosing users

56. Can‘t scale vertically endlessly!
2.68s Load Time
94.09% CPU
Bound
Early 2015: Monolithic App

57. Front End
to Cloud
Scale Backend
in Containers!
Proposal: Service approach!

58. 7:00 a.m.
Low Load and Service running
on minimum redundancy
12:00 p.m.
Scaled up service during peak load
with failover of problematic node
7:00 p.m.
Scaled down again to lower load
and move to different geo location
Testing the Backend Service alone scales well …

59. GO LIVE DAY – 7:00 a.m.

60. GO LIVE DAY – 12:00 p.m.

61. What Went Wrong?

62. 26.7s Load Time
5kB Payload
33! Service Calls
99kB - 3kB for each call!
171!Total SQL Count
Architecture Violation
Direct access to DB from frontend service
Single search query end-to-end

63. 2.5s (vs 26.7)
5kB Payload
1! (vs 33!) Service Call
3kB (vs 99) Payload!
3!(vs 177) Total
SQL Count
The fixed end-to-end use case
“Re-architect” vs. “Migrate” to Service-Orientation

64. Key Metrics
# of Service Calls
Payload of Service Calls
# of Involved Threads
1+N Service Call Pattern!

66. You measure it! from Dev (to) Ops

67. Build 17 testNewsAlert OK
testSearch OK
Build # Use Case Stat # API Calls # SQL Payload CPU
1 5 2kb 70ms
1 40 5kb 120ms
Use Case Tests and Monitors Service & App Metrics
Build 26 testNewsAlert OK
testSearch OK
Build 25 testNewsAlert OK
testSearch OK
1 4 1kb 60ms
34 171 104kb 550ms
Ops
#ServInst Usage RT
1 0.5% 7.2s
1 63% 5.2s
1 4 1kb 60ms
2 3 8kb 100ms
1 0.6% 4.2s
5 75% 2.5s
Build 35 testNewsAlert -
testSearch OK
- - - -
2 3 10kb 150ms
- - -
8 80% 2.0s
Metrics from and for Dev(to)Ops
Re-architecture into „Services“ + Performance Fixes
Scenario: Monolithic App with 2 Key Features

68. your tool of choice
Functional Result + # SQLs Executed by Application
Fail the build early!
Performance Hero

69. Questions
Slides: slideshare.net/grabnerandi
Get Tools: bit.ly/dtpersonal
YouTube Tutorials: bit.ly/dttutorials
Contact Me: agrabner@dynatrace.com
Follow Me: @grabnerandi
Read More: blog.dynatrace.com

70. Andreas Grabner
Dynatrace Developer Advocate
@grabnerandi
http://blog.dynatrace.com