Partners:
Hints on complex system performance
optimization
Ainars Galvans
Exigen Services Latvia

December 2, 2013

www.ExigenServic...
System Performance
Introduction: context, terms and basics

3

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenServices.com
Terminology

4

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenServices.com
Performance is:

Requirement
Number of users
Hardware spec.
“Response time”

Measurement
Throughput
CPU utilization
Respon...
A simple 3-tier Web Application

Client

App.

DB

Prepare request
Response time

User Experience

Parse, process
Select o...
A simple 3-tier Web Application

App.
Client1

7

DB

Client2

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenServices.com
With 3rd party WebServices

Client1

App.

DB

Client2

8

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenServices.com
Other Apps share the same DB?

App.

9

DB

CONFIDENTIAL – NOT FOR DISTRIBUTION

Other
(Corp.)
App.

www.ExigenServices.co...
Performance is:

Requirement
Number of users
Hardware spec.
“Response time”

10

Measurement
Throughput
CPU utilization
Re...
Rules of thumb

 Response time
– >2 seconds: is it all CPU time?
– 0.5-2.0 seconds: why?
– <0.5 seconds : OK

 Concurren...
My issue localization techniques
 First step: where (what host) is the bottleneck?
– Single user repeat the same action (...
System Performance
Performance optimization stories

13

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenServices.com
Locating database call issues

App.
Client1

14

DB

Client2

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenServices.com
Story #1: Just a few tables
 Problem description
– Web Page shows a number of tables
– Page response time is a bit above ...
Locating WebService call issues

Client1

App.

DB

Client2

16

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenServices.c...
Story #2: One call only takes 0.5 secs.
 Problem description
– Web Page shows user’s data overview
– Typical customer pag...
Application Logic (HTML processing)

App.
Client1

18

DB

Client2

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenService...
Story #3: Tables, again: .NET
 Problem description:
– Page shows main page and a dynamic table with hot links
– Response ...
Client Side (rare case though)

App.
Client1

20

DB

Client2

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenServices.com
Story #4: Client side
 Problem
– Visually page loads 8-12 seconds and vary
– Looks like response time depend on client wo...
… everything. The business itself

App.
Client1

22

DB

Client2

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenServices....
Story #5. The reports and queries
 Production problem
– Special feature “accounting reports” usually work 1-2 mins
– Some...
Conclusions
Localizing the problem

24

CONFIDENTIAL – NOT FOR DISTRIBUTION

www.ExigenServices.com
My issue localization techniques
 First step: where (what host) is the bottleneck?
– Single user repeat the same action (...
Happy to share more details

Skype name: ainarsgalvans
Phone: +371 29432698
Facebook: Ainars Galvans
Email: Ainars.Galvans...
Feedback

  
Partners:
Upcoming SlideShare
Loading in …5
×

Ainars Galvans - Hints on complex system performance optimization

311 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
311
On SlideShare
0
From Embeds
0
Number of Embeds
44
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • OribkiZatoplenieZatorTransport
  • Ainars Galvans - Hints on complex system performance optimization

    1. 1. Partners:
    2. 2. Hints on complex system performance optimization Ainars Galvans Exigen Services Latvia December 2, 2013 www.ExigenServices.com
    3. 3. System Performance Introduction: context, terms and basics 3 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    4. 4. Terminology 4 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    5. 5. Performance is: Requirement Number of users Hardware spec. “Response time” Measurement Throughput CPU utilization Response time i.e. time user wait for page to load in the browser 5 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    6. 6. A simple 3-tier Web Application Client App. DB Prepare request Response time User Experience Parse, process Select or update More Logic Select or update Render HTML Render UI Repeated 6 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    7. 7. A simple 3-tier Web Application App. Client1 7 DB Client2 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    8. 8. With 3rd party WebServices Client1 App. DB Client2 8 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    9. 9. Other Apps share the same DB? App. 9 DB CONFIDENTIAL – NOT FOR DISTRIBUTION Other (Corp.) App. www.ExigenServices.com
    10. 10. Performance is: Requirement Number of users Hardware spec. “Response time” 10 Measurement Throughput CPU utilization Response time CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    11. 11. Rules of thumb  Response time – >2 seconds: is it all CPU time? – 0.5-2.0 seconds: why? – <0.5 seconds : OK  Concurrent users – The term is misleading – What matters technically is throughput  Resource utilization: – DB CPU should be below 50% 11 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    12. 12. My issue localization techniques  First step: where (what host) is the bottleneck? – Single user repeat the same action (i.e. with Selenium) – Monitor “CPU time” delta (secs.), not CPU (%) – Use excel to understand how it scales  If it’s the App Server: analyze HTML – FireFox “Save Page As” -> complete page – Look for large, unused data  If it’s DB or Web Services: analyze traffic, logs – Put milliseconds to log files, see what takes most of it – Look for repeated requests for single page rendering  If it’s client (browser): profile JavaScript – IE -> Dev. Tools (F12) -> Profiler – Are all the JavaScripts cached (or compiled each time)? 12 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    13. 13. System Performance Performance optimization stories 13 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    14. 14. Locating database call issues App. Client1 14 DB Client2 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    15. 15. Story #1: Just a few tables  Problem description – Web Page shows a number of tables – Page response time is a bit above 2 seconds – Under load DB becomes overloaded quickly – All DB queries takes <0.1 sec  Investigation results: – Single page load cause about 100 select executions – Each took 0.02 secs – There were two distinct statements  Solution – After adjusting DB caching – Response times reduced to 0.4 – Almost no load to DB under load 15 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    16. 16. Locating WebService call issues Client1 App. DB Client2 16 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    17. 17. Story #2: One call only takes 0.5 secs.  Problem description – Web Page shows user’s data overview – Typical customer page response time 5 seconds – Response time up to 30 seconds for specific customers – Under load response times don’t increase too much  Investigation results: – Data in tables is requested via web services – Separate call made on each business item – Call response time is 0.5 secs  Solution – Protocol adjusted to request all info with one call – Response times reduced to 0.9 17 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    18. 18. Application Logic (HTML processing) App. Client1 18 DB Client2 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    19. 19. Story #3: Tables, again: .NET  Problem description: – Page shows main page and a dynamic table with hot links – Response times >5 secs. – App server loaded significantly – Under load response times increase  Investigation – In page source (HTML) I’ve discovered huge callbackstate  Solution – http://www.devexpress.com/Support/Center/Question/Details/Q42 5597 [Browsable(false)] – Response times < 3secs. now 19 public string UnboundLargeData { get; set; } CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    20. 20. Client Side (rare case though) App. Client1 20 DB Client2 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    21. 21. Story #4: Client side  Problem – Visually page loads 8-12 seconds and vary – Looks like response time depend on client workstation  Investigation – Server response time is only 3 seconds – It takes browser 1-2 seconds to send request – It takes browser 3-4 seconds to render response – It takes more if workstation CPU is slow (<3 GHz)  Solution – Optimize JavaScript and remove features from page – Browser choice recommendation – Unfortunately: optimal client CPU recommendation 21 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    22. 22. … everything. The business itself App. Client1 22 DB Client2 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    23. 23. Story #5. The reports and queries  Production problem – Special feature “accounting reports” usually work 1-2 mins – Sometimes 10 times as much – Suspected reason a “morning lag” (all users)  Investigation – One report loads DB CPU to 100% for 1-2 minutes – Only technical solution: rewrite whole app, changing DB structure and logic  The actual solution – Nightly batch jobs on demand – The default nightly batch jobs – Daily execution only with special privileges and only one at a time 23 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    24. 24. Conclusions Localizing the problem 24 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    25. 25. My issue localization techniques  First step: where (what host) is the bottleneck? – Single user repeat the same action (i.e. with Selenium) – Monitor “CPU time” delta (secs.), not CPU (%) – Use excel to understand how it scales  If it’s the App Server: analyze HTML – FireFox “Save Page As” -> complete page – Look for large, unused data  If it’s DB or Web Services: analyze traffic, logs – Put milliseconds to log files, see what takes most of it – Look for repeated requests for single page rendering  If it’s client (browser): profile JavaScript – IE -> Dev. Tools (F12) -> Profiler – Are all the JavaScripts cached (or compiled each time)? 25 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    26. 26. Happy to share more details Skype name: ainarsgalvans Phone: +371 29432698 Facebook: Ainars Galvans Email: Ainars.Galvans@gmail.com Blog: find me at SoftwareTestingClub.com 26 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
    27. 27. Feedback   
    28. 28. Partners:

    ×