Serghei Radov Principal Performance Engineer Oracle UGBU

1. Workload Analysis

2. Serghei Radov
Current position:
Principal Performance Engineer
Oracle UGBU
Contacts :
sergey.radov@gmail.com
Github: github.com/sergeyradov
Telegram: @sradov
Skype : serghei.radov

3. 1. Amazon: 100-millisecond increase in page speed translated to a 1%
increase in its revenue.
1. Shopzilla improved their revenues by 5% to 12% after drop their page
load times from 6 to 9 seconds down to 1.2 seconds.
1. The Aberdeen Group in 2015 showing that performance issues impact
revenues by up to 9%. An average loss of $117 million per year.
1. Strangeloop (bought by Radware in 2013) showed that a 2-second delay
in load time during checkout could result in abandonment rates reaching
up to 87%.
Why it matters?

4. Why it matters?

5. ● Define acceptance criteria
● Select tools for monitoring and testing
● Workload Characterization
● PitFalls in Analysis
● Report to stakeholders

6. Define performance tests SLA
● Throughput
● Expected Response times
● User count
● Thread limitation
● Resource Usage
● Time-out and e.t.c

7. Issues as SLA
● Failure
● Network issues
● Denial of service
● Scheduled maintenance

8. NRQL - NewRelic query language
SELECT uniqueCount(session) FROM PageView SINCE 1 week ago
SELECT uniqueCount(session) FROM PageView SINCE 1 week ago
COMPARE WITH 1 week ago
SELECT count(*) FROM PageView SINCE 1 day ago COMPARE
WITH 1 day ago TIMESERIES AUTO
SELECT uniqueCount(uuid) FROM MobileSession FACET osVersion
SINCE 7 days ago

9. Gathering response times

10. Transactions throughput

11. Finding peak load
● Target PEAK load will be 1.14K RPM
● Lowest point will be 430 RPM

12. ● Define acceptance criteria
● Select tools for monitoring and testing
● Workload Characterization
● PitFalls in Analysis
● Report to stakeholders

13. Monitoring targets
● Response times
● Resource utilisation at SUT
● Resource utilisation at Test Tool
● Exceptions
● Workload behaviour

14. CPU usage per 1 server (DataDog)

15. Grafana

16. Resource usage

17. to catch exceptions

18. ● Define acceptance criteria
● Select tools for monitoring and testing
● Workload Characterization
● PitFalls in Analysis
● Report to stakeholders

19. Workload Characterization
● Catch traffic patterns
● Resource utilisation
● Distribution of response times
● Distribution of response sizes
● Characterizations of users
behaviour
● Analyse input data
● Use performance analysis
toolkit

20. Traffic patterns
“Keep workload as real as possible.”

21. Transactions throughput

22. - Ramp up to 430
RPM slowly to 700
RPM in 4 hours
- Run test for 6 hours
- Ramp up to 1.14K
rpm
- Run test for 11
hours
- Ramp down slowly
Scenario per one server

23. Characterize user behaviour
Investigate user actions by help of
- New Relic Browser (session+funnel functions)
- Universal Analytics with User behaviour path
- Mixpanel.com (needs code injection)
- Server’s logs at NGINX
- (http requests, REST calls)
- Sumo-logic (apache access logs)
- Server’s App logs (HP ALM has QC sense)
- DB activity logs (applied solution)

24. ● Define acceptance criteria
● Select tools for monitoring and testing
● Workload Characterization
● PitFalls in Analysis
● Report to stakeholders

25. Pitfalls during performance testing
Pitfall 1 : 90% percentile matches to prod.
Pitfall 2 : Extrapolation on horizontal scale
Pitfall 3 : Use a Small Amount of Hard Coded Data
Pitfall 5 : Run Tests from One Location
Pitfall 4 : Focus on a Single Use Case

26. Does the "90 percentile" really work ?

27. Does the "90 percentile" really work ?

28. Does the "90 percentile" really work ?

29. ● Define acceptance criteria
● Select tools for monitoring and testing
● Workload Characterization
● PitFalls in Analysis
● Report to stakeholders

30. Reports
● Goals & achievements (e.g 150% of Daily RPM is reached)
● Side effects are found (DB connections limit reached due to quick ramp
up)
● Exceptions caught during testing (e.g. ELB lost connections)
● Run-time notes and fixes made by DevOps (EC2 change during the test
iterations)
● Observations ( CPU usage was critical resource during RPM increase)
● Recommendations ( EC2 - add more VM, add more Shards DB)

31. Thank you!

32. Questions and Answers

33. Books

34. Books