1. A webshop was experiencing slow product pages and loggers deadlocking. Observations found high response times and warnings about increased GC usage, memory, and thread counts. 2. An initial hypothesis was formed that new functionality going live, including for products, and Oracle patches caused the product catalog to respond slowly to calls, triggering a circuit breaker that returned nulls and caused logging failures. 3. Experiments found that Oracle queries had slowed down with an update, causing extra queries to execute on the same thread and slowing the system, triggering the circuit breaker, log failures, and locks. Wrapping up, the focus should be on structuring diagnosis, formulating hypotheses, and iterating experiments.

1. I WANT TO GO FAST!I WANT TO GO FAST!
Carst Tankink (ctankink@bol.com)

2. SPEED MATTERSSPEED MATTERS
How fast is the webshop?
Requests/second: 1400
Requests/talk: 2.5 million
Average response !me: 200 ms
“Fastest e-tailer in NL & BE” – Webshop
manifesto

3. DIAGNOSE BOTTLENECKSDIAGNOSE BOTTLENECKS

6. THE EMPIRICAL CYCLETHE EMPIRICAL CYCLE
Observation
Evaluation Induction
Testing Deduction

7. THE EMPIRICAL CYCLETHE EMPIRICAL CYCLE
Observa!on
Induc!on: form hypothesis
Deduc!on: create experiment
Tes!ng: conduct experiment
Evalua!on: accept or reject hypothesis

8. OBSERVATIONOBSERVATION

9. WHAT’S THE PROBLEM?WHAT’S THE PROBLEM?
Slow (product) pages
Loggers deadlocking

11. OBSERVATIONSOBSERVATIONS
High response !mes
Some!mes, warnings:
GC usage/count up
Memory going up
Too much threads in use

12. REACT TO OBSERVATIONSREACT TO OBSERVATIONS
Triage: is there a direct problem?
Mi!gate

13. START INVESTIGATIONSTART INVESTIGATION
Communicate, and keep communica!ng
Hipchat, Jira !cket, whiteboard…
Find/create grafana dashboard

14. INDUCTIONINDUCTION

15. FORMULATE HYPOTHESISFORMULATE HYPOTHESIS
As specific as possible
Focus on diffs
Try to build a !meline
Write down, communicate

16. New func!onality went live, including for products
Oracle patches

17. New code calls the product catalog differently1.
Product catalog responds slowly to these calls2.
A circuit breaker kicks in, returning null3.
All calls that need a valid product start logging
Log4j starts buckling under the load
4.

18. ALTERNATIVE HYPOTHESISALTERNATIVE HYPOTHESIS
Oracle patches somehow changed something1.
FindProduct calls use Oracle queries which
somehow became more slow
2.
Circuit breaker kicks in, returning null3.
All calls that need a valid product start logging
Log4j starts buckling under the load
4.

19. DEDUCTIONDEDUCTION

20. DESIGN EXPERIMENTSDESIGN EXPERIMENTS
Focus on one hypothesis at the !me
Performance tests
Run the script locally, if possible
Add code if needed

21. SELECTING THE RIGHT HYPOTHESISSELECTING THE RIGHT HYPOTHESIS

22. OCCAM’S RAZOROCCAM’S RAZOR
Helps in keeping experiments small
Related: Su#on’s law, most likely first
When presented with compe!ng
hypothe!cal answers to a problem, one
should select the answer that makes the
fewest assump!ons

24. NEWTON’S FLAMING LASER SWORDNEWTON’S FLAMING LASER SWORD
(Alder’s razor)
Don’t bother explaining it with jvm/compiler bugs,
unless you know the workaround
What cannot be se"led by experiment is
not worth deba!ng

25. HITCHEN’S RAZORHITCHEN’S RAZOR
If you want to help out, and have a new hypothesis,
back it up
What can be asserted without evidence
can be dismissed without evidence

26. Pick hypothesis 1
The “Oracle hypothesis” was too vague, less likely
Revert webshop to produc!on version, rerun
performance test

27. TESTINGTESTING

28. EVALUATIONEVALUATION

29. EVALUATIONEVALUATION
Was the hypothesis correct?
If yes: fix the cause
If no:
Iterate!
More observa!ons?
Maybe another hypothesis?

31. ITERATEITERATE

33. AJP THREADSAJP THREADS
at sun.misc.Unsafe.park
at java.util.concurrent.locks.LockSupport.park
...
at com.bol...PCSAdapter.getProductsRepositoryItems
at com.bol...RemoteCatalogRepository.getPCSItems

34. NING THREADSNING THREADS
at java.net.SocketInputStream.socketRead0
...
at oracle.jdbc.driver.OracleStatementWrapper.executeQuery
...
at com.bol...getProductsCustomMultiProperty
at
com.bol...CustomPropertiesLoader.getProductsCustomMultiProperty
at
com.bol...PCSAdapter.lambda$transformProductListToRepoItems$17

35. NEW HYPOTHESISNEW HYPOTHESIS
Oracle queries slowed down with update1.
Product catalog calls execute extra queries on same
thread
2.
Slowdown causes circuit breaker, log failure, system
lock
3.

36. NEW EXPERIMENTNEW EXPERIMENT

38. WRAPPING UPWRAPPING UP

39. TAKE HOMETAKE HOME
Structure your diagnosis
Formulate hypotheses
Focus on diffs
Build !melines
Run one experiment at the !me
Locally, if possible
Iterate experiments if needed
Communicate, communicate, communicate

41. REMAINING MAGICREMAINING MAGIC
Training coming up end of summer!
Together with An Ngo
Cover both GC/performance tuning and analysis
tools

42. THANKSTHANKS
ctankink@bol.com