This document discusses advice for dealing with "fires" or urgent performance issues based on the author's experience as a consultant. The key points are: 1. Analyze issues using "the box" model to consider potential causes at different levels from the network to the application code. 2. Stay calm and collected when dealing with fires rather than losing your head or getting frightened. Make sure to double check your own work before others can blame you. 3. Set up monitoring tools and logging to get advance warnings of potential problems and gather the right diagnostic information when issues do occur. 4. Know when to remove yourself from a situation if there is too much ongoing stress from attempting to address fires. Your

1. Seeing through smoke
@LAFK_pl Consultant @
Tomasz Borek

2. GeeCON 2015 Kraków!
@LAFK_pl
http://2015.geecon.org/register/

3. I'll take ALL feedback I can get.
@LAFK_pl, #testwell
@LAFK_pl

4. Symentis
@LAFK_pl

5. Symentis
@LAFK_pl
Jarosław Pałka Kuba Marchwicki

6. Joining criteria?
@LAFK_pl
Easy!

7. Joining criteria
@LAFK_pl
● Conference speaker
● Open-source committer
● Recommended by somebody on the team

8. Who are you?
@LAFK_pl
● Programmer?
● Tester?
● Build person?
● Hosting/Ops?
● Manager?
● Other?

9. Today?
@LAFK_pl

10. T
o
o
l
s
@LAFK_pl

11. Putting out fires
@LAFK_pl

12. Details
● Where this came from
● Diagnostic methodology for performance problems
● Keep calm
● Monitoring tools
● Logging 101
● How not to fly solo
● Advice for when there's too much smoke
@LAFK_pl

13. Where this came from?
@LAFK_pl
So you know where I stand

14. Where this came from?
● plug-a-hole fireman: 2 years
● heroic fireman: 2 years
● Fire drills I've did and had done to me
@LAFK_pl

15. Advice #1
@LAFK_pl
Don't be a fireman for half a year, not to mention
for years

16. Tool #1
@LAFK_pl

17. The ”box”
● Performance problem?
● Heinz Kabutz
● Kirk Pepperdine
● Sets priorities straight
Net, hardware, etc.
JVM / OS
Application itself
Users!
THE BOX:
@LAFK_pl

18. @LAFK_pl

19. How not to loose your head
● Praemonitus – praemunitus!
● Advanced warning
● Knowledge is power
● Don't get (too) involved
● You're the expert, Anderson!
@LAFK_pl

20. What gives warnings in advance?
● What comes in code?
● What the code runs on?
● Can our X hold Y Zs at same time?
● How long till A gets exhausted?
● Alert emails mailbox full?
● Which Selenium tests are least likely to give
false-positive or flop on you due to some glitch?
@LAFK_pl

21. Sun Tzu, Art of War
@LAFK_pl
If you know the enemy and know yourself, you
need not fear the result of a hundred battles. If
you know yourself but not the enemy, for every
victory gained you will also suffer a defeat. If you
know neither the enemy nor yourself, you will
succumb in every battle.

22. Advice #3
@LAFK_pl
If you know the enemy and know yourself, you
need not fear the result of a hundred battles. If
you know yourself but not the enemy, for every
victory gained you will also suffer a defeat. If you
know neither the enemy nor yourself, you will
succumb in every battle.

23. Don't be involved
@LAFK_pl
Best advice and outlook have those outside

24. You're the expert, Anderson
@LAFK_pl

25. Double, triple check yours
@LAFK_pl
So when someone shouts ”it's your fault! Your
responsibility! You fix it!” you will have a retort
right away

26. @LAFK_pl
Never let them shout at you
Advice #4

27. @LAFK_pl
Never let them shout at you
Advice #4

28. Preparation
● What can go wrong, will
● Toolchain
● Limit your SPOFs in advance
● Cover them if you can't limit them!
@LAFK_pl

29. How to fight fright in a fire
● Who's frightful?
● What's urgent?
● What's DR timeline for the urgent items and for
the frightful people?
● RPO?
● RTO?
@LAFK_pl

30. Monitoring tools
● Performance co-pilot
● Kibana / Splunk
● AppDynamics
● New Relic
● Nagios
● Ganglia + Graphite (ish)
● SHELL DWARVES EVERYTHING FOR FIRES
@LAFK_pl

31. Logging 101
● What
● The relevant
● The crucial
● Reload on-the-fly
● Storage
● Throughput
@LAFK_pl

32. Logging levels
@LAFK_pl
Not those, that you think of

33. Level 3
@LAFK_pl
We have logs, humans look at them sometimes

34. Level 3
@LAFK_pl
We have logs, humans look at them sometimes
* when there's a fire

35. Level 2
@LAFK_pl
We have logs, we need to pull out more, more
often

36. Level 1
@LAFK_pl
We log, we parse logs, we take action

37. Tools for logging
● ELK – ElasticSearch + LogStash + Kibana
● Or the equivalent / improvement
● Available to hosting, devs, testers and support
● Loggly, Splunk
● Log levels defineable on the go by Hosting /
key others
@LAFK_pl

38. How not to
● Request
● Ask
● Plead
● Beg
● Bargain
● Blackmail
● Ok threaten perhaps
@LAFK_pl

39. How not to
● Request
● Ask
● Plead
● Beg
● Bargain
● Blackmail
● Ok threaten perhaps
@LAFK_pl

40. @LAFK_pl
You do NOT want to be a hero.
Advice #5

41. Too much smoke
@LAFK_pl
Quit. Best like a boss.

42. Why should I?
@LAFK_pl
Physical recovery: 6
weeks.
Psychological
recovery: 6 months.
Capacity ... unlikely.

43. When would that be?
@LAFK_pl
When you hurt / complain too much.
Keep a trend!

44. Today
● Tool #1? The box. You know, for performance
problems.
● Keep calm: don't lose your head, don't be
frightened and don't let them shout at you.
● Monitoring tools: have your own set.
● Logging 101: aim for high level, visualize
● Fly Solo, if you have a Chewbacca next to you.
● Too much smoke... quit. Trend yourself.
@LAFK_pl