Performance Management at Employment Hero

1. Performance management
Part 1: Setting the right expectation
Luong Vo - coolie @ Employment hero

2. I am not an expert :)

3. I inspire
Not instruct

4. Today’s Agenda
1. Why ?
2. Why measure?
3. Why monitor?
4. What to monitor?
5. How to monitor?

5. Today’s Goals
Everyone leaves with at
least one idea about
performance management.
Everyone knows where and
what to look for in
performance monitoring.

6. 1. Why ?
Pinterest increased search engine traffic and sign-ups
by 15% when they reduced perceived wait times by 40%.
COOK increased conversions by 7%, decreased bounce rates
by 7%, and increased pages per session by 10% when they
reduced average page load time by 850 milliseconds.
The BBC found they lost an additional 10% of users for
every additional second their site took to load.
DoubleClick by Google found 53% of mobile site visits
were abandoned if a page took longer than 3 seconds to
load.

7. 1. Why ?
It’s about users , expenses, outages
Then business, income, adoption, retention….
Source: https://idlewords.com/talks/website_obesity.htm , https://developers.google.com/web/fundamentals/performance/why-performance-matters

8. 2. Why measure?
Measure the actual
performance of your
application, and focus on
the 20% of your code that
is the worst performance
offender.
If you're applying equal
effort to all areas, you
are wasting your time.

9. I do not optimize
anything in my
application until my
metrics tell me so.

10. 2. Why Monitor?
Random boss in AU:
“The site feels slow”
Luong:
“Really!? It works like a
charm here”
Luong’s colleague:
“Look good on my machine”
Is my API, web or mobile
application available?
“Yes”
Is it functioning
correctly?
“Yes”
Is it fast?
“Hm”

11. Monitoring is DEfinitely a
good place to start with.
(although there are other options)

12. 3. WHat to monitor?
External metrics: latency,
throughput, error rate.
Internal metrics: CPU,
Load Average, Memory,
Network.

13. Individual Brainstorming:
Scenario 1
What will we look for if some
of our users tell us that the
app is slow ?

14. Individual Brainstorming:
Scenario 2
How can we be aware of outage
before our users?

15. Individual Brainstorming:
Scenario 3
How do we know how many
servers we need or how much
load we can handle in peak
time?

16. 4. How to Monitor?
Mindset:
- Calculate
- Store
- Query
- Which layer?
- How to integrate?
Tools:
- Stackify
- Datadog
- New Relic
- Prometheus
- ...

17. 4. How to Monitor?

18. 4.1. Prometheus overview
Model: Push/Pull

19. 4.1. Prometheus overview
Push Gateway

20. 4.1. Prometheus overview
Exporters

21. 4.1. Prometheus overview
Review

22. 4.2. Instrument your application
Step 1: Decide which type
of metrics you should use:
- Gauge
- Counter
- Histogram
- Summary
Is it only rising?
“Counter”
Is it rising and
decreasing?
“Gauge”
Is it based on
observations and
timeframe?
“Histogram and Summary”

23. Individual Brainstorming:
Scenario 1
What type of metric will we
use for Memory usage?

24. Individual Brainstorming:
Scenario 2
What type of metric will we
use for number of requests
served?

25. Individual Brainstorming:
Scenario 3
What type of metric will we
use for average latency?

26. 4.2. Instrument your application
Step 2: Choose your metric name and labels, start using it

27. 4.2. Instrument your application
Or: Integrate with your
application
- Eh Monitoring
- Middleware
- Required files
- Hooks
- Metric container
We wrap Ruby Prometheus
library to handle labeling
and automatic integrations
with rails, sinatra,
sequel, active record….

28. 4.2. Instrument your application
Step 3: Start Querying
- PromQL
- Metric name
http://prometheus.ehrocks.com
http://prometheus.staging.ehrock
s.com

29. 4.2. Instrument your application
Step 4: Start visualizing
http://monitor.ehrocks.com

30. 4.2. Instrument your application
Step 2.5: Actually....

31. 4.2. Instrument your application
If you want to go fast, use
cloud monitoring as a
service

32. 4.2. Instrument your application
Step 4: Start Alerting

34. Why measure?
¯_(ツ)_/¯ Noi dao ly thi thuong song nhu … ?

35. REsults from Sep 2019

36. REsults Nowadays

37. Always set your Goals.
Make sure you got the right
metrics to back it up
(because reality sucks)