Talk #2 at Time Series Estonia #2 by James Bach (Site Reliability Engineer at Transferwise) on The problem with not being observable

1. The Problems with not
being Observable
A TransferWise Journey

2. Observability
Team @ TW
Est. Late 2018
London / Tallinn
6 People
TransferWise is hiring
https://www.transferwise.jobs/
Toomas

3. What is Observability?

4. Vendor Definition
- ‘Three Pillars’
- Distributed systems
problems are pure tech
- Describes your problems
through aggregations or
Firehosing
- Investigation requires prior
knowledge of your stack
- Monitoring != Observable
- Distributed systems
problems are sociotechnical
problems
- Describes your problems
through aggressive
sampling
- Investigate new problems
without shipping new code
Pure Definition
- Charity Majors,
@mipsytipsy

5. Vendor Definition
- ‘Three Pillars’
- Distributed systems
problems are pure tech
- Describes your problems
through aggregations
- Investigation requires prior
knowledge of your stack
- Monitoring != Observable
- Distributed systems
problems are sociotechnical
problems
- Describes your problems
through aggressive
sampling
- Investigate new problems
without shipping new code
Pure Definition
- Charity Majors,
@mipsytipsy

6. So what does TransferWise look
like architecturally?

7. 2011-2017

8. 2017-2019

9. 2019-2020

10. 2020-202X?

11. TL;DR
I actually literally have no idea

12. But it’s
probably
like this
- Monzo, We built network
isolation for 1,500 services to
make Monzo more secure

13. But it’s
probably
like this
- Monzo, We built network
isolation for 1,500 services to
make Monzo more secure
Yours
probably
looks like this

14. Here are some things we
learned along the way

15. Distributed systems are hard
and asks really random
questions

16. For example:

17. “Why does our site break every
first monday of the month?”

24. What did we do to resolve it?

25. Explore what data we had
Instrument user flows for
visibility
Make more data driven
decisions

26. “Why does our site break every
first monday of the month?”

27. Answer:
First monday is pay-day.
It is one bad query on one
service.

28. Tip 1:
Monitoring for Unknown
Unknowns is technically foolish

29. Tip 2:
Engineers need Production
access to be effective

30. Tip 3:
No Observability
==
Missed Opportunities

31. Example:

32. “Why do we have a spike of
customers using the site, but not
a spike of money being
transferred?”

33. Parliament
Votes on Brexit

34. UK General
Election 2019

35. Tip 5:
More Observability == more
opportunity for Production
Excellence

37. Tip 6:
Transitioning from Monitoring to
Observability is hard

38. Talk to your colleagues. Find
someone who can champion
your cause. Make them care
about it.

39. Tip 7:
Look at what other people are
doing.

40. Thank You
James Bach
@qweety_