Anders Iderström, Engineering Manager for Live Operations and Observability, recently gave a presentation at Load Up 2016 in New York, entitled "Aiming for four nines with Apica Synthetic". Anders talked about how we monitor and keep business critical systems up and running here at Klarna, as well as how we handle incidents while minimising the impact on our end users.

1. Aiming for four nines with Apica
Synthetic
Anders Iderström
Engineering Manager
Live Operations & Observability

2. • Unique offering
• Fast growth
=
• Speed is king
• Money is not an issue
About Klarna

3. • What we provide
• What we rely on
Live Operations

4. We provide fast and consistent professional incident management.
We act as an internal communications bridge and provide support on monitoring
and log analysis.
We identify and share knowledge about risks in an ever changing environment
and usage scenario.
We aim to be the source of truth and provide IT operations expertise.
Live Operations

6. Our everyday tools:
Apica WPM, op5 Monitor, built-in alerting mechanisms, Splunk,
Watchdog/opstat (Pentaho/HighCharts), Monks, Graphite, Grafana, Jarmon,
Dashing.
Also used at Klarna:
New Relic, Kermit, OpsGenie, Oracle-related, finance IT related
= if we don’t have it, we’re probably in the process of getting or writing or buying
it. 
Live Operations

7. • IT-operations Monitoring
• Panoptic development team
• Personal experience from monitoring
implementation in about 150 organisations
• My development, training and coach
• A hundred people in ITops
• A couple of hundred other engineers
(developers)
My other powers

8. Mid 2012 through late 2013

9. 2014 and 2015 - way better

10. Monthly KRED (KPM) availability
2014 and 2015
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2014 2015
99.8%
99.9%
99.95%
Availability
(Uptime/month)
Downtime
Minutes/month
2014 average
11 min down/month
22 sec down/day
99.975% availability
2015 average
11 min down/month
22 sec down/day
99.975% availability
0
22
44
66
88
110
132
154
176
198
220
Yes, averages are identical for 2014 and 2015
99.975%
100.00%
Q4 2015 at average 99.997% for full quarter

11. Q4 2015 KCO availability
KCO RedFlow Sweden
Oct Nov Dec
2015
99.8%
99.9%
99.95%
Availability
(Uptime/month)
Downtime
Minutes/month
Q4 2015 average
RedFlow SE
4.7 min down/month
9 sec down/day
99.99 % availability (full quarter)
100% availability in Nov and Dec
99.975%
100.00%0
22
44
66
88
110
132
154
176
198
220

13. • How did we reach 99,975?
• We integrated Apica WPM API with Opstat/Seqailizer/Petahoo (wallboarding
last 2 minutes / last 2 hours, number of timeouts, avail-numbers, ect)
• And ”Defined downtime”: ext/int: x number of timeouts of y number of
minutes AND/OR other checks we can include, like Klarna Online or
backoffice being unavailable
2014 and 2015 - continued

14. Implementing end-to-end
responsibility in Product
Development

15. • Measurements become more sensitve too, so you need to remember when
your providers providers failed, or make a NOTE of it and exclude that from
your measurements 
• Added to a new feature
• Requested at Apica Kundforum last year
• Implemented, thanks Apica!
Polishing the truth (feature req)

16. from advanced integration and simple check, to advanced checks and no integration
- It takes some time to get right
- It requires tons of approvals in modern fintech systems
+ It’s quick and simple to consume
+ It’s fairly “new” (new type of checks, or at least newly completed for all big product
versions/regions)
+ We need a rolling 12 month view now, as a complement to hour-to-hour
wallboards
Shifting focus

17. The 8 step checks and how we see which one has failed:
Zebratester (a.k.a Proxysniffer):
Merchant Create Order (merchant_create_order_v2)
Merchant Read Order (merchant_read_order_v2)
Client Read Order (client_read_pre_purchase_order_v1)
Client Update Order - Sending challenge information
(client_update_pre_purchase_order_v1)
Client Update Order - Sending billing
information (client_update_pre_purchase_order_v1)
Select Payment Method (client_update_pre_purchase_order_v1)
PGW loading iframe - WebPage (client_update_pre_purchase_order_v1)
PGW loading iframe - API details (client_update_pre_purchase_order_v1)
Shifting focus

18. Providing an additional 6-7 minutes uptime per month (99,975->99,990%)
Communication platforms starts to become cost efficient
Company goal: 99,97
IT-operations OKRs: Q1: 99,97, Q2: 99,98, Q3/Q4: 99,99
2016 company availability goal and
2016 ITops OKRs

19. And the progress? (2016)

20. High pace
= we quickly forget

22. • Three decimals needed at reporting/pdf-e-mail-report level for proper
precision in reporting numbers later combined with other figures to track top-
level company goals
• 10 decimals available in the API, but we’re not integrating at the moment 
• Needed
• Requested
• Implemented (actually with four decimals) (thanks Apica!) 
Actually seeing the last decimal -
from suggestion to practical use
(feature req)

23. - Gradually remove dependencies
- Let the experts handle stuff you heavily depend
on and/or create specialized teams (networking,
message bus, certificates)
- Never repeat mistakes
- Implement modern architecture
- Serve your existing customers from your new
shiny platform (LOCO)
Reaching above 99.975%

24. • Implement end-to-end responsibility
• Get started on a shift in architecture
• microservices in cloud platform
• graceful degradation
• but do it piece by piece (breaking pieces of the monolith)
• Keep centralized Incident Management (operations land, OPs
knowledge)
• Do continuous improvement/feedback - on all levels (liveops, dev-teams,
retros, incident reports and planned actions)
• Save minutes/seconds in communications
• Actually service all your customers from the new platform (solve the
tech dept through massive added value)
TAKEAWAYS

25. Anders Iderström
Thank you!