1. Nothing Good Ever
Happens After 2am
Reversim 2019

2. Daniel Korn
Engineering Team Lead at BigPanda
korndaniel1

5. BigPanda’s
Outage Procedure

6. Roles and responsibilities
On-call Incident Manager
On-Call (IMOC)
Tech Lead
On-Call (TLOC)
Support
On-Call (SOC)

7. Incident Priority Definitions
Priority Affect Outage Resolution
P1
• Core feature
• Multiple customers
24/7
P2
• Core feature
• Single customer
24/7
P3
• Secondary feature
• No workaround
Next business day

8. Tools

9. Tools
• Alerting

10. Tools
• Alerting
• Communication

11. Tools
• Alerting
• Communication
• Observability

12. Alert/Support
notifies On-call
IMOC asses impact,
determine P1/P2/P3
On-call performs
simple mitigation
On-call escalate
to IMOC
IMOC escalate to
TLOC and SOC
1
2
3
4
5

13. 6
7
8
9
10
On-call If (P1) {
StatusPage;
dedicated channel;
}
SOC update
customers
R&D mitigate till
solved, update
StatusPage
IMOC Verifies resolved,
summary in channel
IMOC postmortem,
share with stakeholders

14. The Long Night

15. THIS IS A TRUE STORY.
The events depicted in this postmortem
took place in Tel Aviv and San Francisco
in 2018.
Despite the request of the survivors, the
names have not been changed.
Out of respect for our customers, the
story has been told exactly as it occurred.

16. Michal
On-call
Almog & Pini
TLOCs
Daniel (Me)
TLOC

17. Shmeff Andru
SOC Support
Julio
Support

18. Background
• REMINDER: BigPanda’s SLA
• New Access Control (RBAC) service
• Not all customers migrated
• Sunday: Multi-service deployment

19. [MON 05:03 PM] SOC
multiple tickets:“cannot
update environments”
[05:05 PM] On-call
Asks SOC for details, opens a
dedicated Slack channel
[05:08 PM] On-call
Identifies as Auth-related,
notifies TLOCs

22. [05:35 PM] On-call
“we think it’s related to a
deploy, working on a fix”
[05:33 PM] SOC
considers opening a status
page, but “might be a P3”
[06:16 PM] SOC
Opens status page

23. Stick to the Plan
TA
K
EAW
AY

24. [07:41 PM] TLOCs
Deploy fix to production
[06:50-07:30 PM] TLOCs
Fix is tested, not reproduced
debate fix or revert
[07:45-08:05 PM] SOC
Verifies together with TLOCs
the issue is resolved
[08:10 PM] SOC
Closes status page
On-call and TLOCs leaving

25. REVERT FIRST
Rule of Thumb
TA
K
EAW
AY

26. [12:57 AM] SOC
“So it appears to be just a
UI issue”. Notifies On-call
[12:45 AM] Support
“Some customers can’t see
roles in the env editor”
[12:59 AM] On-call
Notifies TLOC
[01:01 AM] TLOC
Starts investigating the issue

28. – Someone smart
If it looks like an outage, and (support)
sounds like an outage, then it might
be just a bug“

29. Do not Assume
an Outage
TA
K
EAW
AY

30. [01:54 AM] TLOCs
Deploy fix to production,
ask SOC to verify with customers
[01:20 AM] TLOCs
Identifying the cause,
starting to work on a fix

31. If you think this has a
happy ending, you haven’t
been paying attention.
— Ramsay Bolton
“

32. [02:00 AM] SOC + Support
Debating on StatusPage re-open
[01:57 AM] Support
customers reporting the initial issue -
“cannot update environments”
[02:03 AM] TLOCs
Start investigating the issue

33. [02:15-02:51 AM] TLOC
Manually adds missing
permissions to customers DB
[02:10 AM] TLOCs
Identifying the cause - lack of
permissions (migration)

35. Time to Call it
a Night
TA
K
EAW
AY

36. [02:56 AM] SOC
Verifies this customer is
facing the issue
[02:52 AM] TLOC
Having problems with a
specific customer
[02:56-03:25 AM] TLOCs
Identify the problem - edge case
involving FT and manual customizations
[03:25 PM] SOC
Asks TLOC to discuss the
situation on a phone call

37. [-04:07 AM] SOC+TLOC
SOC asks TLOC to
commit to fix by EOD
[03:29- AM] SOC + TLOC
Sensitive customer, no
changes ,issue remains
[09:30 AM - 05:12 PM] TLOCs
Implemented a fix, deploy to production,
ask SOC to verify
[05:25 PM] SOC
Verifies issue resolved

38. Do not Commit
to Action Items
TA
K
EAW
AY

39. [19:00 PM] CS + R&D + PM
Joint postmortem,
Preparing customer’s updates
[WED 11:00 AM] R&D
Conduct a postmortem,
Share with R&D and CS

40. Chaos isn’t a pit.
Chaos is a ladder.
— Petyr “Littlefinger” Baelish
“

41. Recap

42. • Stick to the plan
• Rule of thumb: REVERT FIRST
• Do not assume an outage
• Time to call it a night
• Do not commit to action items