1. An organization experienced three separate outages over 72 hours affecting their API and applications due to server reboots. 2. The first incident caused slower response times while the second and third incidents were full outages. 3. The outages were caused by human error in dismissing an alert combined with a bug in a database driver, resulting in all front-ends being unavailable after servers rebooted.

1. How It All Goes Down
Your next service outage will be during
@ctosummit at 1:50pm.
Daniel Doubrovkine
db@artsy.net
@dblockdotorg

3. Do you have a Syrian domain?
somewhere here there’s your .sy TLD authority

4. Are you using MongoDB?

5. Do you use a PAAS?

6. Is there any logic involved?
1. Restart Unicorns, no improvement.
2. Restart a server, no improvement.
3. Found an EC2 maintenance note buried in email.
4. Patch responsible for 5x slower response times!
5. Do a DB query from a production console.
6. Identify a pattern (slow, then fast).
7. Look in MongoDB logs.

7. Are there humans involved?

8. Are you using a beta version of a driver?

9. Post-Mortem

10. Subject

11. Summary
We have experienced 3 separate
outages over the last 72 hours that
have affected api.artsy.net which is
the backbone of all our
applications, including Admin,
CMS, artsy.net, m.artsy.net, etc.
The first incident was slower
response time 9/28 2AM-10AM
EST.
The second and third incidents
were two outages, 9/29 11PM-
12AM EST and 9/30 4AM-6:30AM
EST.

12. Cause
1. Trigger
Servers rebooted.
2. Unexpected behavior.
Reboots should have been handled just fine by software.
3. Cause.
Human error + bug in the driver.
4. Consequence.
All front-ends down.

13. Resolution
1. Ticket
Woke up a contractor.
2. Manual intervention.
Kick servers.

14. Post-Mortem
1. Human Error Preventable
The dismissal of the alert was wrong.
2. Failure to Plan.
Reboot schedule needed a human to monitor it.

15. Outage History

16. Thanks!
Daniel Doubrovkine
db@artsy.net
@dblockdotorg