The Theory and Practice, Practice, Practice of AWS Operations - AWS Summit Sydney

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The theory and practice, practice,
practice of AWS Operations
Colm MacCárthaigh
Senior Principal Engineer
Amazon Web Services

Do you carry a pager?

Industry-wide recovery rate

What is operations?

What is operations?
Operations is the doing of things
Over and over, better and better
How do we form good habits, and prevent bad habits

What is operations?
Great operations is built on humility
We build robust systems and designs, but anticipate that
there will be something that we didn’t think of
Healthy paranoia helps too

What is operations?
Every operational action is reviewed
”Two Person Rule” for anything that is not routine
Constant strive to automate routine actions away

What are we going to cover?
What is different at scale?
What is operational risk?
Compartmentalisation
Deployment Safety
The Operational Mindset
Staying SAFE when things go
wrong

Something is “broken” all of the time
The stakes are higher
The number of people involved is larger

There are greater opportunities to perfect
automation and operational practices
There is more experience to learn from
The number of people involved is larger

How we think about operational risk
Operational risk Is really related to change
• Key types of change
• Component failure
• Environmental failure
• Increases in load
• New code paths being exercised
• New modes of operation

ap-southeast-2
us-east-2
eu-west-1
…

ap-southeast-2b
us-east-2b
eu-west-1b
…
ap-southeast-2a ap-southeast-2c
us-east-2c
eu-west-1ceu-west-1a
us-east-2a

Deployment Safety

Deployment Safety
New code means risk, so we are incredibly
paranoid about deploying it
CI/CD staged deployment process
Promotion testing and monitoring at every
stage, with automated rollback
Fast and reliable rollback

Deployment Safety
1. Code-review
2. Check-in
3. Pre-Production
4. One Box
5. One Availability Zone
6. One Region
7. Onwards …

Deployment Safety
Reliable and fast rollback is key
Random Selection is useful to avoid repeat issues
Services have to be designed with phased deployment
and rollback in mind
What if we’re making backwards-incompatible changes?

The operational mindset
Services are not simply code in an editor
Services are live running systems that respond to input
Services change over time even if you don’t change the code
There is no “done”

Every week at AWS we have an ‘All Hands’ operations meeting
We dig into any COEs (Corrections of Error) from the previous week
We choose some services at random and dive into their operational
metrics
We look at operational sustainability too

Every team has a corresponding weekly operations meeting:
review the previous weeks metrics
We pay close attention to any alarms that fired, and any
tickets that were cut
On-call report from the engineers who were on-call

Staying SAFE when things go wrong
Every team has an active on-call engineer
On-call engineers are automatically engaged for most issues
CloudWatch Alarms -> tracking ticket -> page

Staying SAFE when things go wrong
For larger issues, or when there is elevated risk, we use voice conference
calls to coordinate
Every call has a designated “Call Leader” and a ”facilitator” from our
technical operations team
Call leaders are experienced and tenured AWS staff and are empowered
to make decisions

When
doesn’t apply

The Theory and Practice, Practice,
Practice of AWS Operations
Stay calm
Assess the situation
Focus on mitigation
Escalate early and often

Stay calm
Big events can be overwhelming, but cooler heads prevail. Keep a sense
of urgency, but be methodical and avoid a frenetic energy. Most of all,
don’t panic! Remember that we have a 100% success rate of mitigating
operational events, so take a quick breath if you need to, we will solve
each and every challenge. Try to avoid team huddles that are not on a
conference call with an experienced designated leader, as they split focus
and detract from resolution. Call-leaders can help you prioritise and
engage others. Use the ticket and chime or IRC for communication too.

Assess the situation
If you join a call, first read the associated ticket or SIM, especially the
summary. If the call has a large number of participants then please don’t
interrupt it to check in, instead use the ticket/SIM. Assess your own
dashboards, alarms and alerts and be prepared to report the summary
for your own area, service or team. Do this continuously for the duration
of the call. Every call participant should be active. If you can, take notes
for yourself as you go; the names of other people on the call, key pieces
of technology or information you might not be familiar with, etc …

Focus on mitigation, not root cause analysis
Most issues can be remediated much more quickly with rollbacks, data
center flips, database failover, throttling and other operational actions
that essentially quench the source of the problem rather than truly
“fixing” it in a deeper sense (i.e. patching the code). At Amazon we are
happy to roll back speculatively, on the basis that it might fix the
problem. More often than not, it is not necessary to truly understand the
problem, so defer root cause analysis until you have either exhausted
traditional rollback/flipping steps for your service, or for when you can
do remediation actions and root cause analysis in parallel.

Escalate early and often
The absolute minimum knowledge for every on-call engineer is to know
how to escalate: be able to page your secondary, your manager, and any
other escalation channels appropriate for your service. It is always ok to
escalate! If you yourself are a secondary or an escalation person, be
grateful and supportive too for getting paged. If an LSE is impacting your
service, and if recovery steps are likely, escalate early and get help!
Paging early has been shown to save tens of minutes of impact time
during typical events.

Takeaways
• At AWS teams own their
business, development, and
operations
• We think of operational risk
in terms of change
• Compartmentalisation limits
the blast radius of issues
• Steady deployment safety
wrangles risks associated
with software and
configuration changes
• AWS company culture values
operational excellence
• SAFE when things go wrong

Thank you!
Colm MacCárthaigh
colmmacc@amazon.com

The Theory and Practice, Practice, Practice of AWS Operations - AWS Summit Sydney

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Similar to The Theory and Practice, Practice, Practice of AWS Operations - AWS Summit Sydney

Similar to The Theory and Practice, Practice, Practice of AWS Operations - AWS Summit Sydney (20)

More from Amazon Web Services

More from Amazon Web Services (20)

The Theory and Practice, Practice, Practice of AWS Operations - AWS Summit Sydney