Anurag Gupta spoke at the AWS Big Data Meetup in Palo Alto and described the AWS DevOps culture. In the talk he gives pointers on how service owners can setup monitoring that will continually reduce operational burden.

1. Operations @ Scale
Anurag Gupta, VP
AWS Database Services

2. Dev/Ops
How I learned to stop worrying and love my pager
Dev/Ops - your dev org is your ops org
I get a pager! You get a pager! Everyone gets a pager!
Why would I possibly want this?
It motivates design for operability
It aligns your interests w/ your customer experience
It improves the feedback loop to customer needs

3. Monitor everything
Every API call to your service,
Every API call you make to a dependent service
Canary traffic for things that vary (eg SQL statements)
Most of the metrics won’t be meaningful. That’s OK
Page on your high signal-to-noise metrics
Monitor these metrics during deployments
Median/Average, Fleet-wide, coarse time grain are obscuring
Measure TP90, TP99 (99th percentile response time)
Measure at finer and finer grain
Evaluate per-customer metrics
Look for the needles in the haystack

4. Correction-of-Error (COE) Reporting
Meet weekly on operations (execs, service operators)
Review each issue that happened.
“Spin the wheel” to review a service’s metrics
Support a “truth-seeking” culture
Looking for data, process improvements
COE
- Customer impact
- Timeline: incidence to detection to response to resolution
- 5 Whys? Get to actionable changes to extinguish cause
- Actions

5. Ops is Dev
Humans are fallible
circa 1% defect injection rate
Error rate changes based on time of day (3am vs 3pm)
New ones show up, have unique issues
Limit human access to machines
Use code/scripts/tools instead
Scripts are code
unit test, code review, deploy, automate

6. Ops load correlates to business growth
As your business does well, your
operations needs to become great
Growing 100-200% YoY is hard.
Improving ops 100-200% YoY is really
hard.
Improving ops 2% each week is possible.
Use Pareto analysis to prioritize work
Bonus – each customer gets a better
experience even as your own ops load
stays constant
Amazon Redshift has grown rapidly since it became generally
available in February 2013. While our guiding principles have
served us well over the past two years, we now manage many
thousands of database instances and below offer some lessons we
have learned from operating databases at scale.
Design escalators, not elevators: Failures are common when
operating large fleets with many service dependencies. A key
lesson for us has been to design systems that degrade on failures
rather than losing outright availability. These are a common
design pattern when working with hardware failures, for example,
replicating data blocks to mask issues with disks. They are less
common when working with software or service dependencies,
though still necessary when operating in a dynamic environment.
Amazon overall (including AWS) had 50 million code
deployments over the past 12 months. Inevitably, at this scale, a
small number of regressions will occur and cause issues until
reverted. It is helpful to make one’s own service resilient to an
underlying service outage. For example, we support the ability to
preconfigure nodes in each data center, allowing us to continue to
provision and replace nodes for a period of time if there is an
Amazon EC2 provisioning interruption. One can locally increase
replication to withstand an Amazon S3 or network interruption.
We are adding similar mitigation strategies for other external
understanding that, even if not a widespread concern, each issue is
meaningful to the customer experiencing it. In Figure 5, Sev 2
refers to a severity 2 alarm that causes an engineer to get paged.
This means operational load roughly correlates to business
success. Within Amazon Redshift, we collect error logs across our
fleet and monitor tickets to understand top ten causes of error,
with the aim of extinguishing one of the top ten causes of error
each week.
Figure 5: Tickets per cluster over time
Pareto analysis is equally useful in understanding customer
functional requirements. However, it is more difficult to collect.

7. Escalators, not elevators
Failures happen.
Durability failures are “easy”
mirroring, quorums, well understood techniques
Availability failures are “hard” –
want to degrade on unavailability not cascade failures
tolerate 1-2 hours of unavailability (time to detect, fix)
- eg caching IP addresses when DNS is unavailable
- eg maintaining instance warm pools rather than provisioning
- eg losing the ability to restore a backup, not lose writes

8. Ship often
Continuous delivery should be to the
customer
Benefits
Customers prefer small patches
Rollback is easier
Rollback is less likely
Faster response to customer issues
We push a new database engine
version, including both features and
bug fixes, every two weeks.
dependencies that can fail independently from the database itself.
Continuous delivery should be to the customer: Many
engineering organizations now use continuous build and
automated test pipelines to a releasable staging environment.
However, few actually push the release itself at a frequent pace.
While customers would prefer small patches to large ones for the
same reasons engineering organizations prefer to build and test
continuously, patching is an onerous process. This often leads to
special-case, one-off patches per customer that are limited in
scope – while necessary, they make patching yet more fragile.
Figure 4: Cumulative features deployed over time
Amazon Redshift is set up to automatically patch customer
clusters on a weekly basis in a 30-minute window specified by the
Cumulative features deployed over time