This document summarizes lessons learned from an outage at the DC Public Library's website, which was hosted on Amazon Web Services. The outage occurred in April 2011 when an AWS engineer mistakenly rerouted internal traffic, overwhelming the system. Key lessons included that AWS' multiple availability zones are not truly independent, poor communication from AWS exacerbated the problems, and the library needed to have a recent backup of their website and database in a completely separate facility to avoid being dependent solely on AWS. The library now takes daily backups to separate servers at Rackspace.

1. AMAZON FAIL
DC Public Library’s Lessons Learned from the Amazon Cloud Outage
Friday, June 24, 2011

2. BACKGROUND
• DClibrary.org was first major DC Government website to use
cloud-based hosting beginning circa June 2009
• Initial architecture designed to leverage low cost of large instances
Amazon Web Services (AWS) servers for database operations
and lower cost small and mid servers for WWW services
• DClibrary.org Content Management System is Drupal 6
• Bonus: Experimental Drupal 7 amazon machine instance available
on our website; currently undergoing user testing
Friday, June 24, 2011

3. WHAT WENT WRONG
• Background: AWS de-couples the physical hard disk space (called Elastic
Block Storage or EBS) from the CPUs (called “compute instances”)
• late April 2011: an AWS engineer mistakenly routed “backplane” (internal
server traffic) which connects EBS to the CPUS through a system that could
not handle the load
• This triggered an alarm; since everything in AWS is redundant, the systems
thought the backup EBS drives had all failed simultaneously, causing an
overload as the system tried to compensate
• In a nutshell, it’s almost as if the CPUs no longer had hard drives
Friday, June 24, 2011

4. 2009 ARCHITECTURE
• June 2009 architecture focused
on load balancing and database
replication across Amazon
Availability Zones
• SVN machine was also in cloud
• Too reliant on one service
provider (amazon)
Friday, June 24, 2011

5. PRE-OUTAGE ARCHITECTURE
• AWS began a new service called “RDS” for Relational Data Service in 2010.
This was a managed database service -- mySQL -- that was more powerful
and simpler to administer than us doing so ourselves on large servers
• We migrated to RDS in 2010
• The remaining architecture, with the mid-instance front ends and load
balancers, remained the same
Friday, June 24, 2011

6. KEY LESSONS LEARNED
• Amazon’s multiple availability zones failover are not reliable
• Does not imply separate physical or logical facilities!
• Amazon’s poor communication during the outage compounded this problem
• Due to Amazon’s poor initial incidence response communications, we on the spot decided to
create new machine instances (AMIs) in a different geographic zone (US-West vs. US-East) and
copy over the “offsite” one-day-old SVN and DB backups
• Downtime minimized to 1.5 hours; many websites (Reddit, Quora, Foursquare) were down for
days
• Future Worst Case: Amazon goes completely offline. Means we need a very recent full backup of
both WWW and DB instances in a physically and logically separate facility + ability to load balance/
change DNS quickly
• Solution was to scale up Rackspace instances and make daily copies to those servers
Friday, June 24, 2011

7. 2011 ARCHITECTURE
Friday, June 24, 2011

8. WHAT WE RECOMMEND
• get physically and logically separate backup servers
• do nightly full copy backups to the above servers
• have a clear, written process in place for the following things:
• communicating with superiors about what’s happening
• what steps need to be taken to failover
• when the “worst-case” failover plan is implemented (can be time-based or circumstance-based
or both)
• either implement automatic load balancing or (not as good) have complete control over your DNS
• use a very good alerts monitoring service; some of the best ones are cheap/free. We use
binarycanary.com.
Friday, June 24, 2011