Failure is not an Option - Designing Highly Resilient AWS Systems

P U B L I C S E C T O R
S U M M I T
Wa shingto n, D C

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Failure is not an Option
Designing Highly Resilient AWS
Systems
Tim Griesbach
Manager, Solutions Architecture
AWS WWPS
3 0 2 9 5 5

S U M M I T
Agenda
What are we planning for? Risk and Resiliency requirements.
Think resiliently. Principles of Resiliency
Resilient design. System, Test, and Operations patterns & best practices

S U M M I T
“Everything fails, all the time”
- Werner Vogels
(CTO, Amazon.com)

S U M M I T
Resiliency is the ability for a
system to recover quickly and
continue operating even when a
failure occurs

S U M M I T
Push for resiliency
IT failures lead to broad society impact
• Government Services, Airline, Financial, Communications
Reputation / Legal
• More and more people depend on IT systems for everything.
$$$
• Lost productivity, idle time of people dependent on system
• Lost productivity, time putting out fires and recovering
• Lost revenue from system

S U M M I T
What are we planning for?

S U M M I T
Consider each applications
significance to your business,
and the potential impact if a disruption

S U M M I T
Cause Examples Probability
Operator error Manual operator error HIGH
Deployment induced Software, hardware, network, or configuration deployment
Both automated and manual changes
HIGH
Load induced Change in behavior, either of a specific caller or aggregate
Service reaching a tipping point
Load failures can occur in the network
Denial of service (DDoS)
HIGH
Data induced Data accepted by the system that it can’t process (“poison
pill”)
MED
Credential expiration Expiration of a certificate or credentials MED
Hardware failure Any hardware component in the system, i.e. hosts, storage,
network, or elsewhere.
LOW
Infrastructure Power feed or environmental conditions LOW

S U M M I T
On-premise data center realities
• Traditional DR “weekend tests”
• Connected to the internet? Exposed to same external attacks
• DR site is always “ACTIVE” and hence you are paying for resources
• Data is constantly replicated
• Datacenter security compliance is expensive and resource intensive

S U M M I T
Think Resiliently. Principles of
Resiliency

S U M M I T
A V A I L A B I L I T Y , R E L I A B I L I T Y ,
A N D R E S I L I E N C E
IN 21ST CENTURY ARCHITECTURES
Test recovery procedures
Automatically recover from failure
Scale horizontally to improve availability
Stop guessing capacity
Manage change through automation

S U M M I T
What do those 9’s really mean?
Availability
Max disruption
(per year)
Max disruption
(per month)
Max disruption
(per month)
99% 3 days 15 hours 7.31 hours 14.4 minutes
99.9% 8 hours 45
minutes
43.83 minutes 1.44 minutes
99.95% 4 hours 22
minutes
21.92 minutes 43.2 seconds
99.99% 52 minutes 4.38 minutes 8.64 seconds
99.999% 5 minutes 26.3 seconds 864 milliseconds

S U M M I T
Disaster
Recovery point
Data loss
Recovery time
Down time
Time
Recovery Point and Recovery Time Objective

S U M M I T
Classification
Security Policy
Customer Provided and Managed Controls
Encryption
Governance
ITDaM
ITSM
Monitoring
Operations
Malware
Risk
Management
You control how you manage your own risks
AWS Managed and Audited Controls
SOC 1 SOC 2 PCI-DSS NIST 800-53 ISO 27001
AWS Provided, Customer Configured and Managed Controls
Virtual Private
Cloud
Key
Management
Logging Other AWS features and services
Customer Risk Appetite and Desired Control Environment
Business Risks Sourcing Risks
Technology
Risks
Security Risks ComplianceAWSCustomers

S U M M I T
Design Systems Resiliently

S U M M I T
A complex system that works is invariably found
to have evolved from a simple system that worked.
G A L L ’ S L A W

It’s not binary.
Start somewhere
and scale up.

S U M M I T
“There is no compression algorithm for experience.”
AWS has had 13+ years to build the world’s most reliable, secure, scalable, and
cost-effective infrastructure.
• Your operational DNA has to be crafted for reliability.
• Service SLAs between 99.9% and 100% availability
• Amazon S3 is designed for 99.999999999% durability
• AWS Availability Zones exist on isolated fault lines, flood plains, networks, and local electrical grids to
substantially reduce the chance of simultaneous failure.
• Disaster is inevitable; automation + redundancy = availability.
We are driven to remove any and all causes of failure. Our goal is to make our operational
performance indistinguishable from perfect.

S U M M I T
A W S
R E G I O N A L
E X PA N S I O N
23 Regions and 67 AZs 4 New Regions and 12 AZs
2 GovCloud Regions Today New GovCloud, TS, and Secret Regions
Coming Soon

S U M M I T 25
Amazon Global Network
AWS Region with Multiple Edge Locations
Amazon CloudFront PoPs
AWS Direct Connect Location
96 AWS Direct Connect locations
Customers can reach every public AWS Region from
the local Direct Connect location (except China)
A W S
C O N N E C T I V I T Y
O P T I O N S

S U M M I T
Resilient AWS Cloud
Infrastructure
Regions, AZs, Networking
Service Design
Cell-based architecture
Multi-Az architecture
Micro-service architecture
Distributed systems best practices
Understand the AWS Services scope
Single AZ, Regional, Global, Cross-Reginal
capability Figure 3

S U M M I T
Resilient Networking
Networking is foundation
Packets must get from point-a to point-b
Ensure network supporting your applications is appropriately
redundant, always available, and seamlessly routed.
AWS provides a global infrastructure with 20 Regions and
61 Availability Zones (at the time of publication)
AWS services
Amazon EC2 networking
Amazon Virtual Private Cloud (VPC), VPC Peering, VPC Sharing
AWS Gateways for external, internal and back to on-premise routing (VPN, Transit)
DNS (Route53)
Elastic Load Balancer (ELB)

S U M M I T
Resilient Data
Must have confidence in the resilience of your data
Many forms: filesystem, block storage, databases, in memory caches
Consider how eventual consistency impacts design
AWS services
Amazon S3 cross-region replication
Cross region snapshots (Amazon EBS volumes)
Amazon RDS cross region replicas
AWS Storage & File Gateway
Amazon FSx for Windows and Lustre
Figure 10

S U M M I T
Self-Healing applications
Highly resilient applications must be able to self-heal.
How
Leverage Microservices app architecture
Decouple inter-dependencies, loose coupling
Remove state from app components
AWS services
Elastic Load
Balancing
AWS Auto Scaling Amazon Simple
Queue Service

S U M M I T

S U M M I T
Single Region: Multi AZ
Start here before adopting more complex architecture
Only consider multi-region if requirements dictate
Pros
Availability of AWS region-wide services include
Amazon S3, Amazon DynamoDB, Amazon EFS,
Amazon SQS, Amazon Kinesis
Much less complexity in design, implementation, and
operations.
Cons
If you need >99.9% resiliency, consider multi-region.
May not meet needs of regulators

S U M M I T
Multi-Region: Active-Standby
Traditional DR Pattern
Backup env used in event of failure only
Pros
For Apps which cannot use native AWS features
Least # changes to the application
Cons
Delays while Standby becomes Active (hrs)
RPO limited by replication lag
AWS Services
Amazon RDS Amazon Route 53

S U M M I T
Multi-Region: Active-active
Both stacks active, traffic distributed
Data replication critical, must consider latency impacts
Pros
Zero RTO
Works well for apps that can partition users
Cons
Data replication must be handled by Applications
AWS Services
Storage replication from APN partners
Amazon RDS Amazon DynamoDB Amazon Aurora AWS Database
Migration Service
Amazon Route 53

S U M M I T
Multi-Region: Dual-write
Shared nothing architecture – all TX processed in
duplicate/parallel
Good for legacy applications
Pros
Zero RPO
Little/No change to apps in each region
Cons
Requires checkpointing
Reconciliation jobs to ensure sites in sync
Downstream apps must avoid duplicates
AWS Service
AWS Lambda Amazon Route 53

S U M M I T
Anti-Patterns
• Replicate on-premise problems & patterns to the cloud
• Use of Non-redundant architectures to meet schedules
• Single datacenter (Availability Zones) architectures
• Reusing manual processes
• Data retention practices, Failover & Scaling
• Responding to monitoring alerts and metrics (vs self-healing, auto scaling)
• Assuming data is safe in your data center
Don't sacrifice long-term value
for short-term results

S U M M I T
Resilient operations, often
overlooked

S U M M I T
Operations is key pillar in resiliency
Success in operations means you
• Successful & consistent implementation of changes
• Have insight to operational health
• Have insight to achievement of business outcomes
• Respond in timely and effectively to events impacting the application
How?
• Perform operations as code
• Annotated documentation
• Make frequent, small, reversible changes
• Refine operations procedures frequently
• Anticipate failure
• Learn from operational failures

S U M M I T
Operations monitoring
Must detect failures fast
Applications emit telemetry to detect
Processes defined and understood
AWS Services
Amazon CloudWatch
AWS Personal
Health Dashboard

S U M M I T
Operations automation
“A key mechanism to achieve this is to automate
the management as much as possible, removing
error prone, manual operations.” - Werner Vogels

S U M M I T
Application deployment
Infrastructure as code – integrates infrastructure and application change processes
Examples include staged deployment, canary deployments, isolation zone deployments, and
automatic roll back
AWS Services
AWS CodeBuild
AWS CodeCommit
AWS CodeDeploy
AWS CodePipeline

S U M M I T
Testing enforces resiliency

S U M M I T
Application testing & certification
Resilient system continue to operate successfully in the presence of failures.
Failure Mode Effect Analysis (FMEA) – industry standard technique
Estimate risk priority number (RPN) between 1 and 1000
Rank probability, severity, and observability on a 1-10 scale, where 1 is good and 10 is bad, and multiplying them.
Perfectly low probability, low impact, easy to measure risk has an RPN of 1.
Extremely frequent, permanently damaging, impossible to detect risk has an RPN of 1000
Failure impact analysis
Failure Effect Mitigation Result
Failure of an AZ
Temporary capacity
reduction
Automatic failover to secondary
AZ
Temporary performance
degradation
Total failure of satellite region Data replication offline
Repair/reconfigure replication
using alternate region
No service interruption
Partition of network between regions Data replication offline
Auto recovery when network is
available
No service interruption
Total failure of primary region Service Offline Failover to secondary region
Service restored within two
hours

S U M M I T
Chaos engineering
Cloud has ushered in new method of testing
Principles of Chaos Engineering – “Chaos Engineering can be thought of as the facilitation of
experiments to uncover systemic weaknesses.” https://principlesofchaos.org/
Principles
Building a hypothesis around steady state behavior
Applying variations to simulate real world events
Run experiments in production
Automate the experiments to run continuously
Minimize blast radius of failures

S U M M I T
Continuous Testing of Infrastructure
Regularly execute tests in stable, production & production-like test environments.
Treat Infrastructure as Code
• CI/CD Test in Infrastructure Build Pipeline
• Testing of infrastructure during Integration Test

S U M M I T
Future Considerations
• Consider serverless, reduces maintenance and moves the responsibility of the
resilient design to AWS.
• Take advantage of our distributed systems by building on top of them – Amazon
S3/AWS Lambda/Amazon ECS.
• Break systems down into smaller pieces along logical seams. Reduce the blast
radius of a failure of any individual piece of the system
• Leverage Well Architected tool to assess your applications -
https://aws.amazon.com/well-architected-tool

S U M M I T
Additional Resources
• AWS Well Architected https://aws.amazon.com/architecture/well-architected
• AWS Whitepaper: Building Mission-Critical Financial Services Applications on
AWS, April 2019
• re:Invent 2018: Close Loops & Opening Minds: How to Take Control of Systems,
Big & Small-https://www.youtube.com/watch?v=O8xLxNje30M
• Building Microservices: Designing Fine-Grained Systems
• AWS re:Invent 2018: Architecture Patterns for Multi-Region Active-Active
Applications (ARC209-R2)

Thank you!
S U M M I T
Tim Griesbach
awstim@amazon.com

Failure is not an Option - Designing Highly Resilient AWS Systems

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Failure is not an Option - Designing Highly Resilient AWS Systems

Similar to Failure is not an Option - Designing Highly Resilient AWS Systems (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Failure is not an Option - Designing Highly Resilient AWS Systems