Downloaded 10 times
Uploaded byCobus Bernard
DevConf 2020: Resiliency and availability design patterns for the cloud
The document discusses resiliency and availability design patterns in cloud systems, emphasizing the importance of building resilient software to handle failures effectively. It highlights strategies such as multi-AZ architecture, auto-scaling, and chaos engineering, along with the significant costs associated with application downtime. Overall, it provides insights on leveraging AWS infrastructure to enhance application availability and performance.
More Related Content
Similar to DevConf 2020: Resiliency and availability design patterns for the cloud
More from Cobus Bernard
DevConf 2020: Resiliency and availability design patterns for the cloud
- 1. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. O N L I N E - D E V C O N F Z A Resiliency and availability design patterns for the cloud Cobus Bernard Senior Developer Advocate AmazonWeb Services 02.04.20 @cobusbernard cobusbernard cobusbernard
- 2. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved.
- 3. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Can you guess whatwillhappen?
- 4. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Distributed Systems are hard
- 5. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Failures areagivenand everythingwilleventuallyfail over time. Werner Vogels CTO – Amazon.com “ “
- 6. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Resiliency:Abilityfor asystemtohandle and eventuallyrecover from unexpected conditions conditions
- 7. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Partialfailure mode
- 8. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Why do we build resilient software systems?
- 9. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Thecostof downtime Annual Fortune 1000 application downtime costs (IDC) $1.25 to $2.5B Average cost of a data breach (Ponemon Institute) $3.6M Cost/hr of a critical application failure (IDC) $500K to $1M Average cost/hr of downtime (Ponemon Institute) $474K Average cost per lost or stolen record (Ponemon Institute) $141
- 10. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. How do we build resilient software systems?
- 11. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. People Application Network & Data Infrastructure
- 12. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Let’s talk aboutAvailability
- 13. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Systemavailability Availability = Normal Operation Time Total Time MTBF** MTBF** + MTTR* = * Mean Time To Repair (MTTR) **Mean Time Between Failure (MTBF)
- 14. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Reading homework
- 15. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Availabilityinparallel A = 1 – (1 – Ax)2 Part X Part X
- 16. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Availabilityinparallel Component Availability Downtime X 99% (2-nines) 3 days 15 hours Two X in parallel 99.99% (4-nines) 52 minutes Three X in parallel 99.9999% (6-nines) 31 seconds
- 17. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Componentredundancyincreases availability significantly!
- 18. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Global Infrastructure • 22 Regions with 70 Availability Zones • 5 Regions coming soon: Cape Town Jakarta, Milan, Spain, Osaka • 100Gbps redundant network • 99.99% SLA
- 19. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Fully-scaledAvailabilityZone
- 20. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Highlyredundant regional network
- 21. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. AWS Region and availabilityzones Region Availability zone a Availability zone b Availability zone c data center data center data center 1 or more data centers per AZ 2 or more AZs per region (new regions min 3) data center data center data center data center data center data center
- 22. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. How about a global architecture?
- 23. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Once upon a time … Origin
- 24. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. And Now … Origin ~300ms
- 25. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Origin Improve latency for end-users Origin
- 26. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Improve availabilityand disasterrecovery Applications in US West Applications in US East Users from San Francisco Users from New York Service 1 Service 2 Service 3 Service 4 Service 1 Service 2 Service 3 Service 4
- 27. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. So should we go for a global architecture?
- 28. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Perfect your regional architecture first!
- 29. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Let’s talk about Multi-AZ
- 30. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture Region Availability zone a Availability zone b Availability zone c Instances Instances Instances DB Instance DB instance standby Elastic Load Balancing (ELB)
- 31. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture Region Availability zone a Availability zone b Availability zone c Instances Instances Instances DB Instance DB instance standby Elastic Load Balancing (ELB)
- 32. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture Region Availability zone a Availability zone b Availability zone c Instances Instances Instances DB Instance DB instance standby Elastic Load Balancing (ELB)
- 33. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture Region Availability zone a Availability zone b Availability zone c Instances Instances Instances DB Instance DB instance new master Elastic Load Balancing (ELB)
- 34. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture • Enables fault-tolerant applications • AWS regional services designed to withstand AZ failures • Leveraged by AWS regional services such as Amazon S3, Amazon DynamoDB, Amazon Aurora, Amazon ELBs, etc. Region Availability zone a Availability zone b Availability zone c Instances Instances Instances DB Instance DB instance standby Elastic Load Balancing (ELB)
- 35. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Let’s talk about auto scaling
- 36. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Auto-Scaling FixedVariable
- 37. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Availability zone 1 Auto Scaling group AWS Region Availability zone 2 Auto-scaling for self-healing Elastic Load Balancing (ELB) X
- 38. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Let’s talk about the responsibility models
- 39. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. AWSoperationalresponsibilitymodels On-Premises Cloud Less More Compute Virtual Machine EC2 Elastic Beanstalk AWS LambdaFargate Databases MySQL MySQL on EC2 RDS MySQL RDS Aurora Aurora Serverless DynamoDB Storage Storage S3 Messaging ESBs Amazon MQ Kinesis SQS / SNS Analytics Hadoop Hadoop on EC2 EMR Elasticsearch Service Athena Firehose
- 40. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Let’s talk about databases
- 41. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Common resiliencyissueswithDatabases?? REPLICATION BACKUPSSCALING
- 42. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved.
- 43. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR AZ 1 AZ 2 AZ 3 Network RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR Storage Node Leader PutItem
- 44. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR AZ 1 AZ 2 AZ 3 RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR Storage Node Leader GetItem Network
- 45. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon.com, Nike, Netflix, Duolingo, Lyft, Airbnb, Samsung, Toyota, and Capital One depend on the scale and performance of DynamoDB to support their workloads. 10 trillion requests per day 20 million requests per second
- 46. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Purpose-built databases Relational Key-value Document In-memory Graph Time-series Ledger DynamoDB NeptuneAmazon RDS Aurora CommercialCommunity Timestream QLDBElastiCacheDocumentDB
- 47. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Read-Writeseparation Master Read Replica Read Replica Read Replica Instance InstanceInstance
- 48. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. DatabaseFederation Users DB Products DB Master (Read) Replica Master (Read) Replica Instance InstanceInstance
- 49. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. DatabaseSharding User ShardID 002345 A 002346 B 002347 C 002348 B 002349 A CBA Master (Read) Replica Master (Read) Replica Master (Read) Replica Instance InstanceInstance
- 50. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Let’s talk about backups
- 51. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved.
- 52. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. AWS Backup service
- 53. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. PreventingAccidentalTableDeletion https://aws.amazon.com/blogs/database/preventing-accidental-table-deletion-in-dynamodb/
- 54. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. PreventingAccidentalTableDeletion (sql)
- 55. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Practice and testrecoveryfrom your backups!!
- 56. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Let’s talk about timeouts, backoff & retries!
- 57. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Users App DB Conn Pool INSERT INSERT INSERT INSERT What happens if the DB “slows down”? Timeout client side Timeout backend side ??
- 58. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. User 1 App DB Conn Pool INSERT Timeout client side = 10s Timeout backend side = default = Infinite Retry INSERT Retry INSERT ERROR: Failed to get connection from pool Retry
- 59. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. https://docs.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.timeout
- 60. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved.
- 61. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html
- 62. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. @timeout_decorator.timeout(5, timeout_exception=StopIteration) def timed_get(url): return requests.get(url) https://pypi.org/project/timeout-decorator/
- 63. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. How elsecould wehaveprevented theerror? User 1 DB Conn Pool INSERT Retry INSERT Retry INSERT Retry ERROR: Failed to get connection from pool
- 64. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. User 1 DB Conn Pool INSERT Timeout client side = 10s Timeout backend side = 10s Wait 2s before Retry INSERT INSERT Wait 4s before Retry Wait 8s before Retry Wait 16s before Retry Backing off betweenretries Releasing connectionsBackoff
- 65. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. No jitter With jitter https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ SimpleExponentialBackoffisnotenough:AddJitter
- 66. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Example: add jitter0-1000ms def get_item(self, url, n=1): MAX_TRIES = 12 try: res = requests.get(url) except: if n > MAX_TRIES: return None n += 1 time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0)) return self.get_item(url, n) else: return res
- 67. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Idempotent operation No additional effect if it is called more than once with the same input parameters.
- 68. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Let’s talk about health checking!
- 69. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Auto Scaling group Service A Availability zone 1 Auto Scaling group AWS Region Service A Availability zone 2 Service BService B database Email Probing for health Cluster
- 70. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Email database Shallowhealthcheck Instance Cache node Cluster Are you healthy? yes
- 71. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. database Shallowhealthcheck Instance Cache node Cluster Are you healthy? yes Email
- 72. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Email database Deep healthcheck Instance Cache node Cluster Are you healthy? yes Are you healthy? yes yes yes yes
- 73. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Email database Deep healthcheck Instance Cache node Cluster Are you healthy? no Are you healthy? no yes yes yes
- 74. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Prioritize shallow health checks during hard times. Cache.
- 75. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Let’s talk about load shedding.
- 76. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved.
- 77. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved.
- 78. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Cheaply reject excess work
- 79. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved.
- 80. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Be careful when selecting the right metric
- 81. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Don’tbeoverlyoptimisticandtakeonmorethanyoucan. Findanoperationalmetrictorejectwhatyoucannottakein. Favorcachedandstaticcontent PrioritizeELBhealthcheck(shallow)pings Inanoverloadsituationyouhavepreciousresources,donotletany ofitgotowaste. Load Shedding
- 82. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Let’s talk aboutresiliency (chaos) engineering
- 83. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Fire Drills
- 84. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Chaosengineering https://github.com/Netflix/SimianArmy
- 85. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. “Chaos Engineeringis the discipline of experimentingon a distributedsystem in orderto buildconfidence in the system’s capabilitytowithstand turbulentconditionsin production.” http://principlesofchaos.org
- 86. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Failureinjection • Start small & build confidence • Application level • Host failure • Resource attacks (CPU, memory, …) • Network attacks (dependencies, latency, …) • Region attacks • “Paul” attack https://www.gremlin.comhttps://github.com/Netflix/SimianArmy https://chaostoolkit.org
- 87. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. STEADY STATE HYPOTHESIS RUN EXPERIMENT VERIFY FIX! PhasesofChaosEngineering
- 88. Thank you! © 2020,Amazon Web Services, Inc. or its affiliates. All rights reserved. Cobus Bernard Senior DeveloperAdvocate Amazon Web Services O N L I N E - D E V C O N F Z A @cobusbernard cobusbernard cobusbernard