Chaos engineering & Gameday on AWS

1. Game day & Chaos Engineering Concepts on AWS

2. Gameday & Chaos Engineering Concepts on AWS

3. Game day & Chaos Engineering Concepts on AWS •Famous Disasters •Catastrophes common thread •Ingredients for Catastrophes •Classess of issues •Fragility vs Resilience •Resilience vs Antifragility • GameDay Concept • AWS GameDay with the AWS Well-Architected Framework • Example GameDay Plan • Chaos Experiments • Resource Exhaustion • The Network is Not Reliable • Datastore saturation • DNS Unavailability

4. Game day & Chaos Engineering Concepts on AWS • Chaos Engineering • Examples of inputs for chaos experiments: • How Does Chaos Engineering Differ from Testing? • Software Tools for Chaos Engineering • Chaos Toolkit • Simian Army • Chaos Monkey •Chaos Monkey Basic Demo on AWS

6. Game day & Chaos Engineering Concepts on AWS In February 2017, Cloudflare faced a major software bug that led to sensitive customer data like passwords, cookies and authentication tokens to get leaked from customer websites. Bitcoin Unlimited suffered a serious memory leak which caused several nodes to fall from 800 to about 300. This is almost 70 percent of the nodes run by Bitcoin Unlimited at the time.

7. Game day & Chaos Engineering Concepts on AWS Less than a week into 2016, HSBC became the first bank to suffer a major IT outage. Millions of the bank's customers were unable to access online accounts. Services only returned to normal after a two-day outage. The bank’s chief operating officer Jack Hackett blamed a “complex technical issue” with its internal systems. In June 2015 about 600,000 payments failed to enter the accounts of RBS overnight – including wages and benefit payments. Many took several days to come through. The bank’s chief admin officer said a “technology fault meant we could not ingest a file from a third-party provider”

10. Game day & Chaos Engineering Concepts on AWS Catastrophes

19. Game day & Chaos Engineering Concepts on AWS GameDay Concept • GameDay is a hands-on competition with live data. By setting up a “fake-real” environment for the teams to work in, players are free to try things out in a live environment with no actual risk. It’s unusual to have such a “carefree experience working in a live environment. Ironically, though, the one-day time frame is actually “kind of punitive. In a real situation you’d probably have more time to deal with each problem, but the high-pressure environment adds to the excitement, and AWS GameDay’s goal is to entertain and make the learning experience as fun as possible.

20. Game day & Chaos Engineering Concepts on AWS • How Are You Managing Keys? • How Are You Planning for Disaster Recovery? • How Does Your System Withstand Component Failures? Looking at GameDay through the lens of the Well-Architected Framework, it was obvious that there were many opportunities for improvement. The AWS review team prioritized the findings into two sets: critical and recommended. Most of the findings were classified as recommended—these don’t pose an immediate risk and will be incorporated into our roadmap. However, the three elements that were identified as critical needed to be addressed immediately.

21. Game day & Chaos Engineering Concepts on AWS  Target In any GameDay, an exact target or targets should be specified. Without it, it’s impossible to bring in the right people to run and observe the GameDay.  Time And Place Things to include: • Precise date • War room for in-person attendance (make sure it fits enough people) • Dial-in information - including conferencing link, phone number and code to join  Goals Of The GameDay When planning, we need a goal for the GameDay. This is to ensure that we create relevant test cases. Sometimes the goal is to replay as many previous production impacts as possible, to test whether or not the current systems are more or less resilient.

22. Game day & Chaos Engineering Concepts on AWS  Whiteboarding • With so many great minds present and aligned on the goal of the GameDay, it’s the perfect time to whiteboard out a system’s architecture. This session helps paint a clear picture of what we’re about to break, and makes obvious some areas worth testing.  Test Cases Scoping • Test cases are developed to help answer the question, “What could go wrong?” or “Do we know what will happen if this breaks?” As a team looks at the architecture that’s on the whiteboard, you will start to identify areas of concern.

23. Game day & Chaos Engineering Concepts on AWS  Make a plan • Role play and scope definition • Create the simulation environment • Set a deadline • Create the GameDay Environment • Activate AWS CloudTrail ; Gameplay recording and auditing • Simulate activity

24. Game day & Chaos Engineering Concepts on AWS Resources on computers are finite. A machine/VM/container will inevitably hit a resource limit at some point, and the application will be forced to handle the lack of a resource. Commonly, this is CPU, Memory, or I/O. We can reproduce CPU exhaustion by conducting a chaos experiment. Running this experiment will consume CPU cycles, leaving the application with the same amount of customer-facing work, and less CPU to do it with. As always, we advocate starting small on a single instance, then increasing the blast radius as confidence grows. Common reactions to CPU exhaustion are an increase in errors and latency and a reduction in successful requests to customers.  Attack: CPU / Memory / Disk  Scope: Single instance  Expected Results: Rate of good responses goes down, errors increase at all layers, brownout mode entered (if implemented), alerts fire (if configured at single-instance level), load balancer routes traffic away (if applicable)

25. Game day & Chaos Engineering Concepts on AWS Network Latency Network dependencies are a fact of life in a distributed system, and as distributed systems are growing in adoption AND complexity, chaos engineering becomes an optimal way to test for potential failures on the path to increasing resilience.  Attack: Network Blackhole / Latency  Scope: single instance  Expected Results: Traffic to dependency goes to 0 (or gets slow), startup completes without errors, application-level metrics in steady-state are unaffected, traffic to fallback systems shows up and is successful, dependency alerts and pages may fire (if scoped to single-instance)

26. Game day & Chaos Engineering Concepts on AWS All interesting applications have some sort of storage, and managing the relationship between application and datastore is critical to overall system health. There are a variety of ways that an application may overwhelm a data store (poor queries, lack of indices, bad sharding, upstream caching decisions, etc), but all of them result in what appears to be an unresponsive data layer. It’s important to understand how datastore saturation manifests in your application. There are a few ways of modeling this with a Chaos Experiment. You can blackhole your datastore, making it appear completely unavailable. You can add latency to requests to your datastore, making it appear slow. Finally, you can consume I/O bandwidth to simulate a congested path to the datastore.  Attack: Network Blackhole / Latency / IO  Scope: single instance  Expected Results: Traffic to datastore is reduced or slower, application-level metrics in steady-state are unaffected to the degree possible, traffic to fallback systems shows up and is successful, timeouts and concurrency limits kick in when appropriate, alerts and pages may fire (if scoped to single-instance)

27. Game day & Chaos Engineering Concepts on AWS The best way forward is to induce a DNS outage and understand how your application behaves. If you blackhole DNS traffic on a single instance, it will appear to that instance as if DNS is unavailable. The fixes will vary depending on the issue, but common solutions are to pass around IP addresses instead of hostnames for internal addressing and the use of a backup DNS provider.  Attack: DNS blackhole  Scope: single instance  Expected Results: Inbound traffic may drop, traffic to external systems may fail, startup may not complete successfully

28. Game day & Chaos Engineering Concepts on AWS What happened? Was that expected? What do we do next? After tests are run, it’s good to take some time to wind down, then have a follow up recap. This should be done relatively soon after the GameDay (days, not weeks), as the experience is still fresh for everyone.  How long does it take to detect an event?  Simulate failure situation  Validate assumptions  Prove your architecture  Knowledge of Procedures  How is the communication channel or chain of command during the game?

29. Game day & Chaos Engineering Concepts on AWS Chaos Use Cases  Simulating the failure of an entire region or datacenter.  Partially deleting Kafka topics over a variety of instances to recreate an issue that occurred in production.  Injecting latency between services for a select percentage of traffic over a predetermined period of time.  Function-based chaos (runtime injection): randomly causing functions to throw exceptions.  Code insertion: Adding instructions to the target program and allowing fault injection to occur prior to certain instructions.  Time travel: forcing system clocks out of sync with each other.  Executing a routine in driver code emulating I/O errors.  Maxing out CPU cores on an Elasticsearch cluster.

30. Game day & Chaos Engineering Concepts on AWS The primary difference between Chaos Engineering and these other approaches is that Chaos Engineering is a practice for generating new information, while fault injection is a specific approach to testing one condition. Tests are typically binary, and determine whether a property is true or false. Strictly speaking, this does not generate new knowledge about the system, it just assigns valence to a known property of it. Experimentation generates new knowledge, and often suggests new avenues of exploration.

31. Game day & Chaos Engineering Concepts on AWS 01 02 05 03  VaryReal-World Events  Canary Analysis  Hypothesize about SteadyState  Automate Experiments toRun Continuously 04  Run Experiments in Production

35. Game day & Chaos Engineering Concepts on AWS Chaos Monkey

Chaos engineering & Gameday on AWS

Bilal Aybar

Chaos engineering & Gameday on AWS