This document discusses using "Game Days" to simulate crisis situations and test systems, applications, and teams. It recommends creating a simulated environment to practice deployments, failures, and responses. Key aspects include role playing different scenarios, recording results, and debriefing to identify issues and improve procedures. The goal is to validate architectures and assumptions, prove resilience, and train teams through a fun but realistic learning experience.

1. @caarlco
GAME DAYS
CRASH TEST YOUR APPLICATION…
AND YOUR TEAM

2. 2008
CONTINUOUS DEPLOYMENT

4. THEORY OF CONSTRAINTS
1. THROUGHPUT
2. INVENTORY
3. OPERATIONAL EXPENSE

5. “At Amazon, the goal is to transform
new ideas into high quality production
systems that provide customer value”

6. 1. THROUGHPUT
Number of ideas successfully transformed
into high quality prod systems providing
customer value over a unit of time

7. 2. SOFTWARE INVENTORY
Code you created, time and money
invested. If not deployed into prod, no
return on investment.

8. 3. OPERATIONAL EXPENSE
Get lean: eliminate waste. Every process
that is not automated is waste.

9. FASTER IS ALWAYS
BETTER

10. 11.6s
Mean time between
deployments
(weekday)
1,079
Max number of
deployments in a
single hour
10,000
Mean # of hosts
simultaneously
receiving a
deployment
30,000
Max # of hosts
simultaneously
receiving a
deployment
DEPLOYMENTS AT
AMAZON.COM
(2011)

12. MODULES
& LIBRARIES
LANGUAGE
INTERPRETERS
OBJECT-ORIENTED
PROGRAMMING

13. MODULES
& LIBRARIES
LANGUAGE
INTERPRETERS
OBJECT-ORIENTED
PROGRAMMING
CONTAINERS
VIRTUAL
MACHINES
MICRO SERVICES
ARCHITECTURES

14. SCALABILITY
MAINTAINABILITY
RELIABILITY
DURABILITY
CONFIGURABILITY
...

15. RESILIENCE
Ability to provide and maintain an
acceptable level of service in the face of
faults and challenges to normal operation

16. RESILIENCE
Capacity to anticipate disruptions,
adapt to events, and deliver value

17. RESILIENCE
Ability to cope with change

18. ADAPTABILITY

19. HUMAN OPS
ARE PART OF THE SYSTEM

20. BEST ARCHITECTURES
HAVE THE BEST OPS

21. HOW TO TRAIN
THE BEST OPS?

22. PLAY IN A SIMULATED
ENVIRONMENT

26. IF SOMETHING IS HARD
REPETITION MAKES IT EASIER

27. SIMULATION ENVIRONMENT
FOR CRISIS SITUATIONS

28. GOOD WEATHER DOESN’T MAKE GOOD SAILORS

29. 2000
GAME DAYS

30. 1. PREPARE
SETTING THE STAGE

31. PROCTORS & PLAYERS
ROLE PLAY

32. MAKE A PLAN
WHAT ARE YOU GOING TO TEST?

33. PRE-MORTEM
BRAINSTORMING TECHNIQUE
IMAGINE HOW THINGS CAN FAIL

34. CREATE THE SIMULATION
ENVIRONMENT

35. ON-DEMAND
PAY AS YOU GO
ELASTIC

36. +
=
PROGRAMMABLE PLATFORM
AWS APIs

37. AWS CLOUDFORMATION
STACK-BASED PROVISIONING
SERVICE

38. CLOUDFORMATION
TEMPLATE

39. {
"Description"
:
"Create
RDS
with
username
and
password",
"Resources"
:
{
"MyDB"
:
{
"Type"
:
"AWS::RDS::DBInstance",
"Properties"
:
{
"AllocatedStorage"
:
"500",
"DBInstanceClass"
:
"db.m1.small",
"Engine"
:
"MySQL",
"EngineVersion"
:
"5.5",
"MasterUsername"
:
"MyName",
"MasterUserPassword"
:
"MyPassword"
}
}}}

40. "AWS::CloudFormation::Init"
:
{
"config"
:
{
"packages"
:
{
"yum"
:
{
"mysql"
:
[],
"mysql-­‐server"
:
[],
"httpd"
:
[],
"php"
:
[],
"php-­‐mysql"
:
[]
}},
"sources"
:
{
"/var/www/html"
:
"https://s3.amazonaws.com/my-­‐builds/build-­‐v4.zip"
}}}

41. INFRASTRUCTURE
AS CODE

55. SIMULATE ACTIVITY

56. REPLAY PRE-RECORDED NETWORK ACTIVITY

57. “PRODUCTION IS TRULY
THE ONLY PLACE
YOU CAN VALIDATE YOUR CODE”

58. (THROTTLED) LOAD-TEST
USE AMAZON EC2 SPOT INSTANCES

59. APACHE BENCH – AB
APACHE JMETER – JAVA, DBs
LOCUST.IO – PYTHON
THE GRINDER – JAVA

61. USING AMAZON EC2 TO SIMULATE
2.4 MILLION PLAYERS

62. MAKE IT FUN
SET A DEADLINE

63. 2. PLAY!
THE FUN PART

64. SIMULATE FAILURES

65. REMINDER
COMPARED TO REAL LIFE, YOU’RE
NOT SADISTIC ENOUGH

66. • TERMINATE RESOURCES
• CHANGE SECURITY GROUPS
• CHANGE IAM ROLES
• DISABLE IAM USER
• CHANGE /ETC/HOSTS FILE
• AMAZON RDS FAIL-OVER TEST

68. IF YOU DON’T MEASURE IT
IT DIDN’T HAPPEN

69. RECORD & ANNOTATE
PLAYER’S ACTIONS

70. AWS CLOUDTRAIL
GAMEPLAY RECORDING & AUDITING

72. 3. DEBRIEF
WHERE THE MAGIC HAPPENS

74. HOW LONG DOES IT TAKE
TO DETECT AN EVENT?
TO TAKE ACTION?

75. WHICH LATENT DEFECTS
DID YOU EXPOSE?

76. COMMUNICATION
CHANNELS
ONLINE CHAT, CRIS ROOM,
SHARED DOCS, ETC…

77. ROOKIE DAY
GAME DAY FOR NEW HIRES

78. STAR-LESS DAY
GAME DAY WITHOUT YOUR
ROCK-STAR TEAM MEMBERS

79. VALIDATE YOUR ASSUMPTIONS
PROVE YOUR ARCHITECTURE
KNOW YOUR PROCEDURES
LEARN FROM YOUR FAILURES

80. SUCCESS MAKES PEOPLE
FEEL GOOD
FAILURE MAKES PEOPLE
BETTER

81. LET ME KNOW ABOUT
YOUR GAME DAYS
@caarlco

82. @aws_actus