Come join us as we take a deeper look at Amazon's approach to releasing mission-critical software. In this session, we take a journey through the release process of an AWS Tier 1 service on its way to production. We follow a single code change from idea to release, and we focus on how Amazon updates critical software quickly and safely for its global customers. Throughout the talk, we demonstrate how our internal software release processes map to AWS developer tools, and we highlight how you can leverage AWS CI/CD services to create your own robust release process.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Releasing Mission-Critical
Software at Amazon
D E V 2 0 9 - R
Andy Troutman
Senior Manager, AWS Deployment Services
Amazon Web Services

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why do
you care?

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
To PRODUCTION!

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
It’s more sausage making than magic.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
We’re pushing a lot of change
Executing thousands of
deployments at any given second.
99.996% of deployments execute
without problem.
We find this number unacceptable
and are always pushing to get to
100%

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Much of what enables
Amazon is in our tooling,
and we’re releasing most
of it to AWS customers.
You can build a developer
experience SUPERIOR to
Amazon.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s Meet Our Hero….
hello world.

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon’s Organizational Structure
8 Engineers is still
the average sized
team at Amazon.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DevOps and Ownership
1. A federation of startups
2. Everyone owns the product end to end

11. Amazon’s Service Oriented Architecture
Oooh that feels
niiiice.
EC2IAM

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
My one slide on the local development process
Yeee Ha
• Agile… ish?
• “The Process” will ensure you
don’t cut corners.
• Iterate as quickly as you can.
• Your host is a service.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key Callouts
Fund WinnersBuilding Blocks Marketplace

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our Antagonist
It’s me, THE
PROCESS.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A change is born.
• Replication
• Access Control
• Code Search on Commit
• AWS CodeCommit gives you a lot
of these same advantages (BTW)

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CodeReview and Pull Request
Does this
thing make
me look fat?
• The last step before “the
process” takes over.
• Teams can block commits to
mainline via a variety of
rules.
• You find very little branching
strategy at Amazon.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Build
Global Dependency Closure
Unit Testing
Static Analysis
Bad Software Blocked

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Begin with the Pre-Mortem

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

22. Part 2 – The Journey
Test
Integration
Production
Ugh Oh.

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
20 Regions (5 more coming)
58 Availability Zones
144 CloudFront PoPs
Did I mention we also deploy for Amazon?

24. Release Automation
This is going to be a long road trip.
It’s beautiful.

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release Primitives
Time Windows Test Tasks
Global Andon Cords
Manual Approval
Attribution and Roles
Neat.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Changing Software
Auto-Rollback

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Changing Software
Incremental Release

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Changing Software
Health Tracking

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Changing Software
Change Providence

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Changing Software
Auto-Rollback
Incremental Release
Change Providence
Health Tracking
I don’t feel
trusted.
You’re not!

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Infrastructure Automation
• 1-Day region builds
• Infra and OS
pipelined like
software
• Lock in best practices
with our own
templating engine

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
If All Goes Well…
I Made it!

34. Auditing and Reducing Risk
My job is
never done.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

36. Key Enablers to Our Success
• Amazon standardizes a ton of the software
development process via common
platforms.
• We encourage an open internal marketplace
for extending that platform, and fund
winners.
• We ship so much because we continually
ship everything to production, not because
we go crazy fast.
• Amazon is doubling down on it’s AWS tools
to run it’s own business.

37. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Please leave feedback on the
talk!
Andy Troutman