Come join us as we take a deeper look at Amazon’s approach to releasing mission critical software. In this session, we will take a journey through the release process of an AWS Tier 1 service on its way to production. We’ll follow a single code change throughout the entire process from idea to release, and focus on how Amazon updates critical software quickly and safely for its global customers. Throughout the talk we’ll demonstrate how our internal software release processes map to AWS Developer tools, highlighting how you can leverage AWS’s CI/CD services to create your own robust release process.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
DEV206 Life of a Code Change to a
Tier 1 Service
A n d y T r o u t m a n , S e n i o r M a n a g e r A W S D e p l o y m e n t S e r v i c e s

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why
do
you
care?

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
To
PRODUCTION!

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It is rather impressive…
50 MM Deployments in a year
10’s of thousands of builders.
That comes to more than a software
deployment every second

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It’s more
sausage
making than
magic.

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Much of what enables
Amazon is in our
tooling, and we’re
releasing most of it to
AWS customers.
You can build a
developer experience
SUPERIOR to Amazon.

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s Meet Our Hero….
hello world.

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon’s Service Oriented Architecture
8 Engineers is still
the average sized
team at Amazon.

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DevOps and Ownership
1. A federation of startups
2. Everyone owns the product end to end

11. Amazon’s Service Oriented Architecture
Oooh that feels niiiice.
EC2IAM

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
My one slide on the local development
process
Yeee Ha
• Agile… ish?
• “The Process” will ensure you
don’t cut corners.
• Iterate as quickly as you can.
• Your host is a service.

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building
Blocks
Key Callouts
Fund
Winners
Marketplace

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Our Antagonist
It’s me, THE
PROCESS.

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A change is born.
• Replication
• Access Control
• Code Search on Commit
• AWS CodeCommit gives you a lot
of these same advantages (BTW)

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CodeReview and Pull Request
Does this
thing make me
look fat?
• The last step before “the
process” takes over.
• Teams can block commits to
mainline via a variety of rules.
• You find very little branching
strategy at Amazon.

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Build
Global Dependency Closure
Unit Testing
Static Analysis
Bad Software Blocked

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Begin with the Pre-Mortem

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

22. Part 2 – The Journey
Test
Integration
Production
Ugh Oh.

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
15 Regions
44 Availability Zones
100 Cloudfront PoPs
Did I mention we also deploy for Amazon?

24. Release Automation
This is going to be a long road trip.
It’s beautiful.

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Global Andon Cords
Manual Approval Time Windows Test Tasks
Release Primitives
Attribution and Roles
Neat.

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changing Software
Auto-Rollback

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changing Software
Incremental Release

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changing Software
Health Tracking

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changing Software
Change Providence

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changing Software
Auto-Rollback
Incremental Release
Change Providence
Health Tracking
I don’t feel
trusted.
You’re not!

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Infrastructure Automation
• 1-Day region builds
• Infra and OS pipelined
like software
• Lock in best practices
with our own
templating engine

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
If All Goes Well…
I Made it!

34. Auditing and Reducing Risk
My job is
never done.

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

36. Key Enablers to Our Success
• Amazon standardizes a ton of the software
development process via common platforms.
• We encourage an open internal marketplace
for extending that platform, and fund winners.
• We ship so much because we continually
ship everything to production, not because
we go crazy fast.
• Amazon is doubling down on it’s AWS tools
to run it’s own business.

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!
I ’ l l s t i c k a r o u n d t o a n s w e r q u e s t i o n s .
P l e a s e l e a v e f e e d b a c k o n t h e t a l k !