Organizations around the globe are leveraging the cloud to accomplish world-changing missions. This session will address how AWS can help organizations put more money toward their mission and scale outreach and operations to achieve more with less. Hear some of the most advanced AWS customers on how their organizations handle DevOps, continuous integration, and deployment. Learn how these practices allow them to rapidly develop, iterate, test, and deploy highly scalable web applications and core operational systems on AWS. The discussion will focus on best practices, lessons learned, and the specific technologies and services these customers use.

1. NEW YORK
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

2. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
DevOps, Continuous Integration and
Deployment on AWS
Leo Zhadanovsky, Senior Solutions Architect, AWS
@leozh
Ed Zarecor, DevOps Lead, EdX

3. DevOps

4. What is DevOps?
• « DevOps is the practice of operations
and development engineers participating
together in the entire service lifecycle,
from design through the development
process to production support »
- theagileadmin.com

5. Continuous Integration

6. What is Continuous Integration?
• Changes to code automatically deployed
to mainline branch
– After passing unit and mock tests
• Makes changes to code, and deployments
iterative, not monolithic
• Bugs are detected quickly
• Helps automate deployments
• Allows rapid development and deployment

7. SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
DEVELOPER
PICK
TASKS
SUBMIT
CODE
SCHEDULE
BUILD
RECURRENT
BUILDS
CODE
FETCHCODE QUALITY
TESTS
TEST
RESULTS
BUILD OUTPUT
DOCS
BINARIES
& PACKAGES
DEV FACING
NOTIFICATIONS
CLOUDFORMATION
AMIS or CONTAINERS

9. SOURCE
CODE
REPOSITORY
DNS
CONTINUOUS
INTEGRATION
SERVER
PROJECT
MANAGEMENT
SERVER
BUILDS

10. AWS code services
AWS CodeCommit
Launched Today
AWS CodePipeline
Launched Today
AWS CodeDeploy
Launched Nov 2014

11. Cloud software development lifecycle
10/13/14 11
MonitorProvisionDeployTestBuildCode
AWS Elastic Beanstalk
AWS OpsWorks
Amazon
CloudWatch
AWS
CloudFormation
?

12. Cloud software development lifecycle
10/13/14 12
MonitorProvisionDeployTestBuildCode
AWS Elastic Beanstalk
AWS OpsWorks
CloudWatchCloudFormationCodeDeploy
CodeCommit CodePipeline

13. CODECOMMIT
DNS
CODEPIPELINE
PROJECT
MANAGEMENT
SERVER
BUILDS

14. PAIN POINTS
• UNIT TESTS INCOMPLETE
• MOCK TESTS MAINTENANCE
• EXPENSIVE TEST ENVIRONMENT
• TEST ENVIRONMENT ≠ PRODUCTION
• DEPLOYMENT CYCLES

15. ON-DEMAND
PAY AS YOU GO
ELASTIC

16. =
PROGRAMMABLE PLATFORM

19. IF YOU CAN PROGRAM IT
YOU CAN AUTOMATE IT

20. A lot of options…
• Configuration Management Systems
– Puppet
– Chef
– Saltstack
• Deployment Frameworks
– CodeDeploy
– AWS Elastic Beanstalk
– AWS OpsWorks
– Ansible
– Fabric
– Capistrano
• Infrastructure Management
– CloudFormation
• Containers
– Amazon EC2 Container Service

21. APPLICATION VERSIONS
+
INFRASTRUCTURE
VERSIONS

22. CLOUDFORMATION
TEMPLATE

23. CONTINUOUS
DEPLOYMENT
SMALL, FREQUENT CHANGES
CONSTANTLY INTEGRATING INTO
PRODUCTION.

24. KEY = ITERATION

25. ITERATION
=
MODIFY THE SYSTEM TO BETTER
MEET THE EXPECTATIONS OF
YOUR USERS

26. 11.6s
Mean time
between
deployments
(weekday)
1,079
Max number of
deployments in a
single hour
10,000
Mean number of
hosts
simultaneously
receiving a
deployment
30,000
Max number of
hosts
simultaneously
receiving a
deployment
DEPLOYMENTS AT
AMAZON.COM
(in 2011)

27. SOFTWARE DEPLOY
≠
PRODUCT LAUNCH

28. DATA-DRIVEN
ARCHITECTURES

29. METRICS @ETSY

30. METRICS @OBAMA FOR AMERICA

31. Metrics and Monitoring Options
CloudWatch
… and many more

32. CONTINUOUS
INTEGRATION
CONTINUOUS
DEPLOYMENT

33. CONTINUOUS DEPLOYMENT
=
CONTINUOUS EXPERIMENTATION

34. CONTINUOUS DEPLOYMENT
=
CONTINUOUS IMPROVEMENT

35. INNOVATE

36. SPEED AND AGILITY
Experiment
Often
Fail quickly
at a low
cost
More
Innovation
Experiment
Infrequently
Failure is
expensive
Less
Innovation
“ON-
PREMISES”

37. DevOps@edX
A report from the frontline of an evolving and, hopefully,
continuously improving DevOps organization,
July, 2015

38. *
About edX
‣ Created in May, 2012 by Harvard and MIT
‣ More than 4 million students from every country
‣ More than 12 million course enrollments
‣ 65 prestigious member institutions from around the world

39. *
Our Vision
To democratize and reimagine education by increasing worldwide educational
access and create a culture of continuous, lifelong learning.
Anyone, anywhere with a desire to learn and an Internet connection can take
quality online courses.

40. *
edX, the Open Source Project

41. *
edX on AWS
How we use AWS
‣ Deployed on AWS since the beginning
‣ Use many AWS services, but especially
‣ EC2
‣ RDS
‣ S3
‣ Cloudfront
‣ EMR
‣ Everything in the VPC
‣ Tend to be early adopters
‣ AWS Marketplace image in partnership with Bitnami

42. *
edX on AWS
Why we value AWS
‣ Breadth of offering
‣ Pay-for-what-you-use
‣ “Elastic” everywhere; scales-up and scales-down
‣ A continuum of options from “bare metal” to Elastic Beanstalk
‣ Few tie-ins
‣ Global footprint

43. *
What the hell have you built?
With apologies to @codinghorror

44. **
Key DevOps Learnings
‣ Infrastructure As Code
‣ Immutable Infrastructure

45. **
Infrastructure as Code

46. **
Why
‣ We need to deploy services quickly and consistently
‣ Relationships between services are complex
‣ Each additional service adds incremental operational complexity
Infrastructure as Code

47. **
Each service is complex in its own right
Infrastructure as Code

48. **
How
‣ CloudFormation
‣ Terraform: https://www.terraform.io/
‣ Home-grown solutions
Infrastructure as Code

49. **
Infrastructure as Code
Cloud Migrator
“A simple, opinionated pattern and tools for discrete, scalable, fault-tolerant
services”
Ed

50. **
Infrastructure as Code
How we realize the components in AWS

51. **
Cloud Migrator Design goals:
‣ Ease of adding new services
‣ Minimal additional configuration per service required
‣ Flexible per service configuration supported
‣ Idempotent
‣ D.R.Y.
Infrastructure as Code

52. **
What Cloud Migrator descriptors look like:
‣ yaml
‣ Minimally specify
‣ tags
‣ ports
‣ service CIDR blocks
‣ Anything can be overridden
‣ Easy to extend
Infrastructure as Code

53. **
---
cluster: "notes"
play: "{{ cluster }}"
services_tag: "edx_notes_api"
service_port: 18120
instance_type: "m3.medium"
private_subnet_1: "{{ vpc_class_b }}.110.32/28"
private_subnet_2: "{{ vpc_class_b }}.120.32/28"
private_elb_subnet_1: "{{ vpc_class_b }}.110.128/28"
private_elb_subnet_2: "{{ vpc_class_b }}.120.128/28"
Infrastructure as Code

54. **
Immutable Infrastructure

55. **
Immutable Infrastructure
Why
‣ Traceability, everything in SCM
‣ Pull request workflow
‣ Strong consistency guarantees with identical, tested artifacts
‣ Dead simple auto-scaling
‣ Clear path to automation

56. **
Immutable Infrastructure
How
‣ Our deployment artifacts are pre-baked AMIs
‣ Several versions of application software and config converge to produce our
AMIs
‣ Another homegrown tool, Abbey, builds AMIs
‣ Abbey leverages our configuration management tool of choice, Ansible

57. **
An Aside: Why Ansible
‣ We appreciate its simplicity
‣ It’s built on tried and true tools like SSH
‣ Dynamic inventories using cloud-friendly selectors like tags
‣ It is agentless
‣ We are heavily invested in Python

58. **
Cutting an Image

59. **
Cutting an Image

60. **
Cutting an Image
Need something cut, talk to @alton

61. **
Chatting with @alton

62. **
Chatting with @alton

63. **
Comparing Images

64. **
Deploying

65. **
Blue/Green Deploys

66. **
Whither edX Ops
‣ Continue down the road of decomposing our monolithic application into
independent services
‣ Improve testing, especially automated integration tests
‣ Move to a more fully automated CI/CD pipeline
‣ Expand our geographic reach
‣ Continuously improve our infrastructure
‣ Support our open-source partners
‣ Leverage service discovery
‣ Containers, containers, containers

67. **
Contact Details
‣ Edward Zarecor, DevOps Lead@edX, e0d@edx.org
‣ edX on github: https://github.com/edx
‣ On github I’m e0d

68. NEW YORK