Organizations around the globe are leveraging the cloud to accomplish world-changing missions. This session will address how AWS can help organizations put more money toward their mission and scale outreach and operations to achieve more with less. Hear some of AWS’s most advanced customers on how their organizations handle DevOps, continuous integration and deployment. Learn how these practices allow them to rapidly develop, iterate, test and deploy highly-scalable web applications and core operational systems on AWS. The discussion will focus on best practices, lessons learned, and the specific technologies and services they use.

1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
DevOps, Continuous Integration &
Deployment on AWS
Leo Zhadanovsky, Senior Solutions Architect, AWS
@leozh
Dave Beck, Technical Director, National Novel Writing Month
@DaveB_NaNoWriMo

2. DevOps

3. What is DevOps?
• « DevOps is the practice of operations and
development engineers participating together in
the entire service lifecycle, from design through
the development process to production support
»
- theagileadmin.com

4. Continuous Integration

5. What is Continuous Integration?
• Changes to code automatically deployed to
mainline branch
– After passing unit and mock tests
• Makes changes to code, and deployments
iterative, not monolithic
• Bugs are detected quickly
• Helps automate deployments
• Allows rapid development and deployment

6. SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
DEVELOPER
PICK
TASKS
SUBMIT
CODE
SCHEDULE
BUILD
RECURRENT
BUILDS
CODE
FETCHCODE QUALITY
TESTS
TEST
RESULTS
BUILD OUTPUT
DOCS
BINARIES
& PACKAGES
DEV FACING
NOTIFICATIONS
CLOUDFORMATION
AMIS or CONTAINERS

8. SOURCE CODE
REPOSITORY
DNS
CONTINUOUS
INTEGRATION SERVER
PROJECT
MANAGEMENT SERVER
BUILDS

9. AWS code services
AWS CodeCommit
Coming soon
AWS CodePipeline
Coming soon
AWS CodeDeploy
Launched Nov 2014

10. Cloud software development lifecycle
10/13/14 10
MonitorProvisionDeployTestBuildCode
AWS Elastic Beanstalk
AWS OpsWorks
Amazon
CloudWatch
AWS
CloudFormation
?

11. Source Control
Private Git repositories hosted on Amazon S3
• Full Git compatibility (use with existing tools)
• All the benefits of the cloud
(scalable, durable, reliable, low pay as you go
pricing)
• No size limits on repositories (store binary files)
• Online code tools with browse, edit, diff
10/13/14 11

12. Same Git experience
$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli
Cloning into 'aws-cli'...
Receiving objects: 100% (16032/16032), 5.55 MiB | 1.25 MiB/s, done.
Resolving deltas: 100% (9900/9900), done.
Checking connectivity... done.
$ nano README.rst
$ git commit -am 'updated README'
[master 4fa0318] updated README
1 file changed, 1 insertion(+)
$ git push
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 297 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote:
To https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli
4dacd6d..4fa0318 master -> master

13. Continuous Delivery
Customizable release automation, with integrated build and test
• Model and visualize custom release workflow
(source  build  beta  gamma  prod)
• Automate builds, tests, and deployments
• Enforce custom rules, approvals, and gates
• Integrate with third-party and custom tools
10/13/14 13

15. Deployment
Coordinate software updates to fleets of EC2 instances
• Rolling updates for no downtime
• Deployment health checks and easy rollback
• Auto Scaling integration
• Works with any application
• Reuse existing setup tools
(Bash, Powershell, Chef, Puppet…)
10/13/14 15

16. 9/25/14 Slides not intended for 16

17. Cloud software development lifecycle
10/13/14 17
MonitorProvisionDeployTestBuildCode
AWS Elastic Beanstalk
AWS OpsWorks
CloudWatchCloudFormationCodeDeploy
CodeCommit CodePipeline

18. CODECOMMIT
DNS
CODEPIPELINE
PROJECT
MANAGEMENT SERVER
BUILDS

19. PAIN POINTS
• UNIT TESTS INCOMPLETE
• MOCK TESTS MAINTENANCE
• EXPENSIVE TEST ENVIRONMENT
• TEST ENVIRONMENT ≠
PRODUCTION
• DEPLOYMENT CYCLES

20. ON-DEMAND
PAY AS YOU GO
ELASTIC

21. =
PROGRAMMABLE PLATFORM

24. IF YOU CAN PROGRAM IT
YOU CAN AUTOMATE IT

25. A lot of options…
• Configuration Management Systems
– Puppet
– Chef
– Saltstack
• Deployment Frameworks
– CodeDeploy
– AWS Elastic Beanstalk
– AWS OpsWorks
– Ansible
– Fabric
– Capistrano
• Infrastructure Management
– CloudFormation
• Containers
– Amazon EC2 Container Service

26. Bake an Image Configure dynamically
Time consuming
configuration (startup time)
Static configurations (less
change management)
Bootstrapping

27. Bake an Image Configure dynamically
Continuous deployment
(latest code)
Environment specific (dev-
test-prod)
Bootstrapping

28. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
AWS CLOUDFORMATION
STACK-BASED DEPLOYMENT SERVICE

29. CLOUDFORMATION
TEMPLATE

30. Headers
Parameters
Mappings
Resources
Outputs

31. {
"Description" : "Create RDS with username and password",
"Resources" : {
"MyDB" : {
"Type" : "AWS::RDS::DBInstance",
"Properties" : {
"AllocatedStorage" : "500",
"DBInstanceClass" : "db.m1.small",
"Engine" : "MySQL",
"EngineVersion" : "5.5",
"MasterUsername" : "MyName",
"MasterUserPassword" : "MyPassword"
}
}
}
}

32. "AWS::CloudFormation::Init" : { "config" : {
"packages" : {
"yum" : {
"mysql" : [],
"mysql-server" : [],
"httpd" : [],
"php" : [],
"php-mysql" : []
}
},
"sources" : {
"/var/www/html" :
"https://s3.amazonaws.com/my-builds/build-v4.zip"
}
}

33. {
"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing EC2
KeyPair to enable SSH
access to the instance",
"Type" : "String"
}
},
}

34. CLOUDFORMATION
TEMPLATE
PROCEDURAL
DEFINITION
Create it programmatically
KNOWN
CONFIGURATION
Store stack configuration in
source control
PARAMETER
DRIVEN
Dynamic and user-driven
templates
COLLABORATION
Share templates with ease
as just files

36. APPLICATION VERSIONS
+
INFRASTRUCTURE VERSIONS

37. CLOUDFORMATION
TEMPLATE

38. CONTINUOUS
DEPLOYMENT
SMALL, FREQUENT CHANGES CONSTANTLY
INTEGRATING INTO PRODUCTION.

39. KEY = ITERATION

40. ITERATION
=
MODIFY THE SYSTEM TO BETTER MEET
THE EXPECTATIONS OF YOUR USERS

41. 11.6s
Mean time
between
deployments
(weekday)
1,079
Max number of
deployments in a
single hour
10,000
Mean number of
hosts
simultaneously
receiving a
deployment
30,000
Max number of
hosts
simultaneously
receiving a
deployment
DEPLOYMENTS AT
AMAZON.COM
(in 2011)

42. SOFTWARE DEPLOY
≠
PRODUCT LAUNCH

43. DATA-DRIVEN
ARCHITECTURES

44. METRICS @ETSY

45. METRICS @OBAMA FOR AMERICA

46. Metrics and Monitoring Options
CloudWatch
… and many more

47. CONTINUOUS
INTEGRATION
CONTINUOUS
DEPLOYMENT

48. CONTINUOUS DEPLOYMENT
=
CONTINUOUS EXPERIMENTATION

49. CONTINUOUS DEPLOYMENT
=
CONTINUOUS IMPROVEMENT

50. INNOVATE

51. SPEED AND AGILITY
Experiment
Often
Fail quickly at
a low cost
More
Innovation
Experiment
Infrequently
Failure is
expensive
Less
Innovation
“ON-PREMISE”

52. At a glance
What is NaNoWriMo?
• National
• Novel
• Writing
• Month
We are a community of novelists.
Every November, hundreds of
thousands of writers gather virtually
and in-person to write 50,000 words
each. (The Great Gatsby, by
comparison, is 47,094 words.)
Last year, 180,000 started new
novels, and 40,000 finished.

53. It all started in 1999
Our founder, Chris Baty,
challenged 20 over-caffeinated
friends to write 50,000 words each
in a single month.
Six of them finished.

54. Explosive growth, and no plan to manage it

55. NaNoWriMo now
NaNoWriMo Main: Ruby 1.9.3, Rails 3.2.2, MySQL 5.6
Camp NaNoWriMo: Ruby 1.9.3, Rails 3.2.2, MySQL 5.6
The Young Writers Program: Drupal 5, MySQL 5.6
NaNoWriMo Store: Drupal 5, MySQL 5.6

56. November in detail
• Nov 1: ~500k sessions as people begin
• All month: ~50k forum posts per day
• Nov 30: Last chance to post word counts

57. Annual Site Crash
• Every year from 2003 to 2011 the site would crash on
November 1.
• In 2012 we migrated the sites to Amazon Web Services.
• The result: No more outages!
• Naturally, this is how our participants reacted…

58. Outrage!
They wanted the site to crash; it was
considered a rite of passage. Oh well.

59. Keeping NaNoWriMo Online
• Proxy servers (Varnish) on Debian 7.5 EC2
instances
• Ruby on Rails on EC2 (Nginx)
• Worker servers running Resque on EC2
• Support servers running Resque, Redis,
Sphinx
• MySQL on RDS instances
• Domains and subdomains on Route 53
• Image assets on S3 instances via EBS
• Deployment via Capistrano, Puppet

60. The Year in Instances

61. Next Steps: Goals
 To increase uptime for all services
 To make server management less time-consuming
 To reliably and quickly respond to user-driven events
 To reduce EC2 costs through autoscaling
 To create a faster and more dynamic email system
 To improve site search

62. Next Steps: Goodbye Drupal
What: Porting Drupal sites to Ruby 2.2, Rails 4.2
Why: The old Drupal-based sites are slow and use more
than their share of server resources.

63. Next Steps: Emailing via Amazon SES
What: Migrating outgoing emails from PHPlist/Sendmail to
SES
Why: Event-driven emails on the sites are backgrounded to
support servers, which has proved slow and unreliable.
Mass emails rely on an old and very sluggish application
which has limited layout tools and metrics.

64. Next Steps: Load Balancing
What: Converting proxy servers to Elastic Load Balancers
Why: We will benefit from better AWS dashboard tools,
metrics, and notifications. It is part of our autoscaling
strategy.

65. Next Steps: CodeDeploy
What: Using CodeDeploy for deployment process
Why: Currently deployment relies on many custom shell
scripts and tends to be unreliable. We intend to move to
stateless instance spin-ups as part of our autoscaling
strategy.

66. Next Steps: Autoscaling
What: Implement an Auto Scaling process
Why: In the past we’ve had to predict our server needs
ahead of time. If we underestimated, the sites would
crash; if we overestimated, we weren’t using our small
budget effectively. And the planning process consumed
too much staff time.

67. Next Steps: Amazon SNS Notifications
What: Increased use of Amazon SNS service
Why: Last year SNS event messaging helped us respond
to emerging crises very quickly. With our expanded AWS
presence, we will link event notifications to all of the
services we use.

68. Next Steps: On the Horizon
What: Amazon CloudSearch; AWS Lambda
Why: We are interested in easy-to-manage alternatives to
Sphinx – CloudSearch is one option. Lambda looks
intriguing for a variety of possible applications.

69. Potential future architecture
Internet
Load Balancer
ELB EC2 instances
Code servers
EC2 instances
Databases
RDS DB instances
Search
CloudSearch instance
Outgoing email
Amazon SES service

70. SAN FRANCISCO