AWS Public Sector Symposium 2014 Canberra | Continuous Integration and Deployment Best Practices on AWS

1. AWS Government, Education, & Nonprofits Symposium Canberra, Australia | May 20, 2014 Continuous Integration and Deployment Best Practices on AWS Adrian White Solutions Architect, Amazon Web Services

2. Innovation, Quality, Governance

3. Stacks / Environment(s) An example CI / CD workflow CI/CD tool Issue Tracker SCM Infrastructure automation / deployment Test tools / harnesses

4. CloudFormation Stack(s) An example CI / CD workflow PHPUnit jQuery … Tasks for AWS

5. A CI / CD pipeline Teardown Release Deploy Test Code

6. Get your source under control Prod Phoenix (feature)UAT Code Test Deploy Release Teardown Code Test Deploy Release Teardown Code Test Deploy Release Teardown Code Test Deploy Release Teardown

7. Automated Merging UAT Feature 2Feature 1

8. What does CI give us? •  Test driven promotion (of development change) •  Increasing velocity of feedback cycle through iterative change •  Contain change to reduce risk •  Bugs are detected quickly •  Automated testing reduces size of testing effort

9. What does CD give us? •  Changes are pushed quickly to production •  Immediate feedback from users •  Supports A/B testing or “We test customer reactions to features in production” •  Hardens, de-risks the deployment process •  Gives us a breadth of data points across our applications

10. Deployment approaches •  Deploy in place –  Manage interruption •  Bake –  Discrete environment •  Multiple environments from branches •  Support A/B testing •  “Rolling DNS” Deploy Deploy in-place Bake

11. Deploy in-place Un-baked •  Common baseline AMI •  Customise at instantiation •  Change in-place Your app AMI

17. Bake process 1.  Start a builder instance 2.  Bootstrap / cfn-init, cfn-signal 3.  Bake your AMI 4.  Tag it 5.  Destroy/clean up the builder instance

18. cfn-init "AWS::CloudFormation::Init" : { “cfn” : { "packages" :{ "yum" : { "httpd" : [] } }, "files":{ “/home/ec2-user/myfile.html:{ "source" : { "Fn::Join" : [ "", ["https://s3-ap- southeast-2.amazonaws.com/",{ "Ref" : "S3Bucket" },”/myfile.html”]] }, "mode":"000644", "owner":"root", "group":"root", "authentication":"S3AccessCreds” },

19. cfn-init "services": { "sysvinit" : { "httpd" : { "enabled" : "true", "ensureRunning" : "true" } } } }

20. Release Release Test the new stack Match the traffic between the two stacks Update the “floating” DNS record Send Notification(s) ROLLBACK

21. Blue green deployment awssummit-1.com awssummit-2.com awssummit.com

25. Teardown Teardown Ensure that no traffic is moving though ELB Teardown the CloudFormation Stack Deregister the AMI

26. Advanced Techniques •  Extending CloudFormation with custom resources •  Managing CD sprawl •  Segregation of duties •  Extending your CD tools

27. Stack chaining

28. Stack chaining

29. Stack chaining

30. CloudFormation Merging Git Git CloudFormation Operations Repo Application Repo VPC Subnets Security Groups CloudFormation Frameworks Best Practice Application Code Application CloudFormation Load Balancing Setup

31. CloudFormation Custom Resources •  Change DB schema during deployment •  Extend CloudFormation to support other services - “So You Think You Are An AWS Ninja” talk https://github.com/aws/aws-cfn-resource-bridge https://github.com/awslabs/aws-cfn-custom-resource-examples Parameters Custom resource implementation Git

32. Custom resources – DatabaseSchema "MyDBSchema" : { "Type" : "Custom::DatabaseSchema”, "Version" : "1.0", "Properties" : { "ServiceToken": "arn:aws:sns:us-east-1:12345EXAMPLE:DBSchema", "databaseChangeLog" : [ { "changeSet" : { "id" : "1", "author" : "adamthom", "changes" : [ { "createTable" : { … } } ] } } } }

33. Custom resources – DatabaseSchema "createTable" : { "tableName" : "example", "columns" : [ { "column" : { "name" : "id", "type" : "int", "autoIncrement" : true, "constraints" : { "primaryKey" : true, "nullable" : false } } } ] }

34. Extending your CD tools Tasks for AWS DynamoDB

35. Situational Awareness Burden of Responsibility APIs Tasks for AWS

36. Containerisation •  Build environments for artifacts, don’t update environments with artifacts •  All environments are transient •  Standardisation, abstraction and portability

37. Docker, Amazon Linux and Elastic Beanstalk •  A framework for managing containers •  LXC containers are more lightweight than VMs •  Amazon Linux (2014.03) bundles Docker 0.9 and LXC 0.9 •  Docker containers on Beanstalk are Go!

38. Innovation, Quality, Governance Discrete environments for each branch Automated testing on every commit on every branch Leverage CD tools to provide separation of duties Audit Logs Git approvals process Use custom resources to extend CloudFormation Leverage DNS Interface with the API Environments for artifacts

39. Customer: Open Training Institute

40. Journey to the cloud Steve Mactaggart Manager, Open Training Platform

41. •  Our core Higher Educa/on business –  Open Universi/es Australia is a na/onal leader in online higher educa/on. –  Through Open Universi/es Australia you can study high-‐ quality degrees with leading Australian universi/es. Introduction to Open Universities

42. •  Launch of Open2Study –  Develop Australia’s ﬁrst large scale MOOC –  Enhance our capability to design and deliver online learning –  New technical plaIorm for us to deliver •  Now Open Training Ins/tute –  A new business focused on delivering Voca/onal Educa/on and Training online –  Leveraging our experience of the exis/ng Open business, capabili/es developed for Open2Study –  Project kickoﬀ May 2013, Launch Dec 2013 Our Journey

43. •  New technology stack •  Ambi/ous scope of work •  Lots of compe/ng business objec/ves •  Aggressive /meframes •  Unexpected ini/al load and growth proﬁle •  Geographically separate development team Challenges

44. What does Open2Study look like today 170,000+ students 300,000+ enrolments 4,700,000+ min of video From over 200 countries Web instances scale between 4 -‐-‐> 12 m1.xlarge instances

45. •  Stage 1: Spike it up (POC) –  AWS CLI build AS group and launch •  Stage 2: Automated deployments –  Ini/al version of deployment script, manually triggered by logging on to server •  Stage 3: Deployment pipeline –  Jenkins, ssh and Build pipeline plugin •  Stage 4: Prepare for scale –  Manual AMI crea/on and rollout, Using AWS CLI to reconﬁgure the ASG Our path to successful scale

46. •  Lessons from Ini/al spike –  AWS for prototyping was really painless –  AMI provisioning managed by AS group –  Scale Up/Down rules using CloudWatch –  CPU rules caused early scale up –  Need for becer metrics to trigger scale –  Regular adjustment required •  Lessons from Automated deployments –  Reproducible steps with basic script –  Script needs to be version controlled –  Deployment from git –  AWS looks and feels very similar to our exis/ng infrastructure Key learning’s

47. •  Lessons from Con/nuous delivery –  Manual tasks in pipeline for approval –  Prime the pipeline on every developer change –  Constantly extend ‘deploy’ script –  BASH is a good start, but not the end game •  Lessons from prepare for scale –  Manual AMI crea/on soon became blockage –  Convert process to simple BASH script –  Integrate with deployment pipeline –  ASG to replace applica/on Key learning’s

48. •  Enter OpsWorks –  Provides server conﬁgura/on management –  Lifecycle events to manage deploy process –  Automated Load and Time based scaling •  Lessons from Conﬁgura/on management –  New set of skills required, chef cookbooks –  Re-‐architect the development and deployment process –  Scale up slow (~15min to provision node) –  Python / boto for deployment automa/on control Where we are today

49. •  Repeatable ar/facts promoted between environment –  Solu/on is somewhere between baked AMI 'fixed baked' approach and 'build on demand' process of OpsWorks •  Standard ar/facts with configura/on injec/on –  Configura/on injected via user data –  Read and configured from S3 •  Full automa/on of all applica/on and environment changes –  Flexibility of build from scratch Where we see we are heading

50. THANK YOU Please give us your feedback by filling out the Feedback Forms AWS Government, Education, & Nonprofits Symposium Canberra, Australia | May 20, 2014