How to Triple Your Speed of Development Using Automation

How to Triple Your Speed of Development Using Automation | 26/03/2017
How to Triple Your Speed of
Development Using Automation
Johanan Liebermann
26/03/2017

AWS and GCP Division Overview | Nov. 2016
© 2016 AllCloud
“AWS Managed Service Partners
are skilled at cloud infrastructure
and application migration, and
deliver value to customers by
offering proactive monitoring,
automation, and management of
their customer’s environment.”
https://aws.amazon.com/partners/msp/
http://www.allcloud.io/msp
AWS Next-Gen (v3) Managed Service Partner (MSP)

Johanan Liebermann
Software Developer at AllCloud
johanan.liebermann@allcloud.io
About Me

● Intro
● Using automation to speed up your development
● Two real use cases (instance-based and container-based)
Topics

Intro

● Today’s software world moves fast!
● Tiny startups can challenge giants
● Delivering a product to the masses is easy
● Release cycle takes days, not months
● Time to market can make or break your product
The Need for Speed

How?

Automate Everything!

“A set of practices which make software lifecycle more efficient by
bridging the gap between development and operations and aiming
for automation in every aspect.”
DevOps

● Continuous Integration & Continuous Delivery / Deployment
● A major part of DevOps
● Continually building, testing and deploying your code
● Catch problems as soon as possible
● Work on multiple features in parallel
● Ship code often in small chunks
CI/CD

● Version Control
● Build Process
● Testing
● Configuration Management
● Infrastructure Provisioning
● Deployment & Release
What Areas Can You Automate?

Version Control Managing changes to source
code

● Git hooks
○ Trigger build, test and deploy processes
○ Enforce commit message style
○ Enforce coding standards (can’t commit bad code)
○ Notify developers about code-related events
● Automated handling of branches and pull requests
○ GitHub API
Version Control Automation

Build Process Generating artifacts from
source code

● Build automation tools
○ Make, Maven, Gradle, Grunt, Gulp, Rake...
● Dependency / package managers
○ pip, npm, RubyGems, Conda, Bower...
Build Process Automation

● Your artifacts should be properly-versioned
● Your artifacts should be consistently-available
● Many types of artifacts
○ Language-specific - JAR, Gem
○ OS-specific - RPM, Deb, APK, IPA
○ Images - Docker, Vagrant, AMI
Handling Artifacts

● Dedicated solutions: Artifactory, Nexus, ECR, Docker Hub
● Simple file storage: S3
Artifact Storage Options

Verifying the quality of your
code
Testing

Unit Testing
Types of Testing
Integration Testing
Functional Testing

● Making sure a small, isolated piece of code works properly
● Done as part of the build process
● Unit tests should be an integral part of the code
● A failed unit test should always stop the build process
● Typically done using built-in language features or dedicated
frameworks (JUnit, pytest, rspec…)
● Provides a fast feedback for new code
Unit Testing

● Making sure a couple of components work together properly
● Should be done as part of the deployment / release process
● Usually requires deployment to an environment
● Typically done using dedicated tools or custom scripts
● Typically involves “external” entities such as databases, 3rd party
APIs, managed services etc.
Integration Testing

● End-to-end testing of a product
● Should be done as part of the deployment / release process
● Typically simulates user interaction for front-end applications
using tools such as Selenium and Appium
● Typically simulates API calls for back-end services using tools
such as Ready! API or vREST
Functional Testing

Ensuring your servers are
properly configured to serve
your application
Configuration
Management

● Typically includes:
○ Installing packages
○ Ensuring configuration files are in the desired state
○ Setting OS-level configuration parameters (file permissions, environment
variables...)
○ Configuring services
● Declarative syntax & idempotent in most cases
Configuration Management

● Dedicated tools: Ansible, Chef, Puppet, SaltStack
● Built-in functionality: Dockerfiles, ebextensions
● Custom scripts (not recommended)
Configuration Management Tools

● Two approaches to managing your infrastructure
○ Mutable infrastructure: modifying existing nodes
○ Immutable infrastructure: replacing existing nodes
● Both options are achievable using CM tools
○ Use CM tools to manage your nodes directly
○ Use CM tools to generate images
Mutable vs. Immutable Infrastructure

Infrastructure
Provisioning
Creating, updating and
destroying cloud
infrastructure

● Relevant mainly for IaaS
● Typical resources which need provisioning:
○ Compute resources (EC2, ECS, Lambda)
○ Network & security (VPC, ELB, CloudFront)
○ Databases, cache clusters and queues (RDS, DynamoDB, ElastiCache, SQS)
○ Storage (EBS, EFS, S3)
○ Miscellaneous services (IAM, SNS, SES, CloudWatch...)
Infrastructure Provisioning

● An approach in which infrastructure == code
○ Version-controlled
○ Tested
● Provides reproducibility
● Usually declarative & idempotent
Infrastructure as Code

● Dedicated tools: CloudFormation, Terraform
● Built-in functionality: Elastic Beanstalk, Heroku
Infrastructure Provisioning Tools

Making your code available to
the users
Deployment &
Release

● Back-end services
○ Update or replace instances / containers / Lambda functions
○ Run DB migrations
● Front-end services
○ Run minification & CSS preprocessing
○ Sync S3 buckets
○ Invalidate CDN
Deployment Process

● In-place - simply updating the live nodes
● Canary - deploying to a small amount of nodes first, making sure
everything is fine, then updating all the nodes
● Blue/Green - maintaining two identical production environments
and switching the activity between them
Back-End Deployment Strategies

Sample
Pipeline:
CodeDeploy

● A managed deployment service by AWS
● Handles the deployment process for you
● Minimizes downtime
○ In-place rolling updates
○ Built-in support for Blue/Green deployment
● Automatic or manual rollbacks
● Platform-agnostic: use with any OS, any runtime and any CM
AWS CodeDeploy

● A REST API written in Python on Amazon Linux
● Jenkins as an automation server
● Artifactory for artifact storage
● Ansible for CM
● CodeDeploy for deployment
Pipeline Setup

The Pipeline
2. Run unit tests
3. Build pip package
4. Build Ansible RPM
1. Checkout
5. Store pip & RPM
6. Push new revision
7. Trigger CodeDeploy
8. Deregister from ELB
9. Install RPM using YUM
10. Run Ansible
11. Verify service
12. Register with ELB
Pull RPM & pip - Run pip install
- Modify config files
- Start service

Sample
Pipeline: ECS

● EC2 Container Service
● The native AWS container management service
● Runs on EC2
● Contains a private Docker registry (ECR)
● Native integration with Auto Scaling, ELB and IAM
● ECS is free (but ECR has charges)
AWS ECS

● A REST API written in NodeJS
● Jenkins as an automation server
● ECR for artifact storage
● No need for CM (Docker handles it)
● ECS for deployment
Pipeline Setup

The Pipeline
2. Build Docker image
3. Run unit tests
4. Push image
1. Checkout
5. Update Task
Definition
6. Update Service
7. Replace containers
Pull image

Conclusion

● Implement fully-automated pipelines to leverage the full
potential of the cloud
● Automation will boost your speed of development and allow you
to focus on your differentiating factors
● Relying on managed services wherever possible will reduce the
manpower needed for operating your service and may also
reduce your costs
Conclusion

Thank You!
info@allcloud.io

How to Triple Your Speed of Development Using Automation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to How to Triple Your Speed of Development Using Automation

Similar to How to Triple Your Speed of Development Using Automation (20)

Recently uploaded

Recently uploaded (20)

How to Triple Your Speed of Development Using Automation