A description of the in progress CI/CD pipeline in use at Kelsus featuring AWS and Docker on ECS

1. A battle tested CI/CD Pipeline
Featuring AWS and Docker
with Jon Christensen
of Kelsus
Gluecon 2018

2. We were using Heroku or Elastic Beanstalk
Users were the first to notice issues
We had minimal test coverage
We had disorganized logs
We were hackers
This wasn’t going to cut it at scale

3. • There are too many choices.
• AWS alone adds 5 new features a day.
• So the best way to start CI/CD is
gradually.
• Just like your software itself, CI/CD is
always a work in progress.

4. I truly believe that one of the best benefits of Docker
is that it forces less experienced developers on
teams to confront head-on some of the things they
don’t know about how computers work and learn
them and incorporate them into their body of
knowledge

5. "True" continuous deployment requires no manual
intervention or human involvement.
Requires two pillars:
1. Comprehensive, automated integration tests with
broad coverage
2. The ability to rollback quickly (ideally automated) in
case a bad build gets deployed (because your
verification wasn't complete enough)

6. Dev Test Deploy
Docker Docker Docker
Gitflow Gitflow Gitflow
CircleCI
Volume
Mounts
S3 - KMS
S3 - KMS
AWS ECS
Code
Reviews
AWS ECRUnit Tests
Current CI/CD Tools and Processes

7. Volume Mounts for Hot Reloads
version: "2"
networks:
docker-dev:
external:
name: dev
services:
api:
build: api/
env_file: .env
networks:
default: {}
docker-dev:
aliases:
- my-restful-service
volumes:
- ./shared:/var/www/my-restful-service/shared
ports:
- "8080:80"
links:
- db
db:
build: db/
env_file: .env
ports:
- "32099:5432"
volumes_from:
- data
data:
image: postgres
restart: "no"
command: "true"
volumes:
- /var/lib/postgresql
test:
build: api/
environment:
APP_ENV: test
command: "./run_tests_docker.sh"
volumes:
- ./shared:/var/www/my-restful-service/shared
ports:
- "8080:80"
links:
- test-db
test-db:
build: db/
ports:
- 5432
######################
# The below container is provided
# as a developer convenience.
# It mounts the host volume at the
# project root (so that both container and
# host are reading same files). This
# allows for hot reloading of code changes.
######################
api-dev:
build: api/
env_file: .env
volumes:
- .:/var/www/my-restful-service
ports:
- "8080:80"
links:
- db

8. Testing has to be a moral mandate
In order to succeed at always
writing tests, the organization as a whole
must treat *not* testing as
unthinkable

9. Kelsus Git Flow

10. # Circle CI 2.0 Specification
version: 2
jobs:
build:
# Docker image selected.
docker:
- image: 'kelsus/circleci-node-6.10:1.0'
A walkthrough a CircleCI file
The core thing we do is a build

11. environment:
# The Service Name must be the same you have
# in your package.json, Dockerfile &
# docker-compose.yml.
SERVICE_NAME: customer-dashboard-server
# Git User Email & Name
GIT_USER_EMAIL: ci@kelsus.com
GIT_USER_NAME: CircleCI
A walkthrough a CircleCI file
Then we set some environment variables
like a GitHub account

12. steps:
# Checking out the project from the
# repository.
- checkout
- run:
name: Run tests with docker compose.
command: sh ./tests-up.sh
A walkthrough a CircleCI file
Then we checkout code and run tests!

13. - run:
name: Creating a coverage folder.
command: mkdir -p ./coverage
- run:
name: Copy artifacts from container.
command: docker cp project_test_run_1:/var/www/
$SERVICE_NAME/shared/artifacts/coverage ./coverage
# Prepare Coverage Report on Circle CI
- store_artifacts:
path: ./coverage
destination: coverage
A walkthrough a CircleCI file
Get coverage information and store it

14. - deploy:
name: Deploy if branch is dev or staging
command: ./deploy/deploy.sh
A walkthrough a CircleCI file
Deploy!

15. # When deployed to the server, read the config from the encrypted files on S3
if [[ "$APP_ENV" == "development" || "$APP_ENV" == "staging" || "$APP_ENV" == "production" ]]; then
export AWS_DEFAULT_REGION=$AWS_REGION
S3_CREDENTIALS_BUCKET=$(cat $CONFIGPATH/package.json | jq "(.kelsus.s3_credentials_bucket)" —raw-
output)
# Retrieve encrypted credentials file from S3
aws s3 cp $S3_CREDENTIALS_BUCKET$APP_ENV/$SERVICE_NAME.config $SERVICE_NAME.config.enc
# Decrypt
SERVER_CONFIG_JSON=`aws kms decrypt --ciphertext-blob fileb://$SERVICE_NAME.config.enc --query
Plaintext --output text | base64 --decode`
# Remove tmp file
rm $SERVICE_NAME.config.enc
if [ "$CONFIG_INTO_ENV_VARIABLES" == "true" ]; then
# Extract all entries in the json and put them on environment variables
echo $SERVER_CONFIG_JSON | jq -r '.config | to_entries[] | [.key, .value] | @tsv' > values.txt
while IFS=$'t' read -r key value; do export "$key=$value"; done < values.txt
rm values.txt
else
# Write the server config file on the config/environment folder
echo $SERVER_CONFIG_JSON | jq -r '.config' > ./dist/config/environment/server.json
fi
fi
Grabbing secrets from S3

16. Why ECS?
excerpt from hangops slack

17. CI/CD is always a work in progress
You’re always about half done
We still need to work on
• Improving database migrations
• Getting off of bash scripts
• Updating how we store secrets
• Look at AWS Code Pipeline
• Better tests — always better tests

18. Thank you!
— Jon Christensen
Kelsus Inc
www.kelsus.com
www.prodockertraining.com
@jonxtensen