High Velocity DevOps: Four Ways to Leverage CloudFront in Faster DevOps Workflows (CTD409) - AWS re:Invent 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High Velocity DevOps: Four Ways to Leverage
CloudFront in Faster DevOps Workflows
C T D 4 0 9
Chanka Perera
Solutions Architect
AWS
Acar Erkek
Sr. SDE Cloudfront
AWS
Gourav Tiwari
Principal Software Engineer
Realtor.com

Agenda
Introduction
Build: Programmatically launch and configure your Amazon CloudFront distributions
Test: Confirm your updates deliver the intended result with A/B testing and Lambda@Edge
Release: Continuously manage and deploy your application to CloudFront’s Global Edge Network with AWS CodeStar
Monitor: Uncover actionable insights hiding in your CloudFront logs by leveraging Amazon CloudWatch, Amazon
Athena, or Amazon Marketplace partners for intelligent monitoring and alerting
Customer speaker: Realtor.com

Development transformation at Amazon: 2001-2009
2001
Monolithic architecture +
hierarchical organization
Decoupled services +
two pizza teams
2009

What is at the Edge?
Global DNS Firewall Anti-DDoS Serverless
compute
Content delivery Ingest and proxy

CloudFrontAWS Lambda
Lambda@Edge
AWS Lambda@Edge

CI/CD pipeline and why is it important?
Source Build Test Deploy Monitor

Build

Self-service configuration

AWS code services
Source Build Test Deploy
Software release steps:

Step 1: Creating the AWS CodeStar project

Step 1: Creating the AWS CodeStar project
Users

Step 2: Adding CloudFront
Users

Cache
Viewer
response
Origin
response
Origin
Origin
request
Viewer
request
Client
CloudFront events for Lambda@Edge

Step 2.1: SAM template – Lambda@Edge
Resources:
LambdaEdge:
Type: 'AWS::Serverless::Function'
Properties:
FunctionName: !Sub 'reinvent2018-LambdaEdge-
${Stage}'
CodeUri: lambdaedge/lambdaedge-index.js
Handler: lambdaedge/lambdaedge-index.handler
Runtime: nodejs6.10
# this ensures every update will create a new
version and alias pointing to that version
# e.g. Gamma-Live
AutoPublishAlias: live
CodeUri: .
MemorySize: 128
Timeout: 5
Role: !GetAtt LambdaEdgeRole.Arn
LambdaEdgeRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "lambda.amazonaws.com"
- "edgelambda.amazonaws.com"
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-
role/AWSLambdaBasicExecutionRole

Step 2.2: Lambda@Edge code
'use strict';
const querystring = require('querystring');
exports.handler = (event, context, callback) => {
const request = event.Records[0].cf.request;
const headers = request.headers;
const params = querystring.parse(request.querystring);
if (typeof params.newLook !== 'undefined' && params.newLook == 'true') {
console.log('newLook is being served');
request.uri = 'index-newlook.html';
}
callback(null, request);
};

Step 2.3: SAM template – parameters
AWSTemplateFormatVersion: 2010-09-09
Transform:
- AWS::Serverless-2016-10-31
- AWS::CodeStar
Parameters:
ProjectId:
Type: String
Description: AWS CodeStar projectID used to associate new resources to team members
Stage:
Type: String
Description: Deployment stage
AliasName:
Type: String

Step 2.3: SAM template – CloudFront
myDistribution:
Type: 'AWS::CloudFront::Distribution'
Properties:
DistributionConfig:
Origins:
- DomainName: !Join
- '.'
- - !ImportValue
reinvent2018RestApiId
- "execute-api.us-east-
1.amazonaws.com"
OriginPath: "/Prod"
Id: myCustomOrigin
CustomOriginConfig:
HTTPPort: '80'
HTTPSPort: '443'
OriginProtocolPolicy: https-only
Enabled: 'true'
Comment: reinvent2018
DefaultRootObject: index.html
Aliases:
- !Ref AliasName
DefaultCacheBehavior:
MinTTL: 0
DefaultTTL: 60
MaxTTL: 60
TargetOriginId: myCustomOrigin
ViewerProtocolPolicy: redirect-to-https
ForwardedValues:
QueryString: 'true'
QueryStringCacheKeys:
- '*'
LambdaFunctionAssociations:
- EventType: origin-request
LambdaFunctionARN: !Ref LambdaEdge.Version
PriceClass: PriceClass_100
ViewerCertificate:
CloudFrontDefaultCertificate: 'true'

Step 2.4: Extending Build Logic
buildspec.yml:
<rest of the commands>
# lambdaedge packaging
- aws cloudformation package --template template-cloudfront.yml --s3-bucket $S3_BUCKET --output-
template template-cloudfront-export.yml
artifacts:
type: zip
files:
- template-export.yml
- template-cloudfront-export.yml

Step 2.5: Pipeline changes

Step 2.5: Permission changes – pipeline role
- Action:
- 'cloudformation:DescribeStacks'
- 'cloudformation:DescribeChangeSet'
- 'cloudformation:CreateChangeSet'
- 'cloudformation:DeleteChangeSet'
- 'cloudformation:ExecuteChangeSet'
Resource: !Sub "arn:aws:cloudformation:${AWS::Region}:*:stack/awscodestar-
${ProjectId}-*/*"

Step 2.5: Permission changes – CloudFormation role
- Action:
- 'iam:*'
Resource:
- 'arn:aws:iam::*:role/awscodestar-reinvent2018*'
Effect: Allow
- Action:
- 'lambda:*'
Resource:
- 'arn:aws:lambda:us-east-1:*:function:*reinvent2018*'
Effect: Allow
- Action:
- 'iam:CreateServiceLinkedRole'
- 'cloudfront:*'
Resource:
- '*'
Effect: Allow

Step 2.5: Pipeline changes – generate change set
- Actions:
- ActionTypeId:
Owner: AWS
Category: Deploy
Version: 1
Provider: CloudFormation
Configuration:
TemplatePath: !Sub '${ProjectId}--BuildArtifact::template-cloudfront-export.yml'
ActionMode: CHANGE_SET_REPLACE
Capabilities: CAPABILITY_IAM
ParameterOverrides: !Sub '{"Stage": "Gamma", "ProjectId": "${ProjectId}", "AliasName":
"gamma.acarerkek.com"}'
ChangeSetName: pipeline-changeset
RoleArn: !GetAtt CloudFormationTrustRole.Arn
StackName: !Sub 'awscodestar-${ProjectId}-lambdaedge-Gamma'
InputArtifacts:
- Name: !Sub '${ProjectId}-BuildArtifact'
OutputArtifacts: []
RunOrder: 1
Name: GenerateChangeSet

Step 2.5: Pipeline changes – execute change set
- ActionTypeId:
Owner: AWS
Category: Deploy
Version: 1
Provider: CloudFormation
Configuration:
ActionMode: CHANGE_SET_EXECUTE
ChangeSetName: pipeline-changeset
StackName: !Sub 'awscodestar-${ProjectId}-lambdaedge-Gamma'
InputArtifacts: []
OutputArtifacts: []
RunOrder: 2
Name: ExecuteChangeSet
Name: DeployCloudFront-Gamma

Step 2.5: Pipeline changes - deploying

Test

Step 3: Unit testing

Step 4: Integration testing in gamma

Step 4.1: Integration testing—Code
it('should return 200 and default page', function(done) {
console.log('https://' + process.env.TEST_URL);
https.get('https://' + process.env.TEST_URL, (response) => {
let body = '';
response.on('data', (chunk) => {
body += chunk;
});
response.on('end', () => {
test.number(response.statusCode).is(200);
test.string(body).match(/Congratulations/);
done();
});
});
});
/* other tests */

Step 4.2: Integration testing – build projects

Step 4.2: Integration testing – build projects
CodeBuildProjectGammaTesting:
Condition: CreateCodeBuildResources
DependsOn:
- CodeBuildPolicy
Type: 'AWS::CodeBuild::Project'
Properties:
Artifacts:
Type: codepipeline
Packaging: zip
Description: 'CodeBuild Project for
gamma integration testing'
ServiceRole: !Ref CodeBuildRole
Environment:
Type: LINUX_CONTAINER
EnvironmentVariables:
- Value:
d16rtuhd4dgmx4.cloudfront.net
Name: TEST_URL
Image: 'aws/codebuild/eb-nodejs-
6.10.0-amazonlinux-64:4.0.0'
ComputeType: small
Source:
Type: codepipeline
BuildSpec: integ-
tests/buildspec.yml
Name: !Sub '${ProjectId}-integtest-
Gamma'

Step 4.3: Integration testing - buildspec
version: 0.2
phases:
install:
commands:
# Install dependencies needed for running tests
- npm install
# Upgrade AWS CLI to the latest version
- pip install --upgrade awscli
pre_build:
commands:
# Discover and run unit tests in the 'tests' directory
- npm run-script integ-test

Step 4.4: Integration testing – pipeline
- Actions:
- ActionTypeId:
Owner: AWS
Category: Test
Version: 1
Provider: CodeBuild
Configuration:
ProjectName: !Sub '${ProjectId}-integtest-Gamma'
InputArtifacts:
- Name: !Sub '${ProjectId}-SourceArtifact'
RunOrder: 1
Name: RunIntegTests
Name: IntegTestingGamma

Step 4.4: Integration testing

Release

Step 5: Release

Step 5: Release—Production stages

See how it works
• Default start page: https://dow1m4ilcudkt.cloudfront.net/
• Beta start page: https://dow1m4ilcudkt.cloudfront.net/?newLook=true

Recap
• AWS CodeStar project
• Lambda@Edge and CloudFront in continuous deployment
• Integration testing
• Releasing to production

MONITOR

CloudWatch metrics—CloudFront
• Requests
• BytesDownloaded
• BytesUploaded
• TotalErrorRate
• 4xxErrorRate
• 5xxErrorRate

CloudWatch logs—CloudFront/Lambda@Edge
• Invalid response logs from Lambda@Edge

CloudWatch metrics—AWS WAF & AWS Shield
Advanced
AWS Shield Advanced
• DDoSDetected
• DDoSAttackBitsPerSecond
• DDoSAttackPacketsPerSecond
• DDoSAttackRequestsPerSecond
AWS WAF
• AllowedRequests
• BlockedRequests
• CountedRequest

AWS CloudTrail

Set up a reporting pipeline
Athena
CloudFront
Access Logs
Amazon S3
AWS Glue
AWS Glue Catalog
Data Source
Discovery
Query data
• Amazon CloudFront access logs are
available in Amazon Simple Storage Service
(Amazon S3)
• Use AWS Glue to discover your data
sources, including your log bucket
• Build a data catalog of data and make it
available to Athena
• Use Amazon QuickSight to analyze the data
• Athena will query your S3 bucket
• Simple and serverless!

Export access logs to your own monitoring tools

Gourav Tiwari

Attracts the serious home buyer

Key part of News Corp

Proptiger.com
Housing.com
makaan
Largest global digital RE network

Transformation
• 20-year-old company acquired by News
Corp in November 2014
• 2015: Transform to compete
• Put the customer first
• Iterate on customer feedback
• Improve product velocity and quality
• Rewrite core products

Moving tech-stack to cloud
• DevOps/Agile/Cloud
• Key principles
• Keep it simple
• Design services
• Cloud first (AWS)
• Utilize AWS products and services

Front-end applications on AWS in first 6 months

Use cases

A/B test

A/B test requirements
• Tag the user to new experience from very
first request based on micro-market
• Ability to show variations from two or more
applications(Origins)
• Acceptable latency < 30ms
• Ability to unit test the custom solution
• Easy deployments

A/B test with Lambda@Edge
 Users are tagged from first request with
updated origins
 Serve content from multiple applications
 Acceptable latency < 30ms
 Unit testing
 Jenkins jobs to deploy Lambda functions
and CDN updates

A/B test architecture
Content switch gateway
Rules engine
Conditions
BehaviorsDNS provider
Consumers

New use case

Previous content switch gateway
• 70+ rules for content switching
• Each rule had multiple sub-rules
• Manual changes, manual testing
• Limited deployments ~ one or two times a week
• Limited visibility in monitoring
• Fixed-price model

Migration to CloudFront and Lambda@Edge
• Jenkins pipeline to automate:
• CI/CD
• Continuous testing
• Infrastructure as code
• Configuration versioning via GitHub Enterprise
• Continuous monitoring
• CloudWatch
• AWS X-Ray
• New Relic

CloudFront and Lambda@Edge
Rules engine
Conditions
Behaviors
DNS provider
Consumers
S3 Bucket
access logs

Impact
• Simplified changes for Lambda@Edge
and CloudFront
1. Check-in new changes
2. PR review/comments/merge
3. Total deployment time ~15 minutes
• DevOps team in full control
• Versioned infrastructure to replicate
prod/beta environments

CI/CD with Jenkins

Visibility in monitoring & logging

Result
• Unit tests: 600+
• Average latency: < 20 ms
• Lines of code: ~1200
• Improved velocity
• Check-in to deployment time: ~15 minutes
• Frequent releases
Number of Deployments after
0
1
2
3
1 2 3 4 5 6 7 8
Number of Deployments before
0
2
4
6
8
1 2 3 4 5 6 7 8

Lessons learned / best practices
• Importance of Testing
• Test the AWS WAF rules
• End-to-end performance test vs isolated lambda function’s performance test
• Automation facilitates velocity
• Devops facilitates agility
• No manual changes!
• Leverage tools for better monitoring
• Keep an eye on Lambda Throttling, Cloudwatch metrics are helpful
• Collaboration is key to make DevOps success!

What’s next
• Resize images at run time via CloudFront
• Next domains to migrate
• Lambda blue-green deployment
• ECS blue-green deployment

Key Takeaways
• Consider the Lambda@Edge as part of your application stack
• Automate configuration of the Lambda@Edge
• Build Automation
• Test Automation
• Monitor the Lambda@Edge using CloudTrail and CloudWatch
Check out the new features in:
• AWS Lambda and Lambda@Edge
• AWS API Gateway
• AWS SAM
• AWS WAF and AWS Shield
• AWS CodeStar services
• AWS Cloud9
• AWS CloudFormation
• CloudFront

Thank you!
Chanka Perera
Solutions Architect
AWS
Acar Erkek
Sr. SDE Cloudfront
AWS
Gourav Tiwari
Realtor.com

High Velocity DevOps: Four Ways to Leverage CloudFront in Faster DevOps Workflows (CTD409) - AWS re:Invent 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to High Velocity DevOps: Four Ways to Leverage CloudFront in Faster DevOps Workflows (CTD409) - AWS re:Invent 2018

Similar to High Velocity DevOps: Four Ways to Leverage CloudFront in Faster DevOps Workflows (CTD409) - AWS re:Invent 2018 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

High Velocity DevOps: Four Ways to Leverage CloudFront in Faster DevOps Workflows (CTD409) - AWS re:Invent 2018