Describes the process we use at Fender to achieve low build times while deploying to our Golang AWS Lambda microservices.

1. Build Process:
Fast Deployments To
Multiple Lambda Functions

2. Problem Description
• Multiple Microservices
• Each service has ~50 lambda functions
• Each function handling very specific task
• Build process uses CircleCI, running within containers
• Golang binary to be compiled for each function
– Golang uses all available CPU in build environment
– Compiling in parallel within build environment slows things down
– Build and test was taking ~25 minutes per app
2

3. Potential Solutions
• CircleCI caching
– Caching multiple binaries etc. takes up a lot of space
– More than 5 minutes devoted to save and restore cache
– Does not lower compile times sufficiently
• Parallel compilation
– Each “go build” appears to use all available CPU
– Parallel compile times actually appear slower than sequential
• Build using a build lambda
– Each build runs within an independent environment
– Horizontal scaling is automatic
– Helps to separate concerns during the build process
– DRY build code that can be re-used across all our apps
3

4. Inside the lambda environment
• Based on container technology
• Runs Amazon Linux
• CPU resources tied to configured memory size
• Maximum execution time of 5 minutes
– AWS might be increasing this limit in the future – talk to Support
• Only writable area is /tmp
– Ephemeral
• Files written to /tmp are persistent if the invocation uses an
existing “warm” container
– Limited to 512 MB
• AWS might be increasing this limit in the future – talk to Support
• Lambda handler runs as root user
4

5. Setting up the lambda env for builds
• Set GOPATH and GOROOT to live within /tmp
• Check if Golang is installed, install if required
– It will not be installed in a “cold” lambda container
– If not installed, download and install within /tmp/go
• Check if Apex is required, install if not already done
– We use Apex for older apps developed before native AWS Go
support, using a NodeJS shim around the Golang binary
– Apex is installed only if specified in the lambda’s event JSON
• Checkout application code
– Uses a Github “machine user” token set as an env variable
– Checks out into /tmp
5

6. Testing
• We use “go test” to run unit tests
• Coveralls.io for test/coverage reporting
• Test output is returned as part of the lambda’s response
JSON
– Output is base64-encoded
– CircleCI build script collates outputs from all build-lambda
invocations within a build, then uploads them to Coveralls
• If a test fails, lambda invocation fails
– This failure is detected by the CircleCI build scripts, which take
appropriate actions
6

7. Building
• We use “go build” to build the binary
• Code dependencies are vendored and stored in the
repository
• Built binary is uploaded to an S3 bucket specified in the
event JSON, using the Python AWS SDK
7

8. Invoking the build lambda from the CI
• Depend on convention to separate function code and shared
code
• Iterate through all directories containing code:
– Invoke build-lambda in test-only mode for shared code
– Invoke build-lambda in test+build mode for function code
• Use GNU Parallel to invoke build-lambda simultaneously
– Build AWS CLI commands for each invocation, save to file
– Pipe saved file through GNU Parallel to run commands in parallel
– Halt build if any build-lambda invocation fails
• Use JQ to parse lambda’s invocation result from AWS, and
take appropriate actions:
– Build error if .FunctionError is not null
– Base64-decode test coverage output and append to Coveralls
– Output .LogResult to STDOUT if .FunctionError is not null
8

9. Logging
• The build-lambda logs to Cloudwatch
• The CircleCI script invokes the build-lambda using the
“--log-type Tail” AWS CLI option
– This returns a field in the result named ”.LogResult”
– Contains the last 4K of logs, base64-encoded
• If the invocation failed, then the CircleCI script outputs
the base64-decoded contents of “.LogResult” to
STDOUT
• Build errors can be easily viewed on the CircleCI build
output view
• Further log processing can be done using Cloudwatch
9

10. Deploying built binaries to functions
• Use AWS CLI’s “update-function-code” command
• Directly deploy from S3 without need for file transfers
• Integration tests performed by downloading binaries from
S3 into the CircleCI environment, and running tests
against them
– This ensures that the actual binary that was tested is deployed to
the lambda function
10

11. Q & A
Q & A
11