The document discusses moving a Slack commands application from a single Express app hosting all commands to individual serverless functions (Lambda) per command. It provides examples of the code structure for a sample "user stats" command function, how it is executed by AWS Lambda, and how CloudFormation templates are used to define and deploy the Lambda function and its resources to AWS.

1. from
Big Band Web App
to
Serverless
Bebop
@AnjanaVakil JazzCon.Tech 2018

2. Hello!
I’m @AnjanaVakil
Recurse Center Non-Graduate
Mozilla TechSpeaker & Outreachy Alumna
Engineering Learning & Development Lead, Mapbox

3. location data platform for developers
maps, search, navigation
web, iOS, Android, Unity...
@AnjanaVakil JazzCon.Tech 2018

4. @AnjanaVakil JazzCon.Tech 2018

5. we use to
◍ get user stats
◍ monitor/test unpacker
uploads
◍ ask yoda who’s on call
◍ see who’s out tmrw
◍ pick a random teammate
Place your screenshot here
@AnjanaVakil JazzCon.Tech 2018

6. user-
limits
out
unpackeryoda
stats
Express
slack i/o
auth
our slack-commands app
node
@AnjanaVakil JazzCon.Tech 2018

7. codebase
slack-commands/
├─ readme.md
├─ cloudformation/ #app & cmd config
│ └─ slack-commands.template.js
├─ index.js #express app
├─ commands/
│ ├─ out.js
│ ├─ stats.js
│ └─ ...
└─ test/
├─ index.test.js
├─ out.test.js
├─ stats.test.js
└─ ...
@AnjanaVakil JazzCon.Tech 2018

8. All commands in a single app: Downsides
Security
◍ Permissions?
Secrets?
◍ Least privilege
Maintenance
◍ Many different
teams involved
◍ Ownership?
Support?
◍ Code gumbo
Cost
◍ Always running
◍ No per-command
breakdown
@AnjanaVakil JazzCon.Tech 2018

9. enough
about computers
let’s talk about jazz
@AnjanaVakil JazzCon.Tech 2018

10. “
While swing music tended to feature
orchestrated big band arrangements,
bebop music highlighted improvisation.
@AnjanaVakil JazzCon.Tech 2018
wikipedia

11. Big Band
Glenn Miller Orchestra, c. 1940

12. Big Band
out unpacker
status
yoda
Express
slack i/o
auth
user-
limits
@AnjanaVakil JazzCon.Tech 2018

13. “I kept thinking there's bound to be something
else. I could hear it sometimes. I couldn't play it....
I found that by using the higher intervals of a
chord as a melody line and backing them with
appropriately related changes, I could play the
thing I'd been hearing. It came alive.
- Charlie Parker
@AnjanaVakil JazzCon.Tech 2018
wikipedia

14. Bebop
Tommy Potter, Charlie Parker, Max Roach, Miles Davis, & Duke Jordan, NYC c. 1945

15. “As bebop was not intended for dancing, it
enabled the musicians to play at faster tempos.
Bebop musicians explored advanced harmonies,
complex syncopation, altered chords, extended
chords, chord substitutions, asymmetrical
phrasing, and intricate melodies.
@AnjanaVakil JazzCon.Tech 2018
wikipedia

16. Bebop
out
unpacker
status
yoda user-limits
@AnjanaVakil JazzCon.Tech 2018

17. “there’s bound to be something else”
◍ Assign a single owner/gatekeeper?
◍ Multiple single-command apps?
◍ Separate commands from router app
@AnjanaVakil JazzCon.Tech 2018

18. “serverless” functions to the rescue
AWS Lambda
@AnjanaVakil JazzCon.Tech 2018

19. what do you mean “serverless”
Actual Server
You own a computer
You put code on it
You run it constantly
(and you pay constantly)
You keep it healthy
Cloud Server
AWS owns a computer (or 2)
You put code on it
AWS runs it constantly
(and you pay constantly)
You tell AWS how to keep
it healthy
“No” Server
AWS owns a computer
You put code on it
AWS runs it when you ask
(and you pay only then)
AWS keeps it healthy
@AnjanaVakil JazzCon.Tech 2018

20. ◍ Only concern: Input -> Output
◍ No state maintained between calls
◍ Can be side-effecting, though
(e.g. API call, database write)
◍ Limited resources & exec time (5m)
what do you mean “function”
@AnjanaVakil JazzCon.Tech 2018

21. Lose the server if it’s...
◍ small
◍ short-lived
◍ self-contained
◍ needed occasionally
to lambda or not to lambda
Keep the server if it’s...
◍ heavy
◍ long-running
◍ interdependent
◍ needed constantly
@AnjanaVakil JazzCon.Tech 2018

22. HTTP API
endpoint
AWS
load
balancerEC2s on ECS
JS
JSJS
old architecture
@AnjanaVakil JazzCon.Tech 2018

23. HTTP API
endpoint
router
new architecture
stats
out
unpacker
yoda
@AnjanaVakil JazzCon.Tech 2018

24. old
codebase
slack-commands/
├─ readme.md
├─ cloudformation/ #app & cmd config
│ └─ slack-commands.template.js
├─ index.js #express app code
├─ commands/ #cmd code
│ ├─ out.js
│ ├─ stats.js
│ └─ ...
└─ test/ #app & cmd tests
├─ index.test.js
├─ out.test.js
├─ stats.test.js
└─ ...
@AnjanaVakil JazzCon.Tech 2018

25. new
codebase
slack-commands/
├─ readme.md
├─ cloudformation/ #app config
│ └─ slack-commands.template.js
├─ index.js #express app code
├─ commander.js #invoke lambdas
└─ test/
├─ commander.test.js
└─ index.test.js
slack-command-{cmd}/
├ readme.md
├ cloudformation/ #cmd config
│ └ slack-command-{cmd}.template.js
├ index.js #cmd code
└ test/
└ index.test.js@AnjanaVakil JazzCon.Tech 2018

26. what does a
function
look like?
@AnjanaVakil JazzCon.Tech 2018

27. Place your screenshot here
user stats commmand
@AnjanaVakil JazzCon.Tech 2018

28. module.exports.command = (event, context, callback) => {
const { ApiCoreUrl, MapboxToken } = process.env;
const user = event.args[0];
const from = new Date(event.args[1] || (+new Date - 864e5*30)); // 30d
const to = new Date(event.args[2] || (+new Date));
if (isNaN(from)||isNaN(to)) return callback(new Error('invalid date'));
const requrl = getStatsUrl(user, from, to, ApiCoreUrl, MapboxToken);
request(requrl, (err, res, body) => {
if (err||res.statusCode !== 200) return callback(err||res.statusCode);
try { const data = JSON.parse(body); }
catch(err) { return callback(err); }
return callback(null, formatStatsMessage(user, data));
});
}
@AnjanaVakil JazzCon.Tech 2018

29. module.exports.command = (event, context, callback) => {
const { ApiCoreUrl, MapboxToken } = process.env;
const user = event.args[0];
const from = new Date(event.args[1] || (+new Date - 864e5*30)); // 30d
const to = new Date(event.args[2] || (+new Date));
if (isNaN(from)||isNaN(to)) return callback(new Error('invalid date'));
const requrl = getStatsUrl(user, from, to, ApiCoreUrl, MapboxToken);
request(requrl, (err, res, body) => {
if (err||res.statusCode !== 200) return callback(err||res.statusCode);
try { const data = JSON.parse(body); }
catch(err) { return callback(err); }
return callback(null, formatStatsMessage(user, data));
});
}
handler function
(we tell Lambda its name)
@AnjanaVakil JazzCon.Tech 2018

30. module.exports.command = (event, context, callback) => {
const { ApiCoreUrl, MapboxToken } = process.env;
const user = event.args[0];
const from = new Date(event.args[1] || (+new Date - 864e5*30)); // 30d
const to = new Date(event.args[2] || (+new Date));
if (isNaN(from)||isNaN(to)) return callback(new Error('invalid date'));
const requrl = getStatsUrl(user, from, to, ApiCoreUrl, MapboxToken);
request(requrl, (err, res, body) => {
if (err||res.statusCode !== 200) return callback(err||res.statusCode);
try { const data = JSON.parse(body); }
catch(err) { return callback(err); }
return callback(null, formatStatsMessage(user, data));
});
}
execution environment
we can configure
@AnjanaVakil JazzCon.Tech 2018

31. module.exports.command = (event, context, callback) => {
const { ApiCoreUrl, MapboxToken } = process.env;
const user = event.args[0];
const from = new Date(event.args[1] || (+new Date - 864e5*30)); // 30d
const to = new Date(event.args[2] || (+new Date));
if (isNaN(from)||isNaN(to)) return callback(new Error('invalid date'));
const requrl = getStatsUrl(user, from, to, ApiCoreUrl, MapboxToken);
request(requrl, (err, res, body) => {
if (err||res.statusCode !== 200) return callback(err||res.statusCode);
try { const data = JSON.parse(body); }
catch(err) { return callback(err); }
return callback(null, formatStatsMessage(user, data));
});
}
input { args: [ “vakila”, “1/1”, “3/22”] }
@AnjanaVakil JazzCon.Tech 2018

32. module.exports.command = (event, context, callback) => {
const { ApiCoreUrl, MapboxToken } = process.env;
const user = event.args[0];
const from = new Date(event.args[1] || (+new Date - 864e5*30)); // 30d
const to = new Date(event.args[2] || (+new Date));
if (isNaN(from)||isNaN(to)) return callback(new Error('invalid date'));
const requrl = getStatsUrl(user, from, to, ApiCoreUrl, MapboxToken);
request(requrl, (err, res, body) => {
if (err||res.statusCode !== 200) return callback(err||res.statusCode);
try { const data = JSON.parse(body); }
catch(err) { return callback(err); }
return callback(null, formatStatsMessage(user, data));
});
}
aws runtime info
(e.g. time remaining - ignored here)
@AnjanaVakil JazzCon.Tech 2018

33. module.exports.command = (event, context, callback) => {
const { ApiCoreUrl, MapboxToken } = process.env;
const user = event.args[0];
const from = new Date(event.args[1] || (+new Date - 864e5*30)); // 30d
const to = new Date(event.args[2] || (+new Date));
if (isNaN(from)||isNaN(to)) return callback(new Error('invalid date'));
const requrl = getStatsUrl(user, from, to, ApiCoreUrl, MapboxToken);
request(requrl, (err, res, body) => {
if (err||res.statusCode !== 200) return callback(err||res.statusCode);
try { const data = JSON.parse(body); }
catch(err) { return callback(err); }
return callback(null, formatStatsMessage(user, data));
});
}
“ok, I’m done” function
(AWS passes this when executing)
error data
@AnjanaVakil JazzCon.Tech 2018

34. we wrote a function!
yay
@AnjanaVakil JazzCon.Tech 2018

35. do we get it up there?
how
@AnjanaVakil JazzCon.Tech 2018

36. Cloud::Formation templates
◍ JSON template - “just code”
◍ define your AWS stack components
◍ upload -> AWS builds your stack
@AnjanaVakil JazzCon.Tech 2018

37. Cloud::Formation templates
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "slack-command-stats",
"Parameters": { "ApiCoreUrl": {...}, "MapboxToken": {...}
"Resources": {
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {...}
},
"CommandRole": {
"Type": "AWS::IAM::Role",
"Properties": {...}
}
},
"Outputs": { ... }
}
@AnjanaVakil JazzCon.Tech 2018

38. Cloud::Formation templates
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "slack-command-stats",
"Parameters": { "ApiCoreUrl": {...}, "MapboxToken": {...}
"Resources": {
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {...}
},
"CommandRole": {
"Type": "AWS::IAM::Role",
"Properties": {...}
}
},
"Outputs": { ... }
}
stack
input
params
@AnjanaVakil JazzCon.Tech 2018

39. Cloud::Formation templates
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "slack-command-stats",
"Parameters": { "ApiCoreUrl": {...}, "MapboxToken": {...}
"Resources": {
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {...}
},
"CommandRole": {
"Type": "AWS::IAM::Role",
"Properties": {...}
}
},
"Outputs": { ... }
}
aws
permissions
for function
@AnjanaVakil JazzCon.Tech 2018

40. Cloud::Formation templates
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "slack-command-stats",
"Parameters": { "ApiCoreUrl": {...}, "MapboxToken": {...}
"Resources": {
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {...}
},
"CommandRole": {
"Type": "AWS::IAM::Role",
"Properties": {...}
}
},
"Outputs": { ... }
}
actual
lambda
function
the good stuff
@AnjanaVakil JazzCon.Tech 2018

41. Cloud::Formation templates
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "slack-command-stats-production",
"Code": { "S3Bucket": {...}, "S3Key": {...} },
"Handler": "index.command",
"Environment": { "Variables": {
"ApiCoreUrl": { "Ref": "ApiCoreUrl" },
"MapboxToken": { "Ref": "MapboxToken" } } },
"Role": { "Fn::GetAtt": ["CommandRole","Arn"] },
"Runtime": "nodejs4.3", "MemorySize": 128, "Timeout": 60,
"Tags": [ { "Key": "Team", "Value": "EngOps"} ]
}
}
@AnjanaVakil JazzCon.Tech 2018

42. Cloud::Formation templates
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "slack-command-stats-production",
"Code": { "S3Bucket": {...}, "S3Key": {...} },
"Handler": "index.command",
"Environment": { "Variables": {
"ApiCoreUrl": { "Ref": "ApiCoreUrl" },
"MapboxToken": { "Ref": "MapboxToken" } } },
"Role": { "Fn::GetAtt": ["CommandRole","Arn"] },
"Runtime": "nodejs4.3", "MemorySize": 128, "Timeout": 60,
"Tags": [ { "Key": "Team", "Value": "EngOps"} ]
}
}
@AnjanaVakil JazzCon.Tech 2018
function
name
helps find it later

43. Cloud::Formation templates
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "slack-command-stats-production",
"Code": { "S3Bucket": {...}, "S3Key": {...} },
"Handler": "index.command",
"Environment": { "Variables": {
"ApiCoreUrl": { "Ref": "ApiCoreUrl" },
"MapboxToken": { "Ref": "MapboxToken" } } },
"Role": { "Fn::GetAtt": ["CommandRole","Arn"] },
"Runtime": "nodejs4.3", "MemorySize": 128, "Timeout": 60,
"Tags": [ { "Key": "Team", "Value": "EngOps"} ]
}
}
@AnjanaVakil JazzCon.Tech 2018
code to run
the good stuff

44. Cloud::Formation templates
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "slack-command-stats-production",
"Code": { "S3Bucket": {...}, "S3Key": {...} },
"Handler": "index.command",
"Environment": { "Variables": {
"ApiCoreUrl": { "Ref": "ApiCoreUrl" },
"MapboxToken": { "Ref": "MapboxToken" } } },
"Role": { "Fn::GetAtt": ["CommandRole","Arn"] },
"Runtime": "nodejs4.3", "MemorySize": 128, "Timeout": 60,
"Tags": [ { "Key": "Team", "Value": "EngOps"} ]
}
}
@AnjanaVakil JazzCon.Tech 2018
execution
environment
can pass in params

45. Cloud::Formation templates
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "slack-command-stats-production",
"Code": { "S3Bucket": {...}, "S3Key": {...} },
"Handler": "index.command",
"Environment": { "Variables": {
"ApiCoreUrl": { "Ref": "ApiCoreUrl" },
"MapboxToken": { "Ref": "MapboxToken" } } },
"Role": { "Fn::GetAtt": ["CommandRole","Arn"] },
"Runtime": "nodejs4.3", "MemorySize": 128, "Timeout": 60,
"Tags": [ { "Key": "Team", "Value": "EngOps"} ]
}
}
@AnjanaVakil JazzCon.Tech 2018
permissions
from earlier

46. Cloud::Formation templates
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "slack-command-stats-production",
"Code": { "S3Bucket": {...}, "S3Key": {...} },
"Handler": "index.command",
"Environment": { "Variables": {
"ApiCoreUrl": { "Ref": "ApiCoreUrl" },
"MapboxToken": { "Ref": "MapboxToken" } } },
"Role": { "Fn::GetAtt": ["CommandRole","Arn"] },
"Runtime": "nodejs4.3", "MemorySize": 128, "Timeout": 60,
"Tags": [ { "Key": "Team", "Value": "EngOps"} ]
}
}
@AnjanaVakil JazzCon.Tech 2018
what to run it on
totally not a server

47. Cloud::Formation templates
"Command": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "slack-command-stats-production",
"Code": { "S3Bucket": {...}, "S3Key": {...} },
"Handler": "index.command",
"Environment": { "Variables": {
"ApiCoreUrl": { "Ref": "ApiCoreUrl" },
"MapboxToken": { "Ref": "MapboxToken" } } },
"Role": { "Fn::GetAtt": ["CommandRole","Arn"] },
"Runtime": "nodejs4.3", "MemorySize": 128, "Timeout": 60,
"Tags": [ { "Key": "Team", "Value": "EngOps"} ]
}
}
@AnjanaVakil JazzCon.Tech 2018
arbitrary tags
e.g. who owns this?

48. Mapbox open-source AWS
helpers
github.com/mapbox/
Command-line tools
lambda-cfn create & deploy Node Lambda functions
cfn-config configure/start/update CFN stacks
JS libraries
cloudfriend easily assemble CFN templates in JS
decrypt-kms-env use secret environment vars
@AnjanaVakil JazzCon.Tech 2018

49. we deployed a function!
yay
@AnjanaVakil JazzCon.Tech 2018

50. do we call it?
how
@AnjanaVakil JazzCon.Tech 2018

51. testing in the AWS console (manually)
@AnjanaVakil JazzCon.Tech 2018

52. in Node with AWS SDK
const AWS = require('aws-sdk');
const runCommand = (req, res, next) => {
const [commandName, ...args] = = req.slackText;
const params = {
FunctionName: `slack-command-${commandName}-production`, // our convention
Payload: JSON.stringify({ args: args }), // the `event` Lambda receives
};
const lambda = new AWS.Lambda({ region: 'us-east-1' });
lambda.invoke(params).promise()
.catch((err) => err.message) // pass on error message as response data
.then((data) => res.json(formatForSlack(data)));
};
@AnjanaVakil JazzCon.Tech 2018

53. why
did we do all that?
@AnjanaVakil JazzCon.Tech 2018

54. Before (single app)
◍ Many secrets in one
stack
◍ Updating your code
updates whole stack
◍ No fine-grained cost
analysis
Refactoring to Lambda: Benefits
After (multiple Lambdas)
◍ Each stack only knows
its own secrets
◍ Updating your code
leaves others untouched
◍ Each stack/fn can be
tagged & cost-monitored
@AnjanaVakil JazzCon.Tech 2018

55. HTTP API
endpoint
router
next steps
stats
out
unpacker
yoda
@AnjanaVakil JazzCon.Tech 2018

56. HTTP API
endpoint
next steps
stats
out
unpacker
yoda
@AnjanaVakil JazzCon.Tech 2018
router

57. Merci!
@AnjanaVakil
anjana@mapbox.com
✌ Team Mapbox
Young Hahn, Emily McAfee,
Kelly Young, Jake Pruitt, Andrew Evans
JazzCon.Tech Organizers
Images from Wikimedia
Template by SlidesCarnival.com