Building robust applications for the cloud (AWS lambda / API gateway)

1. @garysieling
CLOUD-NATIVE JAVASCRIPTCLOUD-NATIVE JAVASCRIPT
APPSAPPS

3. @garysieling

4. @garysieling
PROJECTSPROJECTS
Cumulus
Pangeo

5. @garysieling

7. @garysieling

9. @garysieling

10. @garysieling
COSTCOST
DevOps specialists and site reliability
engineers are among the highest paid,
most experienced developers most
satisfied with their jobs, and are
looking for new jobs at the lowest
levels.

11. @garysieling
ROBUST SERVERLESS SOFTWAREROBUST SERVERLESS SOFTWARE
Stable
Secure
Affordable

12. @garysieling
CASE STUDY:CASE STUDY:
SATELLITE IMAGERYSATELLITE IMAGERY
DATA PIPELINEDATA PIPELINE

13. @garysieling

14. @garysieling
DIGITALGLOBEDIGITALGLOBE
80 TB/day
100 PB Stored

16. @garysieling

17. @garysieling

18. @garysieling
LEAFLET.JSLEAFLET.JS
const position = [39.9526, -75.1652 ];
const Map = () =>
<LeafletMap center={position} zoom={11}>
<TileLayer
attribution='© contributors'
url='https://{s}.tile.osm.org/{z}/{x}/{y}.png'
/>
<Marker position={position}>
<Popup>
A <b>popup</b>.
</Popup>
</Marker>
</LeafletMap>
OpenStreetMap

19. HISTORIC AERIAL FOOTAGEHISTORIC AERIAL FOOTAGE

20. @garysieling

21. @garysieling
CLOUD NATIVECLOUD NATIVE

22. @garysieling
DEVELOPMENT PARADIGMSDEVELOPMENT PARADIGMS
Desktop
Mobile
IOT

24. @garysieling

25. @garysieling

26. @garysieling
BRING YOUR OWN VS. MANAGEDBRING YOUR OWN VS. MANAGED
SaaS
Functions
Scaling
Runtime
OS
Virtualization
Hardware

27. @garysieling
STABILITYSTABILITY

28. @garysieling
APP ARCHITECTUREAPP ARCHITECTURE
S3 hosted front-end
JavaScript Microservices (API Gateway + Lambda)
RDS database, with automatic backups
A private link to a customer account

29. @garysieling
GATSBYGATSBY
Wraps webpack
Builds static sites / single page apps
JAMStack - REST apis or Graphql

30. @garysieling
MICROSERVICES - STABILITYMICROSERVICES - STABILITY
Multiple copies of the app
Servers / containers dedicated to modules
Switch out APIs without code changes

31. MAP TILESMAP TILES

32. @garysieling

33. @garysieling
GEOTIFF.JS - TILEGEOTIFF.JS - TILE

34. @garysieling
WEB UI BACKEND - TILESWEB UI BACKEND - TILES
Hosted - Mapbox, OpenStreetMap
Folder structure: zoom + location
S3 - byte ranging

35. @garysieling
DEVOPSDEVOPS
export class MyEcsConstructStack extends core.Stack {
constructor(scope: core.App, id: string, props?: core.StackProps) {
super(scope, id, props);
let vpc = new ec2.Vpc(this, "MyVpc", {});
let cluster = new ecs.Cluster(this, "MyCluster", {vpc});
new ecs_patterns.ApplicationLoadBalancedFargateService(this, {
cluster: cluster,
desiredCount: 6,
taskImageOptions: {
image: ecs.ContainerImage.fromRegistry("amazon/sample")
},
memoryLimitMiB: 2048
});
}
}

36. @garysieling
SECURITYSECURITY

37. @garysieling
SECURITY - TACTICSSECURITY - TACTICS
Limit blast radius
Least privilege
Outsource the hard parts

38. @garysieling
FUNCTION CONTEXTFUNCTION CONTEXT
(x) => x * x

39. @garysieling
FUNCTION CONTEXTFUNCTION CONTEXT
(x) => <h1>{x}</h1>

40. @garysieling
FUNCTION CONTEXTFUNCTION CONTEXT
(x, y) =>
db.query(
`update data set x = $1 where y = $2`,
[x, y])

41. @garysieling
EXECUTION CONTEXTEXECUTION CONTEXT
Logging (Cloudwatch)
Container
Defined memory/CPU/disk
Security groups
ENI

42. @garysieling
AFFORDABILITYAFFORDABILITY

43. @garysieling
FOREST FIREFOREST FIRE

44. @garysieling
DRONE PATHDRONE PATH

45. @garysieling
PROCESSING PIPELINEPROCESSING PIPELINE
Science code (Docker, Python, ...)
JavaScript - Glue code
Web UIs for monitoring

46. @garysieling
SCIENCE CODESCIENCE CODE

47. @garysieling
PROJECTIONPROJECTION

48. @garysieling
GROUND CONTROL POINTGROUND CONTROL POINT

49. @garysieling
const sqs = new AWS.SQS({apiVersion: '2012-11-05'});
const queueURL = "SQS_QUEUE_URL";
const params = { QueueUrl: queueURL };
sqs.receiveMessage(params, (err, data) => {
...CALL YOUR LAMBDA HERE...
const deleteParams = {
QueueUrl: queueURL,
ReceiptHandle: data.Messages[0].ReceiptHandle
};
sqs.deleteMessage(deleteParams, ...);
}})

50. @garysieling
CLOUD - RESOURCESCLOUD - RESOURCES
CPU
Memory
Disk
Network

51. @garysieling
A SOFTWARE ARCHITECT CAN PLANA SOFTWARE ARCHITECT CAN PLAN
USING FUTURE COSTSUSING FUTURE COSTS
Now: Story points (time + risk)
Future: Operational cost (NPV / cost shape)

52. @garysieling
COLLECTIONSCOLLECTIONS

53. @garysieling
METRICS FOR RELIABILITYMETRICS FOR RELIABILITY
How long will our so ware continue to build?
How long will our so ware continue to deploy?
How long will our so ware continue to work?

54. @garysieling
REFERENCESREFERENCES
https://start.jcolemorrison.com/aws-vpc-core-
concepts-analogy-guide/
https://www.jpl.nasa.gov/spaceimages/details.php?
id=PIA14897
http://learningzone.rspsoc.org.uk/index.php/Learning
Materials/Python-Scripting/9.4-Calculate-NDVI-using-
GDAL