The current implementation of AWS Lambda at JUST EAT

1. AWS Lambda at JUST EAT
ANDREW BROWN - Senior Platform Engineer

2. Overview
Just Eat
Early Adoption
Use Cases
Serverless

3. • The world’s leading digital marketplace for takeaway food delivery
• Founded in 2001
• Operate in 13 markets around the globe
JUST EAT
• Platform peak has processed 2,500 orders per
minute
• Thats a lot of Food!

4. TECHNOLOGY
• Run hundreds of EC2 Instances at peak / dinner time
• Scheduled Scaling of Resources - EC2, Dynamo Capacity
• Heavily utilise CloudFormation
• Just Eat Migrated to AWS 5 years ago - All In!

5. AWS - Setup
• Multiple AWS Accounts
• QA Daily Launch and Teardown
• On average an instance less than 3 days old
•
• Approx 150 “features” - Each Feature has a Stack

6. What’s good about our Setup?
Developers can concentrate on writing code
• Packer, CodeDeploy, Consul, Cloudformation
• Developers create package and deploy!
• PaaS, IaaS

7. What’s good about our Setup?
Consistency
• Many accounts, environments, countries, features
• All get the same AWS resources - e.g. Security Groups
• Templated Cloudformations

8. AWS - Lambda
• Re:Invent 2014 announcement
• AWS Lambda runs your code in response to events
•
• “we could use that for x, y, and z!”

9. So Why Use Lambda?
• Lightweight
• Cheaper
• Less to manage
• Quicker to Deploy

10. But Why Should JE Use Lambda?
Let’s review One Feature
• CloudFormation
• EC2 Instances x 3 - Multi AZ
• EBS Volumes
• ELB Endpoint and Health Check
• IAM Roles
• DNS
• Security Groups, Subnets, VPC
• ASG settings, LaunchConfiguration
~ 1600 lines of JSON
$0.246 x 3 x 24 x 365 (m4.large)
$0.10 x 30 x 3 x 12 (gp2)
$0.028 x 24 x 365 (classic)
~= $7k

11. So Why Use Lambda?
What does this Feature do?
• Resets Delivery Time every morning
• Publishes a SNS message about an order
• Provision access for an instance
• Tidy up resources

12. Early Lambdas
● New Technology!
● Multiple languages and frameworks
● Seen as additional parts of a feature
● Using the Console….

13. Early Lambdas
Limitations / Problems
● Consistency (that’s Just Eat not AWS!)
● No VPC Support (Added Feb 2016)
● No Cron in Cloudformation
● Cloudformation Network Still Attached

14. Use Cases - PaaSBot
• CloudFormation Stack per Environment
• Windows EC2 Instance
• Ran a bunch of Scheduled Tasks

15. Use Cases - PaaSBot
Clean up EBS Volumes
• Cron build in Teamcity - one per env
• Scheduled Windows Task (one per account, assumed roles)
• Lambda?
Report Metrics
• T2 Credit Balances
• AWS Limits
• Lambda?

16. Use Cases - PaaSBot
Security Audit
• Check Security Group rules
• Check User access

17. Use Cases - PaaSLambda
Lambdas
• No Windows Instance(s)
• Micro-Scheduled Tasks
• Easier to update
• Better visibility

18. Use Cases - Instance Access
• Recently started using HashiCorp Consul and Vault
• Concerns : “What can my instance access?”
• Consul ACLS and Vault Tokens - limit what key/values you can access
• ACLs Initially provisioned via Deployment Tool

19. Use Cases - Instance Launch
• Lambdas Created with Consul and Vault Cloudformations - nodejs
• ASG Event -> CloudWatch Events -> Trigger Lambda
• Provisions ACL and AppId login
• Removes Access when Instance terminated

20. Use Cases - Instance Launch
Launch
Stack
i-12345
i-12345
i-12345
ASG
CloudWatch
Event
{ “acl” : a1a1a1-b2b2b2 }
{ “token” : secrettoken }

21. Use Cases - Instance Terminate
Delete Stack /
Scale Down
i-12345
i-12345
i-12345
ASG
CloudWatch
Event

22. Different Approach
• Starting to create Lambda Features
• Needed a consistent approach
• Recently Implemented Serverless

23. Serverless
• Still in “Beta”

24. Serverless
Good Points
• Brings Back Consistency
• Ease to deploy
• Defined Dependencies
• Being Quickly Adopted
• Track signal strength
• Calculate cost
• Check capacity
• Start Ad-hoc backup / Prune Log files

25. Serverless
Not so Good Points
• Currently each Feature includes their AWS resources -
previously templated
• Code repeated
• Monitoring & Logging Immature

26. Lessons Learnt & Future Plans
• Difficult to integrate into our Platform - Versatility
• Still a few Bugs -
• Serverless - Based on Initial Uptake
• AWS Lambda & API Gateway
• Will it replace our APIs? - Maybe

27. Any questions?

28. Thank you