The use of Hashicorp tools at JUST EAT is constantly evolving, this is a update on the progress over the past 12 months.

1. Andrew Brown - Senior Platform Engineer
Jan 18th 2016
HashiCorp at JUST EAT
Part II

2. Overview
JUST EAT
Consul
Vault
Summary

3. JUST EAT
“the world’s leading marketplace for online food delivery”
- Amazon Web Services
- Run hundreds of EC2 Instances at peak
- Customers Worldwide
- Microservices > 100
- Multiple accounts, environments
- Using HashiCorp tools in since 2015

4. Consul - Initial Rollout
- Cluster Per Environment
- Seed key/values from GitHub (git2consul)
- Expand keys : true
- Every EC2 Instance runs Consul Agent
- Each microservice registers a service and has custom ACL
- Used Consul-Template to generate .config files

5. Consul
eu-west-1a eu-west-1b eu-west-1c eu-west-1x
GIT2CONSUL

6. Consul Key/Values
Feature Configuration
/config/features/<featureName>/….
Environment Configuration
/config/environment/monitoring/….
Settings
/config/settings/messageRegion/….

7. Consul-Template
Setup
- Consul-Template pre-baked on AMIs (Packer)
- Service installed during deployment
- Consul-Template files bundled with feature package
- Generated *.config files (.Net)
Lessons Learnt
- Replacing static files meant more effort to run locally
- Worked well
- However... web.config and connectionstrings.config cannot be changed without cycling AppPool !

8. Consul Libraries
.NET Libraries
- using https://github.com/PlayFab/consuldotnet
JustEatConsul
- Setup Client
- Setup default endpoints - key prefixes
- Create configuration object recursively checking all key/values (remove prefix)
- Return configuration object:
var defaultTimeout = configurationRoot[“timeout”]

9. Consul Libraries
AppSettings
- Add AppSettings as a configuration source

10. Consul Watches
HttpReload
- Endpoint to trigger rebuild of the ConfigurationRoot
Consul Watch
- Windows (NSSM) Service setup during deployment
- “Watches” key prefixes for the feature deployed
- Invokes a WebRequest to local endpoint
- Only rebuild when a value used by the feature is changed

11. Consul Watches
Lessons Learnt
- Building functionality into the Code vs Static files
- Watch per key prefix…..
- Reliable

12. Consul ACLs
ACL per Feature
- Generated during Deployment
Lessons Learnt
- Adding new endpoints, meant a deployment to get new ACL
- Use events and triggers
- Avoid different rules across types of environment
- Create Ad-hoc ACLs...and tidy them afterwards!
- Clear browser local storage

13. Source of Truth
- Launched a Consul Cluster for Just Eat Source of Truth
- Accounts, Environments, Features, Teams
- UI has transformed JSON files to readable content
- Reliable
Use Case : Centralised Data

14. Use Case : Consul Locks
“Unique” Instance Value
- Require unique node name, used for :
- Consul Node
- Monitoring Stats
- Log Files
- Each EC2 needs an instance position
- Use AWS service SimpleDB - being retired….
Deployment
- Instance Locks key/value
- Finds free position,
- Allocates values
- Releases Lock

15. Lessons Learnt
- Very easy to setup
- Configurable - retries, timeouts, number of locks
- Reliable
Consul Locks
001
003
002

16. Vault
“Why aren’t you using Vault?”
- “We plan to….”
Approach
- Launch separate Vault Clusters
- Keep “config” distinct
- Not held in source control
- Backed by Consul
- Manually added passwords

17. Vault
eu-west-1a
eu-west-1b
eu-west-1c
eu-west-1a
eu-west-1b
eu-west-1c

18. Vault
Challenges
- Mechanism to create “tokens” for EC2 instances
- How to access the secrets
- Keeping it simple vs making it secure
- Backing up the data
- Mechanism to add secrets
- Alerting, Auditing, Logging

19. Vault - Token
Launch
Stack
i-12345
i-12345
i-12345
ASG
CloudWatch
Event
{ “acl” : a1a1a1-b2b2b2 }

20. .NET Library
- using https://github.com/rajanadar/VaultSharp
JustEatVault
- Use token generated during deployment
var password = configuration[“my_password”]
Vault - Accessing Secrets

21. Lessons Learnt
- Vault instances didn’t forward requests…..
- Look after your keys
- Read the docs slowly
Vault

22. - Consul, Packer, Vault, Nomad….
- Use available open source tools
- Newer versions solving workarounds
Summary

23. Summary
- Some roll-out/migration pain - ports, stats
- Read the docs
- Use the features that suit your needs
- Still making the case for Terraform

24. Questions?