Hashicorp Nomad and Lyft Envoy have been brought together as fundamental building blocks for Nelson, a new open source project from Verizon Labs that enables blazing fast continuous delivery, fully automated lifecycle and cleanup for applications, along with dynamic runtime traffic management. Automatically shift your production traffic back and forth between different versions of your immutable service infrastructure without any outages. Encrypt all your traffic transparently and provide best-in-class security without your engineering staff having to learn how your infrastructure and security sausages are made.

1. Nelson
automated, multi-region container deployment

2. Hello.
!
verizon.github.io/nelson/

3. Problem
• Provisioning applications is still too slow (bare metal or cloud).
• Runtime traffic control systems are medieval at best.
• Coupling CI and CD creates monolithic operational systems.
• These systems do everything. This is a distinct problem.
• Current market solutions limited or hard to adopt.
• Most teams have brittle, painful automation nobody wants to use.
• Many teams attempt CD ignorant of the side-effects.

4. Lessons
• Automate every part of the system.
• Testing a distributed system locally is a fable.
• Emergent properties. Scaling issues etc.
• Uniformity is highly desirable and wildly advantageous.
• Beautiful, unique snowflakes are however, inevitable.
• Automated lifecycle management is required.

5. Goals
• Use the minimally powerful components.
• System elements should be awesome at just one thing.
• Reduce overall platform complexity.
• Increase responsibility of engineering teams. Break it, you bought it.
• Decentralize process gatekeepers.
• No build team. No ticket filing for deployments. No configuration
management.

6. Goals
• All application specifications are checked in.
• Build. Deployment. Alerting etc.
• Reduce deployment time 2 minutes or less.
• Support multi-DC topologies from the get-go.
• Automatic credential management and secure-introduction
• Transparent, strong encryption for application I/O on the wire.

7. Build Better.

8. Nomad.

9. Nomad
• Use a farm of servers as a single resource pool: RAM, CPU, etc
• Typically used at larger scale, becoming more common.
• Blazing fast: only placement without provisioning.
• Integration with Vault, so secure-introduction works OOTB.
• Monolithic resource manager & scheduler [1]
• Several open-source & commercial alternatives: Mesos, k8s etc
[1] https://research.google.com/pubs/pub43438.html

10. Envoy.

11. Envoy
• Fast L4 and L7 proxy solving many practical ops concerns.
• Open-sourced end of 2016; blossomed since.
• Lyft, Google, IBM et al all actively contributing.
• Make applications dumb; invest in a single element of routing infra
• Retries, Circuit Breaking, TLS Encryption etc
• Integrate horizontally, not vertically
• Integrate with whatever discovery system you want via APIs.

12. Nelson.

13. – Vice Admiral Horatio Nelson, 1758-1805
“Desperate affairs require desperate remedies.”

14. – Vice Admiral Horatio Nelson, 1758-1805
“Desperate affairs require desperate remedies.”
#opslife

15. Overview
• Github driven developer workflow (.com or enterprise).
• Choose whatever build / CI system you want.
• State of the art runtime routing via Envoy.
• Secure introduction for safe distribution of credentials from Vault.
• Integrated with Nomad; target any datacenter running a scheduler.
• Integrated alert definition with Prometheus.

17. Lifecycle.
Deployment is the easy part.

19. based on
consul

20. typical state

21. user
activated

22. pending
GC

23. pluggable

24. borrowed
time

25. garbage
collection

26. Graph
Pruning

27. X
X
Upgraded!

28. last two
major revsXX X

29. last two
featuresXX X

30. Namespaces.

31. machines

32. scheduler

33. namespaces

34. namespaces
entirely
virtual!

36. root
namespace

37. qa/unstable

38. qa/staging/tim

40. Discovery & Routing.

41. Discovery.
• Discovery protocol written to Consul KV for every stack
• We call this Lighthouse protocol
• Application dependencies are declared a-priori.
• You cannot route to that which you do not tell Nelson about.
• Makes for awesome auditing and security.
• Language implementations need only consume the protocol.

42. Routing.
• Non-prescriptive approach to routing tier implementation.
• Provides a control plane protocol describe routing actions.
• Typically implemented with Envoy, but you can choose.
• Minor application changes required.
• Incentivized these with tracing and context propagation.
• Models traffic shifting as a time vs traffic policy curve.

44. http://timperrett.com/2017/05/13/nomad-with-envoy-and-consul
embeded
envoy

45. http://timperrett.com/2017/05/13/nomad-with-envoy-and-consul
sidecar
envoy

46. http://timperrett.com/2017/05/13/nomad-with-envoy-and-consul
host-based envoy

47. Challenges
• Non-trivial level of investment and execution.
• Tight integration with Hashistack is both pro or con.
• Containerizing legacy applications can be “interesting”.
• Migration can be a challenge if not collocated with “the new world”.
• Small organizations better served by existing solutions.

48. Future Work
• Aim to open-source supporting and complimentary tools.
• Consul / Envoy integration. Cost analysis subsystem.
• Make Nelson easier to extend for third-parties
• eDSL for workflows, externalize policy algebra
• General “plugin” system is a possibility
• Listen to the community feedback.

49. Summary
• Fully automated application lifecycle: no manual housekeeping.
• Choose whatever CI setup best fits your team.
• Secure your deployments.
• Transparent mTLS and rotating credentials.
• Automatic Vault policy management.
• Provide rigor to your application Death Star.

50. EOF
timperrett
verizon.github.io/nelson/