Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Nelson: Rigorous Deployment for a Functional World


Published on

Functional programming finds its roots in mathematics - the pursuit of purity and completeness. We functional programmers look to formalize system behaviors in an algebraic and total manner. Despite this, when it comes time to deploy ones beautiful monadic ivory towers to production, most organizations cast caution to the wind and use a myriad of bash scripts and sticky tape to get the job done. In this talk, the speaker will introduce you to Nelson, an open-source project from Verizon that looks to provide rigor to your large distributed system, whilst offering best-in-class security, runtime traffic shifting and a fully immutable approach to application lifecycle. Nelson itself is entirely composed of free algebras and coproducts, and the speaker will show not only how this has enabled development, but also how it provided a frame with which to reason about solutions to fundamental operational problems.

Published in: Technology
  • Be the first to comment

Nelson: Rigorous Deployment for a Functional World

  1. 1. Nelson automated, multi-region container deployment timperrett
  2. 2. Hello. !
  3. 3. Rig·or·ous. Extremely thorough, exhaustive, or accurate.
  4. 4. Rigorous Systems.
  5. 5. – Dr Tara Smith “Pragmatism is the opposition to principal, on principle.”
  6. 6. Rigorous Systems. • Theory is the foundation. • Category Theory, Combinatorics, Set Theory, Group Theory… • Principles of formal verification becoming more widespread. • Type systems act as proofs about your program. • Amazon S3 (among others) have TLA+ models. • Never fear practical application of rigorous systems. • Invest now to save your sanity tomorrow (i.e. 2am PagerDuty alert).
  7. 7. Rigorous Systems. Time Pain Principled Pragmatic
  8. 8. Rigorous Systems. Time Pain Principled Pragmatic "
  9. 9. Rigorous Systems. Time Pain Principled Pragmatic #
  10. 10. Rigorous Systems. Time Pain Principled Pragmatic $
  11. 11. Deployment. sticky tape, bandages and bash scripts. antithesis 
 of rigor
  12. 12. Problem • Provisioning applications is still too slow (bare metal or cloud). • Runtime traffic control systems are medieval at best. • Coupling CI and CD creates monolithic operational systems. • These systems do everything. This is a distinct problem. • Current market solutions limited or hard to adopt. • Most teams have brittle, painful automation nobody wants to use. • Many teams attempt CD ignorant of the side-effects.
  13. 13. Lessons • Automate every part of the system. • Testing a distributed system locally is a fable. • Emergent properties. Scaling issues etc. • Uniformity is highly desirable and wildly advantageous. • Beautiful, unique snowflakes are however, inevitable. • Automated lifecycle management is required.
  14. 14. Goals • Use the minimally powerful components. • System elements should be awesome at just one thing. • Reduce overall platform complexity. • Increase responsibility of engineering teams. Break it, you bought it. • Decentralize process gatekeepers. • No build team. No ticket filing for deployments. No configuration management.
  15. 15. Goals • All application specifications are checked in. • Build. Deployment. Alerting etc. • Reduce deployment time to 2 minutes or less. • Support multi-DC topologies from the get-go. • Automatic credential management and secure-introduction • Transparent, strong encryption for application I/O on the wire.
  16. 16. Nelson.
  17. 17. – Vice Admiral Horatio Nelson, 1758-1805 “Desperate affairs require desperate remedies.”
  18. 18. – Vice Admiral Horatio Nelson, 1758-1805 “Desperate affairs require desperate remedies.” #opslife
  19. 19. Overview • Provides “best practices” for your organization; focus on shipping. • Github driven developer workflow (.com or enterprise). • Choose whatever build / CI system you want. • State of the art runtime routing via Envoy. • Secure introduction for safe distribution of credentials from Vault. • Target any datacenter running a scheduler (Nomad, Mesos, etc).
  20. 20. Overview. pie of responsibility!
  21. 21. Overview. users
  22. 22. Overview. operators
  23. 23. Features. • Declarative, simplistic YAML manifest per-repository. • Active lifecycle management. • Management of edge ingress (e.g. public load balancers). • Optional routing control plane. • Comprehensive auditing. • Extensible with arbitrary scheduling, routing or artifact systems.
  24. 24. Foundations. graphs and coproducts.
  25. 25. Mathematics. Combinatorics. Graph Theory.
  26. 26. Vertex.
  27. 27. Vertex. vertex[N,L]
  28. 28. A Edge. B
  29. 29. A Directed Edge. B
  30. 30. Directed Edge. LEdge[N,A] Edge[N] A B
  31. 31. Graph.
  32. 32. Monoids. A B CD A F G H (A, A) => A A => A
  33. 33. Monoids. B C D A F G H
  34. 34. A D C B E depends ondepends on depends on depends on
  35. 35. depends ondepends on depends on depends on
  36. 36. depends ondepends on depends on depends on foo@1.2 bar@4.5 qux@2.1 buzz@1.1 klang@1.0
  37. 37. Graphs. • Every application declares their dependencies a-priori. • No routing to disconnected vertices. • No Vault policy attributes for disconnected vertices. • Edge ingress points are the “root” of a graph. • Intersecting graphs with shared or discrete roots. • Provides anchor-point for lifecycle management. • Entire world view falls out of a union of all discrete graphs.
  38. 38. Lifecycle. deployment is the easy part.
  39. 39. Lifecycle. Amazon
 in 2008
  40. 40. based on
  41. 41. typical state
  42. 42. pending
  43. 43. user 
  44. 44. pluggable
  45. 45. borrowed
  46. 46. garbage
  47. 47. Graph 
  48. 48. X X Upgraded!
  49. 49. last two
 major revsXX X
  50. 50. last two
 featuresXX X
  51. 51. Security. that thing you wish you did better. graphs help here too!
  52. 52. Security. • Derivation of security policy, as opposed to proclamation of policy. • Excellent audibility • All dependencies are known - internal and external. • Every single application instance has unique credentials. • Dynamic certificates, dynamic DB access etc. • Compromised containers have the minimal attack surface. • Joined with a service mesh, transparent mTLS for everyone is a breeze.
  53. 53. Namespaces.
  54. 54. machines
  55. 55. scheduler
  56. 56. namespaces
  57. 57. namespaces entirely 
  58. 58. root 
  59. 59. qa/unstable
  60. 60. qa/staging/tim
  61. 61. Discovery & Routing.
  62. 62. Discovery. • Discovery protocol written to Consul KV for every stack • We call this Lighthouse protocol • Application dependencies are declared a-priori. • You cannot route to that which you do not tell Nelson about. • Makes for awesome auditing and security. • Language implementations need only consume the protocol.
  63. 63. Routing. • Non-prescriptive approach to routing tier implementation. • Provides a control plane protocol describe routing actions. • Typically implemented with Envoy, but you can choose. • Minor application changes required. • Incentivized these with tracing and context propagation. • Models traffic shifting as a time vs traffic policy curve.
  64. 64. Workflow. how the sausage is made.
  65. 65. Workflow. HTTPQueue Workflow X Workflow Y End End Executor Pool GitHub
  66. 66. Workflow. and thenand then start replicate
 containers create
 vault policies ??? and then
  67. 67. Workflow. Looks like a graph. Walks like a graph. Must be a… coproduct!
  68. 68. Workflow. Coproduct[F[_], G[_], A](run: F[A] / G[A])
  69. 69. Workflow. type Op0[A] = Coproduct[DockerOp, ConsulOp, A] … type WorkflowOp[A] = Coproduct[SchedulerOp, Op4, A] type WorkflowF[A] = Free.FreeC[WorkflowOp, A]
  70. 70. Workflow. for { _ <- status(id, Pending, "workflow about to start") i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${}'s consul") _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns =, rs = rs) /////////////////////////////// // remove for space limitations /////////////////////////////// l <- launch(i, dc,, vunit, p, hash) _ <- debug(s"response from scheduler $l”) _ <- status(id, getStatus(unit, p), "======> workflow completed <======") } yield () Free monad
  71. 71. Challenges • Non-trivial level of investment and execution. • Tight integration with Hashistack is both pro or con. • Community adding k8s + Istio support % • Containerizing legacy applications can be “interesting”. • Migration can be a challenge if not collocated with “the new world”. • Small organizations better served by existing solutions.
  72. 72. Summary • Provide rigor to your application Death Star. • Fully automated application lifecycle: no manual housekeeping. • Choose whatever CI setup best fits your team. • Secure your deployments. • Transparent mTLS and rotating credentials. • Automatic Vault policy management.
  73. 73. EOF timperrett
  74. 74. Related Links. • Nelson documentation site: • Nelson CLI: • Quiver: Scala port of Haskell FGL • Slipway: integrate any CI system with Nelson