Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2016 Mesosphere, Inc. All Rights Reserved.
CONTAINER
ORCHESTRATION AND
SOFTWARE DEFINED
NETWORK: A FIELD REPORT
@SARGUN
...
© 2016 Mesosphere, Inc. All Rights Reserved.
WHO AM I?
2
© 2016 Mesosphere, Inc. All Rights Reserved. 3
© 2016 Mesosphere, Inc. All Rights Reserved. 4
© 2016 Mesosphere, Inc. All Rights Reserved. 5
© 2016 Mesosphere, Inc. All Rights Reserved. 6
My Mission
© 2016 Mesosphere, Inc. All Rights Reserved.
AGENDA
7
• A brief history of how we got here
• How we’re trying to make it b...
© 2016 Mesosphere, Inc. All Rights Reserved.
A BRIEF
HISTORY OF
NETWORKS
IN THE DC
8
© 2016 Mesosphere, Inc. All Rights Reserved.
ORGANIZATIONS CIRCA 2007
9
•Before DevOps was first heard
•Clear differentiati...
© 2016 Mesosphere, Inc. All Rights Reserved.
SOFTWARE CIRCA 2007
10
•Different services glued together via CORBA, XML-RPC, ...
© 2016 Mesosphere, Inc. All Rights Reserved. 11
SaaS continued to grow at
an incredible rate
© 2016 Mesosphere, Inc. All Rights Reserved. 12
There became a race to ship faster
© 2016 Mesosphere, Inc. All Rights Reserved. 13
We kept the software alive
By feeding it
With Sysadmins
© 2016 Mesosphere, Inc. All Rights Reserved. 14
We kept the machines alive
By feeding them
With Blood
© 2016 Mesosphere, Inc. All Rights Reserved. 15
This wasn’t working
© 2016 Mesosphere, Inc. All Rights Reserved.
ORGANIZATIONS 2008+
16
•We began seeing a gradual shift in the industry where...
© 2016 Mesosphere, Inc. All Rights Reserved.
SOFTWARE CIRCA 2008+
17
•Popularization of Open Source tooling to automate mu...
© 2016 Mesosphere, Inc. All Rights Reserved.
CIRCA 2011
18
•Much of what’s been happening for the past half-decade hits ne...
© 2016 Mesosphere, Inc. All Rights Reserved.
CIRCA 2013
19
• Docker becomes instant hit and brings containers to the foref...
© 2016 Mesosphere, Inc. All Rights Reserved. 20
Everything was changing
© 2016 Mesosphere, Inc. All Rights Reserved. 21
Why?
© 2016 Mesosphere, Inc. All Rights Reserved. 22
Business Value
© 2016 Mesosphere, Inc. All Rights Reserved. 23
© 2016 Mesosphere, Inc. All Rights Reserved.
BENEFITS
24
•Reduction in cost of goods sold
•Smaller engineer to server rati...
© 2016 Mesosphere, Inc. All Rights Reserved. 25
But at what cost?
© 2016 Mesosphere, Inc. All Rights Reserved. 26
Complexity
© 2016 Mesosphere, Inc. All Rights Reserved.
OLD WORLD
27
© 2016 Mesosphere, Inc. All Rights Reserved.
NEW WORLD
28
© 2016 Mesosphere, Inc. All Rights Reserved.
DIVING DEEPER
29
© 2016 Mesosphere, Inc. All Rights Reserved. 30
Paxos?
Raft?
Ω Failure Detector?
Pods?
Wat?
Sidecars?
Etcd?
Zookeeper?
VxL...
© 2016 Mesosphere, Inc. All Rights Reserved. 31
It was time for something completely
new
© 2016 Mesosphere, Inc. All Rights Reserved.
ENTER
DC/OS
32
© 2016 Mesosphere, Inc. All Rights Reserved.
CONTAINERS
WITH
BATTERIES
INCLUDED
33
• Includes everything you need to move ...
© 2016 Mesosphere, Inc. All Rights Reserved. 34
In Your Datacenter
© 2016 Mesosphere, Inc. All Rights Reserved. 35
As a Black Box
© 2016 Mesosphere, Inc. All Rights Reserved. 36
But, our story starts:
~November 2015
(~8 months ago)
© 2016 Mesosphere, Inc. All Rights Reserved. 37
Networking Features
© 2016 Mesosphere, Inc. All Rights Reserved. 38
“Service Discovery”
© 2016 Mesosphere, Inc. All Rights Reserved. 39
Where are my apps running?
© 2016 Mesosphere, Inc. All Rights Reserved. 40
© 2016 Mesosphere, Inc. All Rights Reserved. 41
The Old World
© 2016 Mesosphere, Inc. All Rights Reserved. 42
Let Mesos* Choose Ports
*The Scheduler
© 2016 Mesosphere, Inc. All Rights Reserved. 43
How do you find the tasks?
© 2016 Mesosphere, Inc. All Rights Reserved. 44
A Directory?
© 2016 Mesosphere, Inc. All Rights Reserved. 45
Mesos-DNS
© 2016 Mesosphere, Inc. All Rights Reserved. 46
© 2016 Mesosphere, Inc. All Rights Reserved. 47
…Ish
© 2016 Mesosphere, Inc. All Rights Reserved. 48
DNS SRV Based Service Discovery
© 2016 Mesosphere, Inc. All Rights Reserved. 49
Everyone has DNS right?
© 2016 Mesosphere, Inc. All Rights Reserved. 50
And GLibc even has a bug open for it!
© 2016 Mesosphere, Inc. All Rights Reserved. 51
…Opened in 2005
© 2016 Mesosphere, Inc. All Rights Reserved. 52
We needed a solution that actually
worked
© 2016 Mesosphere, Inc. All Rights Reserved. 53
So, we performed an OODA loop
1.Observe
2. Orient
3. Decide
4. Act
© 2016 Mesosphere, Inc. All Rights Reserved.
OBSERVE: SERVICE DISCOVERY
54
• Existing Dynamic Service Discovery Solutions:...
© 2016 Mesosphere, Inc. All Rights Reserved.
OBSERVE: NETWORKING
55
• Everybody assumes IP per application instance
• Ever...
© 2016 Mesosphere, Inc. All Rights Reserved.
ORIENT:
DC/OS
NETWORKING
CORE TENETS
56
•DC/OS must be agnostic to the underl...
© 2016 Mesosphere, Inc. All Rights Reserved.
DECIDE
57
What do we want to build?
• Load Balancing
• Seamless Service Disco...
© 2016 Mesosphere, Inc. All Rights Reserved.
DC/OS ARCHITECTURE (~NOV ’15): POLYGLOT MICROSERVICES
58
© 2016 Mesosphere, Inc. All Rights Reserved. 59
What Did We Build?
© 2016 Mesosphere, Inc. All Rights Reserved.
ACT:
DEVELOPED
NEW ERLANG
SERVICES
60
• Service Discovery
• Navstar
• Spartan...
© 2016 Mesosphere, Inc. All Rights Reserved.
DC/OS ARCHITECTURE (TODAY): POLYGLOT MICROSERVICES
61
© 2016 Mesosphere, Inc. All Rights Reserved. 62
Network Control Plane:
Navstar
© 2016 Mesosphere, Inc. All Rights Reserved. 63
© 2016 Mesosphere, Inc. All Rights Reserved.
NAVSTAR:
BENEFITS
64
•Fully decentralized control plane
•Scalable
•Extensible...
© 2016 Mesosphere, Inc. All Rights Reserved. 65
DNS:
Spartan
© 2016 Mesosphere, Inc. All Rights Reserved. 66
HA DNS
© 2016 Mesosphere, Inc. All Rights Reserved. 67
/etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
Normal DNS
© 2016 Mesosphere, Inc. All Rights Reserved. 68
Ideal DNS Request
© 2016 Mesosphere, Inc. All Rights Reserved. 69
Ideal DNS Request
© 2016 Mesosphere, Inc. All Rights Reserved. 70
But, sometimes things go wrong
© 2016 Mesosphere, Inc. All Rights Reserved. 71
Not so Ideal DNS Request
© 2016 Mesosphere, Inc. All Rights Reserved. 72
Wait at least 1 second
© 2016 Mesosphere, Inc. All Rights Reserved. 73
Not so Ideal DNS Request
© 2016 Mesosphere, Inc. All Rights Reserved. 74
Wait at least 1 second
© 2016 Mesosphere, Inc. All Rights Reserved. 75
Not so Ideal DNS Request
© 2016 Mesosphere, Inc. All Rights Reserved. 76
Wat?
© 2016 Mesosphere, Inc. All Rights Reserved. 77
Spartan DNS
© 2016 Mesosphere, Inc. All Rights Reserved. 78
Spartan DNS
© 2016 Mesosphere, Inc. All Rights Reserved. 79
Spartan DNS
© 2016 Mesosphere, Inc. All Rights Reserved. 80
Spartan Controls Timeouts
© 2016 Mesosphere, Inc. All Rights Reserved. 81
Edge DNS
© 2016 Mesosphere, Inc. All Rights Reserved. 82
© 2016 Mesosphere, Inc. All Rights Reserved.
SPARTAN:
BENEFITS
83
•Makes DNS highly available in light of failure
•Lowers ...
© 2016 Mesosphere, Inc. All Rights Reserved. 84
Load Balancing:
Minuteman
© 2016 Mesosphere, Inc. All Rights Reserved. 85
© 2016 Mesosphere, Inc. All Rights Reserved. 86
© 2016 Mesosphere, Inc. All Rights Reserved. 87
© 2016 Mesosphere, Inc. All Rights Reserved. 88
© 2016 Mesosphere, Inc. All Rights Reserved. 89
How does it work?
© 2016 Mesosphere, Inc. All Rights Reserved. 90
Setting it up
© 2016 Mesosphere, Inc. All Rights Reserved. 91
Intercepting the
connection
© 2016 Mesosphere, Inc. All Rights Reserved. 92
Remap
© 2016 Mesosphere, Inc. All Rights Reserved. 93
Success
© 2016 Mesosphere, Inc. All Rights Reserved.
MINUTEMAN:
BENEFITS
94
•Appearance of a fixed-load balancer
•Fully distribute...
© 2016 Mesosphere, Inc. All Rights Reserved. 95
How do we tie it all together?
© 2016 Mesosphere, Inc. All Rights Reserved.
GLOBAL STATE
96
• Load Balancer Task Mapping
• DNS Zones
• Virtual Network Ro...
© 2016 Mesosphere, Inc. All Rights Reserved. 97
Computers
© 2016 Mesosphere, Inc. All Rights Reserved. 98
Constant Churn
© 2016 Mesosphere, Inc. All Rights Reserved. 99
Sometimes
They Break
© 2016 Mesosphere, Inc. All Rights Reserved. 100
Sometimes
Many Break
© 2016 Mesosphere, Inc. All Rights Reserved. 101
Sometimes
You don’t know
© 2016 Mesosphere, Inc. All Rights Reserved.
HOW DO YOU
DO
SIGNALING?
102
© 2016 Mesosphere, Inc. All Rights Reserved. 103
Nobody ever got fired for using
Zookeeper
© 2016 Mesosphere, Inc. All Rights Reserved. 104
We know about Zookeeper
© 2016 Mesosphere, Inc. All Rights Reserved. 105
It works…usually
© 2016 Mesosphere, Inc. All Rights Reserved. 106
How else can we do this?
© 2016 Mesosphere, Inc. All Rights Reserved. 107
Naively
(See: Distributed Erlang)
© 2016 Mesosphere, Inc. All Rights Reserved. 108
Massive Amount of Information
© 2016 Mesosphere, Inc. All Rights Reserved. 109
Who else has solved this?
© 2016 Mesosphere, Inc. All Rights Reserved. 110
Academia
© 2016 Mesosphere, Inc. All Rights Reserved. 111
Control Plane:
Lashup
© 2016 Mesosphere, Inc. All Rights Reserved. 112
How do we scale the naive approach?
© 2016 Mesosphere, Inc. All Rights Reserved. 113
All we need is a sparse, connected graph
© 2016 Mesosphere, Inc. All Rights Reserved. 114
But how?
© 2016 Mesosphere, Inc. All Rights Reserved. 115
Enter: HyParView*
Builds a connected graph (overlay), where the degree is...
© 2016 Mesosphere, Inc. All Rights Reserved. 116
Boot Time
© 2016 Mesosphere, Inc. All Rights Reserved. 117
© 2016 Mesosphere, Inc. All Rights Reserved. 118
© 2016 Mesosphere, Inc. All Rights Reserved. 119
© 2016 Mesosphere, Inc. All Rights Reserved. 120
© 2016 Mesosphere, Inc. All Rights Reserved. 121
© 2016 Mesosphere, Inc. All Rights Reserved. 122
© 2016 Mesosphere, Inc. All Rights Reserved. 123
© 2016 Mesosphere, Inc. All Rights Reserved. 124
© 2016 Mesosphere, Inc. All Rights Reserved. 125
© 2016 Mesosphere, Inc. All Rights Reserved. 126
© 2016 Mesosphere, Inc. All Rights Reserved. 127
© 2016 Mesosphere, Inc. All Rights Reserved. 128
© 2016 Mesosphere, Inc. All Rights Reserved. 129
© 2016 Mesosphere, Inc. All Rights Reserved. 130
Connected Graph
Hyparview
© 2016 Mesosphere, Inc. All Rights Reserved. 131
Constant Adaptive
Health Checks*
*Borrowed from SWIM, Gossip Style Failur...
© 2016 Mesosphere, Inc. All Rights Reserved. 132
Dealing with Failure
Hyparview
© 2016 Mesosphere, Inc. All Rights Reserved. 133
Dealing with Failure
Hyparview
© 2016 Mesosphere, Inc. All Rights Reserved. 134
Dealing with Failure
Hyparview
© 2016 Mesosphere, Inc. All Rights Reserved. 135
Dealing with Failure
Hyparview
© 2016 Mesosphere, Inc. All Rights Reserved. 136
Routing*
*Borrowed from Dijkstra, OSPF, IS-IS
© 2016 Mesosphere, Inc. All Rights Reserved. 137
© 2016 Mesosphere, Inc. All Rights Reserved. 138
© 2016 Mesosphere, Inc. All Rights Reserved. 139
Adjacent Node 1 Adjacent Node 2 Adjacent Node 3
A T B C
B T A C
C B A D
D...
© 2016 Mesosphere, Inc. All Rights Reserved. 140
View Changes Gossiped on Change
© 2016 Mesosphere, Inc. All Rights Reserved. 141
Adjacent Node 1 Adjacent Node 2 Adjacent Node 3 Version (epoch)
A T B C 1...
© 2016 Mesosphere, Inc. All Rights Reserved. 142
© 2016 Mesosphere, Inc. All Rights Reserved. 143
Just Run Dijkstra*
*BFS / DFS
© 2016 Mesosphere, Inc. All Rights Reserved. 144
Minimum Spanning Tree, From Sender A
© 2016 Mesosphere, Inc. All Rights Reserved. 145
A Wild Pokemon Appears!*
*CRDT papers
Database
© 2016 Mesosphere, Inc. All Rights Reserved. 146
CRDTS: Semilattices
“Time”
© 2016 Mesosphere, Inc. All Rights Reserved. 147
CRDT KV Store
© 2016 Mesosphere, Inc. All Rights Reserved. 148
“Demo”
© 2016 Mesosphere, Inc. All Rights Reserved. 149
“Demo”
(foo@3c075477e55e)10> nodes().
[baz@3c075477e55e,bar@3c075477e55e]...
© 2016 Mesosphere, Inc. All Rights Reserved. 150
“Demo”
(foo@3c075477e55e)9> lashup_kv:value([erlang_users]).
[]
(bar@3c07...
© 2016 Mesosphere, Inc. All Rights Reserved. 151
“Demo”
(baz@3c075477e55e)10>
net_kernel:allow([foo@3c075477e55e]).
ok
(ba...
© 2016 Mesosphere, Inc. All Rights Reserved. 152
“Demo”
© 2016 Mesosphere, Inc. All Rights Reserved. 153
“Demo”
(bar@3c075477e55e)11>
lashup_kv:request_op([erlang_users], {update...
© 2016 Mesosphere, Inc. All Rights Reserved. 154
“Demo”
(foo@3c075477e55e)12> lashup_kv:value([erlang_users]).
[{{number,r...
© 2016 Mesosphere, Inc. All Rights Reserved. 155
“Demo”
(foo@3c075477e55e)23> erlang:set_cookie(baz@3c075477e55e,
fake).
t...
© 2016 Mesosphere, Inc. All Rights Reserved. 156
“Demo”
© 2016 Mesosphere, Inc. All Rights Reserved. 157
“Demo”
(foo@3c075477e55e)29> lashup_kv:request_op([erlang_users],
{update...
© 2016 Mesosphere, Inc. All Rights Reserved. 158
“Demo”
© 2016 Mesosphere, Inc. All Rights Reserved. 159
“Demo”
(foo@3c075477e55e)1> lashup_kv:value([erlang_users]).
[{{number,ri...
© 2016 Mesosphere, Inc. All Rights Reserved.
LASHUP KV
160
•Datatypes:
•Maps
•Composable
•Sets
•Flags
•Counters
•Registers...
© 2016 Mesosphere, Inc. All Rights Reserved. 161
Does Lashup provide Business Value?
© 2016 Mesosphere, Inc. All Rights Reserved. 162
Prior to Lashup
© 2016 Mesosphere, Inc. All Rights Reserved. 163
After Lashup
© 2016 Mesosphere, Inc. All Rights Reserved.
PROJECT
LASHUP
164
•A novel distributed systems SDK that provides:
•Membershi...
© 2016 Mesosphere, Inc. All Rights Reserved. 165
So,
Why Erlang/OTP?
© 2016 Mesosphere, Inc. All Rights Reserved. 166
Why Not?
(Go)
© 2016 Mesosphere, Inc. All Rights Reserved. 167
The First Version
of Minuteman
Was In
Golang
© 2016 Mesosphere, Inc. All Rights Reserved. 168
Stand on the Shoulders of Giants
© 2016 Mesosphere, Inc. All Rights Reserved. 169
What’s a good language with a healthy
set of distributed systems, and
net...
© 2016 Mesosphere, Inc. All Rights Reserved. 170
What’s a good language with a healthy
set of distributed systems, and
net...
© 2016 Mesosphere, Inc. All Rights Reserved. 171
What’s a good language with a healthy set of
distributed systems, and net...
© 2016 Mesosphere, Inc. All Rights Reserved. 172
It’s a 3AM, who do you want picking up the
phone?
© 2016 Mesosphere, Inc. All Rights Reserved. 173
Answer:
You make it so that your software so it doesn’t
page you at 3 AM.
© 2016 Mesosphere, Inc. All Rights Reserved. 174
Upgrading Software on Mars is Easier
vs.
On-Prem
© 2016 Mesosphere, Inc. All Rights Reserved. 175
Erlang
© 2016 Mesosphere, Inc. All Rights Reserved.
LASHUP KV
NEEDED A
DURABLE STORE
ANSWER:
MNESIA
176
•ACID, Transactional
•20+...
© 2016 Mesosphere, Inc. All Rights Reserved.
LASHUP KV
NEEDED A
CRDT LIBRARY
ANSWER:
RIAK_DT
177
•Riak’s CRDT Library
•Use...
© 2016 Mesosphere, Inc. All Rights Reserved.
LASHUP NEEDED
SECURE
DISTRIBUTION
ANSWER:
DISTERL
178
•SSL with mutual authen...
© 2016 Mesosphere, Inc. All Rights Reserved.
SPARTAN
NEEDED
A DNS SERVER
ANSWER:
ERL-DNS
179
•Runs DNSimple
•Used in produ...
© 2016 Mesosphere, Inc. All Rights Reserved.
MINUTEMAN
NEEDED
A KERNEL API
LIBRARY
ANSWER:
GEN_NETLINK
180
•Liberal licens...
© 2016 Mesosphere, Inc. All Rights Reserved.
LASHUP
NEEDED A
GRAPH LIBRARY
ANSWER:
DIGRAPH
181
•Built-in Library to Erlang...
© 2016 Mesosphere, Inc. All Rights Reserved. 182
So, we built our own.
© 2016 Mesosphere, Inc. All Rights Reserved. 183
On Testing…
© 2016 Mesosphere, Inc. All Rights Reserved. 184
We verified lashup_gm_route against
digraph
© 2016 Mesosphere, Inc. All Rights Reserved. 185
Property-Based Testing:
PropEr
© 2016 Mesosphere, Inc. All Rights Reserved. 186
Automatically Generate Command Set
CSet ={add_node(e), add_node(b),
add_e...
© 2016 Mesosphere, Inc. All Rights Reserved. 187
Verify Equality
Digraph lashup_gm
add_node(e) add_node(e)
Check Equivalen...
© 2016 Mesosphere, Inc. All Rights Reserved. 188
…Tens of Thousands of Times
© 2016 Mesosphere, Inc. All Rights Reserved. 189
Common Test Makes Integration Testing
A Breeze
© 2016 Mesosphere, Inc. All Rights Reserved.
COMMON TEST:
ON EACH
CHECK-IN
(LASHUP)
190
•Spins up 25 virtual nodes
•Tortur...
© 2016 Mesosphere, Inc. All Rights Reserved.
TAKE-AWAYS
191
•The container ecosystem is in is early stages
•Erlang is suit...
© 2016 Mesosphere, Inc. All Rights Reserved.
ERLANG &
CONTAINER
NETWORKING
@SARGUN
192
Upcoming SlideShare
Loading in …5
×

Erlang User Conference 2016: Container Networking: A Field Report

334 views

Published on

Erlang User Conference 2016: Container Networking: A Field Report. This talk speaks to Mesosphere's experience with Erlang, and containers over the past year.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Erlang User Conference 2016: Container Networking: A Field Report

  1. 1. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER ORCHESTRATION AND SOFTWARE DEFINED NETWORK: A FIELD REPORT @SARGUN 1 Sargun Dhillon, 2016
  2. 2. © 2016 Mesosphere, Inc. All Rights Reserved. WHO AM I? 2
  3. 3. © 2016 Mesosphere, Inc. All Rights Reserved. 3
  4. 4. © 2016 Mesosphere, Inc. All Rights Reserved. 4
  5. 5. © 2016 Mesosphere, Inc. All Rights Reserved. 5
  6. 6. © 2016 Mesosphere, Inc. All Rights Reserved. 6 My Mission
  7. 7. © 2016 Mesosphere, Inc. All Rights Reserved. AGENDA 7 • A brief history of how we got here • How we’re trying to make it better • Some systems we’ve built to enable us to better ship • How Erlang helped us get there
  8. 8. © 2016 Mesosphere, Inc. All Rights Reserved. A BRIEF HISTORY OF NETWORKS IN THE DC 8
  9. 9. © 2016 Mesosphere, Inc. All Rights Reserved. ORGANIZATIONS CIRCA 2007 9 •Before DevOps was first heard •Clear differentiation of ownership •The datacenter was owned by a the NOC •Deployment of services was done by sysadmins in the operations group •Developers operated without access to production •Production deployments gated by QA, Operations
  10. 10. © 2016 Mesosphere, Inc. All Rights Reserved. SOFTWARE CIRCA 2007 10 •Different services glued together via CORBA, XML-RPC, SOAP •No one was really consciously doing microservices •Networks were static, giant layer 2 domains •Load Balancing provided by hardware •Firewall provided by hardware •Everyone ran their own datacenter •EC2 in its infancy, only a year prior has the term “Cloud” began to become popular •Systems statically partitioned
  11. 11. © 2016 Mesosphere, Inc. All Rights Reserved. 11 SaaS continued to grow at an incredible rate
  12. 12. © 2016 Mesosphere, Inc. All Rights Reserved. 12 There became a race to ship faster
  13. 13. © 2016 Mesosphere, Inc. All Rights Reserved. 13 We kept the software alive By feeding it With Sysadmins
  14. 14. © 2016 Mesosphere, Inc. All Rights Reserved. 14 We kept the machines alive By feeding them With Blood
  15. 15. © 2016 Mesosphere, Inc. All Rights Reserved. 15 This wasn’t working
  16. 16. © 2016 Mesosphere, Inc. All Rights Reserved. ORGANIZATIONS 2008+ 16 •We began seeing a gradual shift in the industry where lines between QA, Dev, and Ops were blurring •Devops term coined in 2008, first DevOpsDay in 2009 •Gradual adoption of the cloud, fewer organizations owning their own datacenters •Either networking was outsourced to the cloud, or typically remained in a small internal organization •Needed to reduce ratio of operators to servers
  17. 17. © 2016 Mesosphere, Inc. All Rights Reserved. SOFTWARE CIRCA 2008+ 17 •Popularization of Open Source tooling to automate much of traditional operations, and QA •Jenkins / Hudson •Puppet / Chef •Capistrano •Popularization of stacks requiring with more complex operational requirements •Nutch / Hadoop •NoSQLs •Still statically partitioned machines •Networks still sacred territory
  18. 18. © 2016 Mesosphere, Inc. All Rights Reserved. CIRCA 2011 18 •Much of what’s been happening for the past half-decade hits networking •Much of this falls under the term “SDN” (Software Defined Networking) or “NFV” (Network Function Virtualization) •Hastened by the adoption of VMs in the enterprise in the hype cycle •Openflow promises to fix everything •Major adoptions of the cloud by startups as well as enterprise •Virtualization begins to become mainstream as a mechanism of consolidating workloads •The invention of the “private cloud” •DotCloud / Docker funded by Y-combinator a year earlier •Term “Microservice” coined
  19. 19. © 2016 Mesosphere, Inc. All Rights Reserved. CIRCA 2013 19 • Docker becomes instant hit and brings containers to the forefront • Dynamic partitioning begins to make in-roads • Google releases Omega paper • Apache Aurora open sourced • Microservice counts explode, demanding collocation of workloads for efficiency • Mesosphere Founded • Site Reliability Engineering begins to popularize and further blur the lines between Dev, Ops, and QA
  20. 20. © 2016 Mesosphere, Inc. All Rights Reserved. 20 Everything was changing
  21. 21. © 2016 Mesosphere, Inc. All Rights Reserved. 21 Why?
  22. 22. © 2016 Mesosphere, Inc. All Rights Reserved. 22 Business Value
  23. 23. © 2016 Mesosphere, Inc. All Rights Reserved. 23
  24. 24. © 2016 Mesosphere, Inc. All Rights Reserved. BENEFITS 24 •Reduction in cost of goods sold •Smaller engineer to server ratio •Linear, or super linear growth rate of engineering team to servers is unsustainable •Smaller engineer to capability ratio, where capability includes: •Features •Throughput •Better User Experience •Better availability •Quicker release to features
  25. 25. © 2016 Mesosphere, Inc. All Rights Reserved. 25 But at what cost?
  26. 26. © 2016 Mesosphere, Inc. All Rights Reserved. 26 Complexity
  27. 27. © 2016 Mesosphere, Inc. All Rights Reserved. OLD WORLD 27
  28. 28. © 2016 Mesosphere, Inc. All Rights Reserved. NEW WORLD 28
  29. 29. © 2016 Mesosphere, Inc. All Rights Reserved. DIVING DEEPER 29
  30. 30. © 2016 Mesosphere, Inc. All Rights Reserved. 30 Paxos? Raft? Ω Failure Detector? Pods? Wat? Sidecars? Etcd? Zookeeper? VxLan?
  31. 31. © 2016 Mesosphere, Inc. All Rights Reserved. 31 It was time for something completely new
  32. 32. © 2016 Mesosphere, Inc. All Rights Reserved. ENTER DC/OS 32
  33. 33. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINERS WITH BATTERIES INCLUDED 33 • Includes everything you need to move away from 2007 • Networking • Security • Application deployment, and orchestration • Stateful services • Scheduling
  34. 34. © 2016 Mesosphere, Inc. All Rights Reserved. 34 In Your Datacenter
  35. 35. © 2016 Mesosphere, Inc. All Rights Reserved. 35 As a Black Box
  36. 36. © 2016 Mesosphere, Inc. All Rights Reserved. 36 But, our story starts: ~November 2015 (~8 months ago)
  37. 37. © 2016 Mesosphere, Inc. All Rights Reserved. 37 Networking Features
  38. 38. © 2016 Mesosphere, Inc. All Rights Reserved. 38 “Service Discovery”
  39. 39. © 2016 Mesosphere, Inc. All Rights Reserved. 39 Where are my apps running?
  40. 40. © 2016 Mesosphere, Inc. All Rights Reserved. 40
  41. 41. © 2016 Mesosphere, Inc. All Rights Reserved. 41 The Old World
  42. 42. © 2016 Mesosphere, Inc. All Rights Reserved. 42 Let Mesos* Choose Ports *The Scheduler
  43. 43. © 2016 Mesosphere, Inc. All Rights Reserved. 43 How do you find the tasks?
  44. 44. © 2016 Mesosphere, Inc. All Rights Reserved. 44 A Directory?
  45. 45. © 2016 Mesosphere, Inc. All Rights Reserved. 45 Mesos-DNS
  46. 46. © 2016 Mesosphere, Inc. All Rights Reserved. 46
  47. 47. © 2016 Mesosphere, Inc. All Rights Reserved. 47 …Ish
  48. 48. © 2016 Mesosphere, Inc. All Rights Reserved. 48 DNS SRV Based Service Discovery
  49. 49. © 2016 Mesosphere, Inc. All Rights Reserved. 49 Everyone has DNS right?
  50. 50. © 2016 Mesosphere, Inc. All Rights Reserved. 50 And GLibc even has a bug open for it!
  51. 51. © 2016 Mesosphere, Inc. All Rights Reserved. 51 …Opened in 2005
  52. 52. © 2016 Mesosphere, Inc. All Rights Reserved. 52 We needed a solution that actually worked
  53. 53. © 2016 Mesosphere, Inc. All Rights Reserved. 53 So, we performed an OODA loop 1.Observe 2. Orient 3. Decide 4. Act
  54. 54. © 2016 Mesosphere, Inc. All Rights Reserved. OBSERVE: SERVICE DISCOVERY 54 • Existing Dynamic Service Discovery Solutions: • Etcd • Finagle + Zookeeper • Consul • Existing Static Service Discovery Solutions: • Amazon ELB • Hardware Load Balancers • Service Discovery is an afterthought Gathering our data about the field
  55. 55. © 2016 Mesosphere, Inc. All Rights Reserved. OBSERVE: NETWORKING 55 • Everybody assumes IP per application instance • Everybody assumes reliable DNS • Some people want to be fast • Some people want security • Nobody wants to edit application code • Nobody wants to talk to their network engineer Gathering our data about the field
  56. 56. © 2016 Mesosphere, Inc. All Rights Reserved. ORIENT: DC/OS NETWORKING CORE TENETS 56 •DC/OS must be agnostic to the underlying environment •AWS / Azure / GCE / Softlayer as the lowest common denominators •DC/OS should require no to minimal changes to the code in order to work •DC/OS should provide similar services to existing environments •Fixed load balancers •Security •IP/Container •We do not want to require a change in organization procedures •We want to be secure
  57. 57. © 2016 Mesosphere, Inc. All Rights Reserved. DECIDE 57 What do we want to build? • Load Balancing • Seamless Service Discovery • Reliable DNS • External cluster Access • Metrics and Discoverability • IP Per Container
  58. 58. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS ARCHITECTURE (~NOV ’15): POLYGLOT MICROSERVICES 58
  59. 59. © 2016 Mesosphere, Inc. All Rights Reserved. 59 What Did We Build?
  60. 60. © 2016 Mesosphere, Inc. All Rights Reserved. ACT: DEVELOPED NEW ERLANG SERVICES 60 • Service Discovery • Navstar • Spartan • Load Balancing • Minuteman • Networking API • Fishladder • Control Plane • Lashup
  61. 61. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS ARCHITECTURE (TODAY): POLYGLOT MICROSERVICES 61
  62. 62. © 2016 Mesosphere, Inc. All Rights Reserved. 62 Network Control Plane: Navstar
  63. 63. © 2016 Mesosphere, Inc. All Rights Reserved. 63
  64. 64. © 2016 Mesosphere, Inc. All Rights Reserved. NAVSTAR: BENEFITS 64 •Fully decentralized control plane •Scalable •Extensible system to act as a building block •Began as network control plane
  65. 65. © 2016 Mesosphere, Inc. All Rights Reserved. 65 DNS: Spartan
  66. 66. © 2016 Mesosphere, Inc. All Rights Reserved. 66 HA DNS
  67. 67. © 2016 Mesosphere, Inc. All Rights Reserved. 67 /etc/resolv.conf nameserver 8.8.8.8 nameserver 8.8.4.4 Normal DNS
  68. 68. © 2016 Mesosphere, Inc. All Rights Reserved. 68 Ideal DNS Request
  69. 69. © 2016 Mesosphere, Inc. All Rights Reserved. 69 Ideal DNS Request
  70. 70. © 2016 Mesosphere, Inc. All Rights Reserved. 70 But, sometimes things go wrong
  71. 71. © 2016 Mesosphere, Inc. All Rights Reserved. 71 Not so Ideal DNS Request
  72. 72. © 2016 Mesosphere, Inc. All Rights Reserved. 72 Wait at least 1 second
  73. 73. © 2016 Mesosphere, Inc. All Rights Reserved. 73 Not so Ideal DNS Request
  74. 74. © 2016 Mesosphere, Inc. All Rights Reserved. 74 Wait at least 1 second
  75. 75. © 2016 Mesosphere, Inc. All Rights Reserved. 75 Not so Ideal DNS Request
  76. 76. © 2016 Mesosphere, Inc. All Rights Reserved. 76 Wat?
  77. 77. © 2016 Mesosphere, Inc. All Rights Reserved. 77 Spartan DNS
  78. 78. © 2016 Mesosphere, Inc. All Rights Reserved. 78 Spartan DNS
  79. 79. © 2016 Mesosphere, Inc. All Rights Reserved. 79 Spartan DNS
  80. 80. © 2016 Mesosphere, Inc. All Rights Reserved. 80 Spartan Controls Timeouts
  81. 81. © 2016 Mesosphere, Inc. All Rights Reserved. 81 Edge DNS
  82. 82. © 2016 Mesosphere, Inc. All Rights Reserved. 82
  83. 83. © 2016 Mesosphere, Inc. All Rights Reserved. SPARTAN: BENEFITS 83 •Makes DNS highly available in light of failure •Lowers Average Latency •Can control timeouts to sub-second •Edge DNS •Many DNS queries don’t need to go off-node
  84. 84. © 2016 Mesosphere, Inc. All Rights Reserved. 84 Load Balancing: Minuteman
  85. 85. © 2016 Mesosphere, Inc. All Rights Reserved. 85
  86. 86. © 2016 Mesosphere, Inc. All Rights Reserved. 86
  87. 87. © 2016 Mesosphere, Inc. All Rights Reserved. 87
  88. 88. © 2016 Mesosphere, Inc. All Rights Reserved. 88
  89. 89. © 2016 Mesosphere, Inc. All Rights Reserved. 89 How does it work?
  90. 90. © 2016 Mesosphere, Inc. All Rights Reserved. 90 Setting it up
  91. 91. © 2016 Mesosphere, Inc. All Rights Reserved. 91 Intercepting the connection
  92. 92. © 2016 Mesosphere, Inc. All Rights Reserved. 92 Remap
  93. 93. © 2016 Mesosphere, Inc. All Rights Reserved. 93 Success
  94. 94. © 2016 Mesosphere, Inc. All Rights Reserved. MINUTEMAN: BENEFITS 94 •Appearance of a fixed-load balancer •Fully distributed •Other than first packet, the entire lifetime is handled in kernel space •Erlang allows us to maintain latency guarantees
  95. 95. © 2016 Mesosphere, Inc. All Rights Reserved. 95 How do we tie it all together?
  96. 96. © 2016 Mesosphere, Inc. All Rights Reserved. GLOBAL STATE 96 • Load Balancer Task Mapping • DNS Zones • Virtual Network Routing Tables • Reachability • Security ACLs
  97. 97. © 2016 Mesosphere, Inc. All Rights Reserved. 97 Computers
  98. 98. © 2016 Mesosphere, Inc. All Rights Reserved. 98 Constant Churn
  99. 99. © 2016 Mesosphere, Inc. All Rights Reserved. 99 Sometimes They Break
  100. 100. © 2016 Mesosphere, Inc. All Rights Reserved. 100 Sometimes Many Break
  101. 101. © 2016 Mesosphere, Inc. All Rights Reserved. 101 Sometimes You don’t know
  102. 102. © 2016 Mesosphere, Inc. All Rights Reserved. HOW DO YOU DO SIGNALING? 102
  103. 103. © 2016 Mesosphere, Inc. All Rights Reserved. 103 Nobody ever got fired for using Zookeeper
  104. 104. © 2016 Mesosphere, Inc. All Rights Reserved. 104 We know about Zookeeper
  105. 105. © 2016 Mesosphere, Inc. All Rights Reserved. 105 It works…usually
  106. 106. © 2016 Mesosphere, Inc. All Rights Reserved. 106 How else can we do this?
  107. 107. © 2016 Mesosphere, Inc. All Rights Reserved. 107 Naively (See: Distributed Erlang)
  108. 108. © 2016 Mesosphere, Inc. All Rights Reserved. 108 Massive Amount of Information
  109. 109. © 2016 Mesosphere, Inc. All Rights Reserved. 109 Who else has solved this?
  110. 110. © 2016 Mesosphere, Inc. All Rights Reserved. 110 Academia
  111. 111. © 2016 Mesosphere, Inc. All Rights Reserved. 111 Control Plane: Lashup
  112. 112. © 2016 Mesosphere, Inc. All Rights Reserved. 112 How do we scale the naive approach?
  113. 113. © 2016 Mesosphere, Inc. All Rights Reserved. 113 All we need is a sparse, connected graph
  114. 114. © 2016 Mesosphere, Inc. All Rights Reserved. 114 But how?
  115. 115. © 2016 Mesosphere, Inc. All Rights Reserved. 115 Enter: HyParView* Builds a connected graph (overlay), where the degree is <=K *Inspired Protocol
  116. 116. © 2016 Mesosphere, Inc. All Rights Reserved. 116 Boot Time
  117. 117. © 2016 Mesosphere, Inc. All Rights Reserved. 117
  118. 118. © 2016 Mesosphere, Inc. All Rights Reserved. 118
  119. 119. © 2016 Mesosphere, Inc. All Rights Reserved. 119
  120. 120. © 2016 Mesosphere, Inc. All Rights Reserved. 120
  121. 121. © 2016 Mesosphere, Inc. All Rights Reserved. 121
  122. 122. © 2016 Mesosphere, Inc. All Rights Reserved. 122
  123. 123. © 2016 Mesosphere, Inc. All Rights Reserved. 123
  124. 124. © 2016 Mesosphere, Inc. All Rights Reserved. 124
  125. 125. © 2016 Mesosphere, Inc. All Rights Reserved. 125
  126. 126. © 2016 Mesosphere, Inc. All Rights Reserved. 126
  127. 127. © 2016 Mesosphere, Inc. All Rights Reserved. 127
  128. 128. © 2016 Mesosphere, Inc. All Rights Reserved. 128
  129. 129. © 2016 Mesosphere, Inc. All Rights Reserved. 129
  130. 130. © 2016 Mesosphere, Inc. All Rights Reserved. 130 Connected Graph Hyparview
  131. 131. © 2016 Mesosphere, Inc. All Rights Reserved. 131 Constant Adaptive Health Checks* *Borrowed from SWIM, Gossip Style Failure Detector
  132. 132. © 2016 Mesosphere, Inc. All Rights Reserved. 132 Dealing with Failure Hyparview
  133. 133. © 2016 Mesosphere, Inc. All Rights Reserved. 133 Dealing with Failure Hyparview
  134. 134. © 2016 Mesosphere, Inc. All Rights Reserved. 134 Dealing with Failure Hyparview
  135. 135. © 2016 Mesosphere, Inc. All Rights Reserved. 135 Dealing with Failure Hyparview
  136. 136. © 2016 Mesosphere, Inc. All Rights Reserved. 136 Routing* *Borrowed from Dijkstra, OSPF, IS-IS
  137. 137. © 2016 Mesosphere, Inc. All Rights Reserved. 137
  138. 138. © 2016 Mesosphere, Inc. All Rights Reserved. 138
  139. 139. © 2016 Mesosphere, Inc. All Rights Reserved. 139 Adjacent Node 1 Adjacent Node 2 Adjacent Node 3 A T B C B T A C C B A D D C E F E D F H F D E G G F H I H E G I I G H J J I L K K M L J L M K J M L K N N P O M O R P N P Q O N Q S R O R O Q S S T R Q T B A S
  140. 140. © 2016 Mesosphere, Inc. All Rights Reserved. 140 View Changes Gossiped on Change
  141. 141. © 2016 Mesosphere, Inc. All Rights Reserved. 141 Adjacent Node 1 Adjacent Node 2 Adjacent Node 3 Version (epoch) A T B C 1 B T A C 5 C B A D 2 D C E F 7 E D F H 94 F D E G 1 G F H I 415 H E G I 15 I G H J 1 J I L K 3 K M L J 5 L M K J 7 M L K N 88 N P O M 1 O R P N 3 P Q O N 4 Q S R O 6 R O Q S 8 S T R Q 1 T B A S 3
  142. 142. © 2016 Mesosphere, Inc. All Rights Reserved. 142
  143. 143. © 2016 Mesosphere, Inc. All Rights Reserved. 143 Just Run Dijkstra* *BFS / DFS
  144. 144. © 2016 Mesosphere, Inc. All Rights Reserved. 144 Minimum Spanning Tree, From Sender A
  145. 145. © 2016 Mesosphere, Inc. All Rights Reserved. 145 A Wild Pokemon Appears!* *CRDT papers Database
  146. 146. © 2016 Mesosphere, Inc. All Rights Reserved. 146 CRDTS: Semilattices “Time”
  147. 147. © 2016 Mesosphere, Inc. All Rights Reserved. 147 CRDT KV Store
  148. 148. © 2016 Mesosphere, Inc. All Rights Reserved. 148 “Demo”
  149. 149. © 2016 Mesosphere, Inc. All Rights Reserved. 149 “Demo” (foo@3c075477e55e)10> nodes(). [baz@3c075477e55e,bar@3c075477e55e] (bar@3c075477e55e)8> nodes(). [foo@3c075477e55e,baz@3c075477e55e] (baz@3c075477e55e)7> nodes(). [foo@3c075477e55e,bar@3c075477e55e]
  150. 150. © 2016 Mesosphere, Inc. All Rights Reserved. 150 “Demo” (foo@3c075477e55e)9> lashup_kv:value([erlang_users]). [] (bar@3c075477e55e)9> lashup_kv:value([erlang_users]). [] (baz@3c075477e55e)9> lashup_kv:value([erlang_users]). []
  151. 151. © 2016 Mesosphere, Inc. All Rights Reserved. 151 “Demo” (baz@3c075477e55e)10> net_kernel:allow([foo@3c075477e55e]). ok (baz@3c075477e55e)10> net_kernel:disconnect('bar@3c075477e55e'). ok.
  152. 152. © 2016 Mesosphere, Inc. All Rights Reserved. 152 “Demo”
  153. 153. © 2016 Mesosphere, Inc. All Rights Reserved. 153 “Demo” (bar@3c075477e55e)11> lashup_kv:request_op([erlang_users], {update, [{update, {number, riak_dt_pncounter}, {increment, 5}}]}). {ok,[{{number,riak_dt_pncounter},5}]}
  154. 154. © 2016 Mesosphere, Inc. All Rights Reserved. 154 “Demo” (foo@3c075477e55e)12> lashup_kv:value([erlang_users]). [{{number,riak_dt_pncounter},5}] (bar@3c075477e55e)13> lashup_kv:value([erlang_users]). [{{number,riak_dt_pncounter},5}] (baz@3c075477e55e)15> lashup_kv:value([erlang_users]). [{{number,riak_dt_pncounter},5}]
  155. 155. © 2016 Mesosphere, Inc. All Rights Reserved. 155 “Demo” (foo@3c075477e55e)23> erlang:set_cookie(baz@3c075477e55e, fake). true (foo@3c075477e55e)24> erlang:set_cookie(bar@3c075477e55e, fake). true
  156. 156. © 2016 Mesosphere, Inc. All Rights Reserved. 156 “Demo”
  157. 157. © 2016 Mesosphere, Inc. All Rights Reserved. 157 “Demo” (foo@3c075477e55e)29> lashup_kv:request_op([erlang_users], {update, [{update, {number, riak_dt_pncounter}, {increment, 3}}]}). {ok,[{{number,riak_dt_pncounter},8}]} (bar@3c075477e55e)17> lashup_kv:request_op([erlang_users], {update, [{update, {number, riak_dt_pncounter}, {increment, 7}}]}). {ok,[{{number,riak_dt_pncounter},12}]} (baz@3c075477e55e)19> lashup_kv:request_op([erlang_users], {update, [{update, {number, riak_dt_pncounter}, {increment, 11}}]}). {ok,[{{number,riak_dt_pncounter},16}]}
  158. 158. © 2016 Mesosphere, Inc. All Rights Reserved. 158 “Demo”
  159. 159. © 2016 Mesosphere, Inc. All Rights Reserved. 159 “Demo” (foo@3c075477e55e)1> lashup_kv:value([erlang_users]). [{{number,riak_dt_pncounter},26}] (bar@3c075477e55e)25> lashup_kv:value([erlang_users]). [{{number,riak_dt_pncounter},26}] (baz@3c075477e55e)23> lashup_kv:value([erlang_users]). [{{number,riak_dt_pncounter},26}]
  160. 160. © 2016 Mesosphere, Inc. All Rights Reserved. LASHUP KV 160 •Datatypes: •Maps •Composable •Sets •Flags •Counters •Registers •Last-write-wins
  161. 161. © 2016 Mesosphere, Inc. All Rights Reserved. 161 Does Lashup provide Business Value?
  162. 162. © 2016 Mesosphere, Inc. All Rights Reserved. 162 Prior to Lashup
  163. 163. © 2016 Mesosphere, Inc. All Rights Reserved. 163 After Lashup
  164. 164. © 2016 Mesosphere, Inc. All Rights Reserved. PROJECT LASHUP 164 •A novel distributed systems SDK that provides: •Membership •Multicast Delivery •Strongly-eventually consistent data storage •Powers: •Minuteman VIP dissemination •Minuteman node liveness checks •Overlay routing •DNS Synchronization •Open source: github.com/dcos/lashup
  165. 165. © 2016 Mesosphere, Inc. All Rights Reserved. 165 So, Why Erlang/OTP?
  166. 166. © 2016 Mesosphere, Inc. All Rights Reserved. 166 Why Not? (Go)
  167. 167. © 2016 Mesosphere, Inc. All Rights Reserved. 167 The First Version of Minuteman Was In Golang
  168. 168. © 2016 Mesosphere, Inc. All Rights Reserved. 168 Stand on the Shoulders of Giants
  169. 169. © 2016 Mesosphere, Inc. All Rights Reserved. 169 What’s a good language with a healthy set of distributed systems, and networking libraries?
  170. 170. © 2016 Mesosphere, Inc. All Rights Reserved. 170 What’s a good language with a healthy set of distributed systems, and networking libraries? With an ecosystem
  171. 171. © 2016 Mesosphere, Inc. All Rights Reserved. 171 What’s a good language with a healthy set of distributed systems, and networking libraries? With an ecosystem That’s battle tested
  172. 172. © 2016 Mesosphere, Inc. All Rights Reserved. 172 It’s a 3AM, who do you want picking up the phone?
  173. 173. © 2016 Mesosphere, Inc. All Rights Reserved. 173 Answer: You make it so that your software so it doesn’t page you at 3 AM.
  174. 174. © 2016 Mesosphere, Inc. All Rights Reserved. 174 Upgrading Software on Mars is Easier vs. On-Prem
  175. 175. © 2016 Mesosphere, Inc. All Rights Reserved. 175 Erlang
  176. 176. © 2016 Mesosphere, Inc. All Rights Reserved. LASHUP KV NEEDED A DURABLE STORE ANSWER: MNESIA 176 •ACID, Transactional •20+ years old •Battle Tested •Comes built-in •Software Transaction Memory
  177. 177. © 2016 Mesosphere, Inc. All Rights Reserved. LASHUP KV NEEDED A CRDT LIBRARY ANSWER: RIAK_DT 177 •Riak’s CRDT Library •Used in production by others •Battle Tested •Property tested
  178. 178. © 2016 Mesosphere, Inc. All Rights Reserved. LASHUP NEEDED SECURE DISTRIBUTION ANSWER: DISTERL 178 •SSL with mutual authentication is a matter of configuration •Used in production by others •Battle Tested •Comes Built-in
  179. 179. © 2016 Mesosphere, Inc. All Rights Reserved. SPARTAN NEEDED A DNS SERVER ANSWER: ERL-DNS 179 •Runs DNSimple •Used in production •Battle Tested •Some Statistics from DNSimple •Serves 20K+ domains •Does 20k queries/sec+
  180. 180. © 2016 Mesosphere, Inc. All Rights Reserved. MINUTEMAN NEEDED A KERNEL API LIBRARY ANSWER: GEN_NETLINK 180 •Liberal license •Deployed by TravelPing •Used in production •Battle Tested
  181. 181. © 2016 Mesosphere, Inc. All Rights Reserved. LASHUP NEEDED A GRAPH LIBRARY ANSWER: DIGRAPH 181 •Built-in Library to Erlang/OTP •Correctness verified •Simple API •Downsides: •Slow •Heavy
  182. 182. © 2016 Mesosphere, Inc. All Rights Reserved. 182 So, we built our own.
  183. 183. © 2016 Mesosphere, Inc. All Rights Reserved. 183 On Testing…
  184. 184. © 2016 Mesosphere, Inc. All Rights Reserved. 184 We verified lashup_gm_route against digraph
  185. 185. © 2016 Mesosphere, Inc. All Rights Reserved. 185 Property-Based Testing: PropEr
  186. 186. © 2016 Mesosphere, Inc. All Rights Reserved. 186 Automatically Generate Command Set CSet ={add_node(e), add_node(b), add_edge(e,b), …}
  187. 187. © 2016 Mesosphere, Inc. All Rights Reserved. 187 Verify Equality Digraph lashup_gm add_node(e) add_node(e) Check Equivalency add_node(b) add_node(b) Check Equivalency add_edge(e, b) add_edge(e, b) Check Equivalency
  188. 188. © 2016 Mesosphere, Inc. All Rights Reserved. 188 …Tens of Thousands of Times
  189. 189. © 2016 Mesosphere, Inc. All Rights Reserved. 189 Common Test Makes Integration Testing A Breeze
  190. 190. © 2016 Mesosphere, Inc. All Rights Reserved. COMMON TEST: ON EACH CHECK-IN (LASHUP) 190 •Spins up 25 virtual nodes •Torture tests: •Partitions nodes randomly for 5 minutes to see if it can generate a permanent split •Kills nodes randomly to verify global health check convergence •Verifies multicast works even in poor network conditions •Writes divergent keys to all nodes, and reconnects them •~500 lines of tests, 80%+ code coverage
  191. 191. © 2016 Mesosphere, Inc. All Rights Reserved. TAKE-AWAYS 191 •The container ecosystem is in is early stages •Erlang is suitable for building modern, reliable distributed systems •These systems are tricky, so testing is key •We have a healthy ecosystem •There need to be alternatives to consensus
  192. 192. © 2016 Mesosphere, Inc. All Rights Reserved. ERLANG & CONTAINER NETWORKING @SARGUN 192

×