Infrastructure microservices such as Service Discovery and Routing need mechanisms to manage distributed state. The most common answer for this is the use of a central, consistent key value store such as Consul, Etcd or Zookeeper. These systems use consensus-based algorithms, such as Raft or Paxos, to provide consistency and failure tolerance. I believe this is a dangerous direction for our industry, and instead we should be focusing on ease of use and reliabiligy. As such, Weave wants its infrastructure microservices to be decentralized, yet easy to install and run; our approach is inspired by the Internet, which is distributed and operates with no consensus. In this talk, I'll explain how we designed our Service Discovery and Address Management using Convergent Replicated Data Types (CRDTs) and Gossip, review the pros and cons of this concept, and how it compares to alternatives.

1. We don’t need consensus:
all agreed?
tom@weave.works
@tom_wilkie

3. μServices

4. μServices
“In computing, microservices is a software
architecture style in which complex
applications are composed of small,
independent processes communicating with
each other using language-agnostic APIs.”
https://en.wikipedia.org/wiki/Microservices

5. Containerization
Monoliths
Developer Freedom
Services & Replicas
Distributed Systems

6. Infrastructure μServices

7. Routing
Service Discovery
IP Address Management
Load Balancing
Infrastructure Microservices

8. Fallacies of distributed computing
• The network is reliable.
• Latency is zero.
• Bandwidth is infinite.
• The network is secure.
• Topology doesn't
change.
• There is one
administrator.
• Transport cost is zero.
• The network is
homogeneous.
Bill Joy, Peter Deutsch et al., 20 years ago

10. Routing
Service Disco
IPAM
Load Bal.
Flannel, libnetwork
Consul, SkyDNS
Flannel, libnetwork
bakersteet, SmartStack,
kubeproxy
Infrastructure Microservices

11. Our industry’s addiction to this architecture
is leading us to build fragile systems.
This is a dangerous precedent
We should be more concerned about ease
of use and reliability.

12. Ease of use

13. Fragility
Private DC Public Cloud
Where do I put my KV store?

14. Do we even need strong
consistency?

15. Routing
Service Disco
IPAM
Load Bal.
BGP
DNS
ICANN
…
Infrastructure Microservices

16. An alternative approach

17. Gossip
• Consider each node
broadcasting updates:
• O(n
2
) traffic, O(1)
propagation delay
• Alternatively, each node picks
random set of neighbours and
tells them:
• O(log n) traffic, O(log n)
propagation delay on
average

18. Convergent Replicated Data Type
• maintains consistency
without consensus
• nodes merge updates
from other nodes
• merge function must be
commutative, associative
and idempotent
https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type

19. Concrete Application of CRDTs

20. Routing: Weave Net
• Data structure is a
mapping from
container MAC
address to host ID
• We also gossip the
topology, who is
connected to whom

21. IP Address Management
• Data structure is a ring,
mapping IP address segments
to hosts
• Each host only ever
manipulates its own entries.
• When one host wants to grant
some free space to another
host, it splits one of its own
ranges and gossips an update.

22. Service Discovery: WeaveDNS
• Data structure is a mapping
from DNS name to IP address
• Entries also contain the host
and container ID, so multiple
hosts can have an entry for
the same name
• This allows for round-robin
DNS load-spreading

23. Results

25. A demo

26. @weaveworks github.com/weaveworks
Questions?
http://weave.works/product/scope
tom@weave.works
@tom_wilkie

27. Credits
Based on talk by Bryan Boreham, who
cowrote the IPAM implementation with me.
https://www.youtube.com/watch?
v=117gWVShcGU

28. Details
• How do we keep from accumulating an ever-
increasing set of tombstoned entries?
• Tombstoned entries time out after a while
• CRDTs are not something you can pull off-the-shelf
• Data structure must be carefully designed
• How to bootstrap the IP allocator system?
• We use Paxos