Getting Configuration Management in place is a big step in the direction of infrastructure automation. Chef, Puppet and Co. replace error-prone manual changes with periodic system convergence runs controlled by a central database. Even with Puppet’s exported resources and Chef’s search capabilities, the weakness of this approach is that it is rather static. In situations where we need to propagate information quickly, handle failure detection, or tolerate network partitions, other tools might offer better solutions. In this talk, I’m going to present some of these alternatives (e.g. serf, etcd) and how they can be used to allow for more dynamic configuration changes.

1. freistil.it
Dynamic
Infrastructure
Orchestration
Jochen Lillich

2. Jochen Lillich
@geewiz
Sysadmin since 1993
CEO of freistil IT
freistilbox: PaaS for Drupal & WordPress

3. Config Management
Elimination of manual changes
Consistent configuration
Single source of truth
DRY

4. Weaknesses
Periodic convergence runs
Central database

5. Orchestration
Perform changes quickly
Handle failure
Tolerate network partitions

6. serf

7. Basics
Cluster communication tool
Developed by Hashicorp
Simple binary,written in Go
Gossip protocol

8. serf node 1
$ serf agent -node=node1 -bind=srv1
==> Starting Serf agent...
==> Starting Serf agent RPC...
==> Serf agent running!
Node name: 'srv1'
Bind addr: '192.168.0.11:7946'
…
==> Log data will now stream in as it occurs:
2014/04/09 16:05:51 [INFO] agent: Serf agent starting
2014/04/09 16:05:51 [INFO] serf: EventMemberJoin: node1
192.168.0.11
2014/04/09 16:05:52 [INFO] agent: Received event: member-
join

9. serf node 2
$ serf agent -node=node2 -bind=srv2
…
$ serf join srv1
Successfully joined cluster by contacting 1
nodes.
$ serf members
node2 192.168.0.12:7946 alive
node1 192.168.0.11:7946 alive

10. Membership
Cluster management
Service configuration

11. Failure detection
Detects failure quickly
Notifies other nodes
Executes handler scripts

12. Events
member-join
member-leave
member-failed
member-update
member-reap
user
query

13. User events
Deploy application
Perform convergence run
Query information from the cluster

14. Example handler
#!/bin/bash
!
echo
echo "New event: ${SERF_EVENT}. Data
follows..."
while read line; do
printf "${line}n"
done

15. User event
$ serf event deploy-app
!
2014/04/09 17:06:32 [INFO] agent: Received
event: user-event: deploy-app

16. etcd

17. Basics
HA key-value store
Developed by CoreOS
Written in Go
Raft consensus algorithm
Pure HTTPAPI

18. Key-value store
/release = 9607bcfeb48905d26db8f
/cluster
/cluster/node1 = node1.example.com
/cluster/node2 = node2.example.com

19. Quorum
Available

20. Quorum
Available

21. Quorum
Available

22. Quorum
Unavailable

23. Writing
$ etcdctl set /message Hello
Hello
$ curl -L -X PUT
http://127.0.0.1:4001/v2/keys/message
-d value=“Test"
{"action":"set","node":{"key":"/
message","value":"Test","modifiedIndex":
15,"createdIndex":15},"prevNode":{"key":"/
message","value":"Hello","modifiedIndex":
2,"createdIndex":2}}

24. Reading
$ etcdctl get /message
Hello
$ curl -L
http://127.0.0.1:4001/v2/keys/message
{"action":"get","node":{"key":"/
message","value":"Test","modifiedIndex":
15,"createdIndex":15}

25. Deleting
$ etcdctl rm /message
$ curl -L -X DELETE
http://127.0.0.1:4001/v2/keys/message

26. Waiting
$ curl -L http://127.0.0.1:4001/v2/keys/
release?wait=true

27. Waiting
$ curl -L http://127.0.0.1:4001/v2/keys/
release?wait=true
{"action":"set","node":{"key":"/
release","value":"9607bcfe","modifiedIndex":
16,"createdIndex":16},"prevNode":{"key":"/
release","value":"18512199","modifiedIndex":
14,"createdIndex":14}}

28. Applications
Locking
Leader election
Database master
Elastic IP

29. Leader election
etcd
App

30. Leader election
etcd
App
DB?

31. Leader election
etcd
App
DB?
Nope.

32. Leader election
etcd
App
DB?
Nope.
I’ll wait.

33. Leader election
etcd
App

34. Leader election
etcd
App DB X

35. Leader election
etcd
App DB XMaster me!

36. Leader election
etcd
App DB XMaster me!
Ok!

37. Leader election
etcd
App DB X
DB X is master.

38. Leader election
etcd
App DB X
DB X is master.

39. Leader election
etcd
App DB X
DB X is master.

40. Leader election
etcd
App DB X

41. Leader election
etcd
App DB X
DBY

42. Leader election
etcd
App DB X
DBY
Master me!

43. Leader election
etcd
App DB X
DBY
Master me!
No,DB X is.

44. Leader election
etcd
App DB X
DBY

45. Leader election
etcd
App
DBY

46. Leader election
etcd
App
DBY

47. Leader election
etcd
App
DBY

48. Leader election
etcd
App
DBY
You da master!

49. Leader election
etcd
App
DBYDB Y is master now.

50. Leader election
etcd
App
DBY

51. Summary

52. Thanks!
@geewiz
jochen@freistil.it