Mgmt is a next gen config management tool that takes a fresh look at existing automation problems. Three of the main design features of the tool include: * Parallel execution * Event driven mechanism * Distributed architecture This presentation will briefly introduce the tool and spend most of the time presenting and demoing some of the newer features in the project. We'll present some of the new resources (virt, password, etc) new features (libified mgmt, send/recv, DSL) and how these can be used to build autonomous systems. Finally we'll talk about some of the future designs we're planning and make it easy for new users to get involved and help shape the project.

1. 1 JAMES SHUBIN
Next generation
configuration mgmt
James Shubin, a.k.a. @purpleidea
Principal Software Engineer
Storage Engineering Group, Red Hat
OSDC/Netways, Berlin, Germany, 17/May/2017

2. 2 JAMES SHUBIN

3. 3 JAMES SHUBIN
Who am I ?
● Hacker
● Config Mgmt. Architect @ Red Hat
● Technical Blogger: The Technical Blog of James
https://ttboj.wordpress.com/
● Physiologist (Cardiology Specialization)
● DevOps believer

4. 4 JAMES SHUBIN
You might remember some of my puppet hacks...

5. 5 JAMES SHUBIN

6. 6 JAMES SHUBIN
Background

7. 7 JAMES SHUBIN
Background
I started hacking with puppet around ~0.24
I think I got pretty good at it
I wanted to push further, but I ended up needing
to write some outrageous hacks to get there…
I gave a “pushing puppet” talk at LISA 2013, and
DevConf.cz 2015

8. 8 JAMES SHUBIN
Hack: recursion
define recursion($count) {
notify { "count-${count}": }
$minus1 = inline_template('<%= count.to_i-1 %>')
if "${minus1}" != '0' { # recurse
recursion { "count-${minus1}":
count => $minus1,
}
}
}
https://github.com/purpleidea/puppet-pushing/blob/master/manifests/recursion.pp

9. 9 JAMES SHUBIN
Hack: Exec['again']
# notify me to get puppet to run again
exec { 'again':
command => "${vardir}/again/again.py",
logoutput => on_failure,
refreshonly => true,
require => File["${vardir}/again/again.py"],
}
https://github.com/purpleidea/puppet-common/blob/master/manifests/again.pp

10. 10 JAMES SHUBIN
Hack: timers
runonce::exec { 'runonce':
command => "/bin/echo 'Runonce!'",
}
runonce::timer { 'timer':
command => "/bin/echo 'Hello!'",
repeat_on_failure => true,
delta => 120, # time
again => true,
}
https://github.com/purpleidea/puppet-common/blob/master/manifests/again.pp

11. 11 JAMES SHUBIN
Hack: finite state machines
fsm::transition { 'water':
input => 'liquid',
#input => 'solid',
#input => 'gas',
}
https://github.com/purpleidea/puppet-common/blob/master/manifests/again.pp

12. 12 JAMES SHUBIN
Is this the right way to do it?

13. 13 JAMES SHUBIN

14. 14 JAMES SHUBIN
Introducing...

15. 15 JAMES SHUBIN

16. 16 JAMES SHUBIN
Community
IRC:
#mgmtconfig on Freenode
Twitter:
@mgmtconfig & #mgmtconfig
Mailing list:
mgmtconfig-list@redhat.com

17. 17 JAMES SHUBIN
Design triad

18. 18 JAMES SHUBIN
Design triad
Parallel execution
Event driven
Distributed topology

19. 19 JAMES SHUBIN
Design triad
Parallel execution
Event driven
Distributed topology

20. 20 JAMES SHUBIN
Parallel execution: status quo

21. 21 JAMES SHUBIN
Parallel execution: mgmt

22. 22 JAMES SHUBIN
Parallel execution: demo1

23. 23 JAMES SHUBIN
Parallel execution: demo2

24. 24 JAMES SHUBIN
Parallel execution: demo3

25. 25 JAMES SHUBIN
Event driven: status quo

26. 26 JAMES SHUBIN
Event driven: mgmt

27. 27 JAMES SHUBIN
Event driven: demo
● File: inotify / fanotify
● Svc: systemd events (dbus)
● Exec: kernel events
● Pkg: packagekit events
● Virt: libvirtd events

28. 28 JAMES SHUBIN
What is this?

29. 29 JAMES SHUBIN
MONITORING

30. 30 JAMES SHUBIN
Distributed topology: status quo

31. 31 JAMES SHUBIN
Distributed topology: orchestrator

32. 32 JAMES SHUBIN
Distributed topology: mgmt?

33. 33 JAMES SHUBIN
Distributed topology: mgmt

34. 34 JAMES SHUBIN
Distributed topology: mgmt

35. 35 JAMES SHUBIN
Distributed topology: mgmt

36. 36 JAMES SHUBIN
Distributed topology: demo
● Three+ nodes: A, B, C
● Each creates one file which it will export
● A (exports): /tmp/mgmtA/fA
● A (collects): all (exported) files into: /tmp/mgmtA/
● Similarly with B and C, except with the letters B
and C substituted in to the emphasized locations
above.
● Note how fast this converges...

37. 37 JAMES SHUBIN
Distributed topology: example
A B C

38. 38 JAMES SHUBIN
Distributed topology: example
A B C

39. 39 JAMES SHUBIN
MOAR features

40. 40 JAMES SHUBIN
Automatic elastic clustering: example

41. 41 JAMES SHUBIN
Automatic elastic clustering: example

42. 42 JAMES SHUBIN
Automatic elastic clustering: example

43. 43 JAMES SHUBIN
● Add h4, remove h2…
● Needs etcd API improvements for edge cases
● Now ported to etcd V3 beta API
● Help and patch review wanted...
Automatic elastic clustering: demo

44. 44 JAMES SHUBIN
Remote execution: example

45. 45 JAMES SHUBIN
● Executes over SSH
● Exchanges resources through etcd initiator
● Converges together as a cluster
● Responds instantly to configuration changes
Remote execution: demo

46. 46 JAMES SHUBIN
Remote execution: future?

47. 47 JAMES SHUBIN
● Pkg + File + Svc resources: autoedges3.yaml
● Many other resources will be able to use this…
● Other ideas?
Automatic edges: demo

48. 48 JAMES SHUBIN
1) Svc resource could list its config files, eg:
[Config]
File=/etc/foo/thing.conf
Dir=/etc/bar/
Match=/etc/baz/*.conf
2) Svc could determine if running state of service
matches configuration files mentioned via
lastModified time of start/reload checks: eg:
“service needs reload, config file changed”
Automatic edges: enhancements

49. 49 JAMES SHUBIN
Why?
https://github.com/systemd/systemd/issues/4239
Automatic edges: enhancements
/etc/foo.conf food.service

50. 50 JAMES SHUBIN
● Can write to the journal… w00t.
Msg resource: demo

51. 51 JAMES SHUBIN
Automatic grouping: graph

52. 52 JAMES SHUBIN
Automatic grouping: graph

53. 53 JAMES SHUBIN
● Pkg resource: autogroup2.yaml
● File resources could be grouped to save watches
● Higher level, API-bound resources will love this
● Other ideas?
Automatic grouping: demo

54. 54 JAMES SHUBIN
Automatic grouping: data
Bigger is worse...

55. 55 JAMES SHUBIN
● Yeah this exists, and the code is beautiful.
● Not complete, but the core is looking great!
● ./mgmt run --puppet 'notify { “hello world”: }'
● All thanks to Felix Frank (@felis_rex)
Automatic puppet → mgmt compiler

56. 56 JAMES SHUBIN
● Augeas
● Exec
● File
● Hostname
● KV
● Msg
● Noop
Current resources (in some form)
● Nspawn
● Password
● Pkg
● Svc
● Timer
● Virt

57. 57 JAMES SHUBIN
● We have a resource guide. Please help write
new resources, and improve existing ones!
● https://github.com/purpleidea/mgmt/blob/master/
docs/resource-guide.md
Resource guide (internal, upstream)

58. 58 JAMES SHUBIN
● AutoEdge
● AutoGroup
● Noop
● Retry & Delay
● Poll
● Limit & Burst
● Sema
Current metaparams
● Got ideas for more?

59. 59 JAMES SHUBIN
embedding mgmt
libmgmt

60. 60 JAMES SHUBIN
notifications along graph
edges (dependencies)
Refresh

61. 61 JAMES SHUBIN
data transfer along graph
edges (dependencies)
Send->Recv

62. 62 JAMES SHUBIN
Future work

63. 63 JAMES SHUBIN
virt
Future work: demo

64. 64 JAMES SHUBIN
virt + cockpit
Future work: demo

65. 65 JAMES SHUBIN
virt + cockpit + cpu hotplug
Future work: demo

66. 66 JAMES SHUBIN
Future work
● DSL: lexer, parser, declarative, FRP?
● Resources: virt, nspawn, net, http, mount, etc...
● Etcd: clustering improvements
● DHT based file distribution
● Community: it's your tool – Thoughts?

67. 67 JAMES SHUBIN
DFRP language
Upcoming projects...

68. 68 JAMES SHUBIN
Graph FSM and methods
Upcoming projects...

69. 69 JAMES SHUBIN
Glusterd2.x/Gd3 ?
Upcoming projects...

70. 70 JAMES SHUBIN
How can you help?

71. 71 JAMES SHUBIN
How can you help?
● Use it
● Test it
● Patch it
● Share it
● Document it
● Star it
● Blog it
● Tweet it
● Discuss it
● Hack it

72. 72 JAMES SHUBIN
hack it

73. 73 JAMES SHUBIN
Red Hat funds good hackers so that we can...
● Work on open source / free software things...
● Speak at events like this...
● Hack on good products and solutions...
● For access to products, solutions, and support, visit:
https://redhat.com/

74. 74 JAMES SHUBIN
UPSTREAM

75. 75 JAMES SHUBIN
COMMUNITY

76. 76 JAMES SHUBIN
,PROJECT,
NOT PRODUCT

77. 77 JAMES SHUBIN
,PROJECT,
NOT PRODUCT

78. 78 JAMES SHUBIN
Let's recap...

79. 79 JAMES SHUBIN

80. 80 JAMES SHUBIN
Learn more
● The Technical Blog of James:
https://ttboj.wordpress.com/
● Mgmt:
https://github.com/purpleidea/mgmt/
● Technical articles and screencasts about this:
https://ttboj.wordpress.com/?s=mgmtconfig
● Contact me if you have any other questions:
purpleidea @ { irc, twitter, github, gmail, redhat.com }

81. 81 JAMES SHUBIN

82. 82 JAMES SHUBIN
Feedback:
If you really liked this talk,
please take 5 seconds and go
up to Bernd and tell him so!

83. 83 JAMES SHUBIN
IRC:
#mgmtconfig on Freenode
Twitter:
@mgmtconfig & #mgmtconfig
Mailing list:
mgmtconfig-list@redhat.com

84. 84 JAMES SHUBIN
Q & A ?

85. 85 JAMES SHUBIN
Thank you & Happy Hacking !