Introduction to Augeas and examples for how to use it to edit configuration files programmatically from cfgmgmtcamp

1. David Lutterkort
lutter@puppet.com
Augeas
A decade of configuration surgery

2. What's the problem?

3. The problem:
Edit configuration files programmatically

4. Disable PermitRootLogin in sshd_config
$ sed -r 's/PermitRootLogin yes/PermitRootLogin no/'

5. Disable PermitRootLogin in sshd_config
$ sed -r 's/PermitRootLogin yes/PermitRootLogin no/'
$ sed -r 's/(PermitRootLogin[ t]+)yes/1no/'

6. Disable PermitRootLogin in sshd_config
$ sed -r 's/PermitRootLogin yes/PermitRootLogin no/'
$ sed -r 's/(PermitRootLogin[ t]+)yes/1no/'
$ sed -r 's/(PermitRootLogin[ t]+)[a-z]+/1no/'

7. Disable PermitRootLogin in sshd_config
$ sed -r 's/PermitRootLogin yes/PermitRootLogin no/'
$ sed -r 's/(PermitRootLogin[ t]+)yes/1no/'
$ sed -r 's/(PermitRootLogin[ t]+)[a-z]+/1no/'
$ grep PermitRootLogin /etc/ssh/sshd_config
# PermitRootLogin no

8. The real problem:
Large number of config file formats

9. Reading files is easy.
Modifying them is hard.

10. Whole-file management not always
feasible

11. How does Augeas work?

12. Handle config files in place and in
their native format

13. Use the same data structure for all files

14. Preserve 'unimportant' detail
and minimize changes

17. How do you use Augeas?

18. C library with lots of language bindings
(Ruby, Python, Go, Rust, Lua, Node, Haskell, OCaml, …)

19. Included in top-shelf config mgmt systems
(Puppet type, augeasproviders, Salt, Mgmt, …)

20. Getting started: augtool
$ augtool
augtool> help
Admin commands:
context - change how relative paths are interpreted
load - (re)load files under /files
save - save all pending changes
...
Informational commands:
errors - show all errors encountered in processing files
...

21. Getting started: augtool
$ augtool --help
Usage: augtool [OPTIONS] [COMMAND]
-b, --backup preserve originals of modified files with
extension '.augsave'
-r, --root ROOT use ROOT as the root of the filesystem
-t, --transform XFM add a file transform
-l, --load-file FILE load individual FILE in the tree
-f, --file FILE read commands from FILE
-L, --noload do not load any files into the tree on
startup
-A, --noautoload do not autoload modules from the search path

22. Getting started: augmatch (new in 1.10.1)
$ docker pull lutter/augmatch
$ docker run -ti lutter/augmatch
/ # augmatch --help
Usage: augmatch [OPTIONS] FILE
Print the contents of a file as parsed by augeas.
Options:
-l, --lens LENS use LENS to transform the file
-m, --match EXPR start printing where nodes match EXPR
-e, --exact print only exact matches
...

23. Example: /etc/exports
$ cat /etc/exports
/local 207.46.0.0/16(rw,sync)
/home 207.46.0.0/16(rw,root_squash,sync) 192.168.50.2/32(rw,root_squash,sync)
/tmp 207.46.0.0/16(rw,root_squash,sync)
/pub *(ro,insecure,all_squash)

24. Getting started: augmatch (new in 1.10.1)
$ augmatch /etc/exports
dir[1] = /local
dir[1]/client = 207.46.0.0/16
dir[1]/client/option[1] = rw
dir[1]/client/option[2] = sync
dir[2] = /home
dir[2]/client[1] = 207.46.0.0/16
dir[2]/client[1]/option[1] = rw
dir[2]/client[1]/option[2] = root_squash
dir[2]/client[1]/option[3] = sync
dir[2]/client[2] = 192.168.50.2/32
dir[2]/client[2]/option[1] = rw
dir[2]/client[2]/option[2] = root_squash
...

25. Getting started: augmatch (new in 1.10.1)
$ cat /etc/exports
/local 207.46.0.0/16(rw,sync)
/home 207.46.0.0/16(rw,root_squash,sync) 192.168.50.2/32(rw,root_squash,sync)
/tmp 207.46.0.0/16(rw,root_squash,sync)
/pub *(ro,insecure,all_squash)

26. Getting started: augmatch (new in 1.10.1)
$ cat /etc/exports
/local 207.46.0.0/16(rw,sync)
/home 207.46.0.0/16(rw,root_squash,sync) 192.168.50.2/32(rw,root_squash,sync)
/tmp 207.46.0.0/16(rw,root_squash,sync)
/pub *(ro,insecure,all_squash)
# List all clients to which we export a directory
$ augmatch --only-value --exact --match dir/client /etc/exports
207.46.0.0/16
207.46.0.0/16
192.168.50.2/32
207.46.0.0/16
*

27. Getting started: augmatch (new in 1.10.1)
$ cat /etc/exports
/local 207.46.0.0/16(rw,sync)
/home 207.46.0.0/16(rw,root_squash,sync) 192.168.50.2/32(rw,root_squash,sync)
/tmp 207.46.0.0/16(rw,root_squash,sync)
/pub *(ro,insecure,all_squash)

28. Getting started: augmatch (new in 1.10.1)
$ cat /etc/exports
/local 207.46.0.0/16(rw,sync)
/home 207.46.0.0/16(rw,root_squash,sync) 192.168.50.2/32(rw,root_squash,sync)
/tmp 207.46.0.0/16(rw,root_squash,sync)
/pub *(ro,insecure,all_squash)
# List all clients to which we export the /home directory
$ augmatch -eom 'dir["/home"]/client ' /etc/exports
207.46.0.0/16
192.168.50.2/32

29. Getting started: augmatch (new in 1.10.1)
# Find all directories that are exported to at least one client without having
# the 'root_squash' option set
$ cat /etc/exports
/local 207.46.0.0/16(rw,sync)
/home 207.46.0.0/16(rw,root_squash,sync) 192.168.50.2/32(rw,sync)
/tmp 207.46.0.0/16(rw,root_squash,sync)
/pub *(ro,insecure,all_squash)

30. Getting started: augmatch (new in 1.10.1)
# Find all directories that are exported to at least one client without having
# the 'root_squash' option set
$ cat /etc/exports
/local 207.46.0.0/16(rw,sync)
/home 207.46.0.0/16(rw,root_squash,sync) 192.168.50.2/32(rw,sync)
/tmp 207.46.0.0/16(rw,root_squash,sync)
/pub *(ro,insecure,all_squash)
$ augmatch -eom 'dir[client[not(option = "root_squash")]]' /etc/exports
/local
/home
/pub

31. Getting started: augmatch (new in 1.10.1)
# How match dir[client[not(option = "root_squash")]] works
$ augmatch /etc/exports
dir[1] = /local
dir[1]/client = 207.46.0.0/16
dir[1]/client/option[1] = rw
dir[1]/client/option[2] = sync
dir[2] = /home
dir[2]/client[1] = 207.46.0.0/16
dir[2]/client[1]/option[1] = rw
dir[2]/client[1]/option[2] = root_squash
dir[2]/client[1]/option[3] = sync
...

32. Example: Idempotent change
Ensure options of a particular client for a
particular directory

33. Strategy
1. Initialize Augeas and load file(s)
2. Create entry in /scratch
3. Move or append entry

34. Idempotent change
$ augtool print /files/etc/exports/dir["/home"]/client[1]
/files/etc/exports/dir[2]/client[1] = "207.46.0.0/16"
/files/etc/exports/dir[2]/client[1]/option[1] = "rw"
/files/etc/exports/dir[2]/client[1]/option[2] = "root_squash"
/files/etc/exports/dir[2]/client[1]/option[3] = "sync"

35. Idempotent change
def ensure_client(dir, client, opts)
Augeas::open(nil, nil, Augeas::NO_MODL_AUTOLOAD) do |aug|
aug.transform(lens: "Exports.lns", incl: "/etc/exports")
aug.context("/files/etc/exports")
aug.load
aug.set("/scratch/client", client)
options.each {|opt| aug.set("/scratch/client/option[last()+1]", opt)}
if aug.match("dir['#{dir}']").empty?
aug.set("dir[last()+1]", dir)
end
...
aug.mv("/scratch", "dir['#{dir}']/client['#{client}]")
aug.save
end
end

36. Idempotent change
def ensure_client(dir, client, opts)
Augeas::open(nil, nil, Augeas::NO_MODL_AUTOLOAD) do |aug|
aug.transform(lens: "Exports.lns", incl: "/etc/exports")
aug.context("/files/etc/exports")
aug.load
aug.set("/scratch/client", client)
options.each {|opt| aug.set("/scratch/client/option[last()+1]", opt)}
if aug.match("dir['#{dir}']").empty?
aug.set("dir[last()+1]", dir)
end
...
aug.mv("/scratch", "dir['#{dir}']/client['#{client}]")
aug.save
end
end

37. Idempotent change
def ensure_client(dir, client, opts)
Augeas::open(nil, nil, Augeas::NO_MODL_AUTOLOAD) do |aug|
aug.transform(lens: "Exports.lns", incl: "/etc/exports")
aug.context("/files/etc/exports")
aug.load
aug.set("/scratch/client", client)
options.each {|opt| aug.set("/scratch/client/option[last()+1]", opt)}
if aug.match("dir['#{dir}']").empty?
aug.set("dir[last()+1]", dir)
end
...
aug.mv("/scratch", "dir['#{dir}']/client['#{client}]")
aug.save
end
end

38. /files file contents
/augeas metadata
/context current 'directory'
/load file/lens mappings
/root file system root
/save how to save files
/version current version

39. /augeas/files/<path> file metadata
/path path under /files
/mtime file's mtime
/lens lens used to process
/error error detail

40. How can I get involved?

41. Need to reorganize and expand docs
(Want to move to gitbook)

42. Other ideas
- Many language bindings could use
some love
- Write moar lenses
- Make Augeas work on Windows

43. Learn more and get in touch
https://augeas.net/