Managing your Minions with Func

Managing your Minions with Func Daniel Hanks Sr. System Administrator, Omniture

Your humble presenter... ● Daniel Hanks ● Sysadmin @ Omniture* ● B.S. Computer Science, BYU, 2001 ● Linux sysadmin since 1998 ● EagleNet Online: 1997-1999 ~20 servers ● Northsky / About.com / Primedia / United Online: 1999-2008 ~200 servers ● Omniture: 2008 – present ~15,000 servers * I, for one, welcome our new Adobe overlords!

Your humble presenter... ● Blog: http://brainshed.com ● Twitter: @danhanks ● Coming (hopefully) soon: http://opspatterns.com

Why are you here?

The Problem You need to run this: chown 755 /var/www/cgi-bin/my.pl

On this...

The problem ● And gather the output in a useful way*. * How do you know it worked on the 1000 machines you ran it on?

Some history... ● “masshost” script at UOL Web Services ● Written in Expect (later wrapped with Expect.pm) ● SSH under the hood ● Ran serially, host by host ● Most big shops have something like this

You could write your own... ● Deal with Expect (or Expect.pm) ● Handle ssh 'fun' ● Password auth or key auth ● Missing keys (first login) ● Changing keys (after re-imaging a machine) ● Handling parallel/asynchronous execution if your site is large enough ● Handling input / output from all of the above ● Bragging rights

Or you could use Func

What's Func?

Makes it easier to manage your minions...

What's Func? ● Fedora Unified Network Controller ● An easy way to run commands on lots of remote machines ● Written in ● Bindings to talk to it for Java ● But you can talk to it from anything that speaks YAML or JSON

What's Func? ● But it's much more: ● Secure server communications platform ● XML-RPC over SSL ● You could build your own puppet or cfengine with it. ● Build a backend for a server-automation system with a web interface ● Automate 1000s of machines from a single location ● World domination!

Muahahahahahaha!

Func Overview ● Overlord machine(s) distribute commands to Minions:

Func Overview ● Overlord machine(s) distribute commands to Minions: “Install the vim-enhanced package” “Run /usr/bin/detonate”

Func Overview ● Overlord machine(s) distribute commands to Minions: “Stop the countdown-sequence daemon” “Kill all 'intruder*' Processes that are running

Func Overview ● Minions return results to the Overlord: “Here's a list of running processes” “That command Returned status 1”

Getting up and running

Installation ● Easiest way is with yum ● On Fedora: ● yum install func ● Will pull in certmaster too ● On CentOS (Use EPEL): ● rpm -Uvh http://download.fedora.redhat.com/pub/ epel/5/i386/epel-release-5- 3.noarch.rpm ● yum install func

Installation ● On Ubuntu, Debian and others: ● Follow: https://fedorahosted.org/func/wiki/InstallAndSetupGuide

Configuration ● On the overlord: ● /sbin/chkconfig --level 345 certmaster on ● /sbin/service certmaster start

Configuration ● On the minions: ● /etc/certmaster/minion.conf: [main] certmaster = overlord.example.org log_level = DEBUG cert_dir = /etc/pki/certmaster ● Then: – /sbin/chkconfig --level 345 funcd on – /sbin/service funcd start

Configuration ● Back on the overlord: ● certmaster-ca –list ● certmaster-ca –sign minionN.example.com ● All done! ● Now start commanding your minions

Muahahahahahaha!

Using Func from the Command Line

Func from the command-line func minion.example.com call command exists "/bin/detonate"

Func from the command-line Minion Hosts { func minion.example.com call command exists "/bin/detonate" { { { Module Method Arguments

Func from the command-line func minion.example.com call command run "/bin/detonate -c pull_the_lever"

Func from the command-line Globbing to command multiple minions*: func “db*.example.com” call command run "/bin/detonate -c pull_the_lever" *This works best if you haven't named all your minions after Loony Tunes characters...

Func from the command-line Globbing to command multiple minions: func “db*.example.com” … func “db*.example.com;mail.example.com” … func --exclude=”db*” “*.example.com” … func @webhosts … (Define @groups in /etc/func/groups Or extend via API to access remote dbs, ldap, etc.)

Func from the command line ● func minion.example.com call process info “rebel_commandos” ● func minion.example.com call process kill rebel_commandos SIGKILL ● func *.example.com call service start httpd ● func “*” ping # or... ● func “*” list_minions ### will show you all your minions.

Func from the command line What modules are available on my minions? func “*” call system list_modules

Func from the command line What methods are available in a module? func “minion.example.com” call <modulename> list_methods

Other Func Modules ● CopyFile ● SysCtl ● IPTables ● Vlan ● Mount ● Bridge ● Pullfile ● Hardware ● Reboot ● S.M.A.R.T. ● RPM ● Build your own ● Virt (Easy!) ● (More on that later) ● Yum

Func Output

Func Output ● Default: Pretty-printed Python tuples / data structures: ● func minion.example.com call mount list ● {'minion.example.com': [{'device': 'rootfs', 'dir': '/', 'options': 'rw', 'type': 'rootfs'}, {'device': '/proc', 'dir': '/proc', 'options': 'rw', 'type': 'proc'}, ... }]}

Func Output ● Command module output: ● func minion.example.com call command run “/bin/ls /blah” ● ('minion1.example.com', [1, '', 'ls: /blah: No such file or directoryn']) ('minion2.example.com', [1, '', 'ls: /blah: No such file or directoryn']) ● ('<hostname>', [<exit_status>, '<stdout>', '<stderr>'])

Func Output ● If you'd prefer JSON: ● func minion.example.com call --json mount list ● {"minion.example.com": [{"device": "rootfs", "type": "rootfs", "options": "rw", "dir": "/"}, {"device": "/proc", "type": "proc", "options": "rw", "dir": "/proc"}, {"device": "none", "type": "tmpfs", "options": "rw", "dir": "/dev"}, {"device": "/dev/root", "type": "ext2", "options": "rw", "dir": "/"}, {"device": "none", "type": "tmpfs", "options": "rw", "dir": "/dev"}, {"device": "/proc", "type": "proc", "options": "rw", "dir": "/proc"}, {"device": "/sys", "type": "sysfs", "options": "rw", "dir": "/sys"}, {"device": "none", "type": "devpts", "options": "rw", "dir": "/dev/pts"}, {"device": "/dev/cciss/c0d0p1", "type": "ext2", "options": "rw", "dir": "/boot"}, .... }]}

Func Output ● Or XML RPC: func minion.example.com call --xmlrpc mount list <params> <param> <value><struct> <member> <name>minion.example.com</name> <value><array><data> <value><struct> <member> <name>device</name> <value><string>rootfs</string></value> ...

Using Func From Scripts

Func API #!/usr/bin/python import func.overlord.client as fc client = fc.Client(“*.example.org”) results = client.yumcmd.update(“httpd”) results = client.service.start(“httpd”) results = client.sysctl.set('net.ipv4.tcp_syncookies',1)

Func API #!/usr/bin/python import func.overlord.client as fc client = fc.Client(“*.example.org”) results = client.yumcmd.update(“httpd”) results = client.service.start(“httpd”) results = client.sysctl.set('net.ipv4.tcp_syncookies',1) { { { Module Method Arguments

Using Func from Other Add picture here Languages

Using Func from other languages ● Use func-transmit ● Good for anything that can grok JSON or YAML 1.0

Using Func from other languages ● Create a file or string of YAML or JSON: clients: "*" async: False nforks: 1 module: command method: run parameters: "/bin/echo Hello World"

Using Func from other languages ● Pipe it into func-transmit ● func-transmit --yaml < my.yaml ● func-transmit --json < my.json ● Get results back in JSON or YAML

Writing your own modules

Writing your own modules Write the code: #!/usr/bin/python import func_module class Test(func_module.FuncModule): version = "0.1.1" api_version = "0.0.1" description = "A sample module" def mult(self, numb1, numb2): return numb1 * numb2

Write your own modules ● Drop your file into /usr/lib/python$version/site- packages/func/minion/modules ● Restart funcd on your minions ● Done.

Writing your own modules ● func-create-module will generate a basic module template ● Package your modules into rpms for ease of deployment (or push them out with func) ● See existing modules for examples

Writing your own modules ● Can't write modules in other than Python...yet.

Other features ● Delegation ● Multiplexing ● Asynchrony ● Web interface ● ACLs

Cobbler integration ● Kickstart ● Install Func ● Point it to your Overlord ● All from Cobbler! ● See https://fedorahosted.org/cobbler/wiki/ FuncIntegration

Muahahahahahaha!

More Func goodness ● Func Website: ● https://fedorahosted.org/func/ ● IRC: ● #func on freenode ● Mailing list: ● https://www.redhat.com/mailman/listinfo/func- list

Questions?

Thank you!

Photo Credits* ● “gargoyles” by Flickr user 'certified_su,' http://www.flickr.com/photos/certified_su/3502855962/, Creative Commons License BY 2.0 ● “Cubicle Gargoyle” by Flickr user 'Dennis from Atlanta,' http://www.flickr.com/photos/dennis_matheson/3419153723/, CC License BY-SA 2.0 ● “Stacked servers” by Flickr user 'redjar,' http://www.flickr.com/photos/redjar/360113468/, CC License BY-SA 2.0 ● “Happy Gargoyle” by Flickr user 'Wolfrage,' http://www.flickr.com/photos/wolfrage/3931044145/, CC License BY 2.0 ● “Gargoyles” by Flickr user 'Laertes,' http://www.flickr.com/photos/jonhurd/492395/, CC License BY 2.0 ● “blades” by Flickr user 'gothopotam,' http://www.flickr.com/photos/zagrobot/2731084578/, CC License BY 2.0 ● “Table Decoration” by Flickr user 'Kevin,' http://www.flickr.com/photos/kevincollins/75018222/, CC License BY 2.0 ● “Gargoyle” by Flickr user 'Joe Jakeman,' http://www.flickr.com/photos/jojakeman/2553116529/, CC License BY 2.0 ● “Exeter gargoyle” by Flickr user 'aaron.bihari,' http://www.flickr.com/photos/dakima-arts/3456721942/, CC License BY-SA 2.0 ● “Dark gargoyle” by Flickr user 'Bichuas (E. Carton),' http://www.flickr.com/photos/bichuas/442265525/, CC License BY-SA 2.0 ● “IMG_4743” by Flickr user 'beggs,' http://www.flickr.com/photos/beggs/88809549/, CC License BY 2.0 ● “Gothic Gargoyles” by Flickr user 'jakeliefer,' http://www.flickr.com/photos/jakeliefer/283541940/, CC License BY 2.0 ● “Hunting Gargoyles” by Flickr user 'moonrat42,' http://www.flickr.com/photos/moonrat/1444630026/, CC License BY-SA 2.0 ● “Kissing Gargoyle, Leon, Spain” by Flickr user 'AndrewGould,' http://www.flickr.com/photos/andrewgould/3643732025/, CC License BY 2.0 ● “IMG_4781” by Flickr user 'beggs,' http://www.flickr.com/photos/beggs/88810165/, CC License BY 2.0 ● “Thann : gargouille / gargoyle 1” by Flickr user 'OliBac,' http://www.flickr.com/photos/olibac/3451405035/, CC License BY 2.0 ● “(gar)Goyle of my dreams...” by Flickr user 'IntangibleArts,' http://www.flickr.com/photos/intangible/1136971041/, CC License BY 2.0 ● “Gargoyle overlooking Paris” by Flickr user 'aliciat89,' http://www.flickr.com/photos/aliciat/3138416652/, CC License BY 2.0 ● For more info on Creative Commons Licenses see http://creativecommons.org ● This presentation is licensed under a Creative Commons Attribution-Share Alike 2.0 license: http://creativecommons.org/licenses/by-sa/2.0/deed.en *Thank you Creative Commons!

Shell tricks ● My favorite shell trick (at no extra charge): ● {} expansion ● echo db{1,2,3,4}.example.com = “db1.example.com db2.example.com ...” ● Nestable: – mkdir -p rpm/{RPMS/ {i386,i686,noarch},SRPMS,SPECS,BUILD,SOURCES}

Managing your Minions with Func

by danhanks on Oct 12, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 18

Statistics

Managing your Minions with Func — Presentation Transcript