Server Administration in Python with Fabric, Cuisine and Watchdog

Server Administration in Python with Fabric, Cuisine and Watchdog

• 1.
  Fabric, Cuisine & Watchdog Sébastien Pierre, ffunction inc. @Montréal Python, February 2011 www.ffctn.com ffunction inc.
• 2.
  How to usePython for Server Administration Thanks to Fabric Cuisine* & Watchdog* *custom tools ffunction inc.
• 3.
  The way weuse servers has changed ffunction inc.
• 4.
  The era ofdedicated servers Hosted in your server room or in colocation WEB DATABASE EMAIL SERVER SERVER SERVER ffunction inc.
• 5.
  The era ofdedicated servers Hosted in your server room or in colocation WEB DATABASE EMAIL SERVER SERVER SERVER Sysadmins typically Sysadmins typically SSH and configure SSH and configure the servers live the servers live ffunction inc.
• 6.
  The era ofdedicated servers Hosted in your server room or in colocation WEB DATABASE EMAIL SERVER SERVER SERVER The servers are The servers are conservatively managed, conservatively managed, updates are risky updates are risky ffunction inc.
• 7.
  The era ofslices/VPS Linode.com Amazon Ec2 SLICESLICE SLICE 1 1 1 SLICE 1 SLICESLICE 6 1 SLICE SLICE 11 10 SLICE 9 We now have multiple We now have multiple small virtual servers small virtual servers (slices/VPS) (slices/VPS) ffunction inc.
• 8.
  The era ofslices/VPS Linode.com Amazon Ec2 SLICESLICE SLICE 1 1 1 SLICE 1 SLICESLICE 6 1 SLICE SLICE 11 10 SLICE 9 Often located in different Often located in different data-centers data-centers ffunction inc.
• 9.
  The era ofslices/VPS Linode.com Amazon Ec2 SLICESLICE SLICE 1 1 1 SLICE 1 SLICESLICE 6 1 SLICE SLICE 11 10 SLICE 9 ...and sometimes with ...and sometimes with different providers different providers ffunction inc.
• 10.
  The era ofslices/VPS Linode.com Amazon Ec2 SLICESLICE SLICE 1 1 1 SLICE 1 SLICESLICE 6 1 SLICE SLICE 11 10 SLICE 9 IWeb.com We even sometimes DEDICATED DEDICATED We even sometimes still have physical, SERVER 1 SERVER 2 still have physical, dedicated servers dedicated servers ffunction inc.
• 11.
  The challenge ORDER SERVER ffunction inc.
• 12.
  The challenge ORDER SETUP SERVER SERVER ffunction inc.
• 13.
  The challenge Create users, groups Create users, groups Customize config files Customize config files Install base packages Install base packages ORDER SETUP SERVER SERVER ffunction inc.
• 14.
  The challenge ORDER SETUP DEPLOY SERVER SERVER APPLICATION ffunction inc.
• 15.
  The challenge Install app-specific Install app-specific packages packages deploy application deploy application start services start services ORDER SETUP DEPLOY SERVER SERVER APPLICATION ffunction inc.
• 16.
  The challenge ORDER SETUP DEPLOY SERVER SERVER APPLICATION MAKE THIS PROCESS AS FAST (AND SIMPLE) AS POSSIBLE ffunction inc.
• 17.
  The challenge ffunction inc.
• 18.
  The challenge Quickly integrate your Quickly integrate your new server in the new server in the existing architecture existing architecture ffunction inc.
• 19.
  The challenge ...and make sure ...and make sure it's running! it's running! ffunction inc.
• 20.
  Today's menu Interact with your remote machines FABRIC as if they were local ffunction inc.
• 21.
  Today's menu Interact with your remote machines FABRIC as if they were local Takes care of users, group, packages CUISINE and configuration of your new machine ffunction inc.
• 22.
  Today's menu Interact with your remote machines FABRIC as if they were local Takes care of users, group, packages CUISINE and configuration of your new machine Ensures that your servers and services WATCHDOG are up and running ffunction inc.
• 23.
  Today's menu Interact with your remote machines FABRIC as if they were local Takes care of users, group, packages CUISINE Made by Made by and configuration of your new machine Ensures that your servers and services WATCHDOG are up and running ffunction inc.
• 24.
  Part 1 Fabric - http://fabfile.org application deployment & systems administration tasks ffunction inc.
• 25.
  Fabric is aPython library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks. ffunction inc.
• 26.
  Wait... what does Wait... what does that mean ? that mean ? Fabric is a Python library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks. ffunction inc.
• 27.
  Streamlining SSH By hand: version= os.popen(“ssh myserver 'cat /proc/version'”).read() Using Fabric: version = run(“cat /proc/version”) ffunction inc.
• 28.
  Streamlining SSH By hand: version= os.popen(“ssh myserver 'cat /proc/version').read() Using Fabric: from fabric.api import * env.hosts = [“myserver”] version = run(“cat /proc/version”) ffunction inc.
• 29.
  Streamlining SSH By hand: You can specify You can specify multiple hosts and run version = os.popen(“ssh myserver 'cat run multiple hosts and /proc/version').read() the same commands the same commands across them across them Using Fabric: from fabric.api import * env.hosts = [“myserver”] version = run(“cat /proc/version”) ffunction inc.
• 30.
  Streamlining SSH By hand: version= os.popen(“ssh myserver 'cat /proc/version').read() Connections will be Connections will be lazily created and lazily created and pooled pooled Using Fabric: from fabric.api import * env.hosts = [“myserver”] version = run(“cat /proc/version”) ffunction inc.
• 31.
  Streamlining SSH By hand: version= os.popen(“ssh myserver 'cat /proc/version').read() Using Fabric: from fabric.api import * env.hosts = [“myserver”] version = run(“cat /proc/version”) Failures ($STATUS) will Failures ($STATUS) will be handled just like in Make be handled just like in Make ffunction inc.
• 32.
  Example: Installing packages sudo(“aptitudeinstall nginx”) if run("dpkg -s %s | grep 'Status:' ; true" % package).find("installed") == -1: sudo("aptitude install '%s'" % (package) ffunction inc.
• 33.
  Example: Installing packages sudo(“aptitudeinstall nginx”) It's easy to take action It's easy to take action depending on the result depending on the result if run("dpkg -s %s | grep 'Status:' ; true" % package).find("installed") == -1: sudo("aptitude install '%s'" % (package) ffunction inc.
• 34.
  Example: Installing packages Note that we add true Note that we add true sudo(“aptitude install nginx”) so that the run() always so that the run() always succeeds* succeeds* * there are other ways... * there are other ways... if run("dpkg -s %s | grep 'Status:' ; true" % package).find("installed") == -1: sudo("aptitude install '%s'" % (package) ffunction inc.
• 35.
  Example: retrieving systemstatus disk_usage = run(“df -kP”) mem_usage = run(“cat /proc/meminfo”) cpu_usage = run(“cat /proc/stat” print disk_usage, mem_usage, cpu_info ffunction inc.
• 36.
  Example: retrieving systemstatus disk_usage = run(“df -kP”) mem_usage = run(“cat /proc/meminfo”) cpu_usage = run(“cat /proc/stat” print disk_usage, mem_usage, cpu_info Very useful for getting Very useful for getting live information from live information from many different servers many different servers ffunction inc.
• 37.
  Fabfile.py from fabric.api import* from mysetup import * env.host = [“server1.myapp.com”] def setup(): install_packages(“...”) update_configuration() create_users() start_daemons() $ fab setup ffunction inc.
• 38.
  Fabfile.py from fabric.api import* from mysetup import * env.host = [“server1.myapp.com”] def setup(): install_packages(“...”) update_configuration() create_users() start_daemons() Just like Make, you Just like Make, you write rules that do write rules that do something something $ fab setup ffunction inc.
• 39.
  Fabfile.py from fabric.api import* from mysetup import * env.host = [“server1.myapp.com”] def setup(): install_packages(“...”) update_configuration() ...and you can specify create_users() ...and you can specify on which servers the rules start_daemons() on which servers the rules will run will run $ fab setup ffunction inc.
• 40.
  Multiple hosts env.hosts =[ “db1.myapp.com”, “db2.myapp.com”, “db3.myapp.com” ] @hosts(“db1.myapp”) def backup_db(): run(...) ffunction inc.
• 41.
  Roles env.roledefs = { 'web': ['www1', 'www2', 'www3'], 'dns': ['ns1', 'ns2'] } $ fab -R web setup ffunction inc.
• 42.
  Roles env.roledefs = { 'web': ['www1', 'www2', 'www3'], 'dns': ['ns1', 'ns2'] } $ fab -R web setup Will run the setup rule Will run the setup rule only on hosts members only on hosts members of the web role. of the web role. ffunction inc.
• 43.
  Some facts aboutFabric Fabric 1.0 just released! On March, 4th 2011 3 years of development First commit 1161 days ago (on March 10th, 2011) Related Projects Opscode's Chef and Puppet ffunction inc.
• 44.
  What's good aboutFabric? Low-level Basically an ssh() command that returns the result Simple primitives run(), sudo(), get(), put(), local(), prompt(), reboot() No magic No DSL, no abstraction, just a remote command API ffunction inc.
• 45.
  What could beimproved ? Ease common admin tasks User, group creation. Files, directory operations. Abstract primitives Like install package, so that it works with different OS Templates To make creating/updating configuration files easy ffunction inc.
• 46.
  Cuisine: Chef-like functionality forFabric ffunction inc.
• 47.
  Part 2 Cuisine ffunction inc.
• 48.
  What is Opscode'sChef? http://wiki.opscode.com/display/chef/Home Recipes Scripts/packages to install and configure services and applications API A DSL-like Ruby API to interact with the OS (create users, groups, install packages, etc) Architecture Client-server or “solo” mode to push and deploy your new configurations ffunction inc.
• 49.
  What I likedabout Chef Flexible You can use the API or shell commands Structured Helped me have a clear decomposition of the services installed per machine Community Lots of recipes already available from http://cookbooks.opscode.com/ ffunction inc.
• 50.
  What I didn'tlike Too many files and directories Code is spread out, hard to get the big picture Abstraction overload API not very well documented, frequent fall backs to plain shell scripts within the recipe No “smart” recipe Recipes are applied all the time, even when it's not necessary ffunction inc.
• 51.
  The question thatkept coming... sudo aptitude install apache2 python django- python Django recipe: 5 files, 2 directories What it does, in essence ffunction inc.
• 52.
  The question thatkept coming... Is this really necessary Is this really necessary for what I want to do ? sudo aptitude install for what I want to do ? apache2 python django- python Django recipe: 5 files, 2 directories What it does, in essence ffunction inc.
• 53.
  What I lovedabout Fabric Bare metal ssh() function, simple and elegant set of primitives No magic No abstraction, no model, no compilation Two-way communication Easy to change the rule's behaviour according to the output (ex: do not install something that's already installed) ffunction inc.
• 54.
  What I needed Fabric ffunction inc.
• 55.
  What I needed FileI/O File I/O Fabric ffunction inc.
• 56.
  What I needed User/Group User/Group File I/O File I/O Management Management Fabric ffunction inc.
• 57.
  What I needed User/Group User/Group Package Package File I/O File I/O Management Management Management Management Fabric ffunction inc.
• 58.
  What I needed Text processing & Templates Text processing & Templates User/Group User/Group Package Package File I/O File I/O Management Management Management Management Fabric ffunction inc.
• 59.
  How I wantedit Simple “flat” API [object]_[operation] where operation is something in “create”, “read”, “update”, “write”, “remove”, “ensure”, etc... Driven by need Only implement a feature if I have a real need for it No magic Everything is implemented using sh-compatible commands No unnecessary structure Everything fits in one file, no imposed file layout ffunction inc.
• 60.
  Cuisine: Example fabfile.py fromcuisine import * env.host = [“server1.myapp.com”] def setup(): package_ensure(“python”, “apache2”, “python-django”) user_ensure(“admin”, uid=2000) upstart_ensure(“django”) $ fab setup ffunction inc.
• 61.
  Cuisine:Fabric's coreimportedfabfile.py Example functions Fabric's core functions are already are already imported from cuisine import * env.host = [“server1.myapp.com”] def setup(): package_ensure(“python”, “apache2”, “python-django”) user_ensure(“admin”, uid=2000) upstart_ensure(“django”) $ fab setup ffunction inc.
• 62.
  Cuisine: Example fabfile.py fromcuisine import * env.host = [“server1.myapp.com”] def setup(): package_ensure(“python”, “apache2”, “python-django”) user_ensure(“admin”, uid=2000) upstart_ensure(“django”) Cuisine's API $ fab setup Cuisine's API calls calls ffunction inc.
• 63.
  File I/O ffunction inc.
• 64.
  Cuisine : FileI/O ● file_exists does remote file exists? ● file_read reads remote file ● file_write write data to remote file ● file_append appends data to remote file ● file_attribs chmod & chown ● file_remove ffunction inc.
• 65.
  Cuisine : FileI/O Supports owner/group ● file_exists does remote file exists? Supports owner/group and mode change and mode change ● file_read reads remote file ● file_write write data to remote file ● file_append appends data to remote file ● file_attribs chmod & chown ● file_remove ffunction inc.
• 66.
  Cuisine : FileI/O (directories) ● dir_exists does remote file exists? ● dir_ensure ensures that a directory exists ● dir_attribs chmod & chown ● dir_remove ffunction inc.
• 67.
  Cuisine : FileI/O + ● file_update(location, updater=lambda _:_) package_ensure("mongodb-snapshot") def update_configuration( text ): res = [] for line in text.split("n"): if line.strip().startswith("dbpath="): res.append("dbpath=/data/mongodb") elif line.strip().startswith("logpath="): res.append("logpath=/data/logs/mongodb.log") else: res.append(line) return "n".join(res) file_update("/etc/mongodb.conf", update_configuration) ffunction inc.
• 68.
  Cuisine : FileI/O + This replaces the values for This replaces the values for ● file_update(location, updater=lambda _:_) configuration entries configuration entries dbpath and logpath dbpath and logpath package_ensure("mongodb-snapshot") def update_configuration( text ): res = [] for line in text.split("n"): if line.strip().startswith("dbpath="): res.append("dbpath=/data/mongodb") elif line.strip().startswith("logpath="): res.append("logpath=/data/logs/mongodb.log") else: res.append(line) return "n".join(res) file_update("/etc/mongodb.conf", update_configuration) ffunction inc.
• 69.
  Cuisine : FileI/O + ● file_update(location, updater=lambda _:_) package_ensure("mongodb-snapshot") def update_configuration( text ): res = [] The remote file will only be The remote file line in text.split("n"): for will only be changed if the content changed if the content if line.strip().startswith("dbpath="): is different is different res.append("dbpath=/data/mongodb") elif line.strip().startswith("logpath="): res.append("logpath=/data/logs/mongodb.log") else: res.append(line) return "n".join(res) file_update("/etc/mongodb.conf", update_configuration) ffunction inc.
• 70.
  User Management ffunction inc.
• 71.
  Cuisine: User Management ● user_exists does the user exists? ● user_create create the user ● user_ensure create the user if it doesn't exist ffunction inc.
• 72.
  Cuisine: Group Management ● group_exists does the group exists? ● group_create create the group ● group_ensure create the group if it doesn't exist ● group_user_exists does the user belong to the group? ● group_user_add adds the user to the group ● group_user_ensure ffunction inc.
• 73.
  Package Management ffunction inc.
• 74.
  Cuisine: Package Management ● package_exists is the package available ? ● package_installed is it installed ? ● package_install install the package ● package_ensure ... only if it's not installed ● package_upgrade upgrades the/all package(s) ffunction inc.
• 75.
  Text & Templates ffunction inc.
• 76.
  Cuisine: Text transformation text_ensure_line(text,lines) file_update( "/home/user/.profile", lambda _:text_ensure_line(_, "PYTHONPATH=/opt/lib/python:${PYTHONPATH};" "export PYTHONPATH" )) ffunction inc.
• 77.
  Cuisine: Text transformation Ensures that the PYTHONPATH Ensures that the PYTHONPATH variable is set and exported, text_ensure_line(text, lines) variable is set and exported, If not, these lines will be If not, these lines will be appended. appended. file_update( "/home/user/.profile", lambda _:text_ensure_line(_, "PYTHONPATH=/opt/lib/python:${PYTHONPATH};" "export PYTHONPATH" )) ffunction inc.
• 78.
  Cuisine: Text transformation text_replace_line(text,old, new, find=.., process=...) configuration = local_read("server.conf") for key, value in variables.items(): configuration, replaced = text_replace_line( configuration, key + "=", key + "=" + repr(value), process=lambda text:text.split("=")[0].strip() ) ffunction inc.
• 79.
  Cuisine: Text transformation Replaces lines that look like Replaces lines that look like VARIABLE=VALUE text_replace_line(text, old, new, find=.., process=...) VARIABLE=VALUE with the actual values from the with the actual values from the variables dictionary. variables dictionary. configuration = local_read("server.conf") for key, value in variables.items(): configuration, replaced = text_replace_line( configuration, key + "=", key + "=" + repr(value), process=lambda text:text.split("=")[0].strip() ) ffunction inc.
• 80.
  Cuisine: Text transformation text_replace_line(text,old, new, find=..,process lambda transforms The process=...) The process lambda transforms input lines before comparing input lines before comparing them. them. configuration = local_read("server.conf")lines are stripped Here the Here the lines are stripped for key, value in variables.items(): of spaces and of their value. of spaces and of their value. configuration, replaced = text_replace_line( configuration, key + "=", key + "=" + repr(value), process=lambda text:text.split("=")[0].strip() ) ffunction inc.
• 81.
  Cuisine: Text transformation text_strip_margin(text) file_write(".profile",text_strip_margin( """ |export PATH="$HOME/bin":$PATH |set -o vi """ )) ffunction inc.
• 82.
  Cuisine: Text transformation Everything after the | separator Everything after the | separator will be output as content. will be output as content. text_strip_margin(text) It allows to easily embed text It allows to easily embed text templates within functions. templates within functions. file_write(".profile", text_strip_margin( """ |export PATH="$HOME/bin":$PATH |set -o vi """ )) ffunction inc.
• 83.
  Cuisine: Text transformation text_template(text,variables) text_template(text_strip_margin( """ |cd ${DAEMON_PATH} |exec ${DAEMON_EXEC_PATH} """ ), dict( DAEMON_PATH="/opt/mongodb", DAEMON_EXEC_PATH="/opt/mongodb/mongod" )) ffunction inc.
• 84.
  Cuisine: Text transformation This is a simple wrapper text_template(text, variables) This is a simple wrapper around Python (safe) around Python (safe) string.template() function string.template() function text_template(text_strip_margin( """ |cd ${DAEMON_PATH} |exec ${DAEMON_EXEC_PATH} """ ), dict( DAEMON_PATH="/opt/mongodb", DAEMON_EXEC_PATH="/opt/mongodb/mongod" )) ffunction inc.
• 85.
  Cuisine: Goodies ● ssh_keygen generates DSA keys ● ssh_authorize authorizes your key on the remote server ● mode_sudo run() always uses sudo ● upstart_ensure ensures the given daemon is running & more! ffunction inc.
• 86.
  Cuisine Tips: Structuringyour rules BOOTSTRAP ffunction inc.
• 87.
  Cuisine Tips: Structuringyour rules BOOTSTRAP You just received your new You just received your new VPS, and you want to set it VPS, and you want to set it up so that you have a base up so that you have a base system that you can access system that you can access without typing a password without typing a password ffunction inc.
• 88.
  Cuisine Tips: Structuringyour rules BOOTSTRAP SETUP ffunction inc.
• 89.
  Cuisine Tips: Structuringyour rules BOOTSTRAP SETUP You install your users, groups, You install your users, groups, preferred packages and preferred packages and configuration. You also configuration. You also install you applications. install you applications. ffunction inc.
• 90.
  Cuisine Tips: Structuringyour rules BOOTSTRAP SETUP UPDATE ffunction inc.
• 91.
  Cuisine Tips: Structuringyour rules BOOTSTRAP SETUP UPDATE You want to deploy the new You want to deploy the new version of the application version of the application you just built you just built ffunction inc.
• 92.
  Cuisine Tips: Structuringyour rules BOOTSTRAP SETUP UPDATE def bootstrap(): # Secure SSH, create admin user # Authorize SSH public keys # Remove unwanted packages ffunction inc.
• 93.
  Cuisine Tips: Structuringyour rules BOOTSTRAP SETUP UPDATE def setup(): # Create directories (ex: /opt/data, /opt/services, etc) # Create user/groups (ex: apps, services, etc) # Install base tools (ex: screen, fail2ban, zsh, etc) # Edit configuration (ex: profile, inputrc, etc) # Install and run your application ffunction inc.
• 94.
  Cuisine Tips: Structuringyour rules BOOTSTRAP SETUP UPDATE def update(): # Download your application update # Freeze/stop the running application # Install the update # Reload/restart your application # Test that everything is OK ffunction inc.
• 95.
  Why use Cuisine? ● Simple API for remote-server manipulation Files, users, groups, packages ● Shell commands for specific tasks only Avoid problems with your shell commands by only using run() for very specific tasks ● Cuisine tasks are not stupid *_ensure() commands won't do anything if it's not necessary ffunction inc.
• 96.
  Limitations ● Limited to sh-shells Operations will not work under csh ● Only written/tested for Ubuntu Linux Contributors could easily port commands ffunction inc.
• 97.
  Get started ! On Github: http://github.com/sebastien/cuisine 1 short Python file Documented API ffunction inc.
• 98.
  Part 3 Watchdog Server and services monitoring ffunction inc.
• 99.
  The problem ffunction inc.
• 100.
  The problem Low diskspace Low disk space ffunction inc.