A presentation given at Unified Diff in Cardiff in 2013, with the aim of introducing the art & science of systems administration to software developers, based on experiences at the web dev agency.

1. SYSTEMS
ADMINISTRATION FOR
CODERS
Hints & tips to increase reliability & reduce maintenance
time.

2. WHAT IS A SYSTEM?
An assemblage or combination of things or parts forming
a complex or unitary whole.

3. WHAT DOES A SYSTEMS
ADMINISTRATOR DO?
• Introduces new technologies into an environment
• Analyses system logs and identifies potential issues
with a system.
• Plans and performs routine maintenance
• Performs and maintains backups
• Installs and configures new software and hardware

4. WHAT DOES A SYSTEMS
ADMINISTRATOR DO?
• Manages user accounts
• Responsibility for security
• Responsibility for documentation of the system
• Plans systems upgrades and outages to apply
upgrades
• Troubleshooting reported problems
• Deals with, often frustrated, system users... ... etc. etc.

5. A COMPUTER SYSTEM
• Many components working together - software
(application, web server, OS), hardware (disks, RAM,
CPU) & others (networking equipment, switches,
routers, load balancers)
• Provides stability and maintainability that underpins the
entire application.
• Supports your software for its lifetime.
• Can provide parts of your application. Sometimes a
problem has already been solved by some other
software.

6. START AT BEGINNING
• Start sysadmin tasks at the beginning of the project.
• Write tools to aid deployment.
• Write tools to set up environments.
• Iterate over these tools and stabilise for production

7. ENVIRONMENTS
• Dev, QA, Live
• Dev, Test, QA, UAT, Live ~~ Dev, Test, QA, UAT,
Staging, Live
• The nearer they get to live, the closer the should
resemble live.
• Dev environment should at least be the same major
versions, preferably OS version.
• Vagrant is a useful tool for this.

8. SSH
• Probably the most frequently used tool
• Forwarding SSH agent to allow key use remotely (e.g.
git, hopping between servers)
• Tunnels for access to remote resources
• Reverse tunnels for remote access to local resource
• Easy to configure the client

9. SSH-AGENT
• Generate keys >2048 bits (e.g. ssh-keygen -b 4096)
• ssh-add to load default key (~/.ssh/id_rsa)
• ssh-copy-id <server> to copy to remote server
• ssh -A <server> to forward agent back to local
instance.
• Agent runs at login for modern Linux desktop, Mac OS.

10. SSH-TUNNELS
• Local access to remote: ssh -L3307:localhost:3306
<server>
• Remote access to local: ssh -R:3307:localhost:3306
<server>
• SOCKS proxy: ssh -D5050 <server>

11. SSH CLIENT
CONFIGURATION
• Per user configuration: ~/.ssh/config
• Config options can be set per host or via wild card, e.g.
User, ForwardAgent, Hostname & many more
• manpage: ssh_config

12. UNIX/LINUX PRINCIPLES
• Most things in Linux & UNIX are text.
• Each command line tools does one task and does it
well.
• Command line tools process text with relative ease.
• Much of the text is separated into fields - especially
logs, or as key = value pairs.
• There are standard locations for many types of file.

13. BASIC TOOLS
• cat - display text
• grep - find text
• awk - field processing (and more)
• sed - search and replace text
• wc - count
• cut - simple field processing
• head, tail - print first and last lines of text
• sort - sort text

14. LOCATION, LOCATION,
LOCATION
• /etc - configuration
• /usr - read-only user data
• /var - variable length files (caches, logs, temporary files)
• /home - users' home directories
• /opt - optional applications
• /srv - served site specific data
• See the Filesystem Hierarchy Standard. Same across most distros

15. VARIABLE LENGTH FILES
• /var/log - Logs go here
• /var/cache - Cached files
• Watch your permissions
• During normal operation, /usr, /opt should be able to be
mounted read only

16. SOFTWARE DEPLOYMENT
• Use vendor supplied packages whenever possible:
• Reduces risk of misconfigurations
• Easier to seek help
• Usually well tested
• Easier upgrades, timely security fixes
• Building from source will take a fair amount of time, CPU
• Ruby may be an exception. PHP isn't

17. CHOICE OF LINUX
DISTRIBUTION
• Two main camps - Debian and RedHat
• Red Hat Enterprise Linux is rock solid but expensive &
packages tend to be older. CentOS is Enterprise Linux
recompiled from the same source RPMs.
• Debian stable is rock solid but packages tend to be old.
Community/3rd party support only.
• Ubuntu LTS is pretty solid, packages are more recent
than EL. Well supported in the Cloud - AWS,
OpenStack especially.

18. SOURCE OF PACKAGES
• Use as stable, well testing packages as much possible
• Ubuntu main, Debian stable ideally
• For EL distros, EPEL augments core packages well
• For EL, IUS provide recent versions of MySQL, PHP
but is less well tested.
• Avoid one person repos, PPAs if at all possible.

19. BUILDING FROM SOURCE
• Do not build on live servers. Deploy only compiled
code.
• Ideally produce a package.
• Avoid if possible. Increased risk of problems - more
moving parts.

20. DIAGNOSTICS
• Check disk space: df -h 100% full is bad.
• Check logs: /var/log, /var/log/syslog, /var/log/messages
- get to know your logs.
• dmesg for hardware information.
• Check RAM (free -m) and CPU usage with top.
• Install sysstat package early on - sar will gather data.
Also gives you iostat, vmstat, mpstat.

21. SECURITY
• Install denyhosts/fail2ban to help protect SSH.
• Disable SSH in as root, use SSH keys.
• Use host based firewalls, AWS security groups.
• Don’t run your servers as root. Try to split them over
different users with clear paths between them. One
user nginx, one. php-fpm
• Audit trials are useful.

22. BACKUPS
• Databases: Dump the DB, don’t take hot copies of the
DB files,
• Make use of your hosting providers backup services.
• Make sure you can restore. Test regularly.

23. PROCESS
• Repeat manual tasks often
• Try to use the same deployment system across stages
• Get live up early, treat it as UAT and deploy to it
regularly. Avoid 'big bang' deployment
• Use what suits - don't blindly follow trends, assess risks
as suits the type of project.
• Small steps, iterative improvement. Agile, Kanban,
Lean etc.

24. AUTOMATION
• CFEngine, Puppet, Chef can get you quick wins. They
can quickly become hard to manage. Learning curves
are steep.
• Ansible is simple to get going on. Can be hacked at
and still get good results. Data driven. Pretty new, but
growing fast.
• Nothing wrong with shell/Python/Ruby/Perl scripts.
Configuration management tools are not essential.
• Packaging gets you out of a lot of automation tasks.

25. THAT’S A LOT OF STUFF!
• Not touched on DR, monitoring, OS provisioning,
storage, networking...
• Hire a sys-admin :)
• A good sys-admin will work with you...
• ...to let you get on with the job you enjoy.

26. QUESTIONS?
matt@monki.org.uk

27. THANKS!
matt@monki.org.uk