Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Salting new ground one man ops from scratch


Published on

Devops at Rebellion Games

Published in: Technology
  • Login to see the comments

  • Be the first to like this

Salting new ground one man ops from scratch

  1. 1. Salting New Ground One Man Ops from Scratch
  2. 2. Me @PercussiveFix ➔ SysAdmin since 2012 ➔ IT Engineer since 1998 ➔ Coding since BASIC on the ZX Spectrum ➔ Gaming since Pong ➔ “Senior DevOps” Engineer at Rebellion Developments in Oxford ➔ Formerly with Electronic Arts at Playfish in London
  3. 3. The project - all the firsts Theirs ➔ First development focused Operations Engineer in the company ➔ First real development effort on a top tier social game - originally for and Facebook ➔ First foray into DevOps methodology ➔ First use of AWS services - EC2, ELB, RDS, Elasticache, S3, Cloudfront, Route53 ➔ First use of configuration management Mine ➔ First time without a team ➔ First time building a complete application stack from scratch ➔ First time being the big dog
  4. 4. One man crusade ➔ DevOps methodology ◆ Culture - People and process first. Get the mindset right. ◆ Automate - As much as possible. CI/Infrastructure as code. ◆ Lean - Fast and stable ◆ Metrics - Measure everything. Show the improvements. ◆ Sharing - Open information distribution. Collaborate. ➔ Taking the Ops out of Dev - but in a good way ➔ Evangelising all over the company, not just within the project team. ➔ Fingers in many pies - web development, mobile game support, internal IT operations ➔ Push the agenda - but in a good way ➔ Try and sooth the hesitancy to rely on one guy - build accessible tools and automation
  5. 5. Building a stack from scratch (nearly) ➔ Starting from 2 hand-configured web servers ➔ No infrastructure security ➔ No monitoring ➔ No config management ➔ No DR process ➔ No docs ➔ No application logging ➔ No log collection ➔ No scaling strategy ➔ No out of hours support ➔ No database standardisation ➔ No metrics
  6. 6. And now for the science Building a stack from scratch - Config Management System Requirements Quick to get started + Straightforward setup and maintenance + Easy to modify and manage + Modular and expandable =
  7. 7. SaltStack - Salt ➔ Written in Python ➔ First and foremost - a remote execution system ➔ Master/Minions arrangement - can be multi-master or standalone ➔ Secure, encrypted protocol running over ZeroMQ ◆ public keys for authentication with the master, then faster AES encryption for payload communication ➔ Fast & scalable - 10’s to 1000’s of endpoints ➔ Targeted execution via minion name, glob, regex, grains (tags) , IPs, nodegroups etc salt '*' "uptime" or salt -G 'os:Ubuntu' "ps -ef | grep java" or salt 'live-product-app0[0-9]' grains.items
  8. 8. Inherent grainscpu_flags: fpu de tsc msr pae ... cpu_model: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz cpuarch: x86_64 defaultencoding: None defaultlanguage: None domain: fqdn: fqdn_ip4: 10.XXX.XXX.XXX fqdn_ip6: gpus: host: live-product-app00 id: live-product-app00 ip_interfaces: {'lo': [''], 'eth0': ['10.XXX.XXX.XXX']} ipv4: 10.XXX.XXX.XXX ipv6: ::1 feXX::XXXX:3XXX:XX04:49X1 kernel: Linux kernelrelease: 3.2.0-40-virtual localhost: live-product-app00 lsb_distrib_codename: precise lsb_distrib_description: Ubuntu 12.04.2 LTS lsb_distrib_id: Ubuntu lsb_distrib_release: 12.04 master: mem_total: 7450 nodename: live-product-app00 num_cpus: 2 num_gpus: 0 os: Ubuntu os_family: Debian osarch: amd64 oscodename: precise osfinger: Ubuntu-12.04 osfullname: Ubuntu osrelease: 12.04 path: /usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin :/sbin:/bin ps: ps -efH pythonpath: /usr/bin /usr/lib/python2.7 /usr/lib/python2.7/plat-linux2 /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages /usr/lib/pymodules/python2.7 pythonversion: saltpath: /usr/lib/pymodules/python2.7/salt saltversion: 0.17.1 server_id: 224501001 shell: /bin/sh virtual: xen virtual_subtype: Xen PV DomU
  9. 9. SaltStack config management ➔ Using Salt States (c.f. recipes, manifests, playbooks etc) ➔ YAML formatted ➔ Human readable ➔ Jinja templating for logic and conditionals ➔ Simple hierarchical layout >> ◆ top.sls as master tree ➔ One line command runs every state specified in top.sls on every targeted box: salt '*' state.highstate # nginx/init.sls nginx: pkg: - installed service: - running - watch: - pkg: nginx - file: /etc/nginx/nginx.conf /etc/nginx/nginx.conf: file.managed: - source: salt://nginx/nginx.conf - require: - pkg: nginx # top.sls base: '*': - core - python - snmp 'os:Ubuntu': - match: grain - nginx - php 'id:*log*': - match: grain - logstash - elasticsearch etc
  10. 10. SaltCloud instance provisioning ➔ Supporting multiple providers (at least partially): AWS EC2, Digital Ocean, GoGrid, IBM SCE, JoyEnt, Linode, Rackspace, Softlayer ➔ And platforms: CloudStack, OpenStack, Parallels, Saltify, Salty-Vagrant ➔ Templating for providers: ec2-live: securitygroup: - default - live provider: ec2 location: eu-west-1 minion: master: And instances: ec2-live-app: provider: ec2-live image: ami-ce7b6fba size: m1.large ssh_username: ubuntu ➔ One line command to provision a box: salt-cloud -p ec2-live-app live-product-app00
  11. 11. Additional components ➔ Pillar - Global value store for all minions ➔ Events - Listens for, publishes and sends events internally, to the master or to a 3rd Party ➔ Reactor - Logic engine to allow Events to trigger actions ➔ Syndic - Allows multi-master and other complicated setups hierarchies ➔ Scheduler - execution of any salt command on master or minions ➔ Halite - Experimental Web-UI ➔ Mine - used to collect arbitrary data from minions and store it on the master ➔ Virt - Virtual machine management - networking, images, disks etc ➔ SSH - Experimental - uses SSH rather than ZeroMQ and agent (hence slower) ➔ Kitchen-Salt - Experimental provisioner for Test-Kitchen
  12. 12. Moderately clever other stuff ➔ Automated ◆ Route53 configuration using EC2 tags and boto ◆ Monitoring discovery ◆ Deployment configuration using estate intelligence ◆ Assignment of Product/Service/Environment grains based on AWS name tag ➔ RDS/ELB graphing from Cloudwatch metrics using CWGraph ➔ Beaver/Logstash/Elasticsearch/Kibana log aggregation service all Salty
  13. 13. Salty goodness ➔ Vibrant & responsive community ◆ Google groups, IRC, Github issues, SaltConf, meetups ➔ Easy to get started ➔ Under active development - good response to issues ➔ Docs are sometimes patchy/dated/disorganised ➔ Can be complex to configure - lots of loosely coupled modules ➔ Under active development - can be buggy & badness
  14. 14. Places to start Docs Discussion IRC: freenode #salt This Presentation Tutorials - getting started - step by step with nginx - good examples of remote execution tools and multi distro setup Sample States - states github search @PercussiveFix
  15. 15. Other links Good overview slides: overvisaltstack-presentation-clean infrastructure-management-with-saltstack-seth- house