One-Man Ops

One-Man Opswith Puppet & Friends Jos BoumansOperations @ Krux Digital

RIPE NCC

Can I haveanother /8 please? How you know us

Ubuntu Server

10.04 LTS

AWS Integration

Good guys ofData Privacy

Not to be confused with...

Our Trafﬁc• Serving 4000-10000 user & contextual data requests/second• Sub 100 ms response times• Processing ~150 gb of raw data per day• Twitter: Average ~3000 tweets/second

Our Infrastructure• Started small on AWS. Now:• 100 dedicated nodes• +100-200 on demand Map/Reduce nodes• Dozens of local development machines• 20 different types of machines

One-Man Ops team

Sad Panda

Go from here...

... to here

Your Toolkit

Ubuntu 10.04

cloud-initUses AMI user-data to bootstrap puppet on the client https://help.ubuntu.com/community/CloudInit http://www.youtube.com/watch?v=-zL3BdbKyGY

#cloud-conﬁg### Update puppet to 2.6.3apt_sources:- source: "ppa:mathiaz/puppet-backports"apt_update: trueapt_upgrade: truessh-rsa: AAAAB3NzaC.....+ujFHzpuppet: conf: puppetd: server: "puppet.example.com" # certname %i: instanceid, %f: fqdn of the machine certname: "%i.%f" ca_cert: | -----BEGIN CERTIFICATE----- ....

monthly updateshttp://uec-images.ubuntu.com/query/lucid/server/ released.current.txt

you can upgrade the kernel Only AMI that I know that can do thishttp://cloud.ubuntu.com/2011/02/migrating-to-pv- grub-kernels-for-kernel-upgrades/

Updated software for 10.04 Backported builds for Apache, Memcache, Mysql, PHP, etc https://launchpad.net/~ubuntu-server-edgers

I may be biased

<3 Elastic Load BalancerTheyre free and will save you more than oncehttp://aws.amazon.com/elasticloadbalancing/

<3 S3(Simple Storage Service) Great cheap data retention Good poor mans CDN http://aws.amazon.com/s3

Tip: Get ExpanDrive forgreat SSHFS and S3FS Available for Windows and Mac: http://www.expandrive.com/

RDS > Own MySQL Hot Standby - Failover is ~7 minutesRead Replicates - Improve read performance BUT, you cant replicate out of RDS :( http://aws.amazon.com/rds/

Use EBS Root (Elastic Block Storage)You can reboot and stop/start machines and keep state Consider attaching extra EBS for data persistenceTip: Software raid for multiple EBS drives for better IO

</3 Network Partitioning This will happen to you a lotRelying on network connections will decrease availability of your machines

</3 Floating public IPS AWS DHCP server is ﬂaky AWS DNS TTL is 60 secondsLimited amount of ﬁxed public IPs

Sort your DNS AWS offers http://aws.amazon.com/route53/When you go multi data center or have big trafﬁc, seriously consider Dyn: http://dyn.com/dns/

Avoid SinglePoints of Failure Because they WILL fail. Architect for eventually consistent, distributed systems where you can.

Remember him..?

Puppet

Optimize for makingPuppet development EASY Bridge the gap between dev & ops Tip: use a c1.medium at least

Put your Puppet code in VCSI really dont need to explain why, right?

Run multiple Puppet environmentshttp://docs.puppetlabs.com/guides/environment.htmlWe put 1 host of each cluster in puppet environment development, 1 in staging, the rest in production Dont break everything at once :)

Split your Puppet code into modules We use: Forge, Components, Serviceshttp://docs.puppetlabs.com/guides/modules.html

Use seperate init.pp,params.pp & config.ppParams.pp so you can include variables from elsewhere Config.pp lets you specify: kfoo::config { $fqdn } in a service and require: Kfoo::Config[ $fqdn ] in the component http://docs.puppetlabs.com/guides/modules.html

Use a common base classSet up all the plumbing from users, to apt, to ﬁlesystems, to mounts, ntp, sudo, git, monitoring, ssh, and so on. Run it early using run stages

Sample Serviceclass s_webui { include kbase include kapache include kwebui include kredis kwebui { $fqdn: } kapache::vhost { $fqdn: ssl => 443 } kredis::conﬁg { $fqdn: memory => 100M }}

Write tools to makeyou more productiveEnable developers to run their own Puppet master Create new components easily Push changes to production Our code: https://github.com/krux/ops-tools /

Your own Puppet server & manifestspuppet001:puppet-jib$ screen -S jib.puppetmaster bin/run_puppet_master_locally 8180Running: sudo puppet master --no-daemonize --verbose --debug --masterport 8180 --pidﬁle /mnt/tmp/puppetmaster.8180.pid --confdir /data/git/puppet-jib/bin/.......notice: Starting Puppet master version 2.6.3.....

Our Layout$git/ bin/ update_env.pl run_puppet_master_locally.pl new_component.pl env/ development/ forge/ krux-modules/ services/ staging/ ... production/ ...

Use an External Node Classifier Manage your host specific configuration separately from your manifestshttp://docs.puppetlabs.com/guides/external_nodes.htmlOur code: https://github.com/krux/ops-tools /blob/puppet/bin/node_classifier.py

Keep nodeconﬁguration in an editable location We chose S3Git, LDAP, or anything else that works for you.

Sign nodes that have a conﬁguration only Keyed off their certname, run periodically Inspired by:http://ubuntumathiaz.wordpress.com/2010/03/24/using- puppet-in-uecec2-puppet-support-in-ubuntu-images/ Our code: https://github.com/krux/ops-tools /blob/puppet/bin/check_csr.py

Master Puppet.conf[master].......node_terminus = execexternal_nodes = /usr/bin/node_classiﬁer.py --bucket instancesreports = http, store, foreman### different puppet environments: development, staging, production[development]templatedir = $confdir/env/development/templatesmodulepath = $confdir/env/development/krux-modules: $confdir/env/development/forge: $confdir/env/development/services[....]

Sample Conﬁguration{ classes: [s_sandbox::jib], parameters: { zone: us-east-1c, instance_type: c1.medium, instance_id: i-23a3d042, security_group: krux-ops-dev, puppet_environment: development, puppet_master_port: 8180, kredis_save_to_disk: 0 certname: ops-dev003.example.com. 47334fd8-1516-451d-bd5a-8760ab2a36c0,}}

Attend a Puppet Master Training! No, I dont get a kick back :)http://puppetlabs.com/services/training-workshops/

... avoid becoming him

Foreman

Email Reports & Alerts This feature alone is worth installing it. Run it on the same host as your Puppet master for minimal frictionhttp://theforeman.org/projects/foreman/wiki/ Summarized_E-Mail_Reports

Dashboard / Browser

Theoretically: Node Classiﬁerhttp://theforeman.org/projects/foreman/wiki/ External_Nodes We are happy with S3 based solution YMMV though: do look into it!

Theoretically:Initiate Puppetrunhttp://theforeman.org/projects/foreman/wiki/ Puppetrun Couldnt get it to work though :(

Python Boto & s3cmd

$ s3cmd put ﬁle.txt s3://my-bucketGreat for cronjobs, maintenance tasks & ﬁle syncs Consider s3://my-dropbox for your company http://s3tools.org/s3cmd

boto: Full python API access to AWS Boto + AWS + Puppet = Real Infrastructure as Code http://code.google.com/p/boto/

start_instance.py: Launch AWS nodes Manage zone, security group, type ami, puppet class, EBS, hostname Bootstraps the node for puppet, integrates with external node classiﬁerOur code: https://github.com/krux/ops-tools /blob/aws/bin/start_instance.py

$ start_instance.py -t m1.large -z us-east-1a -a 10 -H dev001.example.com -s mycorp-development ami-2ec83147 s_developmentStarting instance of ami ami-2ec83147 - this may take a while......... started i-12345678Attaching 10gb volume to instance i-12345678 - this may take a while..... attached vol-87654321Created these DNS entries: dev001.example.com => ec2-172-131-213-58.compute-1.amazonaws.comWrote conﬁguration to S3 key: s3://instances/dev001.example.com.47334fd8-1516-451d-bd5a-8760ab2a36c0

security_groups.py Manage & Sync Programmatically manage your security groups keep groups in sync across regionsOur code: https://github.com/krux/ops-tools /blob/aws/bin/security_groups.py

Monitoring & Graphing

Free developer account 1 Free node with all features, unlimited nodes with basic features Free: HTTP(S), PING, SSH, DNS, TCPPremium: HTTP JSON(!), Custom plugins, Mysql, Apache mod_status, etc. Get a 2nd free node through referral: https://cloudkick.com/referral/633f0729

Performance Graphs

Puppet classes & conﬁg informationMonitoring & Alerts

Generate your cloudkick.conf from Puppet Use puppet classes, tags, colors as you deﬁne them as cloudkick tagsOur code for doing so: https://gist.github.com/1230044

Cloudkick Gem for parallel-ssh Uses your cloudkick tags to do node selection,which are based straight off your puppet classes & facts https://github.com/cloudkick/cloudkick-gem

Cloudkick pssh$ cloudkick pssh --query node:redis-c* hostname[1] 18:38:23 [SUCCESS] 64.206.11.221redis-c-slave001.example.com[2] 18:38:23 [SUCCESS] 52.13.118.158redis-c-master001.example.com[3] 18:38:24 [SUCCESS] 52.16.34.217redis-c-slave004.example.com[4] 18:38:24 [SUCCESS] 183.71.131.32redis-c-slave002.example.com

Krux Improvements: pscp, listing nodes Get it from our github: https://github.com/krux/cloudkick-gem Fork and contribute!

Cloudkick list$cloudkick list --full --query node:redis-c*# Name IP Type Zoneredis-c-master001 52.13.118.158 m2.4xlarge us-east-1aredis-c-slave001 64.206.11.221 m2.4xlarge us-east-1aredis-c-slave002 183.71.131.32 m2.4xlarge us-east-1bredis-c-slave004 52.16.34.217 m2.4xlarge us-east-1d

Take away:Measure Everything! Further reading: Pagerduty for cell phone/pager/email alerts New Relic for more in depth app monitoringMCollective for more advanced task parallelization

Just one more thing....

Vagrant

VirtualBox + Ubuntu + Puppet = JFDI Use same puppet infrastructure to provision dev machines locallyPut it on a USB stick, be up and running in 30 minutesOur code for doing so: https://gist.github.com/1230221

Thank You!

Slides at: slideshare.net/jiboumans Follow us: @KruxEngineering Were Hiring: kruxdigital.com

http://www.turegano.net	921
http://lanyrd.com	225
http://architects.dzone.com	204
http://www.kruxdigital.com	159
http://www.techgig.com	132
http://www.krux.com	38
http://paper.li	29
http://origin.kruxdigital.com	5
http://krux.bluecoastweb.com	4
http://a0.twimg.com	3
http://translate.googleusercontent.com	3
http://webcache.googleusercontent.com	1
http://twitter.com	1
https://twitter.com	1
http://cloud.dzone.com	1

One-Man Ops

by Jos Boumans on Sep 22, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

15 Embeds 1,727

Statistics

One-Man Ops — Presentation Transcript