Damn Abstraction : Development vs Operational Efficiency

goto; conference Copenhagen 2011 Patrick DeboisWednesday 11 May 2011

http://jedi.be/blog monitoring http://devopsdays.orgWednesday 11 May 2011

Project Task We need performance! 1 million writes /5 secondsWednesday 11 May 2011

Wednesday 11 May 2011

“All problems in computer science can be solved by another level of abstraction”Wednesday 11 May 2011

“All problems in computer science can be solved by another level of indirection” Butler Lampson http://zedshaw.com/essays/indirection_is_not_abstraction.htmlWednesday 11 May 2011

We love Ruby & RailsWednesday 11 May 2011

Performance Testing Apache Jmeter Tsung Benchmark Rely much on OS tuning Limitations (tcp sessions, max fd, sockets,...)Wednesday 11 May 2011

Rails Application Active Record Rails Rack Event Machine Thin Unicorn,Rainbows Threads, Fibers, PIL Ruby (threading) Apache, Passenger Ngnix OS (threading) FS, Network tuningWednesday 11 May 2011

Scaling Web Server Multi Loadbalancer Apache/Nginx Instance Database PoolWednesday 11 May 2011

“... except for the problem of too many layers of indirection” David WheelerWednesday 11 May 2011

Magic?Wednesday 11 May 2011

But Seriously?? curl http:// npmjs.org/ install.sh | sh https://github.com/jordansissel/fpm/wiki/ ConvertingNodeNPMWednesday 11 May 2011

RVM bash < <(curl -s https:// rvm.beginrescueend.com/install/rvm) https://github.com/jordansissel/rvm-packagingWednesday 11 May 2011

Brew install curl -LsSf https://github.com/ mxcl/homebrew/tarball/master | sudo /usr/bin/tar xvz -C/usr/ local --strip 1Wednesday 11 May 2011

gosh: Security theater for piped shell scripts https://github.com/ sstephenson/goshWednesday 11 May 2011

Select Database?Wednesday 11 May 2011

Logging https://github.com/indexzero/winstonhttp://log4r.rubyforge.org/http://logging.apache.org/log4j/ https://github.com/cloudera/ﬂume https://github.com/facebook/scribe http://www.loggly.com/Wednesday 11 May 2011

Dependency Mgmt Maven Bundler Warbler Fat Jar Cargo Plugin CapistranoWednesday 11 May 2011

“But that usually create another problem” David Wheeler http://www.dmst.aueb.gr/dds/pubs/inbook/beautiful_code/html/Spi07g.htmlWednesday 11 May 2011

“Every abstraction layer adds (a little) overhead to the CPUWednesday 11 May 2011

Dark Side of Abstraction but also to the poor human who has to read that code http://blog.billyang.me/2009/10/dark-side-of-abstraction.htmlWednesday 11 May 2011

Improvements VS +++++++ +Wednesday 11 May 2011

Runtime AbstractionWednesday 11 May 2011

Database AbstractionWednesday 11 May 2011

Do compilers, Languages, and frames make dumber programmers? http://stackoverﬂow.com/questions/428571/do-smarter-compilers-languages-and-frameworks-make-dumber-programmersWednesday 11 May 2011

They dont make good programmers dumber, they make them more productive; http://stackoverﬂow.com/questions/428571/do-smarter-compilers-languages-and-frameworks-make-dumber-programmersWednesday 11 May 2011

But they make dumber people be able to program and think they are great programmer http://stackoverﬂow.com/questions/428571/do-smarter-compilers-languages-and-frameworks-make-dumber-programmersWednesday 11 May 2011

Sysadmin LandWednesday 11 May 2011

Hardware Abstraction (OS/Drivers)Wednesday 11 May 2011

Disk AbstractionWednesday 11 May 2011

Filesystem Abstraction ext2 VFAT reiserfs NTFS Iscsi NASWednesday 11 May 2011

VM/Storage Abstraction OpenVZ Xen Usermode Linux Qemu Virtualbox LXC Vmware http://libvirt.org/Wednesday 11 May 2011

Disk ImagesWednesday 11 May 2011

Cloud Abstraction https://github.com/geemus/fog https://github.com/jclouds/jcloudsWednesday 11 May 2011

Abstraction FusionWednesday 11 May 2011

Conﬁg Management AbstractionWednesday 11 May 2011

A model is an abstraction ... for a purpose http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htmWednesday 11 May 2011

If all you have is a hammer Puppet Cloudpack http://docs.puppetlabs.com/guides/cloud_pack_getting_started.html http://www.amusei.com/images/hammer-in-the-head1.jpg Knife ec2 Bootstrap https://github.com/opscode/knife-ec2Wednesday 11 May 2011

Orchestration http://docs.puppetlabs.com/mcollective/ http://rundeck.org/Wednesday 11 May 2011

http://www.iinteractive.com/yapc/slides.html Thank you @jtimbermanWednesday 11 May 2011

Vagrant http://vagrantup.com/ http://vagrantbox.es/ https://github.com/jedi4ever/veewee http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/ https://github.com/jedi4ever/sahara https://github.com/jedi4ever/mccloudWednesday 11 May 2011

Packages Boundaries gem deb npm rpm jar tgz Efﬁng Package ManagementWednesday 11 May 2011

Source/Binary wget ... tar -xzvf ... cd ... package install ./conﬁgure make make install External Download => Local Repo Compiling => Do it only onceWednesday 11 May 2011

IdempotentWednesday 11 May 2011

“Abstraction is selective ignorance” Andrew König http://en.wikipedia.org/wiki/Andrew_Koenig_(programmer)Wednesday 11 May 2011

IAASWednesday 11 May 2011

PAAS Google App EngineWednesday 11 May 2011

Cloud StacksWednesday 11 May 2011

Cloud API StandardisationWednesday 11 May 2011

Result is suboptimization http://www.michaelnygard.com/blog/2004/04/too_much_abstraction.htmlWednesday 11 May 2011

Too much abstraction (1) Do the Simplest Thing that could possibly work TDD http://c2.com/cgi/wiki?TooMuchAbstractionWednesday 11 May 2011

Too much abstraction (2) Abstraction should be based on Practical Experience Premature Optimization http://c2.com/cgi/wiki?TooMuchAbstractionWednesday 11 May 2011

Abstraction is deciding which aspects of a problem to consider and which ones to ignore http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htmWednesday 11 May 2011

Sound a lot like Management (without domain knowledge?)Wednesday 11 May 2011

Organization AbstractionsWednesday 11 May 2011

Something’s Fishy http://weblog.raganwald.com/2007/12/somethings-ﬁshy.htmlWednesday 11 May 2011

Devops http://mdzlog.alcor.net/2010/06/08/devops-and-cloud/Wednesday 11 May 2011

Devops http://rationalautomation.wordpress.com/2010/02/04/dogs-and-cats-living-together/Wednesday 11 May 2011

Devops http://www.rajiv.com/blog/2009/03/17/technology-department/Wednesday 11 May 2011

Generalist vs SpecialistWednesday 11 May 2011

Consider Specialization First ... And Generalization SecondWednesday 11 May 2011

T-shaped People They have a principal skill that describes the vertical leg of the T But they are so empathetic that they can branch out into other skills, such as anthropology, and do them as well. http://www.noop.nl/2010/06/t-shaped-people.htmlWednesday 11 May 2011

http://images.icanhascheezburger.com/completestore/2008/9/11/128656414813278195.jpgWednesday 11 May 2011

Wednesday 11 May 2011

NOOPS = Human Abstraction? http://rlv.zcache.com/i_love_noops_t_shirt-p235186382091096346trlf_400.jpgWednesday 11 May 2011

abstraction is selective ignorance http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htmWednesday 11 May 2011

Amazon.com’s real problem isn’t the outage, it’s the communication http://blog.rightscale.com/2011/04/25/amazon-ec2-outage-summary-and-lessons-learned/ http://aws.amazon.com/message/65648/Wednesday 11 May 2011

The Chaos Monkey’s job is to randomly kill instances and services within our architecture If we aren’t constantly tes0ng our ability to succeed despite failure, then it isn’t likely to work when it ma;ers most – in the event of an unexpected outage. http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.htmlWednesday 11 May 2011

Chaos Monkey __ w c(..)o ( __(-) __) https://github.com/mattbasta/chaos-chimp / ( https://github.com/simonmunro/ChaosMonkey /(_)___) w /| | m mWednesday 11 May 2011

All non-trivial abstractions, to some degree, are leaky http://www.joelonsoftware.com/articles/LeakyAbstractions.htmlWednesday 11 May 2011

When you drive a car, thinking about how the engine works is a distraction http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htmWednesday 11 May 2011

When you repair the car, thinking about how the engine works is essential. http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htmWednesday 11 May 2011

Questions?Wednesday 11 May 2011

Programming is a human activity; forget that and all is lost Meta-advice http://www.cs.princeton.edu/courses/archive/spr99/cs333/pdf/21.pdfWednesday 11 May 2011

http://www.jedi.be	1660
http://ruby.dzone.com	1063
http://java.dzone.com	525
http://www.planetdevops.net	350
http://jedi.be	307
http://localhost:4331	62
http://feeds.feedburner.com	52
http://architects.dzone.com	26
http://css.dzone.com	9
http://a0.twimg.com	2
http://lef99.jedi.be	2
http://www.slideshare.net	1
https://twitter.com	1
http://twitter.com	1
http://www.javaoracleblog.com	1
http://webcache.googleusercontent.com	1
http://laz.jedi.be	1
http://www.dzone.com	1
http://paper.li	1

Damn Abstraction : Development vs Operational Efficiency

by Patrick Debois on May 11, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

19 Embeds 4,066

Statistics

Damn Abstraction : Development vs Operational Efficiency — Presentation Transcript