Damn Abstraction : Development vs Operational Efficiency

  • 22,424 views
Uploaded on

Abstraction, Abstraction, Abstraction. …

Abstraction, Abstraction, Abstraction.

The presentation I wanted to give at GotoCon Copenhagen but some bugs decided otherwise.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
22,424
On Slideshare
0
From Embeds
0
Number of Embeds
9

Actions

Shares
Downloads
73
Comments
0
Likes
7

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. goto; conference Copenhagen 2011 Patrick DeboisWednesday 11 May 2011
  • 2. http://jedi.be/blog monitoring http://devopsdays.orgWednesday 11 May 2011
  • 3. Project Task We need performance! 1 million writes /5 secondsWednesday 11 May 2011
  • 4. Wednesday 11 May 2011
  • 5. “All problems in computer science can be solved by another level of abstraction”Wednesday 11 May 2011
  • 6. “All problems in computer science can be solved by another level of indirection” Butler Lampson http://zedshaw.com/essays/indirection_is_not_abstraction.htmlWednesday 11 May 2011
  • 7. We love Ruby & RailsWednesday 11 May 2011
  • 8. Performance Testing Apache Jmeter Tsung Benchmark Rely much on OS tuning Limitations (tcp sessions, max fd, sockets,...)Wednesday 11 May 2011
  • 9. Rails Application Active Record Rails Rack Event Machine Thin Unicorn,Rainbows Threads, Fibers, PIL Ruby (threading) Apache, Passenger Ngnix OS (threading) FS, Network tuningWednesday 11 May 2011
  • 10. Scaling Web Server Multi Loadbalancer Apache/Nginx Instance Database PoolWednesday 11 May 2011
  • 11. “All problems in computer science can be solved by another level of abstraction”Wednesday 11 May 2011
  • 12. “... except for the problem of too many layers of indirection” David WheelerWednesday 11 May 2011
  • 13. Magic?Wednesday 11 May 2011
  • 14. But Seriously?? curl http:// npmjs.org/ install.sh | sh https://github.com/jordansissel/fpm/wiki/ ConvertingNodeNPMWednesday 11 May 2011
  • 15. RVM bash < <(curl -s https:// rvm.beginrescueend.com/install/rvm) https://github.com/jordansissel/rvm-packagingWednesday 11 May 2011
  • 16. Brew install curl -LsSf https://github.com/ mxcl/homebrew/tarball/master | sudo /usr/bin/tar xvz -C/usr/ local --strip 1Wednesday 11 May 2011
  • 17. gosh: Security theater for piped shell scripts https://github.com/ sstephenson/goshWednesday 11 May 2011
  • 18. Select Database?Wednesday 11 May 2011
  • 19. Logging https://github.com/indexzero/winstonhttp://log4r.rubyforge.org/http://logging.apache.org/log4j/ https://github.com/cloudera/flume https://github.com/facebook/scribe http://www.loggly.com/Wednesday 11 May 2011
  • 20. Dependency Mgmt Maven Bundler Warbler Fat Jar Cargo Plugin CapistranoWednesday 11 May 2011
  • 21. “All problems in computer science can be solved by another level of abstraction”Wednesday 11 May 2011
  • 22. “But that usually create another problem” David Wheeler http://www.dmst.aueb.gr/dds/pubs/inbook/beautiful_code/html/Spi07g.htmlWednesday 11 May 2011
  • 23. “Every abstraction layer adds (a little) overhead to the CPUWednesday 11 May 2011
  • 24. Dark Side of Abstraction but also to the poor human who has to read that code http://blog.billyang.me/2009/10/dark-side-of-abstraction.htmlWednesday 11 May 2011
  • 25. Improvements VS +++++++ +Wednesday 11 May 2011
  • 26. Runtime AbstractionWednesday 11 May 2011
  • 27. Database AbstractionWednesday 11 May 2011
  • 28. Do compilers, Languages, and frames make dumber programmers? http://stackoverflow.com/questions/428571/do-smarter-compilers-languages-and-frameworks-make-dumber-programmersWednesday 11 May 2011
  • 29. They dont make good programmers dumber, they make them more productive; http://stackoverflow.com/questions/428571/do-smarter-compilers-languages-and-frameworks-make-dumber-programmersWednesday 11 May 2011
  • 30. But they make dumber people be able to program and think they are great programmer http://stackoverflow.com/questions/428571/do-smarter-compilers-languages-and-frameworks-make-dumber-programmersWednesday 11 May 2011
  • 31. Sysadmin LandWednesday 11 May 2011
  • 32. Hardware Abstraction (OS/Drivers)Wednesday 11 May 2011
  • 33. Disk AbstractionWednesday 11 May 2011
  • 34. Filesystem Abstraction ext2 VFAT reiserfs NTFS Iscsi NASWednesday 11 May 2011
  • 35. VM/Storage Abstraction OpenVZ Xen Usermode Linux Qemu Virtualbox LXC Vmware http://libvirt.org/Wednesday 11 May 2011
  • 36. Disk ImagesWednesday 11 May 2011
  • 37. Cloud Abstraction https://github.com/geemus/fog https://github.com/jclouds/jcloudsWednesday 11 May 2011
  • 38. Abstraction FusionWednesday 11 May 2011
  • 39. Config Management AbstractionWednesday 11 May 2011
  • 40. A model is an abstraction ... for a purpose http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htmWednesday 11 May 2011
  • 41. If all you have is a hammer Puppet Cloudpack http://docs.puppetlabs.com/guides/cloud_pack_getting_started.html http://www.amusei.com/images/hammer-in-the-head1.jpg Knife ec2 Bootstrap https://github.com/opscode/knife-ec2Wednesday 11 May 2011
  • 42. Orchestration http://docs.puppetlabs.com/mcollective/ http://rundeck.org/Wednesday 11 May 2011
  • 43. http://www.iinteractive.com/yapc/slides.html Thank you @jtimbermanWednesday 11 May 2011
  • 44. Vagrant http://vagrantup.com/ http://vagrantbox.es/ https://github.com/jedi4ever/veewee http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/ https://github.com/jedi4ever/sahara https://github.com/jedi4ever/mccloudWednesday 11 May 2011
  • 45. Packages Boundaries gem deb npm rpm jar tgz Effing Package ManagementWednesday 11 May 2011
  • 46. Source/Binary wget ... tar -xzvf ... cd ... package install ./configure make make install External Download => Local Repo Compiling => Do it only onceWednesday 11 May 2011
  • 47. IdempotentWednesday 11 May 2011
  • 48. “Abstraction is selective ignorance” Andrew König http://en.wikipedia.org/wiki/Andrew_Koenig_(programmer)Wednesday 11 May 2011
  • 49. IAASWednesday 11 May 2011
  • 50. PAAS Google App EngineWednesday 11 May 2011
  • 51. Cloud StacksWednesday 11 May 2011
  • 52. Cloud API StandardisationWednesday 11 May 2011
  • 53. Result is suboptimization http://www.michaelnygard.com/blog/2004/04/too_much_abstraction.htmlWednesday 11 May 2011
  • 54. Too much abstraction (1) Do the Simplest Thing that could possibly work TDD http://c2.com/cgi/wiki?TooMuchAbstractionWednesday 11 May 2011
  • 55. Too much abstraction (2) Abstraction should be based on Practical Experience Premature Optimization http://c2.com/cgi/wiki?TooMuchAbstractionWednesday 11 May 2011
  • 56. Abstraction is deciding which aspects of a problem to consider and which ones to ignore http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htmWednesday 11 May 2011
  • 57. Sound a lot like Management (without domain knowledge?)Wednesday 11 May 2011
  • 58. Organization AbstractionsWednesday 11 May 2011
  • 59. Something’s Fishy http://weblog.raganwald.com/2007/12/somethings-fishy.htmlWednesday 11 May 2011
  • 60. Devops http://mdzlog.alcor.net/2010/06/08/devops-and-cloud/Wednesday 11 May 2011
  • 61. Devops http://rationalautomation.wordpress.com/2010/02/04/dogs-and-cats-living-together/Wednesday 11 May 2011
  • 62. Devops http://www.rajiv.com/blog/2009/03/17/technology-department/Wednesday 11 May 2011
  • 63. Generalist vs SpecialistWednesday 11 May 2011
  • 64. Consider Specialization First ... And Generalization SecondWednesday 11 May 2011
  • 65. T-shaped People They have a principal skill that describes the vertical leg of the T But they are so empathetic that they can branch out into other skills, such as anthropology, and do them as well. http://www.noop.nl/2010/06/t-shaped-people.htmlWednesday 11 May 2011
  • 66. http://images.icanhascheezburger.com/completestore/2008/9/11/128656414813278195.jpgWednesday 11 May 2011
  • 67. Wednesday 11 May 2011
  • 68. NOOPS = Human Abstraction? http://rlv.zcache.com/i_love_noops_t_shirt-p235186382091096346trlf_400.jpgWednesday 11 May 2011
  • 69. abstraction is selective ignorance http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htmWednesday 11 May 2011
  • 70. Amazon.com’s real problem isn’t the outage, it’s the communication http://blog.rightscale.com/2011/04/25/amazon-ec2-outage-summary-and-lessons-learned/ http://aws.amazon.com/message/65648/Wednesday 11 May 2011
  • 71. The  Chaos  Monkey’s  job  is  to   randomly  kill  instances  and  services   within  our  architecture If  we  aren’t  constantly  tes0ng  our  ability  to   succeed  despite  failure,  then  it  isn’t  likely  to   work  when  it  ma;ers  most    –  in  the  event  of  an   unexpected  outage. http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.htmlWednesday 11 May 2011
  • 72. Chaos Monkey __ w c(..)o ( __(-) __) https://github.com/mattbasta/chaos-chimp / ( https://github.com/simonmunro/ChaosMonkey /(_)___) w /| | m mWednesday 11 May 2011
  • 73. All non-trivial abstractions, to some degree, are leaky http://www.joelonsoftware.com/articles/LeakyAbstractions.htmlWednesday 11 May 2011
  • 74. When you drive a car, thinking about how the engine works is a distraction http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htmWednesday 11 May 2011
  • 75. When you repair the car, thinking about how the engine works is essential. http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htmWednesday 11 May 2011
  • 76. Questions?Wednesday 11 May 2011
  • 77. Programming is a human activity; forget that and all is lost Meta-advice http://www.cs.princeton.edu/courses/archive/spr99/cs333/pdf/21.pdfWednesday 11 May 2011