Ignite@DevOpsDays - Why devs need ops

Michael Brunton-Spall
Michael Brunton-SpallSoftware Developer at Guardian.co.uk
Why Devs needs
                          Ops
                            Or MP’s, Crowds and failure




Saturday, 15 October 2011
Copyright © Steve Bell 2009




                             MP’s Expenses
Saturday, 15 October 2011
Saturday, 15 October 2011
Simon Willison
             “You can now build working
             software in less time than it
             takes to have the meeting to
                      describe it”
            Photo courtesy of Tom Coates




Saturday, 15 October 2011
Prototyping
                    • 1 Meeting = 6 people * 1 hour = 6 hours
                    • 1 prototype = 1 dev * 6 hours = 6 hours!
                    • New Software Development Stacks for
                            Rapid development
                            • Django, Rails, Node.js, Lift, Moustache,
                              Play, .NET MVC Scalatra...



Saturday, 15 October 2011
Example
                    • MP’s Expenses
                     • 1 Developer/ 1 Week
                     • 1 Designer / 2 Days
                     • 1 SysAdmin / 1 Day
                     • EC2 - £50
                     • Live 10 days after work started
Saturday, 15 October 2011
Not always roses

                    • Prototype code does not scale
                    • Prototyping devs don’t think of everything
                    • ...(or even anything!)


Saturday, 15 October 2011
Crash #1
                    • Default Configs
                     • Our default
                       • 50 apache children
                       • 30 MySQL connections
                     • This isn’t going to work well!

Saturday, 15 October 2011
Saturday, 15 October 2011
Saturday, 15 October 2011
ORM’s suck mkay
                    • Pages.object.filter
                            (votes__isnull=True).distinct.count
                    • SELECT COUNT(DISTINCT ...) FROM
                            ‘expenses’ LEFT OUTER JOIN ‘vote’ ON
                            (...)
                            WHERE ‘vote.id’ IS NULL



Saturday, 15 October 2011
Crash #2
                    • The code was bad, the SQL was really bad
                    • but it takes a sysadmin to point out...
                    • DB using 135% of CPU
                    • DB and Apaches on the same box = bad
                            idea



Saturday, 15 October 2011
How to migrate a DB
                    • There’s a lot of tools
                    • It could be quite an effort
                    • It’s really broken now
                    • SysAdmins are really evil (especially
                            @pnasrat)



Saturday, 15 October 2011
How to migrate a DB

                    • ssh mps-live “mysqldump mp_expenses” |
                            sed ‘s/ENGINE=MyISAM/ENGINE=InnoDB/
                            g’ | sed ‘s/CHARSET=latin1/
                            CHARSET=utf8/g’ | ssh mysql-big “mysql -u
                            root mp_expenses”




Saturday, 15 October 2011
Next time

                    • Things our sysadmin would have suggested
                            if he’d heard about the project prior to the
                            day of release




Saturday, 15 October 2011
httpperf /
                            apachebench / curl
                                +forloop


Saturday, 15 October 2011
Squid/Varnish



Saturday, 15 October 2011
So why didn’t we?

                    • “SysAdmins slow us down”
                    • “SysAdmins ask difficult questions”
                    • “SysAdmins want to puppet everything”


Saturday, 15 October 2011
What do we know now

                    • “SysAdmins ask the right questions”
                    • “SysAdmins want repeatability”
                    • “SysAdmins know some gnarly stuff”
                    • “SysAdmins can make us go faster”

Saturday, 15 October 2011
DevOps
                    • Sometimes what you need is not a
                            “SysAdmin”
                            • Uptime is important
                    • What you need is “Crash Mat Arranger”
                     • Recovering from failure fast is important

Saturday, 15 October 2011
1 of 20

Recommended

Pantheon @ Drupal Down Under by
Pantheon @ Drupal Down UnderPantheon @ Drupal Down Under
Pantheon @ Drupal Down UnderChapter Three
1.5K views25 slides
MySQL Sandbox - A toolkit for productive laziness by
MySQL Sandbox - A toolkit for productive lazinessMySQL Sandbox - A toolkit for productive laziness
MySQL Sandbox - A toolkit for productive lazinessGiuseppe Maxia
1.4K views74 slides
Core Data in Motion by
Core Data in MotionCore Data in Motion
Core Data in MotionLori Olson
2.7K views75 slides
Vlad Zelinschi - Embrace Native JavaScript (the anti-plugins talk) - Codecamp... by
Vlad Zelinschi - Embrace Native JavaScript (the anti-plugins talk) - Codecamp...Vlad Zelinschi - Embrace Native JavaScript (the anti-plugins talk) - Codecamp...
Vlad Zelinschi - Embrace Native JavaScript (the anti-plugins talk) - Codecamp...Codecamp Romania
469 views33 slides
Devops With Boxfuse and Shippable by
Devops With Boxfuse and ShippableDevops With Boxfuse and Shippable
Devops With Boxfuse and ShippableAndrew Schwabe
2.1K views27 slides
Core Data in RubyMotion #inspect by
Core Data in RubyMotion #inspectCore Data in RubyMotion #inspect
Core Data in RubyMotion #inspectLori Olson
878 views87 slides

More Related Content

What's hot

DownTheRabbitHole.js – How to Stay Sane in an Insane Ecosystem by
DownTheRabbitHole.js – How to Stay Sane in an Insane EcosystemDownTheRabbitHole.js – How to Stay Sane in an Insane Ecosystem
DownTheRabbitHole.js – How to Stay Sane in an Insane EcosystemFITC
673 views29 slides
Python to go by
Python to goPython to go
Python to goWeng Wei
8.5K views50 slides
PuppetCamp SEA 1 - The State of Puppet by
PuppetCamp SEA 1 - The State of PuppetPuppetCamp SEA 1 - The State of Puppet
PuppetCamp SEA 1 - The State of PuppetWalter Heck
479 views44 slides
[Rakuten TechConf2014] [C-2] Big Data for eBooks and eReaders by
[Rakuten TechConf2014] [C-2] Big Data for eBooks and eReaders[Rakuten TechConf2014] [C-2] Big Data for eBooks and eReaders
[Rakuten TechConf2014] [C-2] Big Data for eBooks and eReadersRakuten Group, Inc.
2.2K views36 slides
Ten years later by
Ten years laterTen years later
Ten years laterCristian Vat
61 views39 slides
Godoggo by
GodoggoGodoggo
Godoggomskmoorthy
365 views21 slides

What's hot(20)

DownTheRabbitHole.js – How to Stay Sane in an Insane Ecosystem by FITC
DownTheRabbitHole.js – How to Stay Sane in an Insane EcosystemDownTheRabbitHole.js – How to Stay Sane in an Insane Ecosystem
DownTheRabbitHole.js – How to Stay Sane in an Insane Ecosystem
FITC673 views
Python to go by Weng Wei
Python to goPython to go
Python to go
Weng Wei8.5K views
PuppetCamp SEA 1 - The State of Puppet by Walter Heck
PuppetCamp SEA 1 - The State of PuppetPuppetCamp SEA 1 - The State of Puppet
PuppetCamp SEA 1 - The State of Puppet
Walter Heck479 views
[Rakuten TechConf2014] [C-2] Big Data for eBooks and eReaders by Rakuten Group, Inc.
[Rakuten TechConf2014] [C-2] Big Data for eBooks and eReaders[Rakuten TechConf2014] [C-2] Big Data for eBooks and eReaders
[Rakuten TechConf2014] [C-2] Big Data for eBooks and eReaders
Rakuten Group, Inc.2.2K views
Clojure slides by mcohen01
Clojure slidesClojure slides
Clojure slides
mcohen01434 views
How I Learned To Stop Worrying & Love HTML5 by Dale Cruse
How I Learned To Stop Worrying & Love HTML5How I Learned To Stop Worrying & Love HTML5
How I Learned To Stop Worrying & Love HTML5
Dale Cruse624 views
Through Meteor to the stars - Developing full-stack SPA's with meteor.js by Wekoslav Stefanovski
Through Meteor to the stars - Developing full-stack SPA's with meteor.jsThrough Meteor to the stars - Developing full-stack SPA's with meteor.js
Through Meteor to the stars - Developing full-stack SPA's with meteor.js
Things you should know if you plan to ship a game by DevGAMM Conference
Things you should know if you plan to ship a gameThings you should know if you plan to ship a game
Things you should know if you plan to ship a game
Customizing the custom loop wordcamp 2012-jeff by Jeff Marx
Customizing the custom loop   wordcamp 2012-jeffCustomizing the custom loop   wordcamp 2012-jeff
Customizing the custom loop wordcamp 2012-jeff
Jeff Marx337 views
WordPress Development Environments by Josh Cummings
WordPress Development EnvironmentsWordPress Development Environments
WordPress Development Environments
Josh Cummings476 views
EclipseCon Europe 2011 by Sigasi
EclipseCon Europe 2011EclipseCon Europe 2011
EclipseCon Europe 2011
Sigasi481 views

Similar to Ignite@DevOpsDays - Why devs need ops

Node.js Patterns and Opinions by
Node.js Patterns and OpinionsNode.js Patterns and Opinions
Node.js Patterns and OpinionsIsaacSchlueter
4K views51 slides
Using AWS, Terraform, and Ansible to Automate Splunk at Scale by
Using AWS, Terraform, and Ansible to Automate Splunk at ScaleUsing AWS, Terraform, and Ansible to Automate Splunk at Scale
Using AWS, Terraform, and Ansible to Automate Splunk at ScaleData Works MD
447 views17 slides
Observability will not fix your Broken Monitoring ,Ignite by
Observability will not fix your Broken Monitoring ,IgniteObservability will not fix your Broken Monitoring ,Ignite
Observability will not fix your Broken Monitoring ,IgniteKris Buytaert
167 views20 slides
Scaling a High Traffic Web Application: Our Journey from Java to PHP by
Scaling a High Traffic Web Application: Our Journey from Java to PHPScaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHP120bi
15.2K views70 slides
Scaling High Traffic Web Applications by
Scaling High Traffic Web ApplicationsScaling High Traffic Web Applications
Scaling High Traffic Web ApplicationsAchievers Tech
4.2K views70 slides
Yet Another Dan Kaminsky Talk (Black Ops 2014) by
Yet Another Dan Kaminsky Talk (Black Ops 2014)Yet Another Dan Kaminsky Talk (Black Ops 2014)
Yet Another Dan Kaminsky Talk (Black Ops 2014)Dan Kaminsky
12.7K views80 slides

Similar to Ignite@DevOpsDays - Why devs need ops(20)

Using AWS, Terraform, and Ansible to Automate Splunk at Scale by Data Works MD
Using AWS, Terraform, and Ansible to Automate Splunk at ScaleUsing AWS, Terraform, and Ansible to Automate Splunk at Scale
Using AWS, Terraform, and Ansible to Automate Splunk at Scale
Data Works MD447 views
Observability will not fix your Broken Monitoring ,Ignite by Kris Buytaert
Observability will not fix your Broken Monitoring ,IgniteObservability will not fix your Broken Monitoring ,Ignite
Observability will not fix your Broken Monitoring ,Ignite
Kris Buytaert167 views
Scaling a High Traffic Web Application: Our Journey from Java to PHP by 120bi
Scaling a High Traffic Web Application: Our Journey from Java to PHPScaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHP
120bi15.2K views
Scaling High Traffic Web Applications by Achievers Tech
Scaling High Traffic Web ApplicationsScaling High Traffic Web Applications
Scaling High Traffic Web Applications
Achievers Tech4.2K views
Yet Another Dan Kaminsky Talk (Black Ops 2014) by Dan Kaminsky
Yet Another Dan Kaminsky Talk (Black Ops 2014)Yet Another Dan Kaminsky Talk (Black Ops 2014)
Yet Another Dan Kaminsky Talk (Black Ops 2014)
Dan Kaminsky12.7K views
All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan... by Ernie Souhrada
All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...
All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...
Ernie Souhrada15K views
Dev ops lessons learned - Michael Collins by Devopsdays
Dev ops lessons learned  - Michael CollinsDev ops lessons learned  - Michael Collins
Dev ops lessons learned - Michael Collins
Devopsdays2.9K views
Monitoring is easy, why are we so bad at it presentation by Theo Schlossnagle
Monitoring is easy, why are we so bad at it  presentationMonitoring is easy, why are we so bad at it  presentation
Monitoring is easy, why are we so bad at it presentation
Theo Schlossnagle1.3K views
Php johannesburg meetup - talk 2014 - scaling php in the enterprise by Sarel van der Walt
Php johannesburg   meetup - talk 2014 - scaling php in the enterprisePhp johannesburg   meetup - talk 2014 - scaling php in the enterprise
Php johannesburg meetup - talk 2014 - scaling php in the enterprise
Sarel van der Walt845 views
Serverless Toronto helps Startups by Daniel Zivkovic
Serverless Toronto helps StartupsServerless Toronto helps Startups
Serverless Toronto helps Startups
Daniel Zivkovic598 views
Load testing, Lessons learnt and Loadzen - Martin Buhr at DevTank - 31st Janu... by Loadzen
Load testing, Lessons learnt and Loadzen - Martin Buhr at DevTank - 31st Janu...Load testing, Lessons learnt and Loadzen - Martin Buhr at DevTank - 31st Janu...
Load testing, Lessons learnt and Loadzen - Martin Buhr at DevTank - 31st Janu...
Loadzen1.1K views
Keeping MongoDB Data Safe by Tony Tam
Keeping MongoDB Data SafeKeeping MongoDB Data Safe
Keeping MongoDB Data Safe
Tony Tam6K views
PyData Texas 2015 Keynote by Peter Wang
PyData Texas 2015 KeynotePyData Texas 2015 Keynote
PyData Texas 2015 Keynote
Peter Wang2.6K views
Hadoop Demystified + Automation Smackdown! Austin JUG June 24 2014 by datafundamentals
Hadoop Demystified + Automation Smackdown!  Austin JUG June 24 2014Hadoop Demystified + Automation Smackdown!  Austin JUG June 24 2014
Hadoop Demystified + Automation Smackdown! Austin JUG June 24 2014
datafundamentals942 views
Hacklu2011 tricaud by stricaud
Hacklu2011 tricaudHacklu2011 tricaud
Hacklu2011 tricaud
stricaud558 views
My site is slow by hernanibf
My site is slowMy site is slow
My site is slow
hernanibf2.2K views
What Drove Wordnik Non-Relational? by DATAVERSITY
What Drove Wordnik Non-Relational?What Drove Wordnik Non-Relational?
What Drove Wordnik Non-Relational?
DATAVERSITY510 views

Recently uploaded

Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ... by
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...ShapeBlue
119 views17 slides
DRBD Deep Dive - Philipp Reisner - LINBIT by
DRBD Deep Dive - Philipp Reisner - LINBITDRBD Deep Dive - Philipp Reisner - LINBIT
DRBD Deep Dive - Philipp Reisner - LINBITShapeBlue
180 views21 slides
Business Analyst Series 2023 - Week 4 Session 8 by
Business Analyst Series 2023 -  Week 4 Session 8Business Analyst Series 2023 -  Week 4 Session 8
Business Analyst Series 2023 - Week 4 Session 8DianaGray10
123 views13 slides
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ... by
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...ShapeBlue
184 views12 slides
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R... by
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...ShapeBlue
173 views15 slides
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue by
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlueElevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlueShapeBlue
222 views7 slides

Recently uploaded(20)

Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ... by ShapeBlue
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
ShapeBlue119 views
DRBD Deep Dive - Philipp Reisner - LINBIT by ShapeBlue
DRBD Deep Dive - Philipp Reisner - LINBITDRBD Deep Dive - Philipp Reisner - LINBIT
DRBD Deep Dive - Philipp Reisner - LINBIT
ShapeBlue180 views
Business Analyst Series 2023 - Week 4 Session 8 by DianaGray10
Business Analyst Series 2023 -  Week 4 Session 8Business Analyst Series 2023 -  Week 4 Session 8
Business Analyst Series 2023 - Week 4 Session 8
DianaGray10123 views
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ... by ShapeBlue
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
ShapeBlue184 views
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R... by ShapeBlue
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
ShapeBlue173 views
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue by ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlueElevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
ShapeBlue222 views
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit... by ShapeBlue
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
ShapeBlue159 views
Future of AR - Facebook Presentation by Rob McCarty
Future of AR - Facebook PresentationFuture of AR - Facebook Presentation
Future of AR - Facebook Presentation
Rob McCarty64 views
The Role of Patterns in the Era of Large Language Models by Yunyao Li
The Role of Patterns in the Era of Large Language ModelsThe Role of Patterns in the Era of Large Language Models
The Role of Patterns in the Era of Large Language Models
Yunyao Li85 views
Business Analyst Series 2023 - Week 4 Session 7 by DianaGray10
Business Analyst Series 2023 -  Week 4 Session 7Business Analyst Series 2023 -  Week 4 Session 7
Business Analyst Series 2023 - Week 4 Session 7
DianaGray10139 views
State of the Union - Rohit Yadav - Apache CloudStack by ShapeBlue
State of the Union - Rohit Yadav - Apache CloudStackState of the Union - Rohit Yadav - Apache CloudStack
State of the Union - Rohit Yadav - Apache CloudStack
ShapeBlue297 views
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ... by ShapeBlue
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
ShapeBlue126 views
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f... by TrustArc
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc170 views
"Surviving highload with Node.js", Andrii Shumada by Fwdays
"Surviving highload with Node.js", Andrii Shumada "Surviving highload with Node.js", Andrii Shumada
"Surviving highload with Node.js", Andrii Shumada
Fwdays56 views
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P... by ShapeBlue
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...
ShapeBlue194 views
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N... by James Anderson
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
James Anderson160 views
Digital Personal Data Protection (DPDP) Practical Approach For CISOs by Priyanka Aash
Digital Personal Data Protection (DPDP) Practical Approach For CISOsDigital Personal Data Protection (DPDP) Practical Approach For CISOs
Digital Personal Data Protection (DPDP) Practical Approach For CISOs
Priyanka Aash158 views
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda... by ShapeBlue
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
ShapeBlue161 views
Webinar : Desperately Seeking Transformation - Part 2: Insights from leading... by The Digital Insurer
Webinar : Desperately Seeking Transformation - Part 2:  Insights from leading...Webinar : Desperately Seeking Transformation - Part 2:  Insights from leading...
Webinar : Desperately Seeking Transformation - Part 2: Insights from leading...

Ignite@DevOpsDays - Why devs need ops

  • 1. Why Devs needs Ops Or MP’s, Crowds and failure Saturday, 15 October 2011
  • 2. Copyright © Steve Bell 2009 MP’s Expenses Saturday, 15 October 2011
  • 4. Simon Willison “You can now build working software in less time than it takes to have the meeting to describe it” Photo courtesy of Tom Coates Saturday, 15 October 2011
  • 5. Prototyping • 1 Meeting = 6 people * 1 hour = 6 hours • 1 prototype = 1 dev * 6 hours = 6 hours! • New Software Development Stacks for Rapid development • Django, Rails, Node.js, Lift, Moustache, Play, .NET MVC Scalatra... Saturday, 15 October 2011
  • 6. Example • MP’s Expenses • 1 Developer/ 1 Week • 1 Designer / 2 Days • 1 SysAdmin / 1 Day • EC2 - £50 • Live 10 days after work started Saturday, 15 October 2011
  • 7. Not always roses • Prototype code does not scale • Prototyping devs don’t think of everything • ...(or even anything!) Saturday, 15 October 2011
  • 8. Crash #1 • Default Configs • Our default • 50 apache children • 30 MySQL connections • This isn’t going to work well! Saturday, 15 October 2011
  • 11. ORM’s suck mkay • Pages.object.filter (votes__isnull=True).distinct.count • SELECT COUNT(DISTINCT ...) FROM ‘expenses’ LEFT OUTER JOIN ‘vote’ ON (...) WHERE ‘vote.id’ IS NULL Saturday, 15 October 2011
  • 12. Crash #2 • The code was bad, the SQL was really bad • but it takes a sysadmin to point out... • DB using 135% of CPU • DB and Apaches on the same box = bad idea Saturday, 15 October 2011
  • 13. How to migrate a DB • There’s a lot of tools • It could be quite an effort • It’s really broken now • SysAdmins are really evil (especially @pnasrat) Saturday, 15 October 2011
  • 14. How to migrate a DB • ssh mps-live “mysqldump mp_expenses” | sed ‘s/ENGINE=MyISAM/ENGINE=InnoDB/ g’ | sed ‘s/CHARSET=latin1/ CHARSET=utf8/g’ | ssh mysql-big “mysql -u root mp_expenses” Saturday, 15 October 2011
  • 15. Next time • Things our sysadmin would have suggested if he’d heard about the project prior to the day of release Saturday, 15 October 2011
  • 16. httpperf / apachebench / curl +forloop Saturday, 15 October 2011
  • 18. So why didn’t we? • “SysAdmins slow us down” • “SysAdmins ask difficult questions” • “SysAdmins want to puppet everything” Saturday, 15 October 2011
  • 19. What do we know now • “SysAdmins ask the right questions” • “SysAdmins want repeatability” • “SysAdmins know some gnarly stuff” • “SysAdmins can make us go faster” Saturday, 15 October 2011
  • 20. DevOps • Sometimes what you need is not a “SysAdmin” • Uptime is important • What you need is “Crash Mat Arranger” • Recovering from failure fast is important Saturday, 15 October 2011