How not to release software          Laura Thomson        laura@mozilla.com               @lxt
Scaling Mozilla’s Websites     Laura Thomson   laura@mozilla.com
Writing Maintainable        PHP        Laura Thomson    PHP Quebec Conference       16th March 2007
PHP and MySQL Best Practices          Luke Welling     http://lukewelling.com         Laura Thomson http://www.laurathomso...
Building theaddons.mozilla.org API         Laura Thomson       laura@mozilla.com
Write Beautiful Code         Laura Thomson    laura@laurathomson.com
PHPBest Practices         Laura Thomson    laura@laurathomson.com http://www.laurathomson.com    © 2007 Luke Welling and L...
Rewrite or RefactorWhen to declare technical bankruptcyLaura Thomson (laura@mozilla.com)OSCON - July 22, 2010             ...
FirefoxCrashReportinglaura@ mozilla.com@lxt
Conference speakers and book authors arepeople too. Despite our smug demeanor,we also fail.Frequently.
Conference speakers and book authors arepeople too. Despite our smug demeanor,we also fail.Frequently.
This talk sparked by the release-that-shall-not-be named.It went so badly that the version numberwas retired.After 24 hour...
This talk sparked by the release-that-shall-not-be named.It went so badly that the version numberwas retired.After 24 hour...
This talk sparked by the release-that-shall-not-be named.It went so badly that the version numberwas retired.After 24 hour...
Thus I present this catalogue of failure, acompendium of things not to do when youwant to release any kind of software.
Methodology
Agile fail #1Avoid estimation by working in short sprints- surely we can estimate the next twoweeks, right?Right?
Agile fail #1Cynical basis of agile: estimation is intractableAvoid estimation by working in short sprints- surely we can ...
Agile fail #1Cynical basis of agile: estimation is intractableAvoid estimation by working in short sprints- surely we can ...
Agile fail #1Cynical basis of agile: estimation is intractableAvoid estimation by working in short sprints- surely we can ...
Sprint fail #1: rolloverWe will build these five thingies/bugs/storiesin the next two week sprintTwo are more complex than ...
Sprint fail #1: rolloverWe will build these five thingies/bugs/storiesin the next two week sprintTwo are more complex than ...
Sprint fail #1: rolloverWe will build these five thingies/bugs/storiesin the next two week sprintTwo are more complex than ...
Sprint fail #1: rolloverWe will build these five thingies/bugs/storiesin the next two week sprintTwo are more complex than ...
Sprint fail #1: rolloverWe will build these five thingies/bugs/storiesin the next two week sprintTwo are more complex than ...
Sprint fail #2:         Mara-sprintWe can’t get these five things done. Shouldwe ship with only two?If we just push the dea...
Sprint fail #2:         Mara-sprintWe can’t get these five things done. Shouldwe ship with only two?If we just push the dea...
Sprint fail #2:         Mara-sprintWe can’t get these five things done. Shouldwe ship with only two?If we just push the dea...
Sprint fail #2:         Mara-sprintWe can’t get these five things done. Shouldwe ship with only two?If we just push the dea...
Sprint fail #3:        Death sprintsWe HAVE to get these five things doneThe deadline is an immovable objectWe’ll just work...
Sprint fail #3:        Death sprintsWe HAVE to get these five things doneThe deadline is an immovable objectWe’ll just work...
Sprint fail #3:        Death sprintsWe HAVE to get these five things doneThe deadline is an immovable objectWe’ll just work...
Sprint fail #3:        Death sprintsWe HAVE to get these five things doneThe deadline is an immovable objectWe’ll just work...
Things I’ve learnedPlan the dimension of flex and choose:     Less functionality in the same time     Flexible timelines or...
Coding
Coding fail #1:Build a new bikeshedSitting down to code when you don’t knowwhat you’re doing                    vsAnalysis...
Coding fail #1:Build a new bikeshedSitting down to code when you don’t knowwhat you’re doing                      vsAnalys...
Coding fail #2:      One True RingThere’s one right way to solve this problem
Coding fail #2:      One True RingThere’s one right way to solve this problemComplex
Coding fail #2:      One True RingThere’s one right way to solve this problemComplexGoing to take a long time
Coding fail #2:      One True RingThere’s one right way to solve this problemComplexGoing to take a long timeScope keeps e...
Coding fail #3:    Over-engineeringClosely related to the One True RingBeautiful class hierarchy and architectureLovely di...
Coding fail #3:    Over-engineeringClosely related to the One True RingBeautiful class hierarchy and architectureLovely di...
Coding fail #3:    Over-engineeringClosely related to the One True RingBeautiful class hierarchy and architectureLovely di...
Coding fail #3:    Over-engineeringClosely related to the One True RingBeautiful class hierarchy and architectureLovely di...
Things I’ve learnedDifferent people work in different ways.Some people deal better with uncertainty.Those people should bu...
Testing
Test fail #1:The Null HypothesisHave no tests.
Test fail #2:     Epic confidenceHave some tests, but no idea aboutcoverage.Ensures you have a great level of over-confidenc...
Test fail #2:     Epic confidenceHave some tests, but no idea aboutcoverage.Ensures you have a great level of over-confidenc...
Test fail #3:UR DOING IT RONGTest the wrong things.99% unit test coverage, no load tests.Push something out and don’t real...
Test fail #3:UR DOING IT RONGTest the wrong things.99% unit test coverage, no load tests.Push something out and don’t real...
Test fail #3:UR DOING IT RONGTest the wrong things.99% unit test coverage, no load tests.Push something out and don’t real...
Things I’ve learnedKnow your coverage and your limitationsUse CIUse browser testingUse load and smoke testingGet QA’s opin...
Release anddeployment
Deployment fail #1:   Version chaosPush master/trunk/head.Have no other branches.Have no tags.Have things that aren’t unde...
Deployment fail #1:   Version chaosPush master/trunk/head.Have no other branches.Have no tags.Have things that aren’t unde...
Deployment fail #1:   Version chaosPush master/trunk/head.Have no other branches.Have no tags.Have things that aren’t unde...
Deployment fail #1:   Version chaosPush master/trunk/head/.Have no other branches.Have no tags.Have things that aren’t und...
Deployment fail #2:Fail forward, fail backHave no rollback plan.Have no ability to fail forward, either.If failing forward...
Deployment fail #2:Fail forward, fail backHave no rollback plan.Have no ability to fail forward, either.If failing forward...
Deployment fail #2:Fail forward, fail backHave no rollback plan.Have no ability to fail forward, either.If failing forward...
Deployment fail #3:   Manual pushesRelease by manually logging into eachmachine and updating a checkout.This is even bette...
Deployment fail #3:   Manual pushesRelease by manually logging into eachmachine and updating a checkout.This is even bette...
Deployment fail #3:   Manual pushesRelease by manually logging into eachmachine and updating a checkout.This is even bette...
Deployment fail #4:ALTER TABLE ADD FAILRequire manual changes to databases. Haveno record of the changes in version contro...
Deployment fail #4:ALTER TABLE ADD FAILRequire manual changes to databases. Haveno record of the changes in version contro...
Deployment fail #5:  Config of DoomRequire manual changes to configuration,especially if there are more than twomachines inv...
Deployment fail #6:Single Person Of FailHave only one person that knows how todeploy your stuff.(Don’t document any of it.)
Deployment fail #6:Single Person Of FailHave only one person that knows how todeploy your stuff.(Don’t document any of it.)
Deployment fail #7: It Worked In StagingThis is the upscale version of “it worked onmy laptop”
Deployment fail #7: It Worked In StagingThis is the upscale version of “it worked onmy laptop”Staging not the same as prod:
Deployment fail #7: It Worked In StagingThis is the upscale version of “it worked onmy laptop”Staging not the same as prod...
Deployment fail #7: It Worked In StagingThis is the upscale version of “it worked onmy laptop”Staging not the same as prod...
Things I’ve learnedHave a release management systemBuild the capability to deploy continuouslyAutomate everything: build, ...
Operations
Ops fail #1:         Devops FailOps and Dev don’t speak to, work with, ortrust each otherMost communication takes place th...
Ops fail #1:         Devops FailOps and Dev don’t speak to, work with, ortrust each otherMost communication takes place th...
Ops fail #2:      Dilettante devsStuff breaks at 2amDevs have not documented anything andaren’t available to fix itOps gets...
Ops fail #2:      Dilettante devsStuff breaks at 2amDevs have not documented anything andaren’t available to fix itOps gets...
Ops fail #2:      Dilettante devsStuff breaks at 2amDevs have not documented anything andaren’t available to fix itOps gets...
Ops fail #3: Telepathic troubleshootingStuff breaksDevs have to troubleshoot via remote:Did you look at this log, what did...
Ops fail #3: Telepathic troubleshootingStuff breaksDevs have to troubleshoot via remote:Did you look at this log, what did...
Things I’ve learnedEverybody who works on a system isresponsible for itCross train your teamsOpen up systems to devs for r...
Questions?laura@mozilla.com
How Not To Release Software (Open Source Bridge 2012)
How Not To Release Software (Open Source Bridge 2012)
How Not To Release Software (Open Source Bridge 2012)
How Not To Release Software (Open Source Bridge 2012)
Upcoming SlideShare
Loading in...5
×

How Not To Release Software (Open Source Bridge 2012)

933

Published on

Published in: Technology, News & Politics
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
933
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
25
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • How Not To Release Software (Open Source Bridge 2012)

    1. 1. How not to release software Laura Thomson laura@mozilla.com @lxt
    2. 2. Scaling Mozilla’s Websites Laura Thomson laura@mozilla.com
    3. 3. Writing Maintainable PHP Laura Thomson PHP Quebec Conference 16th March 2007
    4. 4. PHP and MySQL Best Practices Luke Welling http://lukewelling.com Laura Thomson http://www.laurathomson.com OSCON 2007 © Luke Welling and Laura Thomson 8
    5. 5. Building theaddons.mozilla.org API Laura Thomson laura@mozilla.com
    6. 6. Write Beautiful Code Laura Thomson laura@laurathomson.com
    7. 7. PHPBest Practices Laura Thomson laura@laurathomson.com http://www.laurathomson.com © 2007 Luke Welling and Laura Thomson 11
    8. 8. Rewrite or RefactorWhen to declare technical bankruptcyLaura Thomson (laura@mozilla.com)OSCON - July 22, 2010 12
    9. 9. FirefoxCrashReportinglaura@ mozilla.com@lxt
    10. 10. Conference speakers and book authors arepeople too. Despite our smug demeanor,we also fail.Frequently.
    11. 11. Conference speakers and book authors arepeople too. Despite our smug demeanor,we also fail.Frequently.
    12. 12. This talk sparked by the release-that-shall-not-be named.It went so badly that the version numberwas retired.After 24 hours, people started bringing incasseroles.
    13. 13. This talk sparked by the release-that-shall-not-be named.It went so badly that the version numberwas retired.After 24 hours, people started bringing incasseroles.
    14. 14. This talk sparked by the release-that-shall-not-be named.It went so badly that the version numberwas retired.After 24 hours, people started bringing incasseroles.
    15. 15. Thus I present this catalogue of failure, acompendium of things not to do when youwant to release any kind of software.
    16. 16. Methodology
    17. 17. Agile fail #1Avoid estimation by working in short sprints- surely we can estimate the next twoweeks, right?Right?
    18. 18. Agile fail #1Cynical basis of agile: estimation is intractableAvoid estimation by working in short sprints- surely we can estimate the next twoweeks, right?Right?
    19. 19. Agile fail #1Cynical basis of agile: estimation is intractableAvoid estimation by working in short sprints- surely we can estimate the next twoweeks, right?Right?
    20. 20. Agile fail #1Cynical basis of agile: estimation is intractableAvoid estimation by working in short sprints- surely we can estimate the next twoweeks, right?Right?
    21. 21. Sprint fail #1: rolloverWe will build these five thingies/bugs/storiesin the next two week sprintTwo are more complex than anticipatedOne is blockedShip two, roll the other three overRinse, repeat
    22. 22. Sprint fail #1: rolloverWe will build these five thingies/bugs/storiesin the next two week sprintTwo are more complex than anticipatedOne is blockedShip two, roll the other three overRinse, repeat
    23. 23. Sprint fail #1: rolloverWe will build these five thingies/bugs/storiesin the next two week sprintTwo are more complex than anticipatedOne is blockedShip two, roll the other three overRinse, repeat
    24. 24. Sprint fail #1: rolloverWe will build these five thingies/bugs/storiesin the next two week sprintTwo are more complex than anticipatedOne is blockedShip two, roll the other three overRinse, repeat
    25. 25. Sprint fail #1: rolloverWe will build these five thingies/bugs/storiesin the next two week sprintTwo are more complex than anticipatedOne is blockedShip two, roll the other three overRinse, repeat
    26. 26. Sprint fail #2: Mara-sprintWe can’t get these five things done. Shouldwe ship with only two?If we just push the deadline out a week,everything will be fine.And then another.And then another.
    27. 27. Sprint fail #2: Mara-sprintWe can’t get these five things done. Shouldwe ship with only two?If we just push the deadline out a week,everything will be fine.And then another.And then another.
    28. 28. Sprint fail #2: Mara-sprintWe can’t get these five things done. Shouldwe ship with only two?If we just push the deadline out a week,everything will be fine.And then another.And then another.
    29. 29. Sprint fail #2: Mara-sprintWe can’t get these five things done. Shouldwe ship with only two?If we just push the deadline out a week,everything will be fine.And then another.And then another.
    30. 30. Sprint fail #3: Death sprintsWe HAVE to get these five things doneThe deadline is an immovable objectWe’ll just work really really late or all nightevery night for the next two weeksAnd then the same thing again....andagain...and again
    31. 31. Sprint fail #3: Death sprintsWe HAVE to get these five things doneThe deadline is an immovable objectWe’ll just work really really late or all nightevery night for the next two weeksAnd then the same thing again....andagain...and again
    32. 32. Sprint fail #3: Death sprintsWe HAVE to get these five things doneThe deadline is an immovable objectWe’ll just work really really late or all nightevery night for the next two weeksAnd then the same thing again....andagain...and again
    33. 33. Sprint fail #3: Death sprintsWe HAVE to get these five things doneThe deadline is an immovable objectWe’ll just work really really late or all nightevery night for the next two weeksAnd then the same thing again....andagain...and again
    34. 34. Things I’ve learnedPlan the dimension of flex and choose: Less functionality in the same time Flexible timelines or slower velocity Continuous deployment Train model
    35. 35. Coding
    36. 36. Coding fail #1:Build a new bikeshedSitting down to code when you don’t knowwhat you’re doing vsAnalysis paralysis: getting stuck talkingthrough a complex problem, over and over
    37. 37. Coding fail #1:Build a new bikeshedSitting down to code when you don’t knowwhat you’re doing vsAnalysis paralysis/bikeshedding: getting stucktalking through a complex problem, over andover
    38. 38. Coding fail #2: One True RingThere’s one right way to solve this problem
    39. 39. Coding fail #2: One True RingThere’s one right way to solve this problemComplex
    40. 40. Coding fail #2: One True RingThere’s one right way to solve this problemComplexGoing to take a long time
    41. 41. Coding fail #2: One True RingThere’s one right way to solve this problemComplexGoing to take a long timeScope keeps expanding
    42. 42. Coding fail #3: Over-engineeringClosely related to the One True RingBeautiful class hierarchy and architectureLovely diagramsIt doesn’t do anything
    43. 43. Coding fail #3: Over-engineeringClosely related to the One True RingBeautiful class hierarchy and architectureLovely diagramsIt doesn’t do anything
    44. 44. Coding fail #3: Over-engineeringClosely related to the One True RingBeautiful class hierarchy and architectureLovely diagramsIt doesn’t do anything
    45. 45. Coding fail #3: Over-engineeringClosely related to the One True RingBeautiful class hierarchy and architectureLovely diagramsIt doesn’t do anything
    46. 46. Things I’ve learnedDifferent people work in different ways.Some people deal better with uncertainty.Those people should build a prototype.Do The Simplest Thing That Could PossiblyWorkBe happy with 60-80% solutions. Iterate.Even if you’re not a startup, Minimum ViableProduct is valid.
    47. 47. Testing
    48. 48. Test fail #1:The Null HypothesisHave no tests.
    49. 49. Test fail #2: Epic confidenceHave some tests, but no idea aboutcoverage.Ensures you have a great level of over-confidence going into a release.
    50. 50. Test fail #2: Epic confidenceHave some tests, but no idea aboutcoverage.Ensures you have a great level of over-confidence going into a release.
    51. 51. Test fail #3:UR DOING IT RONGTest the wrong things.99% unit test coverage, no load tests.Push something out and don’t realize that it’stoo slow, causes data or user loss, until toolate.
    52. 52. Test fail #3:UR DOING IT RONGTest the wrong things.99% unit test coverage, no load tests.Push something out and don’t realize that it’stoo slow, causes data or user loss, until toolate.
    53. 53. Test fail #3:UR DOING IT RONGTest the wrong things.99% unit test coverage, no load tests.Push something out and don’t realize that it’stoo slow, causes data or user loss, until toolate.
    54. 54. Things I’ve learnedKnow your coverage and your limitationsUse CIUse browser testingUse load and smoke testingGet QA’s opinion on whether you’re readyto ship something
    55. 55. Release anddeployment
    56. 56. Deployment fail #1: Version chaosPush master/trunk/head.Have no other branches.Have no tags.Have things that aren’t under versioncontrol.
    57. 57. Deployment fail #1: Version chaosPush master/trunk/head.Have no other branches.Have no tags.Have things that aren’t under versioncontrol.
    58. 58. Deployment fail #1: Version chaosPush master/trunk/head.Have no other branches.Have no tags.Have things that aren’t under versioncontrol.
    59. 59. Deployment fail #1: Version chaosPush master/trunk/head/.Have no other branches.Have no tags.Have things that aren’t under versioncontrol.
    60. 60. Deployment fail #2:Fail forward, fail backHave no rollback plan.Have no ability to fail forward, either.If failing forward fails, have no backup plan.
    61. 61. Deployment fail #2:Fail forward, fail backHave no rollback plan.Have no ability to fail forward, either.If failing forward fails, have no backup plan.
    62. 62. Deployment fail #2:Fail forward, fail backHave no rollback plan.Have no ability to fail forward, either.If failing forward fails, have no backup plan.
    63. 63. Deployment fail #3: Manual pushesRelease by manually logging into eachmachine and updating a checkout.This is even better under high load.Scaling is something best not talked about.
    64. 64. Deployment fail #3: Manual pushesRelease by manually logging into eachmachine and updating a checkout.This is even better under high load.Scaling is something best not talked about.
    65. 65. Deployment fail #3: Manual pushesRelease by manually logging into eachmachine and updating a checkout.This is even better under high load.Scaling is something best not talked about.
    66. 66. Deployment fail #4:ALTER TABLE ADD FAILRequire manual changes to databases. Haveno record of the changes in version control.Don’t estimate how long these changes willtake, how much they will degradeperformance, or how much downtime will berequired before kicking them off.
    67. 67. Deployment fail #4:ALTER TABLE ADD FAILRequire manual changes to databases. Haveno record of the changes in version control.Don’t estimate how long these changes willtake, how much they will degradeperformance, or how much downtime will berequired before kicking them off.
    68. 68. Deployment fail #5: Config of DoomRequire manual changes to configuration,especially if there are more than twomachines involved - 30 or more is especiallygood.
    69. 69. Deployment fail #6:Single Person Of FailHave only one person that knows how todeploy your stuff.(Don’t document any of it.)
    70. 70. Deployment fail #6:Single Person Of FailHave only one person that knows how todeploy your stuff.(Don’t document any of it.)
    71. 71. Deployment fail #7: It Worked In StagingThis is the upscale version of “it worked onmy laptop”
    72. 72. Deployment fail #7: It Worked In StagingThis is the upscale version of “it worked onmy laptop”Staging not the same as prod:
    73. 73. Deployment fail #7: It Worked In StagingThis is the upscale version of “it worked onmy laptop”Staging not the same as prod:Different OS, packages, hardware
    74. 74. Deployment fail #7: It Worked In StagingThis is the upscale version of “it worked onmy laptop”Staging not the same as prod:Different OS, packages, hardwareOnly one of something where there are >1in prod (classic examples: db replication,memcache)
    75. 75. Things I’ve learnedHave a release management systemBuild the capability to deploy continuouslyAutomate everything: build, test, deployUse configuration managementDocument everythingLoad test when neededEliminate SPOFs, including peopleStaging should reflect production
    76. 76. Operations
    77. 77. Ops fail #1: Devops FailOps and Dev don’t speak to, work with, ortrust each otherMost communication takes place throughaggressive ticket filing and email
    78. 78. Ops fail #1: Devops FailOps and Dev don’t speak to, work with, ortrust each otherMost communication takes place throughaggressive ticket filing and email
    79. 79. Ops fail #2: Dilettante devsStuff breaks at 2amDevs have not documented anything andaren’t available to fix itOps gets the blame for extended downtime
    80. 80. Ops fail #2: Dilettante devsStuff breaks at 2amDevs have not documented anything andaren’t available to fix itOps gets the blame for extended downtime
    81. 81. Ops fail #2: Dilettante devsStuff breaks at 2amDevs have not documented anything andaren’t available to fix itOps gets the blame for extended downtime
    82. 82. Ops fail #3: Telepathic troubleshootingStuff breaksDevs have to troubleshoot via remote:Did you look at this log, what did it say?What is this config value set to?
    83. 83. Ops fail #3: Telepathic troubleshootingStuff breaksDevs have to troubleshoot via remote:Did you look at this log, what did it say?What is this config value set to?
    84. 84. Things I’ve learnedEverybody who works on a system isresponsible for itCross train your teamsOpen up systems to devs for read access toconfig + loggingTroubleshoot together
    85. 85. Questions?laura@mozilla.com
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×