Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Build Quality In: Stop the Line - Peter Antman


Published on

Peter Antman's (@peterantman) presentation from MeetUI 2013, SoapUI's first user conference, in Stockholm, Sweden. Peter is

Published in: Technology
  • Dating direct: ❤❤❤ ❤❤❤
    Are you sure you want to  Yes  No
    Your message goes here
    Are you sure you want to  Yes  No
    Your message goes here
  • How To Make Every Day Your Perfect Day By Raising Your Vibration ▲▲▲
    Are you sure you want to  Yes  No
    Your message goes here

Build Quality In: Stop the Line - Peter Antman

  1. 1. How to build quality in – a talefrom the trenchesPeter Antman2013-05-23
  2. 2. Who am I?Peter Antman 2Background…§  Developer since 1995§  Linux, Open Source andEnterprise Java§  Leader and manager forsoftware developer since 2000§  Head of Product Development,Polopoly Atex 2007 - 2011§  Media business§  Drive: Help organizations andpeople fully realize theirpotential (doing softwaredevelopment)Peter Antman0760 140,
  3. 3. Crisp is an employee owned companyknown for agile courses with internationallyrenowned teachers and experienced agiledevelopers and coaches.h"p://  Persistent improvements
  4. 4. §  Polopoly – Enterprise WebCMS§  International take of 2008§  Started Agile transformation 2007§  Existed for 16 year§  Large code base§  Large user base§  Thousands of editors, millions of users, billions ofpage viewsDevelopment Manager @ PolopolyPeter Antman 4
  5. 5. Organized around leanprinciples
  6. 6. Cult of quality
  7. 7.,_Type_G_2.jpg The Type-G Toyoda Automatic Loom, the worlds first automaticloom with a non-stop shuttle-change motion, was invented bySakichi Toyoda in 1924. This loom automatically stopped whenit detected a problem such as thread breakage.
  8. 8. Time Boxed releases
  9. 9. No junk on trunk9RELTeamDemoedTeam test branch (convinience)Story branchAllways releaseable!Test everything!
  10. 10. Continous integration
  11. 11. §  1.5 million lines of code§  10 000 tests§  PEAR§  PAF§  Upgrade, and more configurations§  Five database vendors * multiple versions§  Four browsers * multiple versions§  Three web containers * multiple versions§  Two JDK:s * multiple versions§  Two core OS:es * multiple versions§  Two EJB containers * multiple versions
  12. 12. Plattform to validateLinux Windows  Mysql   Oracle   Msql   Postgres   Derby  Sun JDK IBM JDKTomcat JBossWeb WS WebJBoss WebsphereChrome Firefox Safari IE1.5  million  10  000  tests  Multipleconfigurations Multiple versions
  13. 13. That’s a lot of tests
  14. 14. Had to build our own test cloud§  Started with RedHat kickstart on oldmachines under our tables–  Fragile, hard to upgrade, electricity out takes§  Went to static installs on blades–  Lots of sys admin. One machine = one setup§  Tried VMWare–  Clock issues (went backwards)§  Started using Amazon–  To slow to upload our builds§  Static Xen on local blades–  One machine = one setup§  Eucalytus (kvm) on local blades, elasticcloud–  Very buggy (more than 6 month to stabilize)History
  15. 15. 596 570For example when merging a bugfix to 9.10
  16. 16. Number of tests (november2011)
  17. 17. Our version of the automaticloom
  18. 18.
  19. 19. Compile class Test class Compile module Test module Build product Test product Acceptance test product Test on specific plattform Test with specific configuration At what stage is it possibleto continue working even ifit’s broken?Incremental compile Software development
  20. 20. Urban decay
  21. 21. Broken WindowsA sociological theory from 1982
  22. 22. The New York Theorem:Fighting big crime by pickinglitter
  23. 23. Broken builds are broken windows
  24. 24. The line must be stopedmanually
  25. 25. ToolsCulture
  26. 26. §  Everything must be tested§  Examine each failed test, always§  New bugs are,–  either fixed when standing on them–  handled as a ”fix next sprint” (first in line)–  put into drawerStop the line and fix it
  27. 27. §  Bugs discovered by non automated tests arefixed in next sprint§  Philosophy: We strive not to produce softwarewith errors§  Policy: If it had been covered by a test it wouldhave been fixed immediately§  Practically: Old bugs tends to never get fixed(adapt to capacity)FixNextSprint – never let awindows be broken
  28. 28. §  Integrate–  first on ticket branch,–  then team test branch,–  then team branch,–  then branch§  Hide known bugs (KnownBugs)§  Run only when changed (600 000 -> 100 000)§  Nightly test check responsibility (rotate between teams)§  Daily summary mail on test faults§  Tickets for test faults§  Indexed in Solr§  Annotate in JenkinsTools and policies are necesary
  29. 29. Analyzing Test Faults
  30. 30. Clean up GoGreen(adapt to capacity)FixNextSprint(Stop-the-line)
  31. 31. It’s a never ending work
  32. 32. Here’s a song to singI keep a close watch on these tests of mineI keep my Jenkins open all the timeI see a defect coming down the lineBecuse youre mine, I stop the line