Turbocharge your automated tests with ci

1,572 views

Published on

Presented at STPcon 2011 on leveraging your automated tests by putting them under CI. Covers value proposition of CI, challenges to implement, and ideas for scaling.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,572
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
30
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • Soon to be in a Spanish language newsletter!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Wrap up should have plenty of time for questions\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • the build on the left for “agenda direcciones” produced the email on the right. \n
  • \n
  • Turns out that is what CI is for, discovering the non obvious!\n
  • \n
  • \n
  • \n
  • \n
  • checkability requires control and visibility into your app\n
  • \n
  • war file deployment is a pain\n
  • \n
  • \n
  • \n
  • \n
  • tester who knows that there are no silly integration bugs delaying work!\n
  • not what we are trying to do. \n
  • \n
  • Human error was behind major industrial disasters such as three mile island. NASA lost the 125 million dollar Mars Climate Orbiter because Lockheed used english units while Nasa used metric when working with navigation data. Clearly not enough integration between the teams!\n
  • Talk about Seymour, the build master, and the big weekend build, and you have to come in. And the time that I forgot to check in all my code. \n
  • \n
  • The CI book makes this point: if something is painful, do it often till you solve the problem.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • bangalore India...\n
  • \n
  • \n
  • \n
  • This is a “glanceable” device, or ambient information.\n
  • \n
  • \n
  • \n
  • \n
  • we need soemthing tell us how far away we are getting from mailine\n
  • \n
  • \n
  • \n
  • \n
  • not just a build loop, it’s a people loop!\n\nJames Shore touched on something really important back in his 2006 blog article "Continuous Integration on a Dollar a Day".\n
  • Mention the jesters cap at VA..\n
  • \n
  • Look at the Panopticode Report in dasher\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • tell about CI system running great for 6 months... only cause the smtp server was rejecting all build status email from being sent!\n
  • \n
  • Remember the Ginsu Knife? “How much would you pay for this?” “But wait! There's more”. Sold between 2 and 3 million sets between 78 and 84 and invented the modern infomerical\n
  • \n
  • \n
  • \n
  • This is really about an existing suite of tests that are NOT under CI.\n
  • \n
  • some folks argue this isn’t really an issue if integration is constantly happening in very small chunks. \n
  • \n
  • \n
  • \n
  • \n
  • Of course you need virtualized servers to make this work\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • not just a build loop, it’s a people loop!\n\nJames Shore touched on something really important back in his 2006 blog article "Continuous Integration on a Dollar a Day".\n
  • \n
  • \n
  • \n
  • market for both open source and commercial has exploded.\n
  • aka as “Hudkins”\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Turbocharge your automated tests with ci

    1. 1. TURBOCHARGE YOUR AUTOMATED TESTS WITH CONTINUOUS INTEGRATION STPCON 2011 1
    2. 2. WHY AM I QUALIFIED TO BE UP HERE?• President of OpenSource Connections• Contributor to CruiseControl and Continuum CI projects• Member of Apache Software Foundation• Presenter at conferences (OSCON, ApacheCON, jTDS, ExpoQA, STPcon 2009!)
    3. 3. WRITER
    4. 4. AUTHOR
    5. 5. FATHER
    6. 6. AGILISTA
    7. 7. AGENDA CI in a Flash Value Proposition of CI Challenges in using CIScaling CI for LARGE suites of tests Wrap Up 7
    8. 8. CI IN A FLASH! 8
    9. 9. DEFINITIONa fully automated and reproducible build, including testing, that runs many times a day - Martin Fowler http://martinfowler.com/articles/continuousIntegration.html
    10. 10. MARTIN FOWLER• Coinedthe term Continuous Integration• Chief Scientist for ThoughtWorks• Speaksand Blogs on software: • http://martinfowler.com/
    11. 11. CONCEPTUAL
    12. 12. MAPPINGdiagram credit to Brian Di Croce @ http://blog.briandicroce.com/2008/03/17/setting-up-a-continuous-integration- environment-preface/
    13. 13. FEEDBACK MECHANISM From: cruisecontrol@opensourceconnections.com Subject: [CruiseControl] agenda_direcciones build 370 fixed Date: October 7, 2008 2:22:50 PM EDT To: epugh@opensourceconnections.com The build has been fixed. CHANGES ------- New revision 370 detected Revision 370 committed by epugh on 2008-10-07 18:22:03 mysql esta mejor que sqlite  M /branches/ERIC_EXPO_08_DEMO/agenda_direcciones/config/database.yml See http://cruise.opensourceconnections.com:3333/builds/agenda_direcciones/370 for details.
    14. 14. VALUE PROPOSITION OF CI 14
    15. 15. ACTIVITYTurn to the person next to you, and see how many non-obvious things you have in common.
    16. 16. LIFE WITH OUT CI• Unstable code, integration is difficult• Many build errors reported• Only one person on the team can build the entire project• Getting “test ready” state is hard• Feedback cycle is long Every day is a struggle to be productive
    17. 17. LIFE WITH CI• The build process is simple and repeatable• Elimination of Human Error• Bugs stay fixed!• Feedback cycle is very rapid Every day you know you can produce better software!
    18. 18. photo credit to Jay Grieves @ www.bigswingingdeveloper.com
    19. 19. RISK REDUCTION• What is in the build?• What has changed between builds?• How do I verify functionality?
    20. 20. REDUCE HUMAN ERROR
    21. 21. MY AH-HAH! MOMENT
    22. 22. REDUCE REPETITIVE PROCESSES• DRY: Don’t Repeat Yourself• Script Everything!• “ant deploy”
    23. 23. MAKE PAINFUL THINGS PAINLESS! photo credit to Paul Duvall @ http://www.integratebutton.com/
    24. 24. PRODUCE DEPLOYABLE SOFTWARE• It’s what we are here to do!• Always in a “Demo Ready” state• No last minute administrivia tasks to get the product ready • Installer is built • Documentation is written • Changelog is written
    25. 25. ENABLE PROJECT VISIBILITY
    26. 26. INCREASE PROJECT CONFIDENCE
    27. 27. photo credit to ThoughtWorks
    28. 28. FACILITATECOMMUNICATION
    29. 29. photo credit to Eric Pugh @ www.opensourceconnections.com
    30. 30. CHALLENGES IN USING CI 34
    31. 31. WHAT PREVENTS TEAMS FROM USING CI?
    32. 32. CULTURAL CHALLENGES
    33. 33. FEAR OF COMMITMENT • Developers and Testers need to commit changes frequently! • Incremental small changes make finding integration issues simplerdrawing credit to Tom Armitage @ http://infovore.org/archives/2007/03/14/the-continuous-integration-barometer/
    34. 34. THIS ISN’T 1984
    35. 35. FEAR OF BLAME
    36. 36. • CI needs a champion who acts as the ambassador to the leaders of the organization.• Thought leaders in the organization who can help encourage team to accept the process changes that CI brings•A very successful test case •A greenfield project if possible
    37. 37. CI ON A DOLLAR A DAY"From now on, our code in revision control will always buildsuccessfully and pass its tests.". - James Shore • Build Computer • Ridiculous Toy • Annoying Bell • Automated Build • Group Agreement http://jamesshore.com/Blog/Continuous-Integration-on-a-Dollar-a-Day.html
    38. 38. credit to Paul Julius @ http://www.anthillpro.com/blogs/anthillpro-blog/2009/09/11/ enterprise_ci_cultural_maturity.html
    39. 39. STRUCTURAL CHALLENGES
    40. 40. TESTABILITY• Modularity? • Perform code complexity analysis to see if some parts• Coupling to underlying are more amenable to platform? testing.• Requiresexternal services to be up and available for testing?
    41. 41. TIGHT COUPLINGphoto credit to Jay Grieves @ www.bigswingingdeveloper.com
    42. 42. 10 MINUTE RULEphoto credit to Jay Grieves @ www.bigswingingdeveloper.com
    43. 43. CUP OF COFFEE METRICWhat happens in the time it takes to get a cup of coffee cantell you a lot about the quality of your continuous integrationsystem. - Paul Julius• Builds shouldn’t take any longer!• Gettinga cup of coffee or drink of water breaks up your day• Givesyou a chance to review your work• Requires a moduler fast build http://pauljulius.com/blog/2009/09/14/cup-of-coffee-metric-for-continuous-integration/
    44. 44. PAUL JULIUS• co-founder of CITCON• Original developer of CruiseControl• Speaksand Blogs on CI extensively • http://pauljulius.com/blog/
    45. 45. 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!!"#$%#& !"#$%&"()*+ ?-#4+!@)1-#)+ 0)/)3*)31A+ 95$1B3C+ ,-&*+D54E+ ,-&*+.#)/" ,-&*+.#)/" :)/4"#45A !"#$%&!(#%)* >2+.3$/"(4#" 0)*1$#)*+ ,-&*+6&-"#)5 .)1-5)+ ,-&*+2$1(3) !"#$%&!(#%)* 643#3-4-"+ .4&*+ ,-&*" .4-51)+643#54&
    46. 46. GETTING TOO FOCUSED ON GREEN BAR Glare of the Green Bar in CI can obscure issues with your product James Bach
    47. 47. TOOL CHALLENGES
    48. 48. INVEST IN CI SERVER X• The CI system must be as well maintained as the Source Control system.• The CI system performs builds very quickly. ✓• Who has responsibility for CI? It’s very important to have a “throat to choke”.• No false alarms. If there are false alarms then the team loses confidence in the CI system.•
    49. 49. MONKEY CIphoto credit to Eric Pugh @ www.opensourceconnections.com
    50. 50. MAKING CI THE CORE OF TOO MANY SYSTEMS
    51. 51. MAKING DECISIONS BASED ON YOUR AVAILABLE TOOLING
    52. 52. !"#$%&() 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8! F)&/)5+.15/#" D-&&A+.15/#)* .)&GH.)5I1)+ L-"(H,-##43+ ?-#4E$#1+ 9)"#+0)/&4A" L54*+0)/&4AE)3# L54*-1#43+ 0)/&4A" ?-#4+0)/&4A+#4+ M$#)*N+?-#4E$#1+ ;3)+!3IJ L54E4#43" 24"#&A+ D-&&A+.#$3*$5*K)*+ .#$3*$5*K)* 0)/&4A"
    53. 53. !"#$!%&( 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8! 944&H,$")*+ .&4)*+:)/45#3C F"#451$&+ 95)3*3C L5)*1#I) :)/45#" :)/45#" 24*)&3C L-%&"(+ 654""H.&4 >"%&#AO+:)/45#+ P$#)"#+:)/45#" 654""+D-31#43+ ?3$&A"" :-33)5 :)/45#+?11)"" ,$"1+ :)/45#+?3$&A""
    54. 54. SCALING CI FOR LARGE SUITES OF TESTS 58
    55. 55. MULTI STAGE CI• Reduce the amount of integration required• Teamsintegrate as modules are marked as ready.• Maven2 was built with this in mind! Damon Poole @ http://damonpoole.blogspot.com/2008/01/advanced-multi-stage-continous.html
    56. 56. BREAK UP TESTS• “preflight” or “smoke” tests• longer integration tests• periodic full site generation and report generation
    57. 57. THROWHARDWARE AT THE PROBLEM
    58. 58. PARALLELISM• Ifyour suite of tests take 24 hours to run, can you spin it up on 24 machines and take an hour?• Testing a web application with Selenium? Selenium Grid!
    59. 59. RUN TO CLOUD!
    60. 60. VIRTUAL SERVERS!
    61. 61. PREDICTIVE TESTING• Some commercial tools support predicting which tests will fail based on previous behavior, and running those first!
    62. 62. DON’T DO IT ALL AT ONCE• Justlike when embarking on a new exercise program, don’t just go full tilt, work up to it!• Start with the tests that are easily put under CI first, then move up the food chain.
    63. 63. SLIDING “WINDOW”• For a test suite that takes a week to run, can you take a sliding chunk, plus some random tests?• Ideas from DNA sequencing! • http://en.wikipedia.org/ wiki/DNA_sequencing
    64. 64. 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!!"#!$%& .4E)++ :)C5)""43+ .4E)+.#$#1+ !@#)3"I)+ RSST+ 9)"#+?-#4E$#43 9)"#"+:-3+Q#(+ ?3$&A""+<")* <3#+9)"#" 9)"#+64I)5$C) ,-&* ?-#4E$#)*+ !@#)3"I)+ D-31#43$&+9)"#" ?-#4E$#)*+ D-31#43+9)"#" :"BH,$")*+ 2$3-$&+9)"#" .)1-5#A+.1$3"
    65. 65. LOTS OF ADVANCED TECHNIQUES !credit to Christine Mitterbauer @ Microdoc
    66. 66. WRAP UP
    67. 67. !P!2!89.+;D+!89!:L:7.!+67 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!!"#$%#& !"#$%&"()*+ ?-#4+!@)1-#)+ 0)/)3*)31A+ 95$1B3C+ ,-&*+D54E+ ,-&*+.#)/" ,-&*+.#)/" :)/4"#45A !"#$%&!(#%)* >2+.3$/"(4#" 0)*1$#)*+ ,-&*+6&-"#)5 .)1-5)+ ,-&*+2$1(3) !"#$%&!(#%)* 643#3-4-"+ .4&*+ ,-&*" .4-51)+643#54&!"#$%&() 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8! F)&/)5+.15/#" D-&&A+.15/#)* .)&GH.)5I1)+ L-"(H,-##43+ ?-#4E$#1+ 9)"#+0)/&4A" L54*+0)/&4AE)3# L54*-1#43+ 0)/&4A" ?-#4+0)/&4A+#4+ M$#)*N+?-#4E$#1+ ;3)+!3IJ L54E4#43" 24"#&A+ D-&&A+.#$3*$5*K)*+ .#$3*$5*K)* 0)/&4A" 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!!"#!$%& .4E)++ :)C5)""43+ .4E)+.#$#1+ !@#)3"I)+ RSST+ 9)"#+?-#4E$#43 9)"#"+:-3+Q#(+ ?3$&A""+<")* <3#+9)"#" 9)"#+64I)5$C) ,-&* ?-#4E$#)*+ !@#)3"I)+ D-31#43$&+9)"#" ?-#4E$#)*+ D-31#43+9)"#" :"BH,$")*+ 2$3-$&+9)"#" .)1-5#A+.1$3"!"#$!%&( 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8! 944&H,$")*+ .&4)*+:)/45#3C F"#451$&+ 95)3*3C L5)*1#I) :)/45#" :)/45#" 24*)&3C L-%&"(+ 654""H.&4 >"%&#AO+:)/45#+ P$#)"#+:)/45#" 654""+D-31#43+ ?3$&A"" :-33)5 :)/45#+?11)"" ,$"1+ :)/45#+?3$&A"" !()*+,"$-!).$/ !"#$%&"! !"#$%&()*&)+(&,%-$(.(/0$&12&3"-+(/-4&3)5$6&78/-$.".$( !""#$%%&&&()"!*++#,-.-/%!"/+%,01-2,.01%&!*"03#(#0,1
    68. 68. Don’t forget the people aspect!credit to Paul Julius @ http://www.anthillpro.com/blogs/anthillpro-blog/2009/09/11/ enterprise_ci_cultural_maturity.html
    69. 69. SELECTING A CI SYSTEM
    70. 70. APPROACHES• Internal Hosted System• Cloud Based Solution• Completely Hosted
    71. 71. CI MATRIXMatrix of differences between CI tools: http://tinyurl.com/39hol7
    72. 72. HUDSON / JENKINS• Hudson, recently forked as Jenkins.
    73. 73. CI IN A BOX• Amazon EC2 Hosted instance• http://www.ciinabox.com/
    74. 74. MIKECI.COM
    75. 75. WHERE TO NEXT?
    76. 76. CITCON• Open Spaces Style• Twice a year, US and International• Portland, Oregon, April 2010• http://www.citconf.com/
    77. 77. CI YAHOO GROUPhttp://tech.groups.yahoo.com/group/citcon/
    78. 78. THE BOOK Continuous Integration:Improving Software Quality
    79. 79. THE CI GUYS http://ci-guys.com/ Free online consulting
    80. 80. THANK YOU!• twitter: dep4b• speakerrate: http://www.speakerrate.com/epugh/• email: epugh@opensourceconnections.com 85

    ×