Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

33rd degree

436 views

Published on

Published in: Technology
  • Be the first to comment

33rd degree

  1. 1. Main sponsorEvery Rose Has Its Thorn Taming automated tests beast Wojciech Seliga
  2. 2. Every Rose Has Its Thorn Taming automated tests beast
  3. 3. About me• 29 years coding• Agile Practices (inc. TDD) since 2003• Certified ScrumMaster, Agile Coach, Trainer, Speaker• 4+ years with Atlassian (JIRA Development Team Lead)• Co-founder of Spartez
  4. 4. The Story
  5. 5. CodebaseAlmost 10 years old
  6. 6. From 2 to about 40 engineers
  7. 7. Obsessed with Quality
  8. 8. Obsessed withAutomated Tests
  9. 9. ... From the Begining
  10. 10. 1.5M lines of code
  11. 11. 1.5M lines of code
  12. 12. Cheating1.5M lines of code
  13. 13. Mixture oftechnologies
  14. 14. XStream OSGi REST Velocity Jersey PicoActive OfBiz EntityEngine GuavaObjects LESS Java OpenSocial OS Workflow QuartzJackson Mixture of Lucene technologies JQuery Spring XML Underscore Soy Seraph OAuth Maven2 JSP Webwork JDBCJavamail Backbone.js SpringDM
  15. 15. Lots of Dependencies
  16. 16. maven dependenciesmvn dependency:list -DincludeScope=compile -o | grep :jar | cut -c 11- | sed s/:provided// | sed s/:compile//| sort -u|wc -l
  17. 17. 554 maven dependenciesmvn dependency:list -DincludeScope=compile -o | grep :jar | cut -c 11- | sed s/:provided// | sed s/:compile//| sort -u|wc -l
  18. 18. Atlassianmaven dependencies
  19. 19. 217 Atlassianmaven dependencies
  20. 20. 217 Atlassianmaven dependencies Atlassian-patchedmaven dependencies
  21. 21. 217 Atlassianmaven dependencies 20 Atlassian-patchedmaven dependencies
  22. 22. 65 modules in one IntelliJ project
  23. 23. 65 modules in one IntelliJ project
  24. 24. 65 modules in one IntelliJ project
  25. 25. 13000 unit tests
  26. 26. Almost 1000Selenium Tests
  27. 27. 4000 Functional and Integration Tests
  28. 28. Atlassian JIRA
  29. 29. Our CI environment
  30. 30. Test frameworks• JUnit 3 and 4• JMock, Easymock, Mockito• Powermock, Hamcrest• QUnit, HtmlUnit• JWebUnit, Selenium, WebDriver• Custom runners, extensions, matchers
  31. 31. Test frameworks• JUnit 3 and 4• JMock, Easymock, Mockito• Powermock, Hamcrest• QUnit, HtmlUnit• JWebUnit, Selenium, WebDriver• Custom runners, extensions, matchers
  32. 32. Bamboo Setup• Dedicated server with 70+ remote agents (including Amazon Elastic)• Build engineers• Bamboo devs
  33. 33. for each main branch
  34. 34. for each main branch
  35. 35. Run first
  36. 36. Run firstRun in parallel in batches
  37. 37. There isMuchMore
  38. 38. Type of Tests• Unit• Functional• Selenium / WebDriver• Integration• Platform• Performance
  39. 39. Platforms
  40. 40. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
  41. 41. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows
  42. 42. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows• Dimension - Java ver.: 1.5, 1.6, 1.7
  43. 43. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit
  44. 44. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
  45. 45. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic• Dimension - Browsers: IE 8+, FF, Chrome,
  46. 46. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic• Dimension - Browsers: IE 8+, FF, Chrome,
  47. 47. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows Coming• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic• Dimension - Browsers: IE 8+, FF, Chrome,
  48. 48. Run Nightly Platforms or Before Release• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows Coming• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic• Dimension - Browsers: IE 8+, FF, Chrome,
  49. 49. Triggering Builds• On Commit (hooks, polling)• Dependent Builds• Nightly Builds• Manual Builds
  50. 50. But...
  51. 51. Slow unit test
  52. 52. Very slow functional tests
  53. 53. Builds Wait in The Queue
  54. 54. Builds Often Fail
  55. 55. Too Often...
  56. 56. It takes time to fix it...
  57. 57. Sometimes very long
  58. 58. You commit at 3 PM
  59. 59. You commit at 3 PMYou get “Unit Test Green” email at 4PM
  60. 60. You commit at 3 PMYou get “Unit Test Green” email at 4PM You happily go home
  61. 61. You commit at 3 PM You get “Unit Test Green” email at 4PM You happily go homeYou get flood of “Red Test X” emails at 4 - 9PM
  62. 62. You commit at 3 PM You get “Unit Test Green” email at 4PM You happily go homeYou get flood of “Red Test X” emails at 4 - 9PM Your colleagues on the other side of the globe
  63. 63. You commit at 3 PM You get “Unit Test Green” email at 4PM You happily go homeYou get flood of “Red Test X” emails at 4 - 9PM Your colleagues on the You other side of the globe
  64. 64. “Slow CI loop and non- deterministic tests arestrong inhibitor of change instead of the catalyst” by W. Seliga
  65. 65. “We probably spend moretime dealing with the JIRA test codebase than the production codebase”
  66. 66. Striving for Coverage 100 75 Test Coverage 50 25 Effort Invested 0
  67. 67. Strange? Relationship 100% 75% Value 50% 25%Investments in automated tests 0%
  68. 68. Outcomes• Development slows down• Devs are afraid of change• Software difficult to release• Significant amount of time spent on analysing test failures• Morale goes down
  69. 69. Feedback Quality Speed `
  70. 70. Feedback Loop Speed• Tiniest change triggers test avalanche• Lack of responsibility syndrome• Devs do not run tests locally (speed)• Before you get the results you are at home
  71. 71. Quality• Non-deterministic tests (races, timeouts)• Catching up with UI changes• 1 red test hides new failures• Ignoring always red tests in dangerous ...
  72. 72. Broken window theory
  73. 73. Long time to fix
  74. 74. Decisions which do not scale
  75. 75. Decisions which do not scale• All unit tests in one maven module
  76. 76. Decisions which do not scale• All unit tests in one maven module• All functional tests in one maven module
  77. 77. Decisions which do not scale• All unit tests in one maven module• All functional tests in one maven module• All Selenium and web-driver tests in one module
  78. 78. Decisions which do not scale• All unit tests in one maven module• All functional tests in one maven module• All Selenium and web-driver tests in one module• Every commit triggers rebuild and re-test of everything
  79. 79. Decisions which do not scale• All unit tests in one maven module• All functional tests in one maven module• All Selenium and web-driver tests in one module• Every commit triggers rebuild and re-test of everything• Monolithic test framework / utils
  80. 80. Decisions which do not scale• All unit tests in one maven module• All functional tests in one maven module• All Selenium and web-driver tests in one module• Every commit triggers rebuild and re-test of everything• Monolithic test framework / utils• Opaque fixtures
  81. 81. Strategies
  82. 82. Test Quality
  83. 83. Problem:Catching up with UI changes
  84. 84. Problem:Catching up with UI changes Solution:
  85. 85. Problem:Catching up with UI changes Solution:Page Objects Pattern
  86. 86. Page Objects Pattern• Page Objects model UI elements (pages, components, dialogs, areas) your tests interact with• Page Objects shield tests from changing internal structure of the page• Page Objects generally do not make assertions• Designed for chaining
  87. 87. Page Objects Examplepublic class AddUserPage extends AbstractJiraPage @Override{ public TimedCondition isAt() { private static final String URI = return and(username.timed().isPresent(), "/secure/admin/user/AddUser!default.jspa"; password.timed().isPresent(), fullName.timed().isPresent()); } @ElementBy(name = "username") private PageElement username; public AddUserPage addUser(final String username, final String password, final String fullName, final @ElementBy(name = "password") String email, final boolean receiveEmail) private PageElement password; { this.username.type(username); @ElementBy(name = "confirm") this.password.type(password); private PageElement passwordConfirmation; this.passwordConfirmation.type(password); this.fullName.type(fullName); @ElementBy(name = "fullname") this.email.type(email); private PageElement fullName; if(receiveEmail) { this.sendEmail.select(); @ElementBy(name = "email") } private PageElement email; return this; } @ElementBy(name = "sendemail") private PageElement sendEmail; public ViewUserPage createUser() { @ElementBy(id = "user-create-submit") return createUser(ViewUserPage.class); private PageElement submit; } @ElementBy (id = "user-create-cancel") private PageElement cancelButton; public <T extends Page> T createUser(Class<T> nextPage, Object...args) @Override { public String getUrl() submit.click(); { return pageBinder.bind(nextPage, args); return URI; } } ...
  88. 88. Using Page Objects@Testpublic void testServerError(){ jira.gotoLoginPage().loginAsSysAdmin(AddUserPage.class) .addUser("username", "mypassword", "My Name", "sample@email.com", false) .createUser(); // assertions here}
  89. 89. Using Page Objects@Testpublic void testImportSampleProject() { final PivotalImporterSetupPage setupPage = getSetupPage(); Assert.assertEquals("1. Connect", setupPage.getActiveTabText()); final PivotalProjectsMappingsPage projectMappingPage = setupPage.next(); Assert.assertEquals("2. Project Mapping", setupPage.getActiveTabText()); Assert.assertTrue("Expecting all project to be selected by default", projectMappingPage.areAllProjectsSelected()); projectMappingPage.setImportAllProjects(false); projectMappingPage.setProjectImported(sampleProject, true); projectMappingPage.createProject(sampleProject, sampleProject, "SAMPLE"); final ImporterFinishedPage importerLogsPage = projectMappingPage.beginImport().waitUntilFinished(); Assert.assertTrue(importerLogsPage.isSuccess()); Assert.assertEquals(0, importerLogsPage.getGlobalErrors().size()); Assert.assertEquals("1", importerLogsPage.getProjectsImported()); // more assertions here}
  90. 90. More on Page Objects• Design for reusability• Design for sharing - libraries of Page Objects• Good support by WebDriver/Selenium 2• Atlassian Selenium 2.0
  91. 91. Problem:Opaque Test Fixtures
  92. 92. Problem:Opaque Test Fixtures Solution:
  93. 93. Problem:Opaque Test Fixtures Solution: REST-based Set-up
  94. 94. REST-based Setup
  95. 95. REST-based Setup@Beforepublic void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml");}
  96. 96. REST-based Setup@Beforepublic void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml");} VS
  97. 97. REST-based Setup@Beforepublic void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml");} VS@Beforepublic void setUpTest() { restClient.restoreEmptyInstance(); restClient.createProject(/* project params */); restClient.createUser(/* user params */); restClient.createUser(/* user params */); restClient.createSomethingElse(/* ... */);}
  98. 98. Problem:Flakey Tests
  99. 99. Problem:Flakey Tests Solution:
  100. 100. Problem:Flakey Tests Solution:Quarantine
  101. 101. Problem: Flakey Tests Solution: QuarantineFix
  102. 102. Problem: Flakey Tests Solution: QuarantineFix Eradicate
  103. 103. Quarantine• @Ignore• @Category• Quarantine on CI server• Recover or Die
  104. 104. Quarantine• @Ignore• @Category• Quarantine on CI server• Recover or Die
  105. 105. Problem:Fixing Flakey Tests
  106. 106. Problem:Fixing Flakey Tests Solution:
  107. 107. Problem:Fixing Flakey Tests Solution:Timed Conditions
  108. 108. Problem:Fixing Flakey Tests Solution: Timed ConditionsTest-friendly Markup
  109. 109. Problem: Fixing Flakey Tests Solution: Timed ConditionsTest-friendly MarkupMock Unreliable Deps
  110. 110. Timed Conditions
  111. 111. Test-friendly Markup• Do not save on IDs• Do not save on CSS classes• XPath is fragile• XPath is expensive• XPath is not readable• i18N
  112. 112. Mock Unreliable Dependencies
  113. 113. SpeedAiming at 10 seconds build
  114. 114. SpeedAiming at 10 seconds build
  115. 115. Splitting Codebase The easiest and most effective improvement
  116. 116. Splitting Codebase• Tests closer to tested code• Less to test• Testing less frequently• Increased team responsibility• Restructuring CI hierarchy - more complicated picture
  117. 117. Less to Test
  118. 118. Less to Test Commit
  119. 119. Less to Test Commit
  120. 120. Less to Test Commit
  121. 121. Less to Test Commit
  122. 122. Less to Test
  123. 123. Less to TestMost of commits happen here
  124. 124. Speed vs Control Workspace Dilemma• Incubation• Maturity• Custom workspaces
  125. 125. Test Execution Time
  126. 126. Execution Time: Test Level
  127. 127. Execution Time: Test Level Unit Tests
  128. 128. Execution Time: Test Level Unit Tests REST API Tests
  129. 129. Execution Time: Test Level Unit Tests REST API TestsJWebUnit/HTMLUnit Tests
  130. 130. Execution Time: Test Level Unit Tests REST API TestsJWebUnit/HTMLUnit TestsSelenium/WebDriver Tests
  131. 131. Execution Time: Test Level Unit Tests REST API TestsJWebUnit/HTMLUnit TestsSelenium/WebDriver Tests
  132. 132. Execution Time: Test LevelSpeed Unit Tests REST API Tests JWebUnit/HTMLUnit Tests Selenium/WebDriver Tests
  133. 133. Execution Time: Test LevelSpeed Unit Tests REST API Tests JWebUnit/HTMLUnit Tests Selenium/WebDriver Tests
  134. 134. Execution Time: Test LevelSpeed Confidence Unit Tests REST API Tests JWebUnit/HTMLUnit Tests Selenium/WebDriver Tests
  135. 135. Execution time - Cont.• Batching• Several tests per single set-up (violation of test isolation)• REST-based assertions• Remove / merge overlapping tests
  136. 136. Execution time - Cont.• IDs over CSS/JQuery Selectors over XPath• JUnit tests running in the container• In-process testing• In-memory DBs• Mocking web servers• Reducing framework initialization time• Test Optimization (Clover)
  137. 137. Waiting time• More build agents• Shorter and smaller tests• No sleep()• Avoiding long fixture setup (hot container, fragmented setup)
  138. 138. Preparation Time• SCM performance• Container set-up• Compilation time (GWT...)• Maven...• Artifacts passing
  139. 139. So how about the goals?Is 10 seconds build realistic?
  140. 140. Realistic Goals (for us*)
  141. 141. Realistic Goals (for us*) *My current personal dreams
  142. 142. Realistic Goals (for us*) Time Type for unit tests for 95% of the 2 min commits for base smoke functional tests 5 min for 95% of the commits for ALL tests for 95% of the 15 min commits on selected platform 30 min for ALL tests for ALL commits *My current personal dreams
  143. 143. Realistic Goals (for us*) Time Type for unit tests for 95% of the 2 min commits for base smoke functional tests 5 min for 95% of the commits for ALL tests for 95% of the 15 min commits on selected platform 30 min for ALL tests for ALL commits *My current personal dreams
  144. 144. Realistic Goals (for us*) Time Type for unit tests for 95% of the 2 min commits for base smoke functional tests 5 min for 95% of the commits for ALL tests for 95% of the 15 min commits on selected platform 30 min for ALL tests for ALL commits *My current personal dreams
  145. 145. Realistic Goals (for us*) Time Type for unit tests for 95% of the 2 min commits for base smoke functional tests 5 min for 95% of the commits for ALL tests for 95% of the 15 min commits on selected platform 30 min for ALL tests for ALL commits *My current personal dreams
  146. 146. Realistic Goals (for us*) Time Type for unit tests for 95% of the 2 min commits for base smoke functional tests 5 min for 95% of the commits for ALL tests for 95% of the 15 min commits on selected platform 30 min for ALL tests for ALL commits *My current personal dreams
  147. 147. Realistic Goals (for us*) p.2
  148. 148. Realistic Goals (for us*) p.2 *My current personal dreams
  149. 149. Realistic Goals (for us*) p.2 Metric Type >98% green unit tests >95% green functional tests <20min average time to fix unit test <2h average time to fix functional test *My current personal dreams
  150. 150. Realistic Goals (for us*) p.2 Metric Type >98% green unit tests >95% green functional tests <20min average time to fix unit test <2h average time to fix functional test *My current personal dreams
  151. 151. Realistic Goals (for us*) p.2 Metric Type >98% green unit tests >95% green functional tests <20min average time to fix unit test <2h average time to fix functional test *My current personal dreams
  152. 152. Realistic Goals (for us*) p.2 Metric Type >98% green unit tests >95% green functional tests <20min average time to fix unit test <2h average time to fix functional test *My current personal dreams
  153. 153. Realistic Goals (for us*) p.2 Metric Type >98% green unit tests >95% green functional tests <20min average time to fix unit test <2h average time to fix functional test *My current personal dreams
  154. 154. Our Possible Future• Further splitting the code-base, incubation and maturity• Finer-grained team responsibilities• Merciless quarantine and purging of flakey tests• More page objects, less old-school Selenium• Refactoring/removal of slow tests• More REST-driven test fixtures and assertions
  155. 155. Take-aways
  156. 156. Automated testing has cumulative benefits
  157. 157. Automated testing has cumulative benefits...and cumulative cost
  158. 158. Splitting codebase is key aspect of short test feedback loop
  159. 159. Test Code is Not Trash
  160. 160. RespectTest Code is Not Trash
  161. 161. RespectDesign Test Code is Not Trash
  162. 162. RespectDesign Test Code is Not Trash Maintain
  163. 163. RespectDesign Test Code is Not Trash Maintain Review
  164. 164. RespectDesign Test Code is Not TrashRefactor Maintain Review
  165. 165. Respect RestructureDesign Test Code is Not TrashRefactor Maintain Review
  166. 166. Respect RestructureDesign Share Test Code is Not TrashRefactor Maintain Review
  167. 167. Respect RestructureDesign Share Test Code is Not TrashRefactor Maintain Review Discuss
  168. 168. Respect RestructureDesign Share Prune Test Code is Not TrashRefactor Maintain Review Discuss
  169. 169. Optimum Balance
  170. 170. Optimum BalanceIsolation
  171. 171. Optimum BalanceIsolation Speed
  172. 172. Optimum BalanceIsolation Speed Coverage
  173. 173. Optimum BalanceIsolation Speed Coverage Level
  174. 174. Optimum BalanceIsolation Speed Coverage Level Structure
  175. 175. Optimum BalanceIsolation Speed Coverage Level Structure Effort
  176. 176. Dangerous to temper with
  177. 177. Dangerous to temper withQuality / Determinism
  178. 178. Dangerous to temper withQuality / Determinism Maintainability
  179. 179. There are no universal rules - silver bullets
  180. 180. There are no universal rules - silver bullets We are expected to findoptimum balance for our specific case
  181. 181. There are no universal rules - silver bullets We are expected to findoptimum balance for our specific caseDefinition of “optimum” constantly changes
  182. 182. Otherwise
  183. 183. Otherwise
  184. 184. Did I mention thatPage Objects pattern?
  185. 185. Credits• Photos: • http://www.flickr.com/photos/toofarnorth/ - Dragon • http://www.flickr.com/photos/striatic - Frustration • http://www.flickr.com/photos/leeadlaf/ - Broken window • http://www.flickr.com/photos/johnloo/ - Lightbulb• Dariusz Kordoński - for Test Improvements Leadership in JIRA
  186. 186. Thank You

×