33rd degree

337 views
285 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
337
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • 2m\n
  • Ask question: how many people write automated tests?\nToday? 1 year ago, 3 years ago? 5 years ago? 10 years ago?\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Joke: 1.2M Java -> 100K Scala -> 10K Perl\n
  • Joke: 1.2M Java -> 100K Scala -> 10K Perl\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • 2 min\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Note: batching helped us to reduce the time from several hours to about 1h\n
  • Note: batching helped us to reduce the time from several hours to about 1h\n
  • Note: batching helped us to reduce the time from several hours to about 1h\n
  • Note: batching helped us to reduce the time from several hours to about 1h\n
  • Note: batching helped us to reduce the time from several hours to about 1h\n
  • Note: batching helped us to reduce the time from several hours to about 1h\n
  • Note: batching helped us to reduce the time from several hours to about 1h\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Mention that Selenium would be to scary to show it :)\n
  • \n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • Joke: and some extra failed builds in the morning next day (from nightly builds)\n
  • \n
  • Question: how many people has this problem?\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Joke\n
  • Joke\n
  • Joke\n
  • Joke\n
  • Joke\n
  • Joke\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Encourages you to write good REST API, REST client\nTell about Backdoor\n
  • Encourages you to write good REST API, REST client\nTell about Backdoor\n
  • Encourages you to write good REST API, REST client\nTell about Backdoor\n
  • \n
  • \n
  • \n
  • \n
  • Quarantine strategies - X days, Max. Y tests. Quarantine after healthy tests\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • VIES example\n
  • \n
  • \n
  • Important: less to test and less frequently -> CI env faster throuput\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • tutaj dodać wagę - obrazek\n
  • \n
  • e.g. QUnit vs Selenium\nTDD makes natural favouring Unit tests.\n
  • e.g. QUnit vs Selenium\nTDD makes natural favouring Unit tests.\n
  • e.g. QUnit vs Selenium\nTDD makes natural favouring Unit tests.\n
  • e.g. QUnit vs Selenium\nTDD makes natural favouring Unit tests.\n
  • e.g. QUnit vs Selenium\nTDD makes natural favouring Unit tests.\n
  • e.g. QUnit vs Selenium\nTDD makes natural favouring Unit tests.\n
  • e.g. QUnit vs Selenium\nTDD makes natural favouring Unit tests.\n
  • e.g. QUnit vs Selenium\nTDD makes natural favouring Unit tests.\n
  • \n
  • \n
  • Story with Sleep for 60 seconds just to see if something did not happen\n
  • Story about GNU C++ compiler producing gigabytes of object files difficult to distribute to CI env (and much time spent on IO)\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Access - all code in one workspace, compiled together, easy to refactor, costly to compile\n
  • Access - all code in one workspace, compiled together, easy to refactor, costly to compile\n
  • Access - all code in one workspace, compiled together, easy to refactor, costly to compile\n
  • Access - all code in one workspace, compiled together, easy to refactor, costly to compile\n
  • Access - all code in one workspace, compiled together, easy to refactor, costly to compile\n
  • Access - all code in one workspace, compiled together, easy to refactor, costly to compile\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • 33rd degree

    1. 1. Main sponsorEvery Rose Has Its Thorn Taming automated tests beast Wojciech Seliga
    2. 2. Every Rose Has Its Thorn Taming automated tests beast
    3. 3. About me• 29 years coding• Agile Practices (inc. TDD) since 2003• Certified ScrumMaster, Agile Coach, Trainer, Speaker• 4+ years with Atlassian (JIRA Development Team Lead)• Co-founder of Spartez
    4. 4. The Story
    5. 5. CodebaseAlmost 10 years old
    6. 6. From 2 to about 40 engineers
    7. 7. Obsessed with Quality
    8. 8. Obsessed withAutomated Tests
    9. 9. ... From the Begining
    10. 10. 1.5M lines of code
    11. 11. 1.5M lines of code
    12. 12. Cheating1.5M lines of code
    13. 13. Mixture oftechnologies
    14. 14. XStream OSGi REST Velocity Jersey PicoActive OfBiz EntityEngine GuavaObjects LESS Java OpenSocial OS Workflow QuartzJackson Mixture of Lucene technologies JQuery Spring XML Underscore Soy Seraph OAuth Maven2 JSP Webwork JDBCJavamail Backbone.js SpringDM
    15. 15. Lots of Dependencies
    16. 16. maven dependenciesmvn dependency:list -DincludeScope=compile -o | grep :jar | cut -c 11- | sed s/:provided// | sed s/:compile//| sort -u|wc -l
    17. 17. 554 maven dependenciesmvn dependency:list -DincludeScope=compile -o | grep :jar | cut -c 11- | sed s/:provided// | sed s/:compile//| sort -u|wc -l
    18. 18. Atlassianmaven dependencies
    19. 19. 217 Atlassianmaven dependencies
    20. 20. 217 Atlassianmaven dependencies Atlassian-patchedmaven dependencies
    21. 21. 217 Atlassianmaven dependencies 20 Atlassian-patchedmaven dependencies
    22. 22. 65 modules in one IntelliJ project
    23. 23. 65 modules in one IntelliJ project
    24. 24. 65 modules in one IntelliJ project
    25. 25. 13000 unit tests
    26. 26. Almost 1000Selenium Tests
    27. 27. 4000 Functional and Integration Tests
    28. 28. Atlassian JIRA
    29. 29. Our CI environment
    30. 30. Test frameworks• JUnit 3 and 4• JMock, Easymock, Mockito• Powermock, Hamcrest• QUnit, HtmlUnit• JWebUnit, Selenium, WebDriver• Custom runners, extensions, matchers
    31. 31. Test frameworks• JUnit 3 and 4• JMock, Easymock, Mockito• Powermock, Hamcrest• QUnit, HtmlUnit• JWebUnit, Selenium, WebDriver• Custom runners, extensions, matchers
    32. 32. Bamboo Setup• Dedicated server with 70+ remote agents (including Amazon Elastic)• Build engineers• Bamboo devs
    33. 33. for each main branch
    34. 34. for each main branch
    35. 35. Run first
    36. 36. Run firstRun in parallel in batches
    37. 37. There isMuchMore
    38. 38. Type of Tests• Unit• Functional• Selenium / WebDriver• Integration• Platform• Performance
    39. 39. Platforms
    40. 40. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
    41. 41. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows
    42. 42. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows• Dimension - Java ver.: 1.5, 1.6, 1.7
    43. 43. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit
    44. 44. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
    45. 45. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic• Dimension - Browsers: IE 8+, FF, Chrome,
    46. 46. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic• Dimension - Browsers: IE 8+, FF, Chrome,
    47. 47. Platforms• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows Coming• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic• Dimension - Browsers: IE 8+, FF, Chrome,
    48. 48. Run Nightly Platforms or Before Release• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle• Dimension - OS: Linux, Windows Coming• Dimension - Java ver.: 1.5, 1.6, 1.7• Dimension - CPU arch.: 32-bit, 64-bit• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic• Dimension - Browsers: IE 8+, FF, Chrome,
    49. 49. Triggering Builds• On Commit (hooks, polling)• Dependent Builds• Nightly Builds• Manual Builds
    50. 50. But...
    51. 51. Slow unit test
    52. 52. Very slow functional tests
    53. 53. Builds Wait in The Queue
    54. 54. Builds Often Fail
    55. 55. Too Often...
    56. 56. It takes time to fix it...
    57. 57. Sometimes very long
    58. 58. You commit at 3 PM
    59. 59. You commit at 3 PMYou get “Unit Test Green” email at 4PM
    60. 60. You commit at 3 PMYou get “Unit Test Green” email at 4PM You happily go home
    61. 61. You commit at 3 PM You get “Unit Test Green” email at 4PM You happily go homeYou get flood of “Red Test X” emails at 4 - 9PM
    62. 62. You commit at 3 PM You get “Unit Test Green” email at 4PM You happily go homeYou get flood of “Red Test X” emails at 4 - 9PM Your colleagues on the other side of the globe
    63. 63. You commit at 3 PM You get “Unit Test Green” email at 4PM You happily go homeYou get flood of “Red Test X” emails at 4 - 9PM Your colleagues on the You other side of the globe
    64. 64. “Slow CI loop and non- deterministic tests arestrong inhibitor of change instead of the catalyst” by W. Seliga
    65. 65. “We probably spend moretime dealing with the JIRA test codebase than the production codebase”
    66. 66. Striving for Coverage 100 75 Test Coverage 50 25 Effort Invested 0
    67. 67. Strange? Relationship 100% 75% Value 50% 25%Investments in automated tests 0%
    68. 68. Outcomes• Development slows down• Devs are afraid of change• Software difficult to release• Significant amount of time spent on analysing test failures• Morale goes down
    69. 69. Feedback Quality Speed `
    70. 70. Feedback Loop Speed• Tiniest change triggers test avalanche• Lack of responsibility syndrome• Devs do not run tests locally (speed)• Before you get the results you are at home
    71. 71. Quality• Non-deterministic tests (races, timeouts)• Catching up with UI changes• 1 red test hides new failures• Ignoring always red tests in dangerous ...
    72. 72. Broken window theory
    73. 73. Long time to fix
    74. 74. Decisions which do not scale
    75. 75. Decisions which do not scale• All unit tests in one maven module
    76. 76. Decisions which do not scale• All unit tests in one maven module• All functional tests in one maven module
    77. 77. Decisions which do not scale• All unit tests in one maven module• All functional tests in one maven module• All Selenium and web-driver tests in one module
    78. 78. Decisions which do not scale• All unit tests in one maven module• All functional tests in one maven module• All Selenium and web-driver tests in one module• Every commit triggers rebuild and re-test of everything
    79. 79. Decisions which do not scale• All unit tests in one maven module• All functional tests in one maven module• All Selenium and web-driver tests in one module• Every commit triggers rebuild and re-test of everything• Monolithic test framework / utils
    80. 80. Decisions which do not scale• All unit tests in one maven module• All functional tests in one maven module• All Selenium and web-driver tests in one module• Every commit triggers rebuild and re-test of everything• Monolithic test framework / utils• Opaque fixtures
    81. 81. Strategies
    82. 82. Test Quality
    83. 83. Problem:Catching up with UI changes
    84. 84. Problem:Catching up with UI changes Solution:
    85. 85. Problem:Catching up with UI changes Solution:Page Objects Pattern
    86. 86. Page Objects Pattern• Page Objects model UI elements (pages, components, dialogs, areas) your tests interact with• Page Objects shield tests from changing internal structure of the page• Page Objects generally do not make assertions• Designed for chaining
    87. 87. Page Objects Examplepublic class AddUserPage extends AbstractJiraPage @Override{ public TimedCondition isAt() { private static final String URI = return and(username.timed().isPresent(), "/secure/admin/user/AddUser!default.jspa"; password.timed().isPresent(), fullName.timed().isPresent()); } @ElementBy(name = "username") private PageElement username; public AddUserPage addUser(final String username, final String password, final String fullName, final @ElementBy(name = "password") String email, final boolean receiveEmail) private PageElement password; { this.username.type(username); @ElementBy(name = "confirm") this.password.type(password); private PageElement passwordConfirmation; this.passwordConfirmation.type(password); this.fullName.type(fullName); @ElementBy(name = "fullname") this.email.type(email); private PageElement fullName; if(receiveEmail) { this.sendEmail.select(); @ElementBy(name = "email") } private PageElement email; return this; } @ElementBy(name = "sendemail") private PageElement sendEmail; public ViewUserPage createUser() { @ElementBy(id = "user-create-submit") return createUser(ViewUserPage.class); private PageElement submit; } @ElementBy (id = "user-create-cancel") private PageElement cancelButton; public <T extends Page> T createUser(Class<T> nextPage, Object...args) @Override { public String getUrl() submit.click(); { return pageBinder.bind(nextPage, args); return URI; } } ...
    88. 88. Using Page Objects@Testpublic void testServerError(){ jira.gotoLoginPage().loginAsSysAdmin(AddUserPage.class) .addUser("username", "mypassword", "My Name", "sample@email.com", false) .createUser(); // assertions here}
    89. 89. Using Page Objects@Testpublic void testImportSampleProject() { final PivotalImporterSetupPage setupPage = getSetupPage(); Assert.assertEquals("1. Connect", setupPage.getActiveTabText()); final PivotalProjectsMappingsPage projectMappingPage = setupPage.next(); Assert.assertEquals("2. Project Mapping", setupPage.getActiveTabText()); Assert.assertTrue("Expecting all project to be selected by default", projectMappingPage.areAllProjectsSelected()); projectMappingPage.setImportAllProjects(false); projectMappingPage.setProjectImported(sampleProject, true); projectMappingPage.createProject(sampleProject, sampleProject, "SAMPLE"); final ImporterFinishedPage importerLogsPage = projectMappingPage.beginImport().waitUntilFinished(); Assert.assertTrue(importerLogsPage.isSuccess()); Assert.assertEquals(0, importerLogsPage.getGlobalErrors().size()); Assert.assertEquals("1", importerLogsPage.getProjectsImported()); // more assertions here}
    90. 90. More on Page Objects• Design for reusability• Design for sharing - libraries of Page Objects• Good support by WebDriver/Selenium 2• Atlassian Selenium 2.0
    91. 91. Problem:Opaque Test Fixtures
    92. 92. Problem:Opaque Test Fixtures Solution:
    93. 93. Problem:Opaque Test Fixtures Solution: REST-based Set-up
    94. 94. REST-based Setup
    95. 95. REST-based Setup@Beforepublic void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml");}
    96. 96. REST-based Setup@Beforepublic void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml");} VS
    97. 97. REST-based Setup@Beforepublic void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml");} VS@Beforepublic void setUpTest() { restClient.restoreEmptyInstance(); restClient.createProject(/* project params */); restClient.createUser(/* user params */); restClient.createUser(/* user params */); restClient.createSomethingElse(/* ... */);}
    98. 98. Problem:Flakey Tests
    99. 99. Problem:Flakey Tests Solution:
    100. 100. Problem:Flakey Tests Solution:Quarantine
    101. 101. Problem: Flakey Tests Solution: QuarantineFix
    102. 102. Problem: Flakey Tests Solution: QuarantineFix Eradicate
    103. 103. Quarantine• @Ignore• @Category• Quarantine on CI server• Recover or Die
    104. 104. Quarantine• @Ignore• @Category• Quarantine on CI server• Recover or Die
    105. 105. Problem:Fixing Flakey Tests
    106. 106. Problem:Fixing Flakey Tests Solution:
    107. 107. Problem:Fixing Flakey Tests Solution:Timed Conditions
    108. 108. Problem:Fixing Flakey Tests Solution: Timed ConditionsTest-friendly Markup
    109. 109. Problem: Fixing Flakey Tests Solution: Timed ConditionsTest-friendly MarkupMock Unreliable Deps
    110. 110. Timed Conditions
    111. 111. Test-friendly Markup• Do not save on IDs• Do not save on CSS classes• XPath is fragile• XPath is expensive• XPath is not readable• i18N
    112. 112. Mock Unreliable Dependencies
    113. 113. SpeedAiming at 10 seconds build
    114. 114. SpeedAiming at 10 seconds build
    115. 115. Splitting Codebase The easiest and most effective improvement
    116. 116. Splitting Codebase• Tests closer to tested code• Less to test• Testing less frequently• Increased team responsibility• Restructuring CI hierarchy - more complicated picture
    117. 117. Less to Test
    118. 118. Less to Test Commit
    119. 119. Less to Test Commit
    120. 120. Less to Test Commit
    121. 121. Less to Test Commit
    122. 122. Less to Test
    123. 123. Less to TestMost of commits happen here
    124. 124. Speed vs Control Workspace Dilemma• Incubation• Maturity• Custom workspaces
    125. 125. Test Execution Time
    126. 126. Execution Time: Test Level
    127. 127. Execution Time: Test Level Unit Tests
    128. 128. Execution Time: Test Level Unit Tests REST API Tests
    129. 129. Execution Time: Test Level Unit Tests REST API TestsJWebUnit/HTMLUnit Tests
    130. 130. Execution Time: Test Level Unit Tests REST API TestsJWebUnit/HTMLUnit TestsSelenium/WebDriver Tests
    131. 131. Execution Time: Test Level Unit Tests REST API TestsJWebUnit/HTMLUnit TestsSelenium/WebDriver Tests
    132. 132. Execution Time: Test LevelSpeed Unit Tests REST API Tests JWebUnit/HTMLUnit Tests Selenium/WebDriver Tests
    133. 133. Execution Time: Test LevelSpeed Unit Tests REST API Tests JWebUnit/HTMLUnit Tests Selenium/WebDriver Tests
    134. 134. Execution Time: Test LevelSpeed Confidence Unit Tests REST API Tests JWebUnit/HTMLUnit Tests Selenium/WebDriver Tests
    135. 135. Execution time - Cont.• Batching• Several tests per single set-up (violation of test isolation)• REST-based assertions• Remove / merge overlapping tests
    136. 136. Execution time - Cont.• IDs over CSS/JQuery Selectors over XPath• JUnit tests running in the container• In-process testing• In-memory DBs• Mocking web servers• Reducing framework initialization time• Test Optimization (Clover)
    137. 137. Waiting time• More build agents• Shorter and smaller tests• No sleep()• Avoiding long fixture setup (hot container, fragmented setup)
    138. 138. Preparation Time• SCM performance• Container set-up• Compilation time (GWT...)• Maven...• Artifacts passing
    139. 139. So how about the goals?Is 10 seconds build realistic?
    140. 140. Realistic Goals (for us*)
    141. 141. Realistic Goals (for us*) *My current personal dreams
    142. 142. Realistic Goals (for us*) Time Type for unit tests for 95% of the 2 min commits for base smoke functional tests 5 min for 95% of the commits for ALL tests for 95% of the 15 min commits on selected platform 30 min for ALL tests for ALL commits *My current personal dreams
    143. 143. Realistic Goals (for us*) Time Type for unit tests for 95% of the 2 min commits for base smoke functional tests 5 min for 95% of the commits for ALL tests for 95% of the 15 min commits on selected platform 30 min for ALL tests for ALL commits *My current personal dreams
    144. 144. Realistic Goals (for us*) Time Type for unit tests for 95% of the 2 min commits for base smoke functional tests 5 min for 95% of the commits for ALL tests for 95% of the 15 min commits on selected platform 30 min for ALL tests for ALL commits *My current personal dreams
    145. 145. Realistic Goals (for us*) Time Type for unit tests for 95% of the 2 min commits for base smoke functional tests 5 min for 95% of the commits for ALL tests for 95% of the 15 min commits on selected platform 30 min for ALL tests for ALL commits *My current personal dreams
    146. 146. Realistic Goals (for us*) Time Type for unit tests for 95% of the 2 min commits for base smoke functional tests 5 min for 95% of the commits for ALL tests for 95% of the 15 min commits on selected platform 30 min for ALL tests for ALL commits *My current personal dreams
    147. 147. Realistic Goals (for us*) p.2
    148. 148. Realistic Goals (for us*) p.2 *My current personal dreams
    149. 149. Realistic Goals (for us*) p.2 Metric Type >98% green unit tests >95% green functional tests <20min average time to fix unit test <2h average time to fix functional test *My current personal dreams
    150. 150. Realistic Goals (for us*) p.2 Metric Type >98% green unit tests >95% green functional tests <20min average time to fix unit test <2h average time to fix functional test *My current personal dreams
    151. 151. Realistic Goals (for us*) p.2 Metric Type >98% green unit tests >95% green functional tests <20min average time to fix unit test <2h average time to fix functional test *My current personal dreams
    152. 152. Realistic Goals (for us*) p.2 Metric Type >98% green unit tests >95% green functional tests <20min average time to fix unit test <2h average time to fix functional test *My current personal dreams
    153. 153. Realistic Goals (for us*) p.2 Metric Type >98% green unit tests >95% green functional tests <20min average time to fix unit test <2h average time to fix functional test *My current personal dreams
    154. 154. Our Possible Future• Further splitting the code-base, incubation and maturity• Finer-grained team responsibilities• Merciless quarantine and purging of flakey tests• More page objects, less old-school Selenium• Refactoring/removal of slow tests• More REST-driven test fixtures and assertions
    155. 155. Take-aways
    156. 156. Automated testing has cumulative benefits
    157. 157. Automated testing has cumulative benefits...and cumulative cost
    158. 158. Splitting codebase is key aspect of short test feedback loop
    159. 159. Test Code is Not Trash
    160. 160. RespectTest Code is Not Trash
    161. 161. RespectDesign Test Code is Not Trash
    162. 162. RespectDesign Test Code is Not Trash Maintain
    163. 163. RespectDesign Test Code is Not Trash Maintain Review
    164. 164. RespectDesign Test Code is Not TrashRefactor Maintain Review
    165. 165. Respect RestructureDesign Test Code is Not TrashRefactor Maintain Review
    166. 166. Respect RestructureDesign Share Test Code is Not TrashRefactor Maintain Review
    167. 167. Respect RestructureDesign Share Test Code is Not TrashRefactor Maintain Review Discuss
    168. 168. Respect RestructureDesign Share Prune Test Code is Not TrashRefactor Maintain Review Discuss
    169. 169. Optimum Balance
    170. 170. Optimum BalanceIsolation
    171. 171. Optimum BalanceIsolation Speed
    172. 172. Optimum BalanceIsolation Speed Coverage
    173. 173. Optimum BalanceIsolation Speed Coverage Level
    174. 174. Optimum BalanceIsolation Speed Coverage Level Structure
    175. 175. Optimum BalanceIsolation Speed Coverage Level Structure Effort
    176. 176. Dangerous to temper with
    177. 177. Dangerous to temper withQuality / Determinism
    178. 178. Dangerous to temper withQuality / Determinism Maintainability
    179. 179. There are no universal rules - silver bullets
    180. 180. There are no universal rules - silver bullets We are expected to findoptimum balance for our specific case
    181. 181. There are no universal rules - silver bullets We are expected to findoptimum balance for our specific caseDefinition of “optimum” constantly changes
    182. 182. Otherwise
    183. 183. Otherwise
    184. 184. Did I mention thatPage Objects pattern?
    185. 185. Credits• Photos: • http://www.flickr.com/photos/toofarnorth/ - Dragon • http://www.flickr.com/photos/striatic - Frustration • http://www.flickr.com/photos/leeadlaf/ - Broken window • http://www.flickr.com/photos/johnloo/ - Lightbulb• Dariusz Kordoński - for Test Improvements Leadership in JIRA
    186. 186. Thank You

    ×