Successfully reported this slideshow.
Your SlideShare is downloading. ×

Beware The Brittle Dragon - A Test Automation Epic Tale

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 38 Ad

Beware The Brittle Dragon - A Test Automation Epic Tale

Download to read offline

"Brittle" or "flaky" tests are fragile and not deterministic tests that might fail due to external conditions.
This situation causes false alarms for hypothetical bugs and, with a reaction chain, other side effect that might zero the advantages of a test automation suite.

"Brittle" or "flaky" tests are fragile and not deterministic tests that might fail due to external conditions.
This situation causes false alarms for hypothetical bugs and, with a reaction chain, other side effect that might zero the advantages of a test automation suite.

Advertisement
Advertisement

More Related Content

Recently uploaded (20)

Advertisement

Beware The Brittle Dragon - A Test Automation Epic Tale

  1. 1. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla
  2. 2. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Moral of the Story
  3. 3. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla What exactly is a brittle test?
  4. 4. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla What exactly is a brittle test? Fragile, flaky, easy to break Not deterministic result Run 1 Input Run 2 Run 3 Run 4 Not due to only the test code
  5. 5. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Raise your hand if you ever had to deal with brittle tests!
  6. 6. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla What are the consequences of brittle tests?
  7. 7. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla High Maintenance Cost Keyword test 1 test 2 test 3 test 4 75% Fail Rollercoaster result trend
  8. 8. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla High Maintenance Cost Debug and manual test Fix broken tests
  9. 9. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Real Example Test 1...50 Precondition Step Assert Nightly regression tests (1000+) Most of the time working This time many tests failed RELEASE DROP What would you do?
  10. 10. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Hidden Fail Test 1 Precondition Step Assert Test 50 Precondition Step Assert Cognitive bias: test fails only because of the precondition What if a real fail has been hidden? Test 1 Precondition Step Assert Test 50 Precondition Step Assert BUG!
  11. 11. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Useless feedback
  12. 12. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Low Trust - Low Morale Worthless feedback ignored Maintenance avoided Tests get even more broken Automation is not trusted V i c i o u s L o o p Effort with no results No result == No commitment
  13. 13. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla What makes our tests brittle?
  14. 14. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla How can we defeat the Dragon?
  15. 15. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Ice Cream Testing Pyramid Real world: Ice Cream Practical Test Pyramid Ham Vocke - Martin Fowler’s blog
  16. 16. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Ice Cream Move Down to the Pyramid Tests Part of tests Setup & Teardown Diamo una chance alle codebase legacy Nicola Mincuzzi
  17. 17. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Multipurpose / Holistic Test 1 Setup Register Assert Registration test: Login Assert ... Assert
  18. 18. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Multipurpose / Holistic “There should never be more than one reason for a class to change” Robert C. Martin (Uncle Bob) “There should never be more than one reason for a test to fail” Single Responsibility Principle
  19. 19. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Multipurpose / Holistic Test 1 Registration Check User Single assert Single Responsibility Setup 1 Checkout Setup 2 Streaming Test 1 Registration Test 2 Login Test one thing
  20. 20. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Poor Design Sleep Best effort automation Quick and Dirty Longer learning curve
  21. 21. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Poor Design Invest in automation Agree on development guidelines (Pekka’s how to write good test cases) How to write good test cases using Robot Framework Pekka Clark
  22. 22. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Poor Design Reduce technical debt Pair development Boy Scout Rule: “Always leave the code you are editing better than you found it” Robert C. Martin (Uncle Bob)
  23. 23. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Good and Bad Technical Debt “Anything about your code that slows you down over the long term. Hard-to-read code, duplication, tangled dependencies, etc.” Good And Bad Technical Debt Henrik Kniberg
  24. 24. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Unreliable and slow environment Undersized Environment Front End Developer Dev - Ops experiment Business test Backend Mad Developer (internal server error hero) Super complicated auto implosion mega query Third part never-working services
  25. 25. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Unreliable and slow environment Dedicated test environment Pay for limitless resources
  26. 26. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Unreliable and slow environment Mock third party systems Strive for performances
  27. 27. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Hard coded data Limited products Test product: 3E24B278-D1EC-358C-ADE5-F51703BBB342
  28. 28. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Hard coded Data Data Preparation Data Targeting Teardown Matters
  29. 29. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla No continuous Testing Test suite needs to be executed People is lazy forget to run Not comparable results
  30. 30. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla No continuous Testing Continuous testing to collect feedback
  31. 31. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Remove/reduce barriers to test execution No continuous Testing
  32. 32. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Daily Standup No continuous Testing Include test feedback in release process Discuss reports together
  33. 33. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Happy Ending
  34. 34. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Automate everything
  35. 35. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Quality assistance Whole team in charge for quality More TDD and pair programming Quality Assistance over Quality Assurance BDD and TDD Outside in
  36. 36. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Robot Framework BDD *** Settings *** Library SeleniumLibrary Resource settings.resource *** Test Cases *** App Game Available Given The Browser When Navigate To The App Game Then Title Should Be Tic Tac Toe *** Keywords *** Given The Browser Open Browser ${EMPTY} ${BROWSER} When Navigate To The App Game Go To ${APP_URL}
  37. 37. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Quality Assistance Team Angelo Caovilla Roberta Massimino Fabrizio Premoli Emanuele Bonanno Art Director Matteo Villa All CHILI ex-colleagues Special Thanks to
  38. 38. 🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla Contacts Angelo Caovilla acaovilla Robot Framework Milano lucagiove Luca Giovenzana @luboxit

×