A Presentation on automated test maintenance and test issues that we can improve. Presented at the Test.IL Northern meetup.

1. @gil_zilberfeld@gil_zilberfeld
Zen and the Art of
Test Maintenance

2. @gil_zilberfeld
Hello!
I AM GIL ZILBERFELD
TestinGil
www.gilzilberfeld.com
www.everydayunittesting.com
@gil_zilberfeld

3. @gil_zilberfeld@gil_zilberfeld
Test Maintenance
What is it good for?

4. @gil_zilberfeld

5. @gil_zilberfeld
The value of our tests
◉ Alignment with product focus
◉ Cover the right areas
◉ Stability
◉ Find bugs
◉ Pinpoint the problems

6. @gil_zilberfeld@gil_zilberfeld
Strategy

7. @gil_zilberfeld

8. @gil_zilberfeld@gil_zilberfeld
Once we got a map
We can create a plan of where we want to go

9. @gil_zilberfeld
High level view
◉ What is important for the product?
◉ What are the risks?
◉ What should we focus on?
◉ At this stage and the short-mid term future

10. @gil_zilberfeld

11. @gil_zilberfeld@gil_zilberfeld
What about Facebook?
Importance, risk, focus

12. @gil_zilberfeld
◉ Important:
◉ Risks
◉ Focus
Ad sales, trust
Losing money, finding out what we
really do, showing relevant
information
Algorithms
Facebook

13. @gil_zilberfeld@gil_zilberfeld
What about Kayak?
Importance, risks, focus

14. @gil_zilberfeld
◉ Important:
◉ Risks:
◉ Focus:
Commerce, Immediacy
Dependency on other systems
Information, reliability, accuracy
Integration
Kayak

15. @gil_zilberfeld

16. @gil_zilberfeld
◉ The tests we have
◉ The workflows we
cover (value)
◉ The risks we answer
◉ The costs incurred by
the tests
◉ The dependencies we
rely on
◉ The architecture
◉ The resources
◉ The skills
Testing cartography

17. @gil_zilberfeld

18. @gil_zilberfeld
◉ “Main” Workflow
coverage
◉ Architectural stability
◉ Workflow stability
◉ Manual regression
testing time
◉ Time to feedback
◉ Escaped bugs that we
could have found(by
customer or internally)
What to track

19. @gil_zilberfeld

20. @gil_zilberfeld
Test suite clean up
◉ Organize sanity, regression and acceptance
◉ Delete “Ignored” tests
◉ Fix grouping for test cohesion

21. @gil_zilberfeld
When adding tests
◉ Add tests at the right level
◉ Move “older stable” area tests to later build
cycles
◉ Move “newer unstable” area tests to earlier
cycles

22. @gil_zilberfeld

23. @gil_zilberfeld

24. @gil_zilberfeld
Uninformative tests
◉ Tests that don’t point to the problem
◉ Not enough information on failure
◉ Checking too many operations
◉ No overlapping between test types
(triangulation)

25. @gil_zilberfeld

26. @gil_zilberfeld
Unreliable tests
◉ Inability to consistently run anywhere, anytime
◉ Flaky tests
◉ Dependency on unstable resources, platforms
◉ Test isolation (resources, initialization, cleanup)

27. @gil_zilberfeld

28. @gil_zilberfeld
◉ Data transformation
◉ Tests that always pass
◉ Unclear test names
◉ Logic in tests
◉ Readability
◉ Code matching
◉ No assertions
Misleading tests

29. @gil_zilberfeld

30. @gil_zilberfeld
Tests that hurt you
◉ Hard to setup
◉ Test run length
◉ Build cycle run length

31. @gil_zilberfeld

32. @gil_zilberfeld
Maintenance issues
◉ Copy-paste, duplication
◉ Where do I add the next test?
◉ Tests that do same thing “just to be on the safe
side”
◉ Verbose and big setup, no framework (Page
object)

33. @gil_zilberfeld

34. @gil_zilberfeld
Lazy creates the norm
◉ Copy-and-paste duplication
◉ Bigger tests over smaller ones
◉ It may not be that good for you

35. @gil_zilberfeld

36. @gil_zilberfeld
Testability
◉ Accessibility
◉ Seams
◉ Big setup and mocks
◉ Shared state
◉ Unknown side effects

37. @gil_zilberfeld

38. @gil_zilberfeld

39. @gil_zilberfeld

40. @gil_zilberfeld
Thanks!
ANY QUESTIONS?
You can find me at:
gil.zilberfeld@gmail.com
@gil_zilberfeld
http://www.GilZilberfeld.com
http://www.EverydayUnitTesting.com