Slides from my talk at SeleniumConf Berlin 2017.

1. The Bumpy Road
Towards
Continuous Delivery
Benjamin Bischoff
Web Test Automation Engineer
SeleniumConf Berlin 2017

2. Topics
1. About trivago
2. Main Product
3. Releases
4. Central Automation Team
5. Challenges
6. Summary
2

3. About trivago

4. trivago
• Founded 2005
• First hotel meta search
in Germany
• Headquartered in
Düsseldorf
4

5. trivago’s Mission Statement
“To be the traveller's first and
independent source of
information for finding the
ideal hotel at the lowest rate”

6. 6
Core Values

7. Main Product

8. www.trivago.de
8

9. 9
We are forwarding you to the booking site deal.
Please double-check the hotel information before making your hotel booking.
OTA
Business Model

10. 55 Markets
10

11. So many things…

12. We Talk About…
...Manual and Automated Testing
...Deployment and Test Pipelines
...Frameworks and Technologies
...Documentation and Knowledge Sharing
12
Why?

13. User Value

14. Hotel Search Releases
The Past

15. 15
The Past
Feature Development
Manual Tests
Merge
Automated Tests / Release

16. 16
Problems
• Repetitive manual work
• Old features could break
• Possible long release cycles
• Frequent rollbacks
• Automated tests did not have any real effect
Feature
Development
Manual tests
Merge
Automated Tests
/ Release

17. Hotel Search Releases
The Goal

18. 18
The Past
Feature Development
Manual Tests
Merge
Automated Tests / Release

19. 19
The Goal
Feature Development
Manual Tests
Automated Tests
Merge / Release

20. 20
Advantages
• Stable new features
• Quicker release cycles
• Easier refactoring
• Critical paths are covered
• New features cannot break older ones
Feature
Development
Manual Tests
Automated Tests
Merge / Release

21. The Team

22. Release & Test Automation Team
• Release Engineering & Test Automation
• Maintain the test framework and pipelines
• Support all teams using our platforms
• Decrease release time
• Remove repetitive work
• Improve stability
22

23. Test Automation Team Mission Statement
“Make automated testing a
widely used and reliable
technique at trivago.”

24. Old Automation Approach
• Old framework was not flexible enough
• No clear guidelines
• No full test suite
• No clear test result visibility
• No test ownership
24

25. RTA TA Decisions
Who writes and owns the tests?
How are the tests structured?
Where should the tests be located?
How / when should the tests be run?
25
 Empower QA
 Clear test guidelines
 Inside the SUT
 Test framework and pipelines

26. Challenges
Technical Challenges

27. 27
Test Framework
• User friendly
• Stable and consistent
• Clearly documented
• Easy to run tests
• Enable parallel execution
• Support application specific
features

28. 28
Test Infrastructure
• In-house Selenium grid
• Lots of maintenance
• Expensive
• External Selenium grid
In-house grid External grid
Hard to set up (static) Instantly available
Lots of maintenance Less maintenance
Hard to scale Scaling built in
Limited OS / Browser
combinations
Huge number of
combinations possible
Less expensive? More expensive?

29. Challenges
Psychological Challenges

30. 30
“Automated tests will replace all manual testing”
”They want to turn QA into developers”
“My test is done, I have nothing else to do.”
”I have no time for automation”
Bias

31. 31
Bias
• Extensive documentation
• Workshops
• “Automation Time”
• Make sure everyone is on the same page!
• Testing is a team effort!
“Automated tests are there
to replace all manual
testing”
”They want to turn QA into
developers”
“My test is done, I have
nothing else to do.”
”I have no time for
automation”

32. 32
Increase Trust In Tests
• Dashboards and reporting
• Extensive logging
• Early flakiness detection
• Test qualification job
• Slack notifications
• Duty managers

33. Project Status

34. 34
What We Achieved
• Test pipelines exist
• Test suite of 200 scenarios
• Automatic tests on PRs and branches
• Kibana dashboards
• Full test reports
• Clear ownership

35. 35
What We Are Working On
• Block PRs
• Even clearer reporting
• Further education
• Test framework architecture and speed
• Working together with feature developers

36. Summary

37. 37
Core Values

38. Key Points
Trust
Unwavering Focus
Power of Proof
Fanatic Learning
38
 Trustful Automation
 Clear Ownership
 Test Result Visibility
 Communication
Why?

39. User Value

40. @BischoffDev
Web Test Automation Engineer
Benjamin Bischoff
Thank you!