Live webinar: The real challenges of bol.com in their journey to radically change the way testing is organized

1. Viktor Clerc, vclerc@xebia.com
Jeroen Ruijgers, jruijgers@bol.com
bol.com's testing
and continuous
delivery journey

2. Agenda
Introduction
Positioning test automation as part of
continuous delivery
- Practice at bol.com
History of testing and test automation at
bol.com
Wrap up
Webinar: bol.com's testing and continuous delivery journey
2

3. Introduction
Viktor Clerc
- Heads Xebia’s Test Automation
team
- Traversed through all phases
and activities of the software
delivery lifecycle
- Supported major organizations
in implementing a test
automation mindset and strategy
Webinar: bol.com's testing and continuous delivery journey
3

4. Introducing bol.com
4
Started by
Bertelsman
Online 2015
bol.com
becomes
separate (and
only) entity
1999
6
employees
#1 online retailer in the
Netherlands and Belgium
> 900 employees
2008
Started with
Scrum
• 5 million customers
• 9 million products
> 35 scrumteams
2001
Webinar: bol.com's testing and continuous delivery journey

5. Introducing Xebia
Xebia explores and creates new frontiers in
IT
- Provide innovative products and services
- Stay one step ahead of our customers’ needs
- Turn new technology trends into business
advantages
Customers choose Xebia for our innovative
solutions, technological depth and
craftsmanship
Webinar: bol.com's testing and continuous delivery journey
5

6. First Principle Agile Manifesto
Webinar: bol.com's testing and continuous delivery journey
6
“Our highest priority is to satisfy the
customer through early and continuous
delivery of valuable software”

7. Continuous Delivery
The three base principles
Base principle 1: Rigorous Automation
Base principle 2: Extreme Feedback
Base principle 3: Continuous Change
Webinar: bol.com's testing and continuous delivery journey
7

8. Transformation to Test Automation
8
SPECIFY DESIGN BUILD TEST INTEGRATE REGRESSION
USER
ACCEPTANCE
RELEASE
“I directly support the
definition of acceptance
criteria”
“The setup and quality of
the tests should be right
to allow for smooth
automation”
Acceptance
Driven Testing
Development = Test
Test = Development
Automate ALL
“Testing becomes a skill
rather than a separate
phase or role…”
User Acceptance
Test
effort
INTEGRATE REGRESSION
USER
ACCEPTANCE

9. CD & TA @ bol.com
Focus on test automation
- Special team with focus on improvements for all Scrum
teams
- Teams do TA themselves
Running tests as often as possible
- Not only once a day, but also after deployments (to test)
- Focus on fast test  Fast feedback  Earlier insight in
regression
Development of MayFly (will be open sourced!)
- User story-based
- Creates separate dev and test environments per
application
9
Webinar: bol.com's testing and continuous delivery journey

10. History of testing @ bol.com
All testing done manually until 2010
2010
- started automating with QFTest
2012
- evaluated other tooling and started working with
- around that time also developed in-house tool for
testing databases
10
Where did we come from?
Webinar: bol.com's testing and continuous delivery journey

11. History of testing @ bol.com
FitNesse is mostly used
- It is Java based and we are a Java company
- The wiki format makes easy reading for business
New tools are introduced
11
Spock
Where are we now?
Webinar: bol.com's testing and continuous delivery journey

12. FitNesse implementation
Started with 1 project containing all tests
- Sharing knowledge
- Structured the same as our use cases, i.e.
WebshopSuite.BusinessAccountSuite.UseCase1500
Nightly runs from the beginning
- Indication by labels (nightly)
- First sequential per application
WebshopSuite
- Later parallel by split running of suites on the lowest
level (a.k.a. suites with only tests)
WebshopSuite.BusinessAccountSuite.UseCase1500
12
Webinar: bol.com's testing and continuous delivery journey

13. Running & Reporting
Maven is used to bootstrap FitNesse
- In combination with a JUnit runner
Specialized Jenkins job for parallel execution
- Knows the structure of FitNesse and fires
corresponding jobs
The tests are executed by Jenkins slaves
- Specialized Linux and Windows hosts with browsers
- Chrome, Firefox, IE8 and IE11
Suites are run and results are sent to the
reporting tool
- Data stored in MySQL database
- Reports generated by BIRT
13
Nightly structure
Webinar: bol.com's testing and continuous delivery journey

14. Webinar: bol.com's testing and continuous delivery journey
14

15. Wrap Up
Best practices
- Focus on optimization of test automation:
feedback, feedback, feedback
- Make testing a team effort, with adequate overall
guidance
And…
- Design test automation at scale
- Without test automation no continuous delivery
- May require novel TA solutions
Webinar: bol.com's testing and continuous delivery journey
15

16. More information?
Visit http://www.xebia.com/test-automation
Meet us (next year)
- TestWorks Conf 2016
- http://www.xebia.com/events
Webinar: bol.com's testing and continuous delivery journey
16

17. Contact
Viktor Clerc
17
vclerc@xebia.com
+31624509719
https://nl.linkedin.com/in/clerc
@viktorclerc