As part of a recent replatforming exercise, Net-A-Porter has worked hard not only to refresh their technology, but to create a great testing culture. As a result, they have come a long way from “throwing tests over the wall.” Testing concerns are now part of their NFRs and technical decisions. Developers are responsible for unit and functional tests, working alongside test specialists who are part of every delivery team for guidance. In this SauceCon 2018 session, Adela Mosincat and James Collins will walk attendees through how Sauce Labs has helped in this process. She will review how the “test-runner” was born to facilitate the consistent and seamless running of their tests on Sauce Labs across teams. The test-runner is a Docker image, whose purpose is to handle the running of the tests both locally and in their CI pipeline. It parallelizes the tests as much as possible, executes intelligent waiting for VMs and retrying for flaky tests, supports test tagging and quarantining and handles reporting to both Sauce Labs and Jenkins. In addition to the test-runner, Adela and Jim will review how Net-A-Porter improved their dev/test culture and CI pipeline, and share what they learned along the way.

1. 1
Slow Flaky and Legacy Tests?
FTFY- Our new testing strategy at Net-A-Porter
1st March 2018
Adela Mosincat, James Collins
Saucecon 2018

2. Who we are and
what we do
2

3. 3

4. AGENDA
– Browser matrix
– Technologies
– Old/Current Architecture
4
– Testing culture
– CI Pipeline
– New Tooling
– Test Runner

5. Browser matrix

6. 49.45% 27.60% 3.43% 2.97%
49.04 % 18.44 %
11.91 %
Device adoption

7. Browser Matrix
7

8. Technologies

9. 9
Old architecture

10. 10

11. CI Pipeline

12. 12
CI PIPELINE

13. Testing Culture

14. 14
Testing Culture - Before
Dev in
test
Developers

15. 15
Testing Culture - After
After

16. New Tooling

17. ● Want tests to be faster, more reliable and easier
● We are still running tests in CI
○ How is it/can it really be different??
● Tech changes (in conjunction with culture) that have made things
possible:
○ Docker, CI Tooling (Docker in CI), JS Developments
● Align Further, Simplify and Containerize...
17
Why?/What’s Different?

18. 18
JS Developments
● Tests in same language as all frontend apps have been aligned to (JS)
○ WebdriverIO Framework
○ Async/Await JS - makes writing asynchronous code (like functional
tests) much nicer than previously with promises
○ Tests also in the same style as JS application unit tests

19. 19
“Promises” Way

20. 20
“Async/Await” Way

21. Docker
21
● Now using Docker in CI
○ Much more flexibility
● Can easily replicate locally
○ Consistent environments/versions
● Use docker-compose to spin up their app
○ and any dependency services
● Helper Images...

22. 22
Helper Images? -What are they?/What do they do?
● Specific functionality
● Contain all the tooling/environment setup
○ Where the time consuming setup work has been done
● Helps keep application repos clean
● Abstract away complexity
● 2 Core ones we use...

23. 23

24. 24
Test-Runner
● One place for all the test “tooling” vs. in every app repo
● “Centralised”
● Handles:
○ Running tests in Local Browsers/SauceLabs
○ Shared config (eg Browser Matrix)
○ Reporting
○ Parallelisation
○ Anti-flakiness

25. 25
Test-Runner
● Based around a pretty heavily modified version of the great “Magellan”
testing framework - https://github.com/TestArmada/magellan

26. 26
Test-Runner
● Teams pull the “latest” image so get updates “for free”
○ Runner has built in browser matrix presets etc.
● Mount the test files into container
○ That live in the application repo
● Handles opening/closing of sauce tunnel
● Tagged builds to user/branch name
● Reports to SauceLabs
● Does quite a lot behind the scenes...

27. Parallelization
27

28. 28
Parallelization
● Parallelized tests, not just per file, but per test case
○ Found most of the JS test frameworks only at best only spun up a
session per file
● Smaller tests to retry if it flakes
● Faster - more VMs you have, quicker tests will run
● Much easier to debug/find issue (in the lovely Saucelabs viewer)
● Named and tagged tests in the SauceLabs UI

29. 29
Reporting

30. Flakiness, and how to beat it
30

31. 31
Anti Flakiness!
● Now mainly single page apps
○ Different challenges
○ Helper functions in tests
● “Helper image” features to reduce flakiness:
○ Intelligent Waiting for browser session (with generous tiemeouts)
○ Easily adjustable retries
○ Consistent data…

32. API-Snapshot
(Test API Caching Proxy Service)

33. 33
API Snapshot
● Docker image/container
● Sits as a proxy between your app and real API
● “Caches” responses as files
● Folder is mounted as a docker volume
● Responses get committed to application repo
○ Live alongside tests
● Can mock error/slow responses
● Can even mock “stateful” responses with a cookie

34. 34
Bringing it all together - Docker-Compose Config

35. 35
Bringing it all together - Inside the build (test stage) job
Jenkins
Docker (Compose)
Git
Repo/Filesystem
Web Application API-Snapshot Test-Runner
API RESPONSES TESTS
API RESPONSES TESTS
DOCKER-COMPOSE.YML

36. 36
Bringing it all together - Inside the build (test stage) job
Jenkins
Docker (Compose)
Git
Repo/Filesystem
Web Application API-Snapshot Test-Runner
API RESPONSES TESTS
API RESPONSES TESTS
Sauce Connect Tunnel
Created by runner)
DOCKER-COMPOSE.YML

37. 37
Bringing it all together - Inside the build (test stage) job
Jenkins
Docker (Compose)
Git
Repo/Filesystem
Web Application API-Snapshot Test-Runner
API RESPONSES TESTS
API RESPONSES TESTS
Sauce Connect Tunnel
Created by runner)
Requests to Server & HTML Payloads
Server Side API Calls
Client-Side API Calls (JSON)
DOCKER-COMPOSE.YML

38. 38
End Results
● Far less test config in app repos than before
● Faster, cleaner, more reliable tests
● Consistent test running environment (all self contained in docker)
● Centralised & Shareable
○ Eg update browsers in one place, all consumers get the new defaults

39. Open Source?

40. Not quite yet… :(

41. Q&A?