1.
No risk, no reward.
The joys of testing in
production.
UXDX Dublin, May 2018
Fabio Cognigni (@faberoh),
Natalia Bartol (@nataliabartol)

2.
Legacy vs. Tech Dreams
?

3.
Legacy vs. Tech Dreams
?

4.
Shipping on top of monolith
staging
prod
QA
devDevs
QA
Ops

5.
1 (huge) system x 4 envs; 3 teams
● No clear ownership
● Slow iterations
● Difficult and risky operations
● Low maneuverability
Deploy: weekly
Ship new features: months
Troubleshooting: days
Shipping on top of monolith

6.
Mobile app services
middleware
wishlist
homepage
customer-
segmentation
CMS

7.
Killing the monolith: strangler pattern

8.
N services x 2 envs (staging, prod); 1 team (Devs)
Power to the Devs via easy/automated deployments. No Ops!
● Deploy small changes, more frequently
● More control (easier troubleshooting)
● Faster iterations
● Higher code (and product) quality
● Ship new features independently
Simplified deployment process

9.
Safe deployments
Phased rollout to production:
● dark canary node
● live canary node
● live nodes
Dark canary
Live canary
Live node 3
Live node 1
Live node 2
Production

10.
Microservices: challenges
N services x 2 envs (staging, prod) - still quite complex!
Maintaining a staging env becomes a headache:
● Infrastructure complexity & cost
● Fragile (never maintained with the same accuracy as prod)
● Bottleneck

11.
Why do we need staging?
● Storing test data separately from production data.
● Separating test traffic/load from prod load.
● Some (3rd party) features are only possible to test in
staging.
● Sometimes testing against 3rd party staging is the only way
to capture breaking changes before they end up in
production.

12.
The way out of staging environments
1. Remove staging environments for “read-only”
operations/domains (e.g. product catalog, inventory,
navigation);
2. Use sandbox / test accounts for everything else
https://mobile-api.hbc.com/organizations/saks/wishlists/...
https://mobile-api.hbc.com/organizations/test/wishlists/...
1. It’s fine to hit production with traffic coming from tests.

13.
Developer-friendly production env
● Make code changes
● Create a Pull Request and get it reviewed
● Merge Pull Request
● Run unit tests
● Run integration tests - against production
● New release
● Validate / Showcase in dark
● Canary deployment

14.
Shipping new features with microservices
N services x 1 env (prod); 1 team
Deploy: daily
Ship new features: weeks
Troubleshoot: hours

15.
What could possibly go wrong?
Not much, if you make sure to have:
● Very good monitoring
● Easy and quick rollback

16.
A Happy Place for Developers
● Own every phase of the SDLC.
● Release a product independently from other services the
product depends upon.
● Test with confidence without the need for staging
environment or stubbed data.
● Deploy at will.