The document discusses implementing QA in production to enhance testing efficiency within an agile software development environment. It highlights the challenges faced with traditional QA methods and proposes solutions like canary releasing and data-driven bug fixing to improve release cycles and bug prioritization. The results demonstrate a significant increase in consistency rates and overall confidence in software releases.

1. Testing in
Production
make your testing process more efficient
without the risks
Istvan Marhefka
Independent Agile Coach, Software Architect
https://istvanmarhefka.com
hello@istvanmarhefka.com
PalinQA meetup, 26/2/2019

2. About the project
❖ Dropbox-clone
❖ 120k+ monthly active users
❖ Product development
❖ International team, startup-like organisation in a large enterprise
❖ Agile software development (TDD, XP, CI/CD, SCRUM)

3. Technology stack
❖ Desktop client (Win & Mac)
❖ Electron (Angular, Typescript)
❖ Java (Sprint Boot, Maven)
❖ Backend: CloudFoundry (PaaS), MongoDB

4. QA process (before)
1. GitFlow (branching model)
2. Code reviews
3. Unit/integration tests (hermetic tests) - commit stage
4. Exploratory tests (TEST env.)
5. Crowdtesting (TEST env.)
6. Release (every 2nd week, SCRUM)

5. Challenges
1. Different OS versions, client configurations, race conditions, timings
2. Impact of a bug?
3. Crowd testing - delay, questionable results
4. How long should we test?
5. Bugs are hard to notice can cause serious issues
we need data from PROD usage
we need data from PROD usage
do we really need it?
how should we know about the existence of those bugs?
we need to “test” further after PROD release

6. Solution: QA in Production
Source: https://martinfowler.com/articles/qa-in-production.html

7. Preparations
for QA in Production
❖ Being able to quickly deploy a new version
❖ Shortening delivery cycle from 2 weeks to 1 week
❖ Automated silent updates
❖ Trunk-based development (no more Gitflow)
❖ Testing against production systems

8. Implementation of
QA in Production

9. Canary releasing
“… slowly rolling out the change to a small
subset of users before … making it
available to everybody.”
Source: https://martinfowler.com/bliki/CanaryRelease.html

10. Canary releasing
❖ Each commit is a release candidate
❖ every commit: employees, friendly users (automatically, if tests passed)
❖ every release:
❖ first 1% of the users (manual trigger)
❖ full rollout (manual trigger)

11. Collecting and analysing errors (and
other data)

12. KPI: consistency rate
Type to enter a caption.

14. Data-driven bugfixing
❖ Impact is known, no more guesses
❖ prioritization is clear

15. Results
❖ Consistency rate goes up to 97% (from 60%)
❖ release cycle of 1 week
❖ stable releases
❖ much more confidence on existing bugs, no more guessing
❖ clear prioritization, less stress
❖ SCRUM: reviews had less importance

16. Change in mindset
❖ Convincing management (Testing in Production??)
❖ Trust is needed towards the development team
❖ Traditional QA
❖ Different release process than in other products (Gitflow, weekly releases)
❖ More responsibility from team members (monitoring & analyzing production
app, implementing custom tools)
❖ Discipline within the team

17. Can you apply it in your situation?
❖ How do you know if your application works as expected in PROD?
❖ What could be your key metrics?
❖ What could you learn from QA in production?

18. What do you think?
Istvan Marhefka
Independent Agile Coach, Software Architect
https://istvanmarhefka.com
hello@istvanmarhefka.com