TestTribe - Testing techniques in business-critical environments

1. © 2023 Thoughtworks | Confidential
Unconventional approaches for
testing in fast-paced,
business-critical environments
Pooja Subramanian
Principal Technologist,
Thoughtworks
Kumar Sankara Iyer
Principal Technologist,
Thoughtworks

2. © 2023 Thoughtworks | Confidential
Unconventional approaches for testing in
2
fast-paced, business-critical
environments

3. © 2023 Thoughtworks | Confidential
What’s unique about
these businesses?
3
● They are digital-first
● They thrive in extremely
competitive landscapes
● Features need to be put in
production rapidly
● Customer experience is of utmost
importance

4. © 2023 Thoughtworks | Confidential 4
But then, the path to
production is not an
easy one!

5. © 2023 Thoughtworks | Confidential
Presenting to you - the value stream..
5
Plan
Code
Validate
Pre-Deploy
Tests
Package
Deploy
Post-Deploy
Tests
Release
Monitor
Trunk-based
Development
Code Style
Unit Testing
Immutable
Artifacts
Ephemeral
Environment
Provisioning
Automated
Integration
Testing
Backwards
Compatibility
Testing
Reporting &
Dashboards
Lightweight
Architecture
Governance
IDE Integration
SAST
Contract
Testing
Image
Scanning
Toggle
Configuration
Automated
Functional
Testing
Production
Verification
Testing
Budget
Management
Developer
Experience
Local Dev
Environments
Static Code
Analysis
Architecture
Fitness
Functions
Artifact
Versioning
Application
Configuration
Automated
Performance
Testing
Sign-offs
Metrics
Pre-commit
Hooks
Software
Composition
Analysis
Standalone
Component
Testing
Secrets
Configuration
Manual
Exploratory
Testing
RASP
Templates &
Quickstarts
Automated
Code Reviews
Mutation
Testing
Service
Virtualization
Chaos Testing
Hotfix
Test-Driven
Design
API
Documentation
Tests
Test Data
Management
Automated
Compliance
Testing
Disaster
Recovery
Refactoring
Schema
Migrations
Zero Downtime
Deployments
Developer
Onboarding
Incident
Management
DAST
Structured
Logging
Zero Impact
Deployments
Distributed
Tracing
Micro
Benchmarking
Lightweight
Domain
Modeling
Release Notes
Synthetic
Monitoring
Synthetic
Monitoring
IAST
Threat
Modeling
Automated
Rollbacks
Accessibility
Tech Debt
Management
Pair
Programming

6. © 2023 Thoughtworks | Confidential
What are they
doing differently?
6

7. © 2023 Thoughtworks | Confidential 7
Gain real-world
insights, boost
confidence, and
iterate faster by
testing in production

8. © 2023 Thoughtworks | Confidential
So should you just test
in production?
8
Yes, as long as:
● There is safety
● There is monitoring
● You complement it with all some of
your existing checks

9. © 2023 Thoughtworks | Confidential
Enter canary testing…
9

10. © 2023 Thoughtworks | Confidential
Canary testing can
minimize unforeseen
negative impact on
end-customers
10
● New features rolled out to a subset
of users
● Actively monitor for any
performance, stability or
functional issues
● Gradually release to a wider
audience as the confidence on the
quality increases

11. © 2023 Thoughtworks | Confidential
The case for parallel
experimentation
11
● Mimic-ing test data is difficult
● Too many parameters to control
● Cannot afford to have any impact
to revenue and other KPIs

12. © 2023 Thoughtworks | Confidential
Forward all requests
Enter Replay testing…
Replay the transactions as and when
they occur. Or store them somewhere
and replay at leisure.
12
Disconnect the responses
Adjust downstreams
Cut-over when “done”

13. © 2023 Thoughtworks | Confidential
Canary Testing vs Replay Testing
Canary
13
Proxy
Existing
New
Replay
Proxy
Existing
New

14. © 2023 Thoughtworks | Confidential 14
Use observability to
measure the quality
of the software
● It is a lot more than traditional
monitoring
● Helps pinpoint issues proactively
● Particularly valuable in complex,
distributed systems, microservices
architectures

15. © 2023 Thoughtworks | Confidential
“If you can’t measure it,
you can’t improve it”
15
Lord Kelvin
15
© 2023 Thoughtworks Photo by Messrs. Dickinson, London, New Bond Street

16. © 2023 Thoughtworks | Confidential
Beware of
Observability
Overload!
16

17. © 2023 Thoughtworks | Confidential 17
Shift-left
performance testing
to eliminate pressure
points earlier on

18. © 2023 Thoughtworks | Confidential
Load tests shifted left
18

19. © 2023 Thoughtworks | Confidential 19
Canary
testing
Replay
testing
Observability
driven testing
Shift-
Left
performance
19
Recap of testing
techniques discussed
so far…
19
© 2022 Thoughtworks

20. © 2023 Thoughtworks | Confidential
Some honourable
mentions too…
20
● Multi-region testing
● Rollback testing
● Snapshot testing
● Chaos testing

21. © 2023 Thoughtworks | Confidential
You don’t have to spend a lot of time
trying to mimic all prod data
How can these
techniques help
you??
21
Quicker time to market due to reduced
time spent in earlier stages
Comprehensive, meaningful
dashboards reduce negative user
impact significantly
We have also used these for testing
certain subsystems that require fast
time to market

22. © 2023 Thoughtworks | Confidential
Questions?
22