In this talk I want to share our own experience testing on prod. Our users help us test and customize our algorithms and do not even know about it! How it works? Let’s discuss…

1. Testing on prod
(without pain and humiliation)
Aliaksei Boole, Juno

2. About the
speaker
Engineer.
5 years of experience.
Worked in:
● Bel. Railway
● Synesis
● Wargaming
I test GIS in Juno.

3. Our test
strategy
We don't perform
testing
on prod!

4. Hmmm...

5. All dream about
factory Feature
Analys
Feature
Development
Feature
Testing
Idea
Feature
Deploy

6. You can't turn
the conveyor Feature
Analys
Feature
Development
Feature
Testing
Idea
Feature
Deploy

7. You can't turn
the conveyor

8. You couldn’t fix
issue without
testing Feature
Analys
Feature
Development
Feature
Testing
Idea
Feature
Deploy
Fix

9. You couldn’t fix
issue without
testing

10. Conclusion
1. You already lost. Bug on prod!
2. You need to roll back the code ASAP
3. You could already have damaged
the data
4. You can make things worse

11. THANK YOU! :)
Алексей Буль (Aliaksei Boole)
aliaksei.boole@gmail.com
http://www.electrictower.ru
Twitter:
@Alex_Koul

12. We should use
another
engineering
Standard
approaches
do not work!

13. Feature flags
https://martinfowler.com/articles/feature-toggles.html
https://featureflags.io/
import yaml
class Feature:
def __init__(self):
with open('features.yaml', 'r') as cfg_f:
self.features = yaml.load(cfg_f)
def is_enabled(self, feature_name):
return self.features[feature_name]
...
if Feature().is_enabled("SEND_TO_ML"):
send_to_ml(data)

14. Canary release
https://martinfowler.com/bliki/CanaryRelease.html
splitter100
25
75
● Id
● Split rule

15. Traffic
replication
Prod
RC
DB
DB
Traffic
Prod
RC
DB
DB
Traffic
Balancer
A)
B)
Diffy: Testing services without writing tests

16. Parallel
execution
APP
DB
Traffic
Prod
RC
DB
Traffic
/x
Balancer
A)
B)
Prod
RC dev/null

17. Option for the
poor
Prod
Stage
DB
Traffic
Tester

18. Hmmm...

19. Metrics and
logs
APP
DB
Traffic
Prod
RC dev/null

20. Our experience
1. Feature flags
2. Canary release
3. Parallel execution
4. Awesome metrics

21. Canary release
16fd2706-8baf-433b-82eb-8c7fada847da
from uuid import UUID
def is_testing_participant(str_uuid, percent):
if percent < 0:
return False
elif percent > 100:
return True
else:
return UUID(str_uuid).node % 100 < percent

22. Our sheme
APP
Coordinates
Prod
RC
APP 2
CDRs

23. Metrics
1. Business (long term)
2. System (medium term)
3. Algorithm specific (short term)

24. Business
metrics
● Providers online
● Drivers online
● Request Rides
● Completed rides
● etc

25. System metrics
● Goroutines
● Memory usage
● GC duration
● Open connections
● RPS
● Requests duration
● etc

26. Algorithm
specific
● Snapping time
● Errors count
● One minute utilization
● Reroutes
● Coordinates flow

27. Right factory
Feature
Analys
Feature
Development
Feature
Testing
Idea
Feature
Deploy
Fix Fix Fix

28. Links
1. AWESOME IMAGE!!!!!!11111
2. Testing in production: Yes, you can (and should)
3. Dodo Pizza QA Meetup
4. Testing in Production? Why it's a good idea
5. Facebook, Amazon, Google, and Microsoft
6. Тесты в продакшене - миф или реальность?

29. THANK YOU! :)
Алексей Буль (Aliaksei Boole)
aliaksei.boole@gmail.com
http://www.electrictower.ru
Twitter:
@Alex_Koul