DevOps has grown in popularity in recent years, particularly in software companies that want to reduce lead time to be measured in days/weeks instead of months/years. If you want to make sure your software does the right things and does things right, you need to test it implacably. The big problem is that companies see the testing phase as the bottleneck of the process slowing down product release. To change that, we need a new way of testing our applications, making the release process of an application a testing process as well and involving the QA from the beginning within the team. QAs are not a separate team anymore: DevTestOps. How is the testing pyramid affected? In this session, we will not only describe but actively demonstrate several techniques that you can use immediately following the session for starting testing in production and speeding up your release cycle to unimaginable limits. Demo1: https://www.youtube.com/watch?v=3mJX38m-D3A Demo2: https://www.youtube.com/watch?v=IcRsb-boWA4 Demo3: https://youtu.be/F15ZeZuqo54

1. Testing in Production.
From DevTestOops to
DevTestOps
Alex Soto Bueno
@alexsotob

2. @alexsotob
!2
Buenas tardes, buenas noches,
señoritas y señores
To be here with you tonight.
Brings me joy, que alegria
— Miguel
“

3. @alexsotob
!3
Who Am I?
Alex Soto

4. @alexsotob4
I think that is the
most stupidest thing
I ever heard it.
— Gideon Grey
“

5. @alexsotob5
https://puppet.com/resources/whitepaper/state-of-devops-report
High-performing
organizations are decisively
outperforming their lower
performing peers ...

6. @alexsotob6
https://puppet.com/resources/whitepaper/state-of-devops-report
46X46X More Deployments

7. @alexsotob7
https://puppet.com/resources/whitepaper/state-of-devops-report
440X440X Shorter Lead Times

8. @alexsotob8
https://puppet.com/resources/whitepaper/state-of-devops-report
96X96X Mean Time to Recover

9. @alexsotob9

10. @alexsotob10

11. @alexsotob11
Change is good.
— Rafiki
“

12. @alexsotob12
OPS PMO DBASEC
BIZ DEV
QA

13. @alexsotob13
Continuous
integration
and testing
Build
Test
Continuous
delivery and
deployment
Release
Deploy
Continuous
operations
Operate
Monitor
Continuous Improvement
Continuous Improvement

14. @alexsotob14
QA Bottleneck

15. @alexsotob15
I don’t want to survive.
I want to live.
— Captain B. McCrea
“

16. @alexsotob16
We are on Production

17. @alexsotob17
Shit! We break Production Environment

18. @alexsotob18
When you find out
you can live without it,
go along not thinking
about it.
— Baloo
“

19. @alexsotob19
Production is not sacrosanct anymore

20. @alexsotob20
- Unit Tests
- Component Tests
- Static Analysis
- Coverage Tests
- Benchmark Tests
- Contract Tests
- Acceptance Tests
- Mutation Tests
- Smoke Tests
- UI/UX Tests
- Penetration Tests
- Integration Tests
- Tap Compare
- Load Tests
- Shadowing
- Config Tests
- Canarying
- Dark Canaries
- Monitoring
- Feature Flagging
- Exception Tracking
- Feature Graduation
- Teeing
- Profiling
- Logs
- Chaos Testing
- Monitoring
- A/B Testing
- Tracing
- Auditing
- OnCall Experience
- Journey tests
Cindy Sridharan
Pre-Production
Testing In Production
Deploy Release Post Release
The New ¿Pyramid?

21. @alexsotob21
There is a lot of grey area in
make me a prince.
— Gene
“

22. @alexsotob22
Regression-free versions (tap compare)

23. @alexsotob23
Diferencia

24. @alexsotob24
Simple Comparison

25. @alexsotob25
Blue-Green Deployments

26. Blue/Green Deployment
CI/CD PRODUCTION ROUTER USERS
BUILD
SCM

27. Blue/Green Deployment
CI/CD PRODUCTION ROUTER USERS
BUILDSCM

28. Blue/Green Deployment
CI/CD PRODUCTION ROUTER USERS
SCM

29. Blue/Green Deployment
CD/CD PRODUCTION ROUTER USERS
SCM

30. @alexsotob30
Canary Release

31. @alexsotob31
CI/CD PRODUCTION ROUTER USERS
SCM

32. @alexsotob32
CI/CD PRODUCTION ROUTER USERS
SCM

33. @alexsotob33
CI/CD PRODUCTION ROUTER USERS
SCM

34. @alexsotob34
CI/CD PRODUCTION ROUTER USERS
SCM

35. @alexsotob35
CI/CD PRODUCTION ROUTER USERS
SCM

36. @alexsotob36
CI/CD PRODUCTION ROUTER USERS
SCM

37. @alexsotob37
CI/CD PRODUCTION ROUTER USERS
SCM

38. @alexsotob @AndyGeeDe38
Dark Launches

39. @alexsotob39
DEVELOPMENT QA STAGING PRODUCTION ROUTER USERS
SCM
INTERNAL USERS
Dark Canaries

40. @alexsotob40
DEVELOPMENT QA STAGING PRODUCTION ROUTER USERS
SCM
Shadowing Traffic

41. @alexsotob41
Feature Graduation

42. @alexsotob42

43. @alexsotob43

44. @alexsotob44
This is the circle of sadness.
Your job is to make sure
that all sadness stays
inside of it.
— Joy
“

45. @alexsotob45
It's time to see
what I can do
To test the limits
and break through.
— Elsa
“

46. @alexsotob46
Istio — ‘Sail’
(Kubernetes — the ‘Helmsman’)

47. @alexsotob47
Put on your Sunday clothes
there's lots of world
out there.
— Wall-E
“
[http://bit.ly/istio-tutorial]
[http://bit.ly/istio-cheat-sheet-v1]

48. @alexsotob48
Every adventure
requires a first step.
— Alice
“

49. @alexsotob49
Now,
you see, the world
is full of temptations .
— Jimmy Cricket
“

50. @alexsotob50
If I go there is just no telling
how far I’ll go.
— Moana
“

51. @alexsotob
Hay un amigo en mí,
cuando salgan a volar,
hay un amigo en mí
— Toy Story
“
@alexsotob
asotobue@redhat.com
http://www.lordofthejars.com/
lordofthejars

52. @alexsotob52
About Images
http://disney.wikia.com/wiki/File:TLK_Simba.png
http://disney.wikia.com/wiki/File:Tiana.12.png
http://insideout.wikia.com/wiki/File:FEAR_Fullbody_Render.png
http://disney.wikia.com/wiki/File:Judy_Hopps_Zootopia.png
https://mbtskoudsalg.com/explore/disney-up-png/#gal_post_3621_up-balloons-png.png
http://parody.wikia.com/wiki/File:Jiminy_cricket_disney.png
http://zootopia.wikia.com/wiki/File:Gideon_Adult_Transprent.png
http://lifeofheroesrp.wikia.com/wiki/File:Aladdin.png
http://disney.wikia.com/wiki/Captain_B._McCrea
http://pixar.wikia.com/wiki/Slim
http://pixar.wikia.com/wiki/Auguste_Gusteau
http://hero.wikia.com/wiki/File:Joy_(Inside_Out).png
http://disney.wikia.com/wiki/File:Diamond_baloo.png
https://www.flickr.com/photos/photogreuhphies/7383113142/in/photolist-cfqojb-pC8jJB-6hCtMr-a4hoxN-9gPWs9-6EHZzs-dV9aSt-81JzLb-85TxfK-qsw4px-81JAH3-9LBbLF-kAeQJ
qdoArQ-pC7bTE-cDWcPJ-doZAgW-aZhdma-oFEs5b-a1865x-23TdcqE-7oLqCo-8TWx33-9Aazdw-5hFSPs-pUCGJQ-oGPNnf-pCa8P9-bAdBVK-51ooTn-nsZnCK-bvV1nu-92f6Ys-p1rKvB-
http://www.stickpng.com/img/comics-and-fantasy/the-incredibles/edna-e-mode
https://www.kisspng.com/png-dory-nemo-palette-surgeonfish-disney-infinity-3-0-916908/
https://www.kisspng.com/png-frozen-olaf-s-quest-elsa-kristoff-anna-anna-frozen-825204/
https://www.kisspng.com/png-eve-wii-wallpaper-wall-e-png-photos-421264/
https://www.kisspng.com/png-rapunzel-free-png-image-63202/
https://www.kisspng.com/png-ratatouille-french-cuisine-film-animation-pixar-ra-631822/
https://www.kisspng.com/png-peter-pan-tinker-bell-wendy-darling-clip-art-fly-655477/
https://gallery.yopriceville.com/Free-Clipart-Pictures/Cartoons-PNG/Transparent_Snow_White_PNG_Clipart#.W_PigJNKgnU
https://www.kisspng.com/png-princess-jasmine-aladdin-genie-youtube-abu-aladdin-961044/
https://www.kisspng.com/png-a-bug-s-life-flik-youtube-film-streaming-media-bug-2538893/
https://www.kisspng.com/png-buzz-lightyear-jessie-toy-story-sheriff-woody-tim-103824/
https://www.kisspng.com/png-alice-s-adventures-in-wonderland-the-mad-hatter-wh-689107/
https://www.kisspng.com/png-russell-youtube-carl-fredricksen-pixar-film-ups-1441690/
http://disney.wikia.com/wiki/File:Bambi_-_Render.png
https://www.clipartmax.com/middle/m2H7H7Z5m2i8G6G6_joy-upside-down-transparent-png-stickpng-inside-out-inside-out-characters-sadness/
https://www.kisspng.com/png-bolt-ariel-the-walt-disney-company-walt-disney-ani-6179624/
https://www.clipartmax.com/middle/m2H7i8b1b1m2A0Z5_4-png-toy-story-slinky-png/
https://dlpng.com/png/115503
https://clipart.info/elsa-frozen-disney-png-6355
http://www.pngall.com/ariel-png
CC 3
Not Commercial
Usage
Not Commercial
Usage
CC 4

53. @alexsotob53
“You’re no one hero.” –Moana (showing simple
things)
“If I go there is just no telling how far i’ll go” –
Moana (showing simple things)