SlideShare a Scribd company logo
1 of 35
Download to read offline
Enrich Your Testing
Portfolio
Me
● Independent consultant
● Backend & Node.js only
● Author of =>
● www.GoldbergYoni.com
● https://github.com/i0natan/no
debestpractices
Problem: an ancient pyramid drives our
modern testing strategy
● Top R&D $$$ are assigned to quality and testing
● How we invest that money? Based on 10 years old model
2014: It started with some provocative
tweets
2018: Leading voices put some doubt
Issue 1: Unit test has lower ROI (per unit)
● Would you paint a house using a mascara brush?
● Focus on exotic cases vs the critical path
Unit
Test
Integration
Tests
Issue2: the sum is greater than the parts
● Integration is often where the dead bodies are covered
● “But it worked on my machine” - Poke holes in reality
?
??
Issue2: the sum is greater than the parts
● Integration is often where the dead bodies are covered
● “But it worked on my machine” - Poke holes in reality
?
?
??
Issue 3: E2E tests are fragile & expensive
● Get confidence for an outrageous prices
● Re-build the entire AWS on your machine is tedious
Issue 4 : One technique fits all?
● Many different architecture - Data pipeline, UI meshup, deep learning
models, IOT systems, etc, etc
● Tests need to solve real-world, not theoretic problems
● Where would you write unit tests here?
Solution #1: The Testing Trophy
Solution #2: Other modernized pyramids
Spotify pyramid Testing quadrat
Solution #3: A rich testing portfolio
● Contextualize - suit tools for problems
● Diversify - rich and holistic approach
The portfolio #1: Component testing
● Problem: Finding the sweet
spot between Unit and E2E
● Good for: Independent
microservices
● Tools: HTTP interceptor (Nock),
API tester (supertest)
● Can replace: Unit tests
Component testing - why I just love it
● Test what you deliver
● Use real internal resources, like db, Stub external collaborators
● Test entire flows -> Better confidence
● Less IO -> Better speed
Component testing: Stub the externals,
boost the internals
Stub the
externalsBoost the
internals
The portfolio #2: Consumer driven contracts
● Problem: many consumers - many
API versions, Swagger is static,
consumers don’t read the manual
● Good for: Integrations with
complex and versioned API, multi-
team microservices
● Framework: PACT
● Can replace: E2E tests
Consumer driven contract: the flow
PACT: The consumer expectations from the
provider microservice
The provider verification code
View integration status within the broker
The portfolio #3: Property-based testing
● Problem: Traditional testing is based
on selective and small set of input
● Good for: Loosely-typed APIs,
serendipity
● Tools: jsverify, testcheck-js
● Can replace/improve: Unit &
integration tests
Traditional testing -> manually code
every input
Property based testing -> automatically
invokes every possible permutation
Property based testing inside test case - example
Invoked x
times
Property based testing demands a shift of mind
Property based testing are great for:
● Generalize your expectations
○ E.g. Never get a price lower than 0
● Validating generic health concerns
○ E.g. HTTP returns only 200, 204, 500
○ Our code throws only valid Errors
○ Process never crash
○ No DDOS
● Great match for loose API
○ E.g. search systems, IOT
● Prepare the ground for unit testing
The portfolio #4: Mutation-based testing
● Problem: Testing coverage
doesn’t measure the test
effectiveness
● Good for: Revealing bugs in
logic-heavy apps
● Tools: Stryker
● Can improve: Unit & integration
test
83% coverage - isn’t this great?
83% coverage and 0 expectations. This isn’t great
Stryker can measure test efficiency
The portfolio #5: Chaos testing
● Problem: Bad things might
come from outside
● Good for: Companies that value
resilience
● Tools: Netflix swiss army knife,
kube-monkey, node-chaos-
monkey
● Can improve: E2E tests
The typical container-killer monkey
Node.js in-process chaos tool
Wrapping up
● Some testing tools suit your needs better
● Lean testing is a great companion for lean teams
● Wanna benefits all these goods? Enrich your testing portfolio
P.s. we haven’t discussed ‘Acceptance testing’, ‘VCR testing’, ‘Database
testing’, ‘Shadow testing’ and others

More Related Content

Recently uploaded

Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Nikki Chapple
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Karmanjay Verma
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditSkynet Technologies
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Jeffrey Haguewood
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 

Recently uploaded (20)

Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance Audit
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 

Yoni Goldberg - enrich your node.js testing portfolio

  • 2. Me ● Independent consultant ● Backend & Node.js only ● Author of => ● www.GoldbergYoni.com ● https://github.com/i0natan/no debestpractices
  • 3. Problem: an ancient pyramid drives our modern testing strategy ● Top R&D $$$ are assigned to quality and testing ● How we invest that money? Based on 10 years old model
  • 4. 2014: It started with some provocative tweets
  • 5. 2018: Leading voices put some doubt
  • 6. Issue 1: Unit test has lower ROI (per unit) ● Would you paint a house using a mascara brush? ● Focus on exotic cases vs the critical path Unit Test Integration Tests
  • 7. Issue2: the sum is greater than the parts ● Integration is often where the dead bodies are covered ● “But it worked on my machine” - Poke holes in reality ? ??
  • 8. Issue2: the sum is greater than the parts ● Integration is often where the dead bodies are covered ● “But it worked on my machine” - Poke holes in reality ? ? ??
  • 9. Issue 3: E2E tests are fragile & expensive ● Get confidence for an outrageous prices ● Re-build the entire AWS on your machine is tedious
  • 10. Issue 4 : One technique fits all? ● Many different architecture - Data pipeline, UI meshup, deep learning models, IOT systems, etc, etc ● Tests need to solve real-world, not theoretic problems ● Where would you write unit tests here?
  • 11. Solution #1: The Testing Trophy
  • 12. Solution #2: Other modernized pyramids Spotify pyramid Testing quadrat
  • 13. Solution #3: A rich testing portfolio ● Contextualize - suit tools for problems ● Diversify - rich and holistic approach
  • 14. The portfolio #1: Component testing ● Problem: Finding the sweet spot between Unit and E2E ● Good for: Independent microservices ● Tools: HTTP interceptor (Nock), API tester (supertest) ● Can replace: Unit tests
  • 15. Component testing - why I just love it ● Test what you deliver ● Use real internal resources, like db, Stub external collaborators ● Test entire flows -> Better confidence ● Less IO -> Better speed
  • 16. Component testing: Stub the externals, boost the internals Stub the externalsBoost the internals
  • 17. The portfolio #2: Consumer driven contracts ● Problem: many consumers - many API versions, Swagger is static, consumers don’t read the manual ● Good for: Integrations with complex and versioned API, multi- team microservices ● Framework: PACT ● Can replace: E2E tests
  • 19. PACT: The consumer expectations from the provider microservice
  • 21. View integration status within the broker
  • 22. The portfolio #3: Property-based testing ● Problem: Traditional testing is based on selective and small set of input ● Good for: Loosely-typed APIs, serendipity ● Tools: jsverify, testcheck-js ● Can replace/improve: Unit & integration tests
  • 23. Traditional testing -> manually code every input
  • 24. Property based testing -> automatically invokes every possible permutation
  • 25. Property based testing inside test case - example Invoked x times
  • 26. Property based testing demands a shift of mind
  • 27. Property based testing are great for: ● Generalize your expectations ○ E.g. Never get a price lower than 0 ● Validating generic health concerns ○ E.g. HTTP returns only 200, 204, 500 ○ Our code throws only valid Errors ○ Process never crash ○ No DDOS ● Great match for loose API ○ E.g. search systems, IOT ● Prepare the ground for unit testing
  • 28. The portfolio #4: Mutation-based testing ● Problem: Testing coverage doesn’t measure the test effectiveness ● Good for: Revealing bugs in logic-heavy apps ● Tools: Stryker ● Can improve: Unit & integration test
  • 29. 83% coverage - isn’t this great?
  • 30. 83% coverage and 0 expectations. This isn’t great
  • 31. Stryker can measure test efficiency
  • 32. The portfolio #5: Chaos testing ● Problem: Bad things might come from outside ● Good for: Companies that value resilience ● Tools: Netflix swiss army knife, kube-monkey, node-chaos- monkey ● Can improve: E2E tests
  • 35. Wrapping up ● Some testing tools suit your needs better ● Lean testing is a great companion for lean teams ● Wanna benefits all these goods? Enrich your testing portfolio P.s. we haven’t discussed ‘Acceptance testing’, ‘VCR testing’, ‘Database testing’, ‘Shadow testing’ and others

Editor's Notes

  1. Goal: It’s about confidence Spice: True story about WU TOC: We test for confidence Story where I spent time Maybe start with the critical path?
  2. Goal: Contextualize - different needs Spice: Have you heard about the montioring pyramid TOC: TBD
  3. Goal: Like it but no context and diversify Spice: Static is very rich, license check, complexity check TOC: TBD
  4. P1: add pyramids
  5. Goal: Beyond pyramid - rich and contextualize Spice: 99% unit test succeed but then db migration, or browser + let me show you now TOC: From a great blog First it’s rich - anything is reliability, will show example You need to contextualize Separate the lab from reality - each Let me show you
  6. Goal: The sweet spot between two big Spice: What do you do when something is too small and other is too big? TOC: The most promising technique Unit is too small but e2e is too fragile Test
  7. Goal: The sweet spot between two big Spice: What do you do when something is too small and other is too big? TOC: The most promising technique Unit is too small but e2e is too fragile Test
  8. Goal: The right focus and speed Spice: I ran 500 component test with DB in 50 seconds TOC: TBD P1: show stats
  9. Goal: Enjoy TDD, but test everything Spice: In-memory DB TOC: TBD P1: fix code P2: in memory DB
  10. Goal: Understand the swagger problem Spice: Cox2m story with customer complains TOC: TBD
  11. Goal: How the failure we get caught Spice: Fall early TOC: TBD
  12. Goal: the problem of multiple consumers TOC: Spice: Argue but don’t read the manual
  13. Goal: the problem of multiple consumers TOC: Spice: Argue but don’t read the manual
  14. Goal: the problem of multiple consumers TOC: Spice: Can replace E2E testing
  15. Goal: Test the unknown Punch: 1000 permutations, we have energy for 3 TOC: TBD TOC: Unit&end&component are about predictable flows, what if… Spice: ? Quote about best effort How to measure test effective, coverage is a lier (demo) Mutation will highlight when our test are not covering (show coverage) Is it OK now? What is the weakness? Our input... But what if we don’t have time. Or we rely on syntethic input Demo PBT P1: put code
  16. Goal: Manual input can be inefficient Punch: Regex and DB TOC: We work for every input and reach ususally only 3 Regex can fall, DB can fall
  17. Goal: Manual input can be inefficient Punch: What a confidence if it passes 10000 TOC: P2: Put the real code...
  18. Goal: Problem - what to expect? Punch: How can you predict the output if you don’t know the input TOC: Tricky: one test, multiple runs, one failure Tell the types Do you see the problem? P2: add something about shrinking
  19. Goal: critical decision TOC: All tutorials & methodology are sold out, code example, curation, Ruby sticker, Spice:
  20. Goal: Make it useful Punch: Never throw wrong errors or crash TOC:
  21. Goal: Coverage insufficient Punch: A man goes to work Unit&end&component are about predictable flows, what if… Spice: ? Quote about best effort Mutation based testing Coverage is a liar See high coverage but many bugs MBT is about your testing effectiveness, free meal, see How to measure test effective, coverage is a lier (demo) Mutation will highlight when our test are not covering (show coverage) Is it OK now? What is the weakness? Our input... But what if we don’t have time. Or we rely on syntethic input Demo PBT P1: Put code