Misleading validations:
Be aware of green
Anand Ramdeo
Agile Testing Days
2013

anand@TestingGeek.com | Blog: TestingGeek....
Disclaimer - Say no to

!

Test Automation
ATDD / BDD
Continuous Integration
Continuous Delivery

anand@TestingGeek.com | ...
Embrace
Test Automation
ATDD / BDD
Continuous Integration
Continuous Delivery

“To exist is to change,
to change is to mat...
Embrace
Test Automation
ATDD / BDD
Continuous Integration

With
Caution!!

Continuous Delivery

One line can change it all...
The Black Swan - Nassim Nicholas Taleb
(NNT)
anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpic...
Mediocristan
Number of test
cases
Defects found
Defects fixed
Time between
failed builds

anand@TestingGeek.com | Blog: Tes...
Extrimistan

One defect can change everything
anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpi...
Software
Testing
Problem to Discover?
or
Problem to Prove?

anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek |...
Right way to develop
Software

anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
9
Men are disturbed not by
things, but by the views which
they take of them.
Epictetus

anand@TestingGeek.com | Blog: Testin...
TDD
Right way
to develop
Software.

ATDD/BDD
Build the
thing PO
wants.

ATDD + TDD = Happy Customer?
anand@TestingGeek.com...
Information or
Theory?
From
Red is bad, green is good.
To
It’s Green, so it must be good.

anand@TestingGeek.com | Blog: T...
Be aware of confirmation bias, understand what green
really means.

anand@TestingGeek.com | Blog: TestingGeek.com | @testin...
So I have a theory
(Green is Good)

and
I am suffering from confirmation bias
(Pre-defined tests are passing in CI)

but
Wha...
Tunnel Vision
What are we delivering in this
Sprint?

anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play:...
System view
is extremely
Important.

anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
1...
Lucid Fallacy
100% code coverage
Continuous Integration in
place
Mandatory code reviews
Acceptance tests are part of
the b...
First Valentine

Second Valentine

Third Valentine

With every passing year,
we are getting closer :-)

anand@TestingGeek....
Marriage!!
anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
19
So I have a theory
(Green is Good)

and
I am suffering from confirmation bias
(Pre-defined tests are passing in CI)

and
Tea...
Project was progressing well and delivered on time,
however customer is not happy.

OR
Project was progressing well and de...
So I have a theory
(Green is Good)

and
I am suffering from confirmation bias
(Pre-defined tests are passing in CI)

and
Tea...
Be aware
Our job is to prevent
teams from taking uninformed risks.

anand@TestingGeek.com | Blog: TestingGeek.com | @testi...
So does green work?

anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
24
The value of any
practice depends on its
context.

anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: Te...
Questions / Discussions?
Given
That time runs out or participants are bored
When
Speaker have nothing more to say or can n...
Upcoming SlideShare
Loading in …5
×

Misleading validations - Be aware of green

1,478 views

Published on

Millions of test cases executed thousands of times mean nothing when a catastrophic defect surfaces and threatens the value of the product. In software testing, what is not tested is more important than what has been tested. However, with continuous adoption of agile and automation, the focus has shifted to what has been tested. In an agile environment, where results of continuous integration are visible to everyone, it’s easy to get fooled.

In this talk, I will explore how validations we seek with testing are affected by fallacies and biases and why green reported by continuous integration might not be good enough. I will try to demonstrate limitations of the acceptance criteria by showing examples of defects in open-source projects.

I will conclude with the discussion on how a context-driven approach can safe-guard us from the inferences we may make from the green build. If teams are not context-driven, all the benefits we see from adopting agile, automation and continuous integration can be short-lived. Projects are not judged by the presence of automation or green build; projects are judged by how they work in production. Remember the case of Ariane 501, where the project was progressing well and the team celebrated the first 36 seconds of the launch. However, the spacecraft crashed in the 37th second and the project became unsuccessful.

This paper is based on an excellent book by Nassim Nicholas Tale – “The Black Swan” and relates limitations of validation to software testing.

Published in: Technology, News & Politics
  • Be the first to comment

  • Be the first to like this

Misleading validations - Be aware of green

  1. 1. Misleading validations: Be aware of green Anand Ramdeo Agile Testing Days 2013 anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 1
  2. 2. Disclaimer - Say no to ! Test Automation ATDD / BDD Continuous Integration Continuous Delivery anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 2
  3. 3. Embrace Test Automation ATDD / BDD Continuous Integration Continuous Delivery “To exist is to change, to change is to mature, to mature is to go on creating oneself endlessly” Henri Bergson anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 3
  4. 4. Embrace Test Automation ATDD / BDD Continuous Integration With Caution!! Continuous Delivery One line can change it all. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 4
  5. 5. The Black Swan - Nassim Nicholas Taleb (NNT) anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 5
  6. 6. Mediocristan Number of test cases Defects found Defects fixed Time between failed builds anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 6
  7. 7. Extrimistan One defect can change everything anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 7
  8. 8. Software Testing Problem to Discover? or Problem to Prove? anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 8
  9. 9. Right way to develop Software anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 9
  10. 10. Men are disturbed not by things, but by the views which they take of them. Epictetus anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 10
  11. 11. TDD Right way to develop Software. ATDD/BDD Build the thing PO wants. ATDD + TDD = Happy Customer? anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 11
  12. 12. Information or Theory? From Red is bad, green is good. To It’s Green, so it must be good. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 12
  13. 13. Be aware of confirmation bias, understand what green really means. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 13
  14. 14. So I have a theory (Green is Good) and I am suffering from confirmation bias (Pre-defined tests are passing in CI) but What about other members in the team? anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 14
  15. 15. Tunnel Vision What are we delivering in this Sprint? anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 15
  16. 16. System view is extremely Important. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 16
  17. 17. Lucid Fallacy 100% code coverage Continuous Integration in place Mandatory code reviews Acceptance tests are part of the build anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 17
  18. 18. First Valentine Second Valentine Third Valentine With every passing year, we are getting closer :-) anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 18
  19. 19. Marriage!! anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 19
  20. 20. So I have a theory (Green is Good) and I am suffering from confirmation bias (Pre-defined tests are passing in CI) and Team members have the same tunnel vision (Focus is on current sprint) and We are affected by lucid fallacy (No red build from past seven days!!) but Surely, we can learn from failures, when it fails. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 20
  21. 21. Project was progressing well and delivered on time, however customer is not happy. OR Project was progressing well and delivered on time, however customer is not happy because upgraded version corrupted their existing data. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 21
  22. 22. So I have a theory (Green is Good) and I am suffering from confirmation bias (Pre-defined tests are passing in CI) and Team members have the same tunnel vision (Focus is on current sprint) and We are affected by lucid fallacy (No red build from past seven days!!) and We can’t learn from failure because of narrative fallacy (Something failed in production, investigate, fix, root cause analysis etc.) anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 22
  23. 23. Be aware Our job is to prevent teams from taking uninformed risks. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 23
  24. 24. So does green work? anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 24
  25. 25. The value of any practice depends on its context. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 25
  26. 26. Questions / Discussions? Given That time runs out or participants are bored When Speaker have nothing more to say or can not answer Than Speaker say thank you for your time and attentions. “Please evaluate my presentation and use for this the AgileTD Mobile App which you can find atwww.touchmyconference.com/ATD2013. I would appreciate your feedbacks. Thank you very much!” anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 26

×