Your SlideShare is downloading. ×
Misleading validations - Be aware of green
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Misleading validations - Be aware of green

885
views

Published on

Millions of test cases executed thousands of times mean nothing when a catastrophic defect surfaces and threatens the value of the product. In software testing, what is not tested is more important …

Millions of test cases executed thousands of times mean nothing when a catastrophic defect surfaces and threatens the value of the product. In software testing, what is not tested is more important than what has been tested. However, with continuous adoption of agile and automation, the focus has shifted to what has been tested. In an agile environment, where results of continuous integration are visible to everyone, it’s easy to get fooled.

In this talk, I will explore how validations we seek with testing are affected by fallacies and biases and why green reported by continuous integration might not be good enough. I will try to demonstrate limitations of the acceptance criteria by showing examples of defects in open-source projects.

I will conclude with the discussion on how a context-driven approach can safe-guard us from the inferences we may make from the green build. If teams are not context-driven, all the benefits we see from adopting agile, automation and continuous integration can be short-lived. Projects are not judged by the presence of automation or green build; projects are judged by how they work in production. Remember the case of Ariane 501, where the project was progressing well and the team celebrated the first 36 seconds of the launch. However, the spacecraft crashed in the 37th second and the project became unsuccessful.

This paper is based on an excellent book by Nassim Nicholas Tale – “The Black Swan” and relates limitations of validation to software testing.

Published in: Technology, News & Politics

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
885
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Misleading validations: Be aware of green Anand Ramdeo Agile Testing Days 2013 anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 1
  • 2. Disclaimer - Say no to ! Test Automation ATDD / BDD Continuous Integration Continuous Delivery anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 2
  • 3. Embrace Test Automation ATDD / BDD Continuous Integration Continuous Delivery “To exist is to change, to change is to mature, to mature is to go on creating oneself endlessly” Henri Bergson anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 3
  • 4. Embrace Test Automation ATDD / BDD Continuous Integration With Caution!! Continuous Delivery One line can change it all. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 4
  • 5. The Black Swan - Nassim Nicholas Taleb (NNT) anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 5
  • 6. Mediocristan Number of test cases Defects found Defects fixed Time between failed builds anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 6
  • 7. Extrimistan One defect can change everything anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 7
  • 8. Software Testing Problem to Discover? or Problem to Prove? anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 8
  • 9. Right way to develop Software anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 9
  • 10. Men are disturbed not by things, but by the views which they take of them. Epictetus anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 10
  • 11. TDD Right way to develop Software. ATDD/BDD Build the thing PO wants. ATDD + TDD = Happy Customer? anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 11
  • 12. Information or Theory? From Red is bad, green is good. To It’s Green, so it must be good. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 12
  • 13. Be aware of confirmation bias, understand what green really means. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 13
  • 14. So I have a theory (Green is Good) and I am suffering from confirmation bias (Pre-defined tests are passing in CI) but What about other members in the team? anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 14
  • 15. Tunnel Vision What are we delivering in this Sprint? anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 15
  • 16. System view is extremely Important. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 16
  • 17. Lucid Fallacy 100% code coverage Continuous Integration in place Mandatory code reviews Acceptance tests are part of the build anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 17
  • 18. First Valentine Second Valentine Third Valentine With every passing year, we are getting closer :-) anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 18
  • 19. Marriage!! anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 19
  • 20. So I have a theory (Green is Good) and I am suffering from confirmation bias (Pre-defined tests are passing in CI) and Team members have the same tunnel vision (Focus is on current sprint) and We are affected by lucid fallacy (No red build from past seven days!!) but Surely, we can learn from failures, when it fails. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 20
  • 21. Project was progressing well and delivered on time, however customer is not happy. OR Project was progressing well and delivered on time, however customer is not happy because upgraded version corrupted their existing data. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 21
  • 22. So I have a theory (Green is Good) and I am suffering from confirmation bias (Pre-defined tests are passing in CI) and Team members have the same tunnel vision (Focus is on current sprint) and We are affected by lucid fallacy (No red build from past seven days!!) and We can’t learn from failure because of narrative fallacy (Something failed in production, investigate, fix, root cause analysis etc.) anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 22
  • 23. Be aware Our job is to prevent teams from taking uninformed risks. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 23
  • 24. So does green work? anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 24
  • 25. The value of any practice depends on its context. anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 25
  • 26. Questions / Discussions? Given That time runs out or participants are bored When Speaker have nothing more to say or can not answer Than Speaker say thank you for your time and attentions. “Please evaluate my presentation and use for this the AgileTD Mobile App which you can find atwww.touchmyconference.com/ATD2013. I would appreciate your feedbacks. Thank you very much!” anand@TestingGeek.com | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com 26

×