Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

10 signs your testing is not enough


Published on

Доклад Mitko Mitev на конференции SQA Days-20. 24-26 ноября 2016. Минск

Published in: Education
  • Be the first to comment

  • Be the first to like this

10 signs your testing is not enough

  1. 1. 10 signs your testing is not enough
  2. 2. How do we know? How do we know if we tested enough? • Frequently asked question • Fiascos • Assuring the minimum acceptable level of quality is a must • Need to analyze if you should be investing more resources, time, or money in your software testing • “Not enough” vs “Not Good Enough”
  3. 3. Background
  4. 4. Background
  5. 5. Background
  6. 6. Another horror story Knight Capital • Only one defect in the trading algorithm for Knight Capital caused a loss of $ 440 million in about 45 minutes. • That loss is 3 times the company’s annual earnings. • The shock that followed caused Knight Capital’s stock to lose 75% of its value in 1 business day. Infinium shut their algorithm down after just three seconds of trading, but it had already lost more than $1 million.
  7. 7. Lack of understanding Management misunderstanding • We don’t have bugs! • Testing is expensive! • It is an expense • What is the alternative? • Preventive by nature • We don’t need testers! • We do DevOps! • We do Agile! • Testing and Dev – 2 different skill sets
  8. 8. Positive trend Budget for software testing grows
  9. 9. How good is your testing?
  10. 10. 1: Glitches Excessive amount of Production bugs: • Unknown • How long have they been there? 1, 2 or more Test cycles? • Why not discovered earlier? • Not only investing more in testing but also considering changing Testing Strategy • Known – Why? • If found by Customers - Worst from business point of view • Bad PR • May have disastrous effects • Most expensive to fix
  11. 11. 2: Bugs in UAT Too much bugs discovered in Acceptance Testing: • UAT supposed to build confidence in product • Last stage before releasing • goal is to find deviations from intended design • Goal is NOT to find bugs
  12. 12. 3: Bug counts grow Are you aware of such trends? • If bug counts grow you should analyze asap to find reasons • If you don’t take measures your project/product might go in a Death Spiral But do you have the Metrics to see trends? • Basic Metrics are: • easy to apply • Great for analyzing trends
  13. 13. Quality Metrics Quality Metrics Project – Level Metrics 1. Requirements and Requirement Coverage – How to measure the coverage? 2. Defect Open and Close Rates
  14. 14. Quality Metrics Quality Metrics Department – Level Metrics 1. Mean Time To Detect (MTTD) and Mean Time To Repair – MTTD measures how long it takes QA professionals to find a problem, while the MTTR demonstrates the amount of time needed to effectively address it.
  15. 15. Quality Metrics Quality Metrics Company – Level Metrics 1. Number of Issues reported by Customers
  16. 16. 4: Not investing enough Investment in Testing is below industry average • Always good to know what competition is doing • You are investing less than similar companies
  17. 17. 5: No clear criteria No definition of what “enough” testing is: • Coverage • Execution • Acceptable levels of defects • Major • Minor • Plan for fixing known defects • etc
  18. 18. 6: Listen to Testers’ NO You receive negative advice from Testers to go ahead: • Always listen to employees • Value of good tester • Analyze the legitimated issues raised
  19. 19. 7: Weak Prevention Get the Team involved in: • Requirements assessments • Definition of coding standards/ good practices • Reviews/Inspections – Static Testing
  20. 20. Prevention
  21. 21. 8: No Dev involvement You don’t have Unit testing in place? • Software quality is a team responsibility • Unit testing is natural to developers • Find bugs earlier
  22. 22. Picture so far Choices and effects
  23. 23. 9: Time problems
  24. 24. 9: Time problems Testing periods are regularly reduced
  25. 25. 9: Time problems Deadlines problems regularly? • How often does it happen that testing is squeezed • Lack of time for testing is a symptom of deeper project, process and organizational issues. • Make sure you make enough time for Testing
  26. 26. 10: Testers keep quitting Dissatisfaction of Test Team: • Ultimate sign! • Happy employees – better performance • Can be financial • Can be also professional • Not enough time for testing • Not enough resources • Frustration levels grow up
  27. 27. Testing is enough! Make it even better! 1. Use testing as a high – level risk management 2. Test reusability 3. Automate 
  28. 28. Challenges in the testing…
  29. 29. Questions?