Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Тест-дизайн в тестировании ПО. Задача "Треугольник"
Next
Download to read offline and view in fullscreen.

0

Share

Download to read offline

Как найти побольше багов? (Особенно, если времени нет)

Download to read offline

SQA Days 11. День 1. Секция А
Ярон Цубери
Глава ISTQB, CEO, Smartest Technologies Ltd.
Гиват-Зеев, Израиль

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Как найти побольше багов? (Особенно, если времени нет)

  1. 1. Smartest Technologies Ltd. Providing the SMART solutions How to find more bugs? (Especially if you have no time) 11th SQA Days & ELT Ukraine April 2012Yaron TsuberyWebsite: www.dSmartest.comEmail: YaronTs@dSmartest.ComMobile: +972 52 854925525 April 2012 Smartest Technologies (c) 2010 1
  2. 2. IntroductionPresentation objectivesThe challenges that we haveTake control over the processSummary
  3. 3. Introduction Yaron Tsubery Homeland: Israel More than 19 years experience in Software Development and Testing, Director QA & Testing at Comverse, managed large testing groups and projects deployed to customers located in Israel, USA, Europe, and to the Fareast countries and... Currently, Managing Director of Smartest. President of ITCB (Israeli Testing Certification Board) and a formal member in ISTQB, President of ISTQB (International Software Testing Qualifications Board).25 April 2012 Smartest Technologies (c) 2010 3
  4. 4. IntroductionPresentation objectivesThe challenges that we haveTake control over the processSummary
  5. 5. ObjectivesWe will walkthrough some theoretical material and see the match between theory and reality,Present the challenges and their results,Show a way to improve your efficiency and effectiveness, through a mixed Approaches,The presentation is aimed at stimulating your mind and opening new views to the subject,25 April 2012 Smartest Technologies (c) 2010 5
  6. 6. Set ExpectationsThis presentation focuses on the testing process related to the Execution phase,Let’s have a dynamic and interactive session25 April 2012 Smartest Technologies (c) 2010 6
  7. 7. IntroductionPresentation objectivesThe challenges that we haveTake control over the processSummary
  8. 8. What Were TheChallenges That We Had?25 April 2012 Smartest Technologies (c) 2010 8
  9. 9. Challenges Very large and complex systems Systems that required to function 24/7 Frequent requests for changes Market change and required fast delivery Continue to keep the high quality level required for our systems Competitors…25 April 2012 Smartest Technologies (c) 2010 9
  10. 10. Background25 April 2012 Smartest Technologies (c) 2010 10
  11. 11. Life-cycles Models (SDLCs) Time box N Big Bang Progress Orientation preparation Steering committee HL plan N+1 Fine planning N+1 Plan updates reviews Updated Plan Requirement Test Development Design definition Water fall design Implement code analyze MicroUnit test proces Test execution prelude s Integration test Spiral System test Reorganization Daily builds – smoke tests Optional release Integration testes V-Model release start Result delivery Time-box and Agile development model25 April 2012 Smartest Technologies (c) 2010 11
  12. 12. Were Usually We Put The Efforts In The Testing Process? How much time are you investing in…? Planning Design Preparations Execution (manual Reporting25 April 2012 Smartest Technologies (c) 2010 12
  13. 13. Where Do We Invest Most of Our Time at The Execution Stage?25 April 2012 Smartest Technologies (c) 2010 13
  14. 14. The Evolution Of Regression Using Risk- Partial based regression Testing tests set approach Full testing including execution ATP cycles Full testing including a execution repeated cycles - regression including all regression tests set tests25 April 2012 Smartest Technologies (c) 2010 14
  15. 15. What Are We Looking For?25 April 2012 Smartest Technologies (c) 2010 15
  16. 16. 25 April 2012 Smartest Technologies (c) 2010 16
  17. 17. Stages That We Can Improve Defect Detection Test Execution Code Requirements25 April 2012 Smartest Technologies (c) 2010 17
  18. 18. Nothing New So Far… (-;25 April 2012 Smartest Technologies (c) 2010 18
  19. 19. IntroductionPresentation objectivesThe challenges that we haveTake control over the processSummary
  20. 20. So, What Did We Do?How Our Strategy &Plan Looked Like?25 April 2012 Smartest Technologies (c) 2010 20
  21. 21. StrategyFocused plan at the Execution CyclesUse risk-based approachRapidly track and analyze the execution resultsManage through defect detection analysis.25 April 2012 Smartest Technologies (c) 2010 21
  22. 22. Planning Main dimensions to consider at execution:  Concentrate on each build we’re getting from the development  Design tests using quality risk categories analysis  Divide each build to 3 parts – following risk-based approach, using priority (High, Medium and Low)  What to cover in each part?  Which coverage level should we have? How can we benefit from previous builds and cycles? How can we benefit from the current build? …and even from the current part we’re testing!25 April 2012 Smartest Technologies (c) 2010 22
  23. 23. Level Of TestingRisk analysis and priorities of the features – taken from the initial plan we builtEvaluation of level according to sanity tests resultsDependencies between features (new & current)Coverage we had in previous build:  Did we cover that part?  How much and what did we cover?  What were the results of previous tests?  How many defects did we had? Their severity and priority!25 April 2012 Smartest Technologies (c) 2010 23
  24. 24. Level Of Testing – cont’Don’t forget the Platform and Infrastructure areas:  New hardware  New Kernel  O/S changes  Changes or New 3rd party parts  Configurations  Etc’And additional parts that fits to your systemI’m sure that you have now concerns about the duration which these activities may take…25 April 2012 Smartest Technologies (c) 2010 24
  25. 25. Coverage AreasDefine what to cover per each set of priority (High, Medium and Low)SanityNew functionalityInfrastructure & system related testsRegressions:  Related to new functionality  Bug fixes / retests  Related to bug fixesATP related tests25 April 2012 Smartest Technologies (c) 2010 25
  26. 26. Adding The Focused Exploratory Part25 April 2012 Smartest Technologies (c) 2010 26
  27. 27. Focused Exploratory TestingAfter we have the plan per each set of priority test, considering all elements of coverage and level of coverage,Add to each set of priority test – at the end – a Focused Exploratory Testing partLet’s have a look at the Gantt25 April 2012 Smartest Technologies (c) 2010 27
  28. 28. Gantt Sample25 April 2012 Smartest Technologies (c) 2010 28
  29. 29. Well… How To Leverage The Efficiency of Exploratory Testing?25 April 2012 Smartest Technologies (c) 2010 29
  30. 30. Leveraging The Exp’ TestingPerform an analysis of the defects density  Severity  PriorityDon’t delay the analysis to the end of the testing cycle, perform that while you execute the test priority set!After analyzing the defects density and the infected areas, chose the most infected areas and focus your exploratory testing there!Now you’re running focused exploratory testing25 April 2012 Smartest Technologies (c) 2010 30
  31. 31. Let’s Simplify The Defect Detection Analysis?25 April 2012 Smartest Technologies (c) 2010 31
  32. 32. Defects Density Per AreaAssumptions:  We have 2 new functionality areas: A & B  We have 2 areas for regressions: L & MWe saw the following defect density behavior Severity Severity Severity Total Total Area Test Group (Critical) (Major) (Minor) (C+Mj) (Mj+Mn) Total A A.1 3 0 3 3 A A.2 5 0 5 5 A A.3 1 8 1 9 9 A A.4 7 0 7 7 A A.5 1 8 1 9 9 B B.1 1 1 1 2 2 B B.2 1 2 3 3 5 6 B B.3 1 2 1 3 3C = Critical L L.1 1 2 3 2 3 L L.2 2 3 4 5 7 9Mj = Major L L.3 1 2 3 2 3Mn = Minor L L.4 1 1 4 2 5 6 M M.1 1 3 1 4 4 M M.2 1 1 1 125 April 2012 Smartest Technologies (c) 2010 32
  33. 33. Divide To Priority SetsOrder the priority using the defects density tableCreate Priority Sets:  High  Medium Severity Severity Severity Total Total  Low Area L Test Group L.2 (Critical) (Major) (Minor) (C+Mj) (Mj+Mn) Total B B.2 L L.1 L L.3 L L.4 A A.3 A A.5 M M.1 B B.3 B B.1 M M.2 A A.4 A A.2 A A.125 April 2012 Smartest Technologies (c) 2010 33
  34. 34. More Complex Analysis TableAnalysis per tested module or featurePerform deep analysis per the various tests executed Risk Defects Defects Coverage Level in Infrastructure Final Module/Feature Level (Sanity) (Dependencies) Previous Build Impact Avarage Priority Module A Medium Module B High Functionality A High Functionality B Low25 April 2012 Smartest Technologies (c) 2010 34
  35. 35. More Complex Analysis TableMeasure the infected area depending on defects detection ratioPrioritize the areas to put the focus when performing Exploratory Testing Risk Defects Defects Coverage Level in Infrastructure Final Module/Feature Level (Sanity) (Dependencies) Previous Build Impact Avarage Priority Module A Medium Module B High Functionality A High Functionality B Low25 April 2012 Smartest Technologies (c) 2010 35
  36. 36. Defect Detection RatioNormal S-Curve S-Curve S-Curve open S-Curve, defects ratio25 April 2012 Smartest Technologies (c) 2010 36
  37. 37. Improved Defect Detection Ratio S-Curve using Focused Exploratory Testing The S-Curve with focused Exploratory S-Curve Ratio of regular S-Curve25 April 2012 Smartest Technologies (c) 2010 37
  38. 38. Where Do We See The Impact?25 April 2012 Smartest Technologies (c) 2010 38
  39. 39. The ImpactApproach method: Defects driven approachProject: delivery timelineQuality: product’s quality raisedEfficiency: less time invested at defect detectionPositioning: higher quality of the Testing Team/s – less escaped defects25 April 2012 Smartest Technologies (c) 2010 39
  40. 40. When To Expand The Focused Exploratory Tests?25 April 2012 Smartest Technologies (c) 2010 40
  41. 41. When To Expand? At the time that we have many features to test and the probability to have defects is high At the time that we have less time for detailed description of the test cases At the time that we have many change requests that we didn’t cover with test cases Recommendations:  Document the skeleton of the Exploratory Tests you executed  Describe in more details those who encountered bugs (- :25 April 2012 Smartest Technologies (c) 2010 41
  42. 42. When To Reduce Or Not Perform The FocusedExploratory Tests?25 April 2012 Smartest Technologies (c) 2010 42
  43. 43. When To Reduce Or Not Perform?At the time that we have only minor areas to cover,At the time that we see defect convergence – less bugs open,At the time that we see that there are less areas that are getting “infected”,At the time that the ROI is less effective.25 April 2012 Smartest Technologies (c) 2010 43
  44. 44. IntroductionPresentation objectivesThe challenges that we haveTake control over the processSummary
  45. 45. Before The Summary… What are the Next Steps?25 April 2012 Smartest Technologies (c) 2010 45
  46. 46. Next StepsTry to Implement that in an Agile environment,Start to use Heuristic methods!Where do we find most of our bugs? (ODC analysis) How to reduce over-testing?Etc’ (-;25 April 2012 Smartest Technologies (c) 2010 46
  47. 47. SummaryAnalyze your testing project and decide if to activate this approachFeel free to Mix some Execution ApproachesFactors for successful implementation:  Initial risk analysis  Prioritize test areas – High, Medium and Low  Daily defect detection analysis – per area  Focus on areas with more potential to have defects  Communicate plan and results with your peers and managersGood Luck!25 April 2012 Smartest Technologies (c) 2010 47
  48. 48. 25 April 2012 SMART Technologies (c) 2010 48
  49. 49. Yaron Tsubery: yaronts@dsmartest.com Office: +972 2 650925525 April 2012 Smartest Technologies (c) 2010 49
  50. 50. System: Physical Architecture25 April 2012 Smartest Technologies (c) 2010 50
  51. 51. System Layout25 April 2012 Smartest Technologies (c) 2010 51
  52. 52. Low-Level Design Each risk to be mitigated via testing will have one or more test cases associated with it. (I’ve not shown all traces, to avoid clutter.)25 April 2012 Smartest Technologies (c) 2010 52 Reference: Rex Black (STE)
  53. 53. Quality risks are potential system problems which could reduce user satisfaction Risk priority number: Aggregate measure of problem risk Tracing information back Business (operational) risk: Impact of the problem to requirements, desig Technical risk: Likelihood of the problem n, or other risk bases Tech. Bus. Risk Extent ofQuality Risk Risk Risk Pri. # Testing TracingRisk Category 1Risk 1Risk 2Risk n A hierarchy of 1 = Very high risk categories 2 = High The product of can help 1-5 = Extensive 3 = Medium technical and organize the 6-10 = Broad 4 = Low business list and jog 11-15 = Cursory 5 = Very low risk, from 1- your memory. 16-20 = Opportunity 25. 25 April 2012 Smartest Technologies (c) 2010 21-25 = Report bugs 53 Reference: Rex Black (STE)

SQA Days 11. День 1. Секция А Ярон Цубери Глава ISTQB, CEO, Smartest Technologies Ltd. Гиват-Зеев, Израиль

Views

Total views

2,189

On Slideshare

0

From embeds

0

Number of embeds

927

Actions

Downloads

51

Shares

0

Comments

0

Likes

0

×