Backward thinking design qa system for quality goals


Published on

carrier grade software quality assurance system design, the overall view

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Backward thinking design qa system for quality goals

  1. 1. Backward ThinkingBackward ThinkingDesign QA System to Meet Quality GoalsDesign QA System to Meet Quality GoalsLiang Gao
  2. 2. Types of TestingTypes of Testing• User testing• Alpha testing• Beta testing• Function testing• Integration testing• Domain testing• Boundary testing• Best representative testing• Logic testing• State-based testing• Path testing• Performance testing 2• Specification-based testing• Requirements-based testing• Combination testing• Regression testing• Scripted testing• Smoke testing• Exploratory testing• Guerilla testing• Scenario testing• Installation testing• Load testing• System testingDon’t Panic
  3. 3. And Lots of Questions Need To Be AnsweredAnd Lots of Questions Need To Be Answered• 产品的组织结构图 产品测试后, 测试人员如何流动• 测试人员的职位和角色的划分 有没有独立的测试设计团队• 测试组织内部的回报关系, 测试人员如何考核 开发和测试的比例• 测试人员的要求,资历,经验等。 质量部对测试团队的监控作用。• 开发流程和开发模式 开发人员给测试人员的质量的保证• 测试工作量如何来评估,如何制定测试计划,依据什么来估计工作量• 开发和测试人员在各个阶段的工作职责测试人员是否参与百合子和黑盒子的测试• 版本发布谁来说了算,如何评估版本的质量。 版本能不能发布要考虑那方面的因素。• 工具和自动化, 仔细介绍一下好的自动化工具。有那些工具。日常测试中用那些工具。是否自己开发的,还是商用工具。• 测试技术用例设计考虑的输入边界值 有否测试类型的划分• 测试用例设计的步骤,需求分析,测试设计从那些角度来考虑,是从用户需求还是从哪些。• 测试策略,一个版本的测试策略,有老特性和新特性。• 如何来自动化,进入维护状态,测试如何进行。 什么组织来负责。是否是独立的组织,人员的交替关系。• 平台的测试和产品的测试的关系,是否有不同的测试团队来负责。测试内容和范围有没有差别• 一个版本,测试是否基于风险,需要考虑那些因素主要问题是在什么阶段发现的。转测试之前还是之后。• 开发人员, SE ,市场人员和服务人员在测试过程中能发挥什么作用 3
  4. 4. Nature of the SoftwareNature of the Software• Continuous Release 4
  5. 5. Product QualityProduct Quality• “Rock Solid Product”, what does it mean?• Your product is only as good as customer’s saying• The ultimate goal is to increase customersatisfaction.• C company publish a customer satisfaction indexevery day on its internal website.• Customer are not satisfied when product is notfunction as they expected to be. 5
  6. 6. Why Not Satisfied?Why Not Satisfied?• Lack of product knowledge – need a TAC, notthe topic today , and not quality problem.• Product has problemo Manufacture problem – not the topic todayo Defects – yes, we need to catch those. 6Internal Found DefectsCustomerFound Defects
  7. 7. Product Quality MetricsProduct Quality Metrics• Product complexity• Numbers of the customer found defects• Customer found defects priority• Number of the unit shippedCan you come up with an equation to calculatethe “Product Quality Index” number?You boss will love to say:” Lets reduce our qualityindex from 4.7 to 3.5 this year”. 7
  8. 8. QA’s goalQA’s goal• Reduce customer found defects!• Reduce high profile customer found defects first.(80 percent of the customer only use 20 percentof the features, 20 percent of the customer make80 percent of your revenue)• So QA strategy is just anther ROI (return oninvestment) 8
  9. 9. 2 Kinds of Testing2 Kinds of TestingOrganizationOrganization• By Feature• By Testing roles 9
  10. 10. 2 Types of Defects -2 Types of Defects -EngineeringEngineering• New feature defects – new code broken• Regression defects – new code break the oldcodeo Enable new codeo Not enable new code 10
  11. 11. 1Type of Defects - Customer1Type of Defects - Customer• Feature interaction defects 11
  12. 12. Why Regression Bug?Why Regression Bug?• Software Modules and Modules are coupled,change one module’s code will affect others.• Network equipments process same packet indifferent place when a packet traverse the box –sequential processing• Protocols are layered (ISO layer)• Protocols depends on one another (one protocoluse another protocol as input) 12
  13. 13. Analysis Your CustomerAnalysis Your CustomerFound BugsFound Bugs• New feature defects (single or multi-featureinteraction)o More testing need to be done on new feature testing.o Your testers need to be trained on how design effectivetest cases to catch new bugs.o Different testing methodologies.o Increase your new feature testing coverage on yourtest plan.o End-to-end customer oriented solution testing• Regression defectso Will not decrease no matter how good your testers areo You need a regression strategyo All code need to be tested whenever release a newversion ( huge work) 13
  14. 14. Combination is endlessCombination is endless• Feature interaction is endless• The way to use your product/system is endless• Lucky you if you can cover them all• If not, you need a strategy• The best strategy is customer’s usage (20/80 rules) 14
  15. 15. To Reduce Customer FoundTo Reduce Customer FoundDefectsDefects• You need a process to test new• You need a process to test old• You need a process to test solution (whatever onthe customers site). 15
  16. 16. Bugs in a Product’s lifeBugs in a Product’s life 16New FeatureBugRegression Bug Regression BugActive Development, MarketShareCash Cow End of LifeProduct RevenueMaintenance, feature enhancement1. 5.06.0
  17. 17. Nowadays in Company CNowadays in Company C• Most of the new feature bugs can be found inthe early development – strong testers, end-to-end system testing, early customer field trial.• Many customer found defects are regressionbugso 10 million lines of code need to be tested whenever a new version isreleasedo Release 2 new versions every week.o Is that possible? Yes, but only via Automation• Many companies use frequent regression tostabilize a code branch. 17
  18. 18. Strategy To Release aStrategy To Release aQuality SoftwareQuality Software• We are talking about carrier grade product.• Test case/test plan design needs to start whencode writing start, and finish when code writingfinish.• Be very strict on review process. (C companyspent multi-million $$ on this)• Closely monitor your bug trend during testingnew.• Repetitive regression to stabilize your codebranch 18
  19. 19. Repetitive regression to stabilize your code branchRepetitive regression to stabilize your code branch• In order to fix regression bugs, new code need tobe added  introduce another regression bugs?• Multiple regression cycles are needed to carveoff regression bus.• The more regression you are able to do beforeFCS, the more stabilize your code branch. 19
  20. 20. Testing Release TimingTesting Release Timing 20Dev Test Regression1 Regression2 Regression3Solution EFTGoldenWeek
  21. 21. Development and Testing At a GlanceDevelopment and Testing At a Glance 21MRD/PRDFunctionalSpecificationCodeDevelopment Bug FixesTest Plan Test CaseCodeReleaseto TestDaily/Frequent Code DropsCode toTestAlphaStartBetaStart FCSVarious Testing ScriptingDevTestCodeControl Opened Bug ID CodeReviewFormGoldenWeek
  22. 22. No Need to Be In SyncNo Need to Be In Sync• Make regression a train• Make System testing and solution a train as youlike.• Only new feature testing need to be in sync withdevelopment and release schedule.• Scheduled regression testing make branch stable• Scheduled system and solution testing makeoptimize your code for performance 22
  23. 23. Branch is QA’s BabyBranch is QA’s Baby• Code branch should be owned by QA.Developer’s checking in new code need QApermission• Design a nightly smoke test suite to monitor thebuild quality.• Test new, and exploratory test new to take thenew code bug out (daily build is necessary)• Close the code branch before final regression.• Only regression commit is permitted duringregression testing• Multiple regression cycle to shrug off theregression bug 23
  24. 24. Automate the Process to Make Your Life EasyAutomate the Process to Make Your Life Easy• When developer check in a code, he canspecify a bug id. Then this bug in the bugtracking system’s state automatically change to“R”• A new code commit automatically trigger asmoke testing, report automatically sent to QAteam. If there are large amount of failures, analarm can be automatically triggered (send aSMS to your cell phone?)• ……… 24
  25. 25. Not Enough TimeNot Enough TimeTesting?Testing?•  More Attributes on the test cases can help• Which functionality is most important to the project’s intended purpose?- Which functionality is most visible to the user?- Which functionality has the largest safety impact?- Which functionality has the largest financial impact on users?- Which aspects of the application are most important to the customer?- Which aspects of the application can be tested early in the development cycle?- Which parts of the code are most complex, and thus most subject to errors?- Which parts of the application were developed in rush or panic mode?- Which aspects of similar/related previous projects caused problems?- Which aspects of similar/related previous projects had large maintenanceexpenses?- Which parts of the requirements and design are unclear or poorly thought out?- What do the developers think are the highest-risk aspects of the application?- What kinds of problems would cause the worst publicity?- What kinds of problems would cause the most customer service complaints?- What kinds of tests could easily cover multiple functionalities?- Which tests will have the best high-risk-coverage to time-required ratio? 25
  26. 26. Ready to ReleaseReady to Release• QA is always asked: can we release it?o QA’s job is to deliver report (a risk assessment),what works, what not. Testing report & BugReport• QA can only answer this question by giving a fullversion testing report• QA use defects to talk to other department andbosses. 26
  27. 27. Big PictureBig Picture 27
  28. 28. Testing TipsTesting Tips• Plan aheado Perform good interview to create strategy and methodology• Discussion is crucial:o Interview & review with right set of people: developers, marketing, sales,manufacturero Use white board as much as possible to discuss new test cases, resolvingtechnical questions• Work with developerso Find out the changes and possible breakageso Resolve ambiguities in varies documentations 28
  29. 29. Testing TipsTesting Tips• Testingo Test the code changes and its dependencieso GUI testing != functionality testing. If possible, do not useGUI testing to test core functionalitieso Use 80/20 rules to focus on the right area to testo Not only ask what to test, but ask what not to test• Remove unnecessary duplicate effortso Types of release can determine the testing, see next slideo Weapon: Schedule. Use it carefullyo Code scale linearly but the complexity grows exponentially 29
  30. 30. Version controlVersion control• Version 1.0, Oct 10th, 2007, Liang Gao• Version 1.1, Oct 29th, 2007, Liang Gao• Version 1.3, Jan 10th, 2008, Liang Gao• Version 1.4, March 13th, 2008, Liang Gao• Version 1.5 June, 2008, Liang Gao 30