1. User Scenario driven testingWhen writing test plan, separate what with How – First, only think about the user scenario – Then, think about how you to test the scenarioWhen write tests, separate what with how – First, write use scenario – Second, implement the test cases which test the scenario
User Scenario Example• Scenario – Block of upgrade SQL Express during VS 2010 Installation (Priority 0)• Action: – When a use install VS 2010 on a AMD64 machine with SSE 2005 install, existing SSE instance can’t upgrade SSE 2008• Success criteria: – Make sure that Setup program detect existing SSE instance correctly. – Make sure that Setup program disable the upgrade/install of SSE correctly. – Make sure SSE is not upgraded if upgrade is not needed. – Existing SSE instance works after setup.
2. Understand the featureKnowing: – how the feature will be implemented. – the risks in term of implementation. – the risks in term of the impact on customer.
Example of “Risk Factors and Mitigation strategies”• We are testing product which is not shipped by SQL Group.• Setup is sensitive to change, and easy to be broken.• Dependence of SSE, such as MSI 4.5 is one of the main factors of causing setup failing.• Huge Test Matrix
3. Define your Test Strategy• Test Strategy – How to address the risk areas? – Manual Test v.s. Automation Test?• Examples: – Align Test tasks with Dev tasks. – For larger dev check-ins, test with a private build before checking-in. – Encourage writing programming based tests instead of writing static Meta tests.
Example of Test Strategy• Manual Testing during Improvement Phase – Avoid huge investigation of test automation – We define high-level user scenario Success criteria and China VAT team help us to verify• Run test as a real customer to find usability issue. – Verify UI, setup option, and no. reboots – Verify localized strings• For bug fix, we drive user scenario and test dimension from the nature of the fix• Long term, will take advance of VS setup automation in SQL Lab
4. Define test patterns• Test Types: – Recorded tests, Scripted tests or Data Driven tests, Declarative Tests – Model-based or static tests• Design Patterns: – Layer Tests – Repository Pattern, Factory Patterns, Visitor pattern• Test Dimensions: – Pair wise or all combination; Parameterized Tests• Verification Pattern: – State Verification; Oracle based Verification; Custom Assertion• Deal with Dependence: – Dependence Injection – Mock Object
Test Case ExampleTest Case: Mismatch between SSE language with VS language if the language of installed SSE instance does not match with the localized SSE 2008, we should not provide options to install SSE.Test Dimension – VS Language: ENU – SSE: SSE 2005, SSE 2008 – SSE Language: JPNTest Case: use pict tool to use generate 3 test casesVerification:• Verify that user can not choose SSE during VS Setup• Verify that VS setup succeed without any error
Conclusion1. User Scenario driven testing2. Understand the Feature to Be Tested3. Define Test Strategy4. From User Scenario to Test Case5. Write good Tests6. Define measurable success critical
Reference: Test Strategy• http://en.wikipedia.org/wiki/Test_strategy• http://www.projectperfect.com.au/info_test_ strategy.php• http://www.satisfice.com/presentations/strat egy.pdf