Database Unit Testing Made Easy with VSTS


Published on

Slide deck from my Database Unit testing talk at STLDODN 2012

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Goals of Software Testing ;Assure the Quality of a Software DeliverableEnsure the software meets requirement specsEnsure the software meets user expectations as well as stakeholder expectationsEnsure that the behaviour of the system is predictable and deterministic -
  • White-box testing is when the tester has access to the internal data structures and algorithms including the code that implements these.Black-box testing treats the software as a "black box"—without any knowledge of internal implementation. The black-box tester has no "bonds" with the code, and a tester's perception is very simple: a code must have bugs. Using the principle, "Ask and you shall receive," black-box testers find bugs where programmers do not. On the other hand, black-box testing has been said to be "like a walk in a dark labyrinth without a flashlight," because the tester doesn't know how the software being tested was actually constructed. As a result, there are situations when (1) a tester writes many test cases to check something that could have been tested by only one test case, and/or (2) some parts of the back-end are not tested at all.Grey-box testinginvolves having knowledge of internal data structures and algorithms for purposes of designing tests, while executing those tests at the user, or black-box level. By knowing the underlying concepts of how the software works, the tester makes better-informed testing choices while testing the software from outside. Grey-box testing implements intelligent test scenarios, based on limited information.
  • Unit testing - also known as component testing, refers to tests that verify the functionality of a specific section of code, usually at the function levelIntegration testing - Integration testing is any type of software testing that seeks to verify the interfaces between components against a software designSystem testing - tests a completely integrated system to verify that it meets its requirementsSystem integration testing -  verifies that a system is integrated to any external or third-party systems defined in the system requirements
  • SandboxA database sandbox is a private environment used for local unit testing and development. The sandbox allows development and testing to be done in isolation from other development activity and ensures repeatable results. Database developers use sandboxes to perform unit testing and validate rollout script behavior. Application developers may use local database sandboxes to test application code without affecting, or being affected by, other development activity. A database sandbox may be created from script, by deploying a VSTSDE database project locally, or by restoring a database backup at a known state.Test CaseA test cast exercises a targeted code path. Success or failure is ascertained after test case execution by asserting that the expected outcome was realized. Multiple assertions are commonly needed to validate the desired outcome of a specific test case.
  • AssertionAn assertion is a simple condition that returns "true" to indicate test success or "false" to indicate failure of a test case. A test case assertion verifies:expected interface output values (e.g. output parameter values, number of resultsets, number of rows, resultset values, etc.)no extraneous results (e.g. errors, extra resultsets, unexpected informational messages, etc.)data modification behavior (e.g. data correctly inserted/updated/deleted)
  • Unit TestA unit test is the collection of all of the test cases that exercise a stored procedure interface and code along with the assertions need to verify correct behavior.Test ScriptA test script is a T-SQL script that executes a test case. A test script may also include assertions implemented in T-SQL, such as checking return code values, output parameter values or a TRY/CATCH to assert an expected RAISERROR of a user-defined error occurred.
  • Multiple related unit tests may be grouped together in a test list within VSTSDE so all can be run as a unit and for logical organizational purposes (a.k.a. test suite).
  • Setup and Cleanup ScriptsSetup and cleanup scripts (a.k.a. fixture and teardown scripts) are T-SQL scripts used to prepare the database for unit testing and return the database back to the initial state afterward. Setup scripts typically insert data needed by a test case while cleanup scripts remove testing artifacts after test case execution. The VSTSDE unit test framework provides common Test Initialize and Test Cleanup scripts that are run before/after each test case within a unit test. This allows the same scripts to be shared by all test cases within a unit test. VSTSDE also includes the ability to run Pre-Test and Post-Test scripts before/after an individual test case. Pre/Post scripts are especially appropriate for data tailored for a specific test case. VSTSDE Data Generation Plans provide an alternative to unit test setup scripts. However, data values must be customized so that test cases can be developed against known data values. Care must be used to ensure DGP changes do not adversely affect existing unit tests and that cleanup scripts restore modified data back to the original state after test case execution.
  • Unit Test explorer replaces the old test view and test results windowUsing simple plugin adapters, third party test frameworks can plug into the test platform layer and get the full experience of running inside of Visual Studio
  • Database Unit Testing Made Easy with VSTS

    1. 1. PlatinumSponsorsGoldSponsorsSilverSponsors
    2. 2. DATABASE UNIT TESTINGMADE EASY WITH VISUALSTUDIOSanil MhatreDatabase Developer, Scottrade
    3. 3. About Me Database Developer –Scottrade (@scottradejobs) MCTS – MS SQL Server 2008 Database Development @SQLSuperGuru MSDN Forums & SQL Saturday #154 | St.Louis | Sept 15
    4. 4. Pre-requisites MS Visual Studio Team system framework & tools MS Visual Studio Team System (VSTS) 2010 & 2012  Ultimate & Premium Editions – create, modify & run database unit tests  Professional Edition – run database unit tests but cannot create or modify them MS Visual Studio Team System 2008 Database
    5. 5. Agenda Software Testing – Basics Unit Testing – Basics Database Unit Testing – Principles, Terminology & Levels Demo – Implement Database Unit Testing with VSTS 2010 & MS SQL Server 2008 R2 New features in VSTS 2012
    6. 6. Software Testing Meet Requirements & Expectations Deterministic & Predictable outcome
    7. 7. Types of Software TestingCriteria Black Box Testing White Box Testing Method in which the Method in which the internal design/ internal design/ implementation of theDefinition implementation of the item being tested is item being tested is NOT known to the known to the tester. tester Mainly applicable to Mainly applicable to higher levels of lower levels ofLevels Applicable To testing:Acceptance testing:Unit Testing Testing Integration Testing System Testing Generally, independent Generally, SoftwareResponsibility Software Testers DevelopersProgramming Not Required RequiredKnowledge
    8. 8. Levels of Software Testing Unit Testing Integration Testing System Testing System Integration Testing
    9. 9. What is Unit Testing ? Validation & Verification Methodology Belongs to the White Box testing category Test individual units of Source Code Unit – smallest testable part of application  Method, Function, Class Tests created before or during development Tests conducted during development
    10. 10. Goals of Database Unit Testing Ensure quality & accuracy of database deliverables Establish a baseline state of your database Verify Subsequent changes made to database objects Identify regression issues throughout the application lifecycle
    11. 11. Database Unit TestingTerminology • Isolated Environment (database) • Local development, testing &SandBox validation • Created by Script, VSTS Project or Restoring backup • Exercises a targeted code pathTestCase • Multiple assertions
    12. 12. Database Unit TestingTerminology • Simple Condition to indicate success or failure • Returns a True or False • Verifies ExpectedAssertion Interface • Verifies extraneous Results • Verifies data modification behavior
    13. 13. Database Unit TestingTerminology • Collection of all test cases Unit for a given stored procedure • Tests interface, code & Test behavior • TSQL Scripts that executes a test case Test • May include assertions like; • Return valuesScript • Output parameters • Expected Errors
    14. 14. Database Unit TestingTerminology • Logical & named grouping of related unit tests • Multiple test lists can be Test created • Can be run as a unit List • A test suite • Builds & Check in Policy • Deprecated in VSTS 2012
    15. 15. Database Unit TestingTerminology • TSQL Scripts • Prepare database for unitSetup testing & • Returns database back to pre test state after testingCleanu • Framework also provides p common Initialize & Cleanup ScriptsScripts • Pre Test and Post Test Scripts • Data Generation Plans
    16. 16. Database Unit Testing PrinciplesIsolation• Outcome of one test must not affect the outcome of any other test(s)• Each test must return the database back to its previous state after executionIndependence• One test case must not depend on any other test case• Each test case must be run independently of any other test case• Test case assertion must be independent of the order of it’s execution
    17. 17. Database Unit Testing PrinciplesCharacteristics of Ideal Unit Test• Exercise all possible code paths• Evaluate all possible conditions• Validate correct behavior for each path/conditionConsiderable efforts for exhaustive testing• Non Trivial stored procedures• Number of input parameters• Permutations of optional parameters• Error code paths
    18. 18. Testable Interfaces of a StoredProcedure Input & • Name Output • Data Type (Length) Parameters • Optionality Return Code • Success/failure Row Count • SET NOCOUNT ON/OFF
    19. 19. Testable Interfaces of a StoredProcedure • One or more SELECT statements • Number of Columns Result Set • Column Ordinal position • Column Name & Data Type Informational • PRINT Messages Messages • RaiseError – Severity 10 & below Error • RaiseError – Severity 10 & above Messages
    20. 20. Unit Test ConditionsPredefined Test conditions in VSTS 2010 Data Checksum Empty ResultSet/Non Empty Result Set Execution Time Expected Schema Row Count Scalar Value Inconclusive
    21. 21. Levels of Unit TestingFactors Affecting the choice of Appropriate Levelof Unit testing Costs of Unit test development & maintenance Added Value of Quality Application Criticality & Frequency of use Project Deadlines
    22. 22. Levels of Unit Testing • Smallest amount of testing • Ensures error free compiles • No code is exercised • Consists of single test caseMinimal with SET FMTONLY ON • Executes with NULL inputs • Intrinsic Assertion – No Errors
    23. 23. Levels of Unit Testing • Exercises commonly used code paths & conditions • One step above Minimal • Tests Empty/Non-EmptyMargina Result sets l • Some assertion of typical behavior & expected interface • Not a thorough & complete test
    24. 24. Levels of Unit Testing • Exercises all code paths & conditions • Test cases for all permutations of optional parametersComplet • Multiple Assertions to completely verify interface & behavior e • Could also verify column names of result set • It’s an exhaustive unit test to verify all but unexpected errors
    25. 25. Levels of Unit Testing Possible to develop and un-testable stored procedure ?  A search stored procedure with 8 optional parameters will need over 2000 test cases  Numerous found/not found conditions  Error validation Possibly an indicator of flawed design Consider refactoring to facilitate testability
    26. 26. Demo Prerequisite : Database Project Exists in VSTS Create a Unit Test Project in VSTS Configure SandBox Create Tests Run Tests
    27. 27. New in VSTS 2012 Redesigned Unit Test Explorer Support for 3rd party testing frameworks Code Analysis improvements Continuous test runner Deprecated Features  Test lists  Generate Unit Test Wizards
    28. 28. Conclusion Software Testing - Basics Unit Testing – Basics Database Unit Testing – Detailed discussion on Principles, Goals & Levels Demo – Implementation of Database Unit Testing with VSTS 2010 & SQL Server 2008 R2
    29. 29. Resources MSDN  us/library/aa833283.aspx  us/library/aa833423.aspx Wikipedia  Image -
    30. 30. THANK YOU !Questions & Feedback