Paper Ps


Published on

Published in: Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Paper Ps

    1. 1. Automated Testing with Team Test Unit, Web, Performance, Load, Manual, and Ordered Tests Code Coverage and Code Analysis
    2. 2. Quality Challenges “ Software bugs, or errors, are so prevalent and so detrimental that they cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic product…an estimated $22.2 billion, could be eliminated by an improved testing infrastructure that enables earlier and more effective identification and removal of software defects.” (Source: NIST 2002) Satisfying users Business interruption risks Ongoing maintenance costs rising Less avail. resources
    3. 3. How Does VSTS/TFS Help? <ul><li>Increased Communication and Integration… </li></ul><ul><li>Code Analysis Tools </li></ul><ul><li>Code Profiling Tools </li></ul><ul><li>Unit Testing and Code Coverage </li></ul><ul><li>Load Testing </li></ul><ul><li>Other Testing Tools </li></ul><ul><li>Test Case Management </li></ul>
    4. 4. Unit Testing <ul><li>Each unit is tested independently </li></ul><ul><li>A large percentage of defects are identified during unit testing </li></ul><ul><li>Automatic and repeatable code testing </li></ul><ul><li>Simplifies integration </li></ul><ul><li>Self documenting </li></ul><ul><li>Auto-generate </li></ul>Isolate small portion of code & determine whether it works correctly
    5. 5. Benefits of Automated Unit Testing <ul><li>Repeatable unit that verifies code still works </li></ul><ul><li>Confirm code is tested </li></ul><ul><li>Build a set of regression tests </li></ul><ul><li>Make changes with confidence </li></ul><ul><li>Aid in understanding code </li></ul><ul><li>Limits: integration, web and Windows UI testing, load, performance testing </li></ul>
    6. 6. Unit Test Types <ul><li>Class (standard) </li></ul><ul><ul><li>Test the properties and methods of a class </li></ul></ul><ul><li>Data-driven </li></ul><ul><ul><li>Bind unit test parameters to a datasource </li></ul></ul><ul><li>ASP.NET </li></ul><ul><ul><li>Test classes (or business logic) inside an ASP.NET application </li></ul></ul><ul><ul><li>Run in the context of the web server (ASP objects available) </li></ul></ul><ul><li>Web Services </li></ul><ul><ul><li>Define a web reference to the service </li></ul></ul>
    7. 7. Extensions of Unit Testing <ul><li>Regression Testing </li></ul><ul><ul><li>Modifications are validated with regression tests </li></ul></ul><ul><ul><li>Write new code and verify system continues to function correctly </li></ul></ul><ul><li>Integration Testing </li></ul><ul><ul><li>Helps validate how tested components interact with one another </li></ul></ul><ul><ul><li>Can identify problems that occur when units are combined </li></ul></ul><ul><li>Scenario Testing </li></ul><ul><ul><li>Combine sequence of unit tests to cover a scenario </li></ul></ul>
    8. 8. Writing Effective Unit Tests <ul><li>Atomic </li></ul><ul><ul><li>Should not require others tests to be run first </li></ul></ul><ul><li>Cover all cases </li></ul><ul><ul><li>One test for each sceneario </li></ul></ul><ul><ul><li>Cover all conditions, exceptions, nulls, etc. </li></ul></ul><ul><li>Able to re-run without configuration </li></ul><ul><ul><li>Database create/read/update/delete without having to modify the database before or after the test is run </li></ul></ul><ul><li>Test a common application state </li></ul>Unit tests should be …
    9. 9. Unit Test Conditions <ul><li>Success baseline </li></ul><ul><ul><li>Common, successful, most-likely call to your code </li></ul></ul><ul><li>Parameter mix </li></ul><ul><ul><li>Pass a varied set of parameters </li></ul></ul><ul><ul><li>Data-driven </li></ul></ul><ul><li>Bounds checking </li></ul><ul><ul><li>Stress the upper and lower limits of your parameters </li></ul></ul><ul><li>Null values </li></ul><ul><ul><li>Determine outcome of passing nulls to parameters </li></ul></ul><ul><li>Error conditions </li></ul><ul><ul><li>Trigger error conditions and validate expected outcomes </li></ul></ul><ul><li>Code coverage scenarios </li></ul><ul><ul><li>Test all conditions in your code </li></ul></ul>
    10. 10. Creating Unit Tests (1 of 2) <ul><li>Attributes are used to denote which methods and classes should be loaded for tests </li></ul><ul><ul><li>TestClass – denotes a class for testing </li></ul></ul><ul><ul><li>TestMethod – denotes a method for testing </li></ul></ul><ul><ul><ul><li>must return void (Sub for VB.NET) and have no parameters </li></ul></ul></ul><ul><li>Certain attributes are associated with initialization </li></ul><ul><ul><li>TestInitialize – run before each test </li></ul></ul><ul><ul><li>TestCleanup – run after each test </li></ul></ul><ul><ul><li>ClassInitialize – run once before running any tests in the class </li></ul></ul><ul><ul><li>ClassCleanup – run once after running all tests in the class </li></ul></ul><ul><li>Assert – static type used for asserting test values </li></ul><ul><li>ExpectedException – attribute used for determining the exception that should be thrown from the test </li></ul>
    11. 11. Creating Unit Tests (2 of 2) <ul><li>Methods on the Assert type </li></ul><ul><ul><li>AreSame / AreNotSame </li></ul></ul><ul><ul><li>AreEqual / AreNotEqual </li></ul></ul><ul><ul><li>IsNull / IsNotNull </li></ul></ul><ul><ul><li>IsInstanceOfType / IsNotInstanceOfType </li></ul></ul><ul><ul><li>IsTrue / IsFalse </li></ul></ul><ul><li>Other Assert types </li></ul><ul><ul><li>CollectionAssert – collection equivalency </li></ul></ul><ul><ul><li>StringAssert – string comparison/regex matching </li></ul></ul><ul><li>Web test types </li></ul><ul><ul><li>HtmlDocument – grants access to tags in the html document </li></ul></ul><ul><ul><li>Validate*/Extract* - types used for reading field values from the request and response </li></ul></ul>
    12. 12. Unit Tests
    13. 13. Code Coverage <ul><li>Improves effectiveness of tests </li></ul><ul><li>Show a measurable indication of code that was covered by unit tests </li></ul><ul><ul><li>Strive for 75% and higher </li></ul></ul><ul><li>Shows the ratio of executed logic to the total logic </li></ul><ul><li>Can be used in conjunction with unit and load testing </li></ul><ul><li>Helps to find unused code </li></ul>Determine code that is exercised by tests
    14. 14. Code Coverage
    15. 15. Static Code Analysis <ul><li>Provides a means to enforce coding standards </li></ul><ul><li>Configurable </li></ul><ul><li>Reduction in code reviews for small issues </li></ul><ul><li>Provides guidance on how to fix </li></ul><ul><li>Define a check-in policy to enforce code analysis pass </li></ul><ul><li>Coverage includes issues with design, globalization, interoperability, maintainability, naming, performance, reliability, security, and usage </li></ul><ul><li>Warning: developers can supress messages with attirbute [SuppressMessage(&quot;AdventureWorks.Rules&quot;, &quot;AW14441&quot;)] </li></ul><ul><li>MORE INFO see, “Writing Quality Code” in TFS documentation </li></ul>
    16. 16. Static Code Analysis
    17. 17. Web Testing <ul><li>Record page navigation and user interactions with web pages </li></ul><ul><li>Run tests back to determine errors </li></ul><ul><li>Seed tests from a database </li></ul><ul><li>Run tests from a group of users </li></ul><ul><li>Validation rules </li></ul><ul><li>Extractions </li></ul>
    18. 18. Web Testing
    19. 19. Load Testing <ul><li>Load Testing: ensure application works under expected, concurrent user load </li></ul><ul><li>Create test cases that simulate real user conditions </li></ul><ul><li>Distribute tests across cases based on user behavior </li></ul><ul><li>Assume standard think times </li></ul><ul><li>Distribute across connection types </li></ul><ul><li>Simulate an application performance in a production environment </li></ul><ul><li>Provide repository to look at a performance trend over time, to see if changes are helping or hurting </li></ul>Simulate multiple users against an application simultaneously
    20. 20. Stress Testing <ul><li>Stress Testing: determine breaking points in your application </li></ul><ul><li>Find scalability issues before the application is deployed </li></ul><ul><li>Step (increase) user load over time, monitor </li></ul><ul><li>Find breaking point and use to monitor application </li></ul><ul><ul><li>Change software </li></ul></ul><ul><ul><li>Add hardware </li></ul></ul>
    21. 21. Performance Testing <ul><li>Test at single user </li></ul><ul><li>Test at normal load </li></ul><ul><li>Test at peak load (1.5x normal load) </li></ul><ul><li>Measure against expected performance </li></ul><ul><ul><li>Peak load should allow for a 20-30% increase in metric </li></ul></ul><ul><li>Set thresholds on the output to avoid digging through data </li></ul>
    22. 22. Load, Stress, and Performance Testing
    23. 23. Manual Test <ul><li>Tracked like other tests when executing test groups </li></ul><ul><li>Presented to the tester to enter and confirm results </li></ul><ul><li>Defined as text file or Word document </li></ul>
    24. 24. Generic Test <ul><li>Wrap existing code and have that code executed as part of the testing process </li></ul><ul><li>Centralized results of all tests </li></ul>
    25. 25. Ordered Test <ul><li>Test to group (and order) other tests </li></ul><ul><li>Results of ordered tests either succeed or fail as a group </li></ul><ul><li>Used for scenario and module-level testing </li></ul><ul><li>Add any test except a load test to an ordered test </li></ul>
    26. 26. Manual Test
    27. 27. Test Case Management <ul><li>Organize tests into lists </li></ul><ul><li>Run tests </li></ul><ul><li>Filter and group the display of the tests </li></ul><ul><li>Import additional tests </li></ul><ul><li>Export tests </li></ul>Organize and manage test cases
    28. 28. The Test Lab <ul><li>Clients </li></ul><ul><ul><li>Tester machine </li></ul></ul><ul><ul><li>Used to create and edit tests / view results </li></ul></ul><ul><li>Servers </li></ul><ul><ul><li>Deployed test environment (web servers, load balancer, database server, etc.) </li></ul></ul><ul><li>Agents </li></ul><ul><ul><li>Computers running tests (multiple-threads, each a user) </li></ul></ul><ul><ul><li>Listen for commands from controller </li></ul></ul><ul><ul><li>Execute tests and pass results back to controller </li></ul></ul><ul><li>Controller </li></ul><ul><ul><li>Central computer that administers agents and collects results </li></ul></ul><ul><ul><li>Distributes load by applying weights to agents </li></ul></ul>
    29. 29. The Test Lab Database <ul><li>Run against a test version of the database </li></ul><ul><li>Test data should be same nature and size as production </li></ul><ul><li>Test a steady state of the database </li></ul><ul><li>Automate the initialization of the database </li></ul><ul><li>Extract data using rules to obfuscate personal information </li></ul><ul><ul><li>Visual Studio Database Developer </li></ul></ul>
    30. 30. Test Lab Database Initialization