Your SlideShare is downloading. ×
Evaluating Test Driven Development And Parameterized Unit Testing In Dot Net Presentation
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Evaluating Test Driven Development And Parameterized Unit Testing In Dot Net Presentation


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Evaluating Test Driven Development [TDD] and Parameterized Unit Test [PUT] in .NET Web Platform
    Prepared By:
    Md. Fazlul Alam Chowdhury,
    Date: 14.04.2010
  • 2. Topics Covered (i)
    Why Test Driven Development?
    TDD in .NET Environment
    • Database Unit Testing - DBUnit Vs. DB Pro
    • 3. Automated Web Testing– Selenium Vs. VS Web Testing
    • 4. Unit Testing - NUnit Vs. MS Test/MVC
    • 5. Load Testing – VS Load Testing
  • 6. Topics Covered (ii)
    MVC .NET Test Suite
    Parameterized Unit Testing (PUT) and Microsoft PEX
    Strengths and Limitation of Testing Tools
    Agile Influence in Visual Studio Team Foundation
    Scope of Future Research and Improvement
  • 7. Abstract (i)
    Web technology has become widely accepted application platform this days. It brings the remote and distributed application integration more successful. It brings parties together from distant places and helps making decisions more collaboratively and effectively. So, to have a reliable and well tested portal application Test Driven Design is obvious.
  • 8. Abstract (ii)
    Application Test Driven Development or simply TDD is an evolutionary approach to produce well tested reliable and shippable code and thus improving the quality of the software through Fault avoidance and Fault detection at early stages and Fault recovery with simpler effort. In Test First Design or TFD approach, codes are being written to fail, executing tests and then updating the functional codes to pass. TDD is basically the combination of TFD and Refactoring.
  • 9. Introduction
    This report is concentrated on Evaluating TDD in.NET environment. Web Application testing comprises of Functionality, Usability, Interface, Compatibility, Performance and Security which is a broad area from testing perspective. Not any of the available toolset performs testing for all the scenarios. So, we tried to narrow down some Web Testing Tools and concepts which contributed to the World Wide acceptance to .NET Applications.
  • 10. What is Quality?
    Quality is a term that stands for wide variety of aspects for a particular product or environment. How well the environment is or how good the product is based on some specific criteria could be regarded as Quality.
    According to Gerald Weinberg: “Quality is a Value to some person”
    By Dr. Tom DeMarco “Product's quality is a function of how much it changes the world for the better”
    Reliability, Scalability, Correctness, Completeness, Maintainability, Extensibility, Fault Tolerance etc. are the key selling points of Software Quality
  • 11. Quality Control Vs. Quality Assurance (i)
    ISO 9001 Definition of
    Quality Control
    The operational techniques and activities that are used to fulfill requirements for quality
    Quality Assurance
    All those planned and systematic activities implemented to provide adequate confidence that an entity will fulfill requirements for quality
  • 12. Quality Control Vs. Quality Assurance (ii)
    QC Vs. QA
    Product Process
    Reactive Proactive
    Line function Staff function
    Find defects Prevent defects
    Walkthrough Quality Audit
    Testing Defining Process
    Inspection Selection of tools
    Checkpoint review Training
  • 13. Test Driven Development
    The term TDD has been around since NASA’ s Project, Mercury in 1960s. But, when Kent Beck and Ward Cunningham introduced TDD as a eXtreme Programming (XP) practice, it started gaining popularity.
    So, What is TDD?
  • 14. What is Test Driven Development?
    Ref: A Survey of Evidence for Test Driven Development in Academia, Chetan Desai, David Janzen, Kyle Savage Computer Science California Polytechnic State University San Luis Obispo, California USA {cdesai, djanzen}
  • 15. TDD – Industrial Case Studies@Microsoft
    Two case studies were done at Microsoft to build empirical knowledge on the efficacy of TDD together with performing Cost-Benefit Economic Analysis on the utility of TDD:
    • Cost-benefit tradeoff between the increase in development time and
    • 16. Resulting improvement in software quality
    Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies, ThirumaleshBhat & NachiappanNagappan
  • 17. Windows networking team case study
    Project A: Context Factor
    Project A: Product Measure
    Project A: Outcome
    Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies, ThirumaleshBhat & NachiappanNagappan
  • 18. MSN Team case study
    Project B: Context Factor
    Project B: Product Measure
    Project B: Outcome
    Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies, ThirumaleshBhat & NachiappanNagappan
  • 19. Can someone write my Test?
  • 20. TDD Tools Used in this Research (i)
    Testing Frameworks:
    • Visual Studio Team System Unit Testing Framework
    • 21. MVC .NET Testing Framework
    • 22. Microsoft PEX
    Unit Testing:
    • NUnit
    • 23. MVC .NET with MOCK
    • 24. MS Test for Visual Studio
  • 25. TDD Tools Used in this Research (ii)
    Database Testing:
    Automated Web Testing
    • Selenium
    • 27. Visual Studio Automated Web Test
    Load/Stress Testing
    • Visual Studio Web Test
    • 28. WCat 6.3 (x86)
  • 29. Database Testing using DBUnit/DB Pro
  • 30. 19
  • 31. What is Database Testing and Why? (i)
    Database testing is a defined process to test database programmability objects. Now, the question comes: what are the database objects that requires testing to produce quality object?
    A Database consists of the following programmability objects:
  • 36. What is Database Unit Testing and Why?(ii)
    So, the database unit tests could be created before hand to test the database programmability objects and make sure that they produce the result as expected and existing functionality are working
    The following types of database unit tests could be performed:
    • Schema Test
    • 37. Feature/Functionality Test
    • 38. Security Test
    • 39. Stock data Test
    • 40. Referential Integrity Constraints
  • 41. What to Test?
    Refactoring Databases: Evolutionary Database Design, by Scott W. Ambler and Pramod J. Sadalage, Addison Wesley Professional    ISBN#:  0-321-29353-3
  • 42. Database Testing with DBUnit
    DBUnit is a Open Source Database Testing Framework which was created by Manuel Laflamme
    DBUnit is a Database Unit Testing Framework to Test Database Driven Classes, Procedures or Functions.
    NUnit, MVC .NET/Visual Studio Testing platforms can be used to build a standard test suite
    Following Operations can be performed using DBUnit
    Other Operations
    Composite Operation
    Transaction Operation
    Identity Insert Operation
  • 43. DBUnit Supported Databases
  • 44. Creating a DBUnit Test Class
    Create a XML Dataset or Export Data from Databases
    If the DBUnit Test Class Name is DbUnitTestCase then the Data Set name has to be DbUnitTestCaseDataset.xml and has to be in the same folder
    Flat XML:
    <?xml version="1.0" encoding="utf-8" ?>
    <table_name col1="value" />
  • 45. Create a Test Class and Extend DBTestCase
    using System;
    using System.Data;
    using System.Data.Sql;
    using System.Data.SqlClient;
    using System.Collections.Generic;
    using System.Reflection;
    using System.Xml;
    using DbUnit.Framework;
    using DbUnit.Core;
    using DbUnit.Dataset;
    using NUnit.Framework;
    using NMock2;
    namespace MyDBUnitProject
    public class MyDBUnitTestCase2 : DbUnitTestCase
    private SqlConnection connection = null;
    Mockery mocks;
    private IDataSet dataset;
  • 46. Implement SetUp and TearDown
    public void SetUpTestCase()
    mocks = new Mockery();
    dataset = new EmbeddedFlatXmlDataSet(this.GetType());
    connection = new SqlConnection("Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=BIteDB;Data Source=localhostsqlexpress");
    public void TearDownTestCase()
    connection.Close(); ;
  • 47. Implement Test Method
    public void TestQuerySQLMessage()
    string _sql = "SELECT AttachmentLink as ReturnValue from Message WHERE CAST(MsgContent AS varchar(4000)) ='My Test'";
    SqlCommand _cmd = new SqlCommand(_sql, connection);
    SqlDataReader _reader = _cmd.ExecuteReader();
    string _AttachmentLink = "";
    while (_reader.Read())
    _AttachmentLink = _reader["ReturnValue"].ToString();
    Assert.AreNotEqual("", _AttachmentLink);
    public void TableNameSetCorrectly()
    Assert.AreEqual("table_name", dataset.Tables[0].Name);
  • 48. Nunit/DBUnit Test Results
  • 49. DBUnit Benefits
    DBUnit provides a stable framework to create and adopt Database Test cases with minimal effort
    It can manage the state of database through the test’s life cycle
    At the end of testing the database can be placed back into its pre-test state
    A database can be populated with a desired data set before a test
    DBUnit test can easily automated by controlling major dependencies
    DBUnit test can be performed utilizing most types of database objects (Tables, Views, Stored Procedures)
    DBUnit Tests do not require manual intervention between tests
    DBUnit Test results doesn’t need manual interpretation
    Code stability increases in a Team environment
    Easy to Learn and Implement
    Wide range of support for standard databases
    DBUnit documentation is pretty well defined
    DBUnit tests are easily manageable
    It is easier to generate the seed data using the dump feature
  • 50. DBUnit Limitations
    Primary test data is XML but it is also possible to create test cases for standard databases
    Test Data generation plan is absent and so have to create and refresh the test dataset when Actual schema changes and so the seed and resulted data could be different
    Primary support for XML Dataset is Flat XML and need to have all columns defined in the first row
    DBUnit does not support multiple Resultset
  • 51. DBUnit – Short Demo
  • 52. Database Testing with MS DB Pro
    The desire on the part of the database developer to use familiar T-SQL scripts to express the unit-test logic and the desire to take advantage of powerful Microsoft® .NET Framework-targeted languages
    Encapsulating the test in T-SQL code and then hosting the T-SQL in C# or Visual Basic .NET code. The unit-test designer generates both, enabling either (or both) to be modified to meet individual needs.
    Database unit testing with Team System provides or supports the features for entire framework and using proper context [Authentication and Security Aspects etc.]
    Apply Test-Driven Development to your Database Projects - Jamie Laflen
  • 53. Two sample joined tables
  • 54. Stored Preocedure
    CREATE PROCEDURE [dbo].[uspGetEmployeeManagers]
    @EmployeeID [int]
    -- Use recursive query to list out all Employees required for a particular Manager
    WITH [EMP_cte]([EmployeeID], [ManagerID], [FirstName], [LastName], [Title], [RecursionLevel]) -- CTE name and columns
    AS (
    SELECT e.[EmployeeID], e.[ManagerID], c.[FirstName], c.[LastName], e.[Title], 0 -- Get the initial Employee
    FROM [HumanResources].[Employee] e
    INNER JOIN [Person].[Contact] c
    ON e.[ContactID] = c.[ContactID]
    WHERE e.[EmployeeID] = @EmployeeID
    SELECT e.[EmployeeID], e.[ManagerID], c.[FirstName], c.[LastName], e.[Title], [RecursionLevel] + 1 -- Join recursive member to anchor
    FROM [HumanResources].[Employee] e
    INNER JOIN [EMP_cte]
    ON e.[EmployeeID] = [EMP_cte].[ManagerID]
    INNER JOIN [Person].[Contact] c
    ON e.[ContactID] = c.[ContactID]
    -- Join back to Employee to return the manager name
    SELECT [EMP_cte].[RecursionLevel], [EMP_cte].[EmployeeID], [EMP_cte].[FirstName], [EMP_cte].[LastName],
    [EMP_cte].[ManagerID], c.[FirstName] AS 'ManagerFirstName', c.[LastName] AS 'ManagerLastName' -- Outer select from the CTE
    FROM [EMP_cte]
    INNER JOIN [HumanResources].[Employee] e
    ON [EMP_cte].[ManagerID] = e.[EmployeeID]
    INNER JOIN [Person].[Contact] c
    ON e.[ContactID] = c.[ContactID]
    ORDER BY [RecursionLevel], [ManagerID], [EmployeeID]
  • 55. DB Pro Test Results
  • 56. Adding a Data Generation Plan
  • 57. Selecting Tables for Test Data
  • 58. Data Column Configuration
  • 59. Generate Test Data before Unit Test
  • 60. DB Pro Benfits
    Feature Tests, Schema Tests, Security Tests, Stock-Data Tests can be performed using DB Pro
    Automatic generation of SQL script stubs of unit tests for stored procedures, functions, and triggers
    DB Pro Provides a set of built-in test conditions to verify results including the rows returned, scalar values, and execution time
    Automatic Generation of Test Data using Data Generation Tool
    It sets the database state through it’s data generation feature
    Same Test Data can be generated repeatedly based on seed value
    Pre-Test, Test and Post Test results could be written, executed and evaluated
  • 61. DB Pro Limitations
    Only supported database is MS SQL Server
    Need Visual Studio Team System - Database Edition to get the DB Pro Test Suite
    Supports C#/VB platforms for unit testing
    Data generation tool needed to use to generate test data and so database sandbox concept is not in there so that may have to work on real data and could be rolled back the data after testing
  • 62. DB Pro – Short Demo
  • 63. Web UI Testing using Selenium/Web Test
  • 64. Selenium for Web UI Testing
    Selenium is an automated web user interface testing tools that provides the environment to write a test case in respective platform [here .NET]. It has two components:
    • Server: A server which automatically launches and kills browsers, and acts as a HTTP proxy for web requests
    • 65. Client: Client libraries to Build a Test Case Using NUnit and Mock
  • 66. Selenium RC (Remote Control)
  • 67. Selenium Feature Matrix
  • 68. Start with Selenium Project in .NET
    Download Selenium from
    Download and Install NUnit or Any other testing suite
    Create a Visual Studio Test Project
    Reference the following DLLs: nmock.dll, nunit.core.dll, nunit. framework.dll, ThoughtWorks.Selenium.Core.dll, ThoughtWorks.Selenium. IntegrationTests.dll and ThoughtWorks.Selenium.UnitTests.dll
    Write a Selenium Test Case
  • 69. Selenium Test Class
    namespace MySeleniumTest
    public class MySeleniumTest1
    private ISelenium selenium;
    private StringBuilderverificationErrors;
    public void SetupTest()
    selenium = new DefaultSelenium("localhost", 2613, "*firefox",
    verificationErrors = new StringBuilder();
    public void TeardownTest()
    catch (Exception)
    // Ignore errors if unable to close the browser
    Assert.AreEqual("", verificationErrors.ToString());
    public void TheNewTest()
  • 70. Selenium Benefits
    Selenium could be used for the functional, regression, load testing of the web based applications. The automation tool could be implemented for post release validation with continuous integration tools like Hudson or CruiseControl.
    C#, Java, Perl, PHP, Python, Ruby are being supported as the programming/testing platform
    Selenium allows simple and powerful Document Object Level (DOM) level testing
    It supports traditional waterfall or Agile environments
    Selenium is a pretty good tool for continuous integration
    Selenium support AJAX
  • 71. Selenium Limitations
    Setting up Selenium Test Suite is complex and step by step documentation is hard to find
    Only supports web application though it has functional, regression, load testing features
    Diagnostics process is cumbersome
    In AJAX implementations, it doesn’t know when AJAX process is completed
  • 72. Selenium – Short Demo
  • 73. Visual Studio Web Test for UI Testing
    The main purpose of Web UI testing is to uncover functional problems of a Web Application. But, there are some key challenges for Web UI Testing:
    • Different Browser/Client Environment
    • 74. Client Side Scripting/Security Settings/Validation
    • 75. Session Variables and Timeouts
  • 76. How it works?
    The test harness system consists of three files (ASP .NET Page, Left.Html and Main.Html)
    ASP .NET Page is loaded into an HTML frame
    Left.Html which contains the test automation JScript code is loaded into another HTML frame
    Main.Html container for the two frames and holds a global variable called timesAppLoaded which tracks how many times the Web application under test has been loaded into its test frame
    Test scenario script uses the timesAppLoaded value to determine exactly what actions to take on the Web app and then reloads the Web application
    This Cycle continues until the system reaches a final state that you specify in the test script, and a pass or fail result is determined.
    Ref: Lightweight UI Test Automation for ASP.NET Web Apps, James McCaffrey,
  • 77. Web Test Container – Main.Html
    <script language="JScript">
    vartimesAppLoaded = 0;
    varscenarioID = "001";
    <frameset cols="40%,*">
    <frame src="left.html" name="left">
    <frame src="color.aspx" name="right" onload="left.updateState();">
  • 78. Visual Studio Web Test
  • 79. Adding Validation Rule
  • 80. Visual Studio Web Test Results
  • 81. Visual Studio Web UI Test Benefits
    Creation of Test Cases for Multiple Browser Support and browsers cross-compatibility can be tested using this tool
    Use different session/data variables for the same test and uncover different feature set according to selected criteria/setting. Let’s say use Admin and Regular user logins
    Support for AJAX/Silverlight
    May call and test other test cases(for an example DBUnit/Nunit )
    Convert Step to Code option allows testers to capture the outline of the test and dig into the code
  • 82. Visual Studio Web UI Test Limitations
    Need Visual Studio Team System to get the Web UI Testing Platform
    Supports C#/VB platforms for unit testing
    Web UI Test is designed for .NET platform
  • 83. MS Web Test – Short Demo
  • 84. Microsoft MVC .NET Test Suite
  • 85. Web Application Testing
    Web portal applications are being developed using MVC pattern paradigm. MVC stands for Model-View-Controller where :
    • Front End Application logics are being hosted in the “View” component
    • 86. All Business Logics are being hosted in “Model” component
    • 87. And Managed by “Controller” component
  • 88. MVC Pattern
  • 89. Web Portal Testing Constraints
    Testing results may vary from Dev, Test and Production environment
    Testing results may vary for users with different security settings or grants
    Blackbox testing is challenging as less control on the environment and html parsing is slow
    Applying TDD is difficult in web applications as the test cases are written to fail and then write the test case to pass followed by refactoring. So, needs a web portal mock up to test small changes time and again
  • 90. MVC Project Layers
    Abstract Layer
    Business Layer
    Data Layer
    MVC project
  • 91. ASP .NET Framework
    Master Pages
  • 92. ASP .NET MVC Framework
    Dynamic Data
  • 93. ASP.NET MVC Features (i)
    Separation of application tasks (input logic, business logic, and UI logic), testability, and test-driven development (TDD) by default
    All core contracts in the MVC framework are interface-based and can be tested by using mock objects, which are simulated objects that imitate the behavior of actual objects in the application
    Unit-test applications without having to run the controllers in an ASP.NET process, which makes unit testing fast and flexible
    Any unit-testing framework can be utilized that is compatible with the .NET Framework.
  • 94. ASP.NET MVC Features (ii)
    An extensible and pluggable framework
    ASP.NET MVC framework components are easily customizable
    The ASP.NET MVC framework also supports the use of Dependency Injection (DI) and Inversion of Control (IOC) container models. DI allows you to inject objects into a class, instead of relying on the class to create the object itself
    Support for existing ASP.NET features. ASP.NET MVC lets you use features such as forms authentication and Windows authentication, URL authorization, membership and roles, output and data caching, session and profile state management, health monitoring, the configuration system, and the provider architecture.
  • 95. Creating .NET Web App
  • 96. Creating a Unit Test Project
  • 97. Model-View-Controller (MVC)
  • 98. MVC Top Level Directories
  • 99. Typical MVC .NET Project Structure
  • 100. Change Password Method
    public ActionResultChangePassword(ChangePasswordModel model)
    if (ModelState.IsValid)
    if (MembershipService.ChangePassword(User.Identity.Name, model.OldPassword, model.NewPassword))
    return RedirectToAction("ChangePasswordSuccess");
    ModelState.AddModelError("", "The current password is incorrect or the new password is invalid.");
    // If we got this far, something failed, redisplay form
    return View(model);
  • 101. Change Password Test Method
    public void ChangePassword_Post_ReturnsRedirectOnSuccess()
    // Arrange
    AccountController controller = GetAccountController();
    ChangePasswordModel model = new ChangePasswordModel()
    OldPassword = "goodOldPassword",
    NewPassword = "goodNewPassword",
    ConfirmPassword = "goodNewPassword"
    // Act
    ActionResult result = controller.ChangePassword(model);
    // Assert
    Assert.IsInstanceOfType(result, typeof(RedirectToRouteResult));
    RedirectToRouteResultredirectResult = (RedirectToRouteResult)result;
    Assert.AreEqual("ChangePasswordSuccess", redirectResult.RouteValues["action"]);
  • 102. Right Click and Create Test Method
  • 103. Membership Service Mocking
    private class MockMembershipService : IMembershipService
    public intMinPasswordLength
    get { return 10; }
    public boolValidateUser(string userName, string password)
    return (userName == "someUser" && password == "goodPassword");
    public MembershipCreateStatusCreateUser(string userName, string password, string email)
    if (userName == "duplicateUser")
    return MembershipCreateStatus.DuplicateUserName;
    // verify that values are what we expected
    Assert.AreEqual("goodPassword", password);
    Assert.AreEqual("goodEmail", email);
    return MembershipCreateStatus.Success;
    public boolChangePassword(string userName, string oldPassword, string newPassword)
    return (userName == "someUser" && oldPassword == "goodOldPassword" && newPassword == "goodNewPassword");
  • 104. Unit Test Results in .NET 3.5
  • 105. ASP.NET MVC Benefits
    Resolving complexity by dividing an application into the model, the view, and the controller
    No view state or server-based forms. So, provides full control over the behavior of an application
    Front Controller pattern processes Web application requests through a single controller
    Creating application that supports a rich routing infrastructure
    Provides better support for test-driven development (TDD)
    Fits into large teams of Developers and Web designers who need a high degree of control over the application behavior.
  • 106. ASP.NET MVC Limitations
    Doesn’t use event model to preserve state over HTTP
    Doesn’t use view state or server-based forms and so hard to manage states
    Doesn’t fit well for small teams to implement RAD
    Complex design for application development, because the components (the Page class, controls, and so on) are tightly integrated and usually require more code than the Traditional Web Application model
  • 107. MVC .Net – Short Demo
  • 108. Parameterized Unit Testing (PUT) and Microsoft Pex
  • 109. Unit Testing At a Glance
    Three sections of Unit Tests:
    void Add() {
    int item = 3;
    var list = new List();
    var count = list.Count;
    Assert.AreEqual(1, count);
    Ref : Parameterized Unit Testing: Principles, Techniques, and Applications in Practice, Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research) Tao Xie (North Carolina State University)
  • 112. Incomplete Test Coverage with Fixed Data
    • Bad data may result in redundant, or worse, incomplete test suites.
    • 113. Fixed data values become invalid when product changes
    • 114. Why choose a value which may not make sense?
    Ref : Parameterized Unit Testing: Principles, Techniques, and Applications in Practice, Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research) Tao Xie (North Carolina State University)
  • 115. What is Parameterized Unit Test?
    Parameterized Unit Test is a Unit Test with Parameters
    Separation of concerns
    Data is generated by a tool
    Developer can focus on functional specification
    void Add(List list, int item) {
    var count = list.Count;
    Assert.AreEqual(count + 1, list.Count);
    Ref : Parameterized Unit Testing: Principles, Techniques, and Applications in Practice, Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research) Tao Xie (North Carolina State University)
  • 116. PUT Sample
    Ref: Parameterized Unit Tests, Nikolai Tillmann (, Wolfram Schulte (, Microsoft Research
    One Microsoft Way, Redmond WA USA
  • 117. Parameterized Unit Testing (PUT)
    Some popular unit testing tools, like Junit, Nunit doesn’t support the automation of creating unit tests
    In some cases it even takes more lines of code to create the unit tests than more implementation being tested
    Some Automatic test generation tools may not even track the divide-by-zero type errors rather than specific error message
    PUT or Parameterized Unit Test accepts behavioral parameters/argument sets for a specific test method
    Ref: Automating Software Testing Using Program Analysis, Patrice Godefroid, Peli de Halleux, Aditya V. Nori, Sriram K. Rajamani, Wolfram Schulte, and Nikolai Tillmann, Microsoft Research Michael Y. Levin, Microsoft Center for Software Excellence
  • 118. PUT Contributions
    They allow unit tests to play a greater role as specifications of program behavior. In fact, PUTs are axiomatic specifications.
    They enable automatic case analysis, which avoids writing implementation-specific unit tests.
    Their generated test cases often result in complete path coverage of the implementation, which amounts to a formal proof of the PUT’s assertions.
    Ref: Parameterized Unit Tests, Nikolai Tillmann (, Wolfram Schulte (, Microsoft Research
    One Microsoft Way, Redmond WA USA
  • 119. PUT Framework
    Symbolic State
    Symbolic Evaluation
    Test Case generation
    Ref: Parameterized Unit Tests, Nikolai Tillmann (, Wolfram Schulte (, Microsoft Research
    One Microsoft Way, Redmond WA USA
  • 120. Symbolic State
    Its is basically the state of program execution. It can contain expressions with symbolic variables.
    Symbolic expressions:
    E = | o object ids, infinite set of potential object identifiers
    | v variables, set of symbolic variable identifiers
    | t types, set of type identifiers
    | f(E bar) function application, set of function symbols
    | For All (v bar).E universal quantification,
    where x to denote lists of items x1, . . . , xn.
    Function symbols: equals(x, y) denotes whether x and y represent the same value for value Types, type(x) denotes the runtime type of object x, and len(x) the length of array x
    Heaps:No of Times a method being called in a nested method call
    Symbolic state: A symbolic state is a 5-tuple S = (O, A, He,Hi, X), where the current set of objects O is a subset of ObjectId, the program stack A is a stack of activation records, He and Hi are expressions denoting the extensional heap and the intentional heap respectively, and finally, X, an object expression, denotes the current exception. Say: O(S) is being written for projection on S
    Ref: Parameterized Unit Tests, Nikolai Tillmann (, Wolfram Schulte (, Microsoft Research
    One Microsoft Way, Redmond WA USA
  • 121. Symbolic Evaluation
    Symbolic Evaluation describes the effect of the current instruction from a given constrained state (S,C).
    Ref: Parameterized Unit Tests, Nikolai Tillmann (, Wolfram Schulte (, Microsoft Research
    One Microsoft Way, Redmond WA USA
  • 122. Axioms
    Axiom is basically of summary of external behavior
    public void TestQuickSort(int[] a)
    Assume.IsTrue(a != null);
    QuickSort.Sort(a, 0, a.Length - 1);
    for (int i = 0; i < a.Length - 1; i++)
    Assert.IsTrue(a[i] <= a[i + 1]);
    Ref: Parameterized Unit Tests, Nikolai Tillmann (, Wolfram Schulte (, Microsoft Research
    One Microsoft Way, Redmond WA USA
  • 123. Test Case Generation
    Each transition sequence (S0,C0) ! (S1,C1) ! · · · represents a unique execution path of the program.
    public void TestQuickSort597()
    int[] ints = new int[2];
    ints[1] = 1;
    public void TestQuickSort512()
    int[] ints = new int[3];
    ints[0] = -2147475453;
    ints[1] = 1073750016;
    ints[2] = 8194;
    Ref: Parameterized Unit Tests, Nikolai Tillmann (, Wolfram Schulte (, Microsoft Research
    One Microsoft Way, Redmond WA USA
  • 124. Constraints
    A constraint on a symbolic state is a pair C = (BG, PC), where BG is the static background, which only depends on the program declarations, and PC is the dynamic path condition, which is built up during symbolic evaluation
    Ref: Parameterized Unit Tests, Nikolai Tillmann (, Wolfram Schulte (, Microsoft Research
    One Microsoft Way, Redmond WA USA
  • 125. What is Pex?
    Pex stands for Program Exploration which is a white box test generation tool developed at Microsoft Research that helps developers to write PUTs in .NET platform and thus generate automated test cases
    For each PUT, Pex uses dynamic test-generation techniques to compute a set of input values that exercise all the statements and assertions in the analyzed program which eventually cover all reachable branches
    Ref: Automating Software Testing Using Program Analysis, Patrice Godefroid, Peli de Halleux, Aditya V. Nori, Sriram K. Rajamani, Wolfram Schulte, and Nikolai Tillmann, Microsoft Research Michael Y. Levin, Microsoft Center for Software Excellence
  • 126. Pex Sample Project
  • 127. Sample Web Method to Test [ServicePoints.asmx]
  • 128. Pex Test Case: Test Axiom [ServicePointsTest.cs]
  • 129. Pex Test Method 1: Null Validation
  • 130. Pex Test Method 2: Validation with valid string value
  • 131. NUnit Test Case
    public class UserTest
    User _user;
    public void Setup()
    _user = new User();
    public void EmailTest()
    _user.Email = “";
    Assert.AreEqual(“", _user.Email);
    public void IsValidTestWhenPasswordsMismatch()
    _user.Password = “SENG";
    _user.Password = “CPSC";
    public void IsValidTestWhenPasswordsMatch()
    _user.FirstName = “Fazlul";
    _user.LastName = “Chowdhury";
    _user.Email = " ";
    _user.Password = " SENG ";
    _user.PasswordConfirmation = " SENG ";
    public void UserNameTest()
    _user.Email = " ";
    Assert.AreEqual(" ", _user.UserName);
    _user.UserName = " ";
    Assert.AreEqual(" ", _user.Email);
  • 132. Instantiating PUT in PEX
    public string ConvertToUpper([PexAssumeUnderTest]MyPexTest1Class target, string Input)
    string result = target.ConvertToUpper(Input);
    return result;
    public void ConvertToUpperAgain(string Input)
    MyPexTest1Class MyP = new MyPexTest1Class();
    var First = MyP.ConvertToUpper(Input);
    var Second = MyP.ConvertToUpper(First);
    Assert.AreEqual(First, Second);
  • 133. PUT & PEX benefits
    PUT specifies the external behavior of the methods for all test arguments and so the tests cover all possible test environment and constraints
    PUTs can be instantiated to re-obtain traditional closed unit tests and to re-verify the result of tests
    PUT enables automatic case analysis, which avoids writing implementation-specific unit tests
    Using PUT in PEX, it is fairly simple to auto generate test cases and cover all possible testing scenarios, constraints and values
  • 134. PUT & PEX limitations
    PEX works for deterministic model of test methods and goes in cycle for non deterministic methods until hits exploration bounds
    PEX doesn’t work in Multi-Thread model
    PEX can analyze any .NET language but the code generation only supports C#
    PEX uses an automatic constraint solver to determine which values are relevant for the test and the code-under-test but which is also limited [Say: floating point arithmetic cannot be reasoned precisely]
    Lack of Tests for Oracle
    Advanced Concepts : PUT in PEX,
  • 135. Microsoft Pex – Short Demo
  • 136. Load Testing
  • 137. Load Testing on Web Applications
    To perform a load test on a web application we need to:
    • Identify the performance-critical scenarios
    • 138. Identify the workload profile for distributing the entire load among the key scenarios
    • 139. Identify the metrics that we want to collect in order to verify them against your performance objectives.
    • 140. Design tests to simulate the load
    • 141. Use tools to implement the load according to the designed tests, and capture the metrics
    • 142. Analyze the metric data captured during the test
    By using an iterative testing process, these steps should help us achieve our performance objectives
    Performance Testing Guidance for Web Applications, J.D. Meier, Carlos Farre, PrashantBansode, Scott Barber, and Dennis Rea
  • 143. Visual Studio Load Testing
  • 144. Visual Studio Load Testing Results
  • 145. VS Load Test – Short Demo
  • 146. Advantages of TDD centric Web Applications design
    Code will be separated from the Data layer
    Easy to understand and code transfer
    It increases the system performance
    Easy to maintain documentation
  • 147. Conclusion (i)
    This research was done to investigate numerous testing tools in .NET platform. We found that various testing tools are specialized in specific testing contexts. NUnit/Visual Studio Testing Tools/MVC .NET and Microsoft PEX are built for unit testing where as DBUnit/DB Pro are specialized in Database Testing, Selenium and MS Web test are for Web UI Testing and VS Load Test and Wcat are being used for Load/Stress testing. But, Microsoft Team System Developer edition comes with a full blown test suite of which VS Test is for Unit Testing, DB pro is for database testing, Web Test for web UI testing and VS Load Test is for Load testing. External testing tools like DBUnit and Nunit test cases can also be generated from Visual Studio Testing suite. MVC .NET implements the Model, View and Controller pattern which makes TDD easier for a Web Application. MVC resolves the Server based forms which were difficult to test. MVC framework fits into large teams of Developers and Web designers who need a high degree of control over the application behavior.
  • 148. Conclusion (ii)
    This presentation also highlighted the concept of Parameterized Unit Tests or simply PUT which is basically a generalization of established closed unit tests. Parameterized Unit Test or PUT has been implemented in Microsoft PEX to generate automated Unit tests. Behavioral parameters can be injected into PUT to cover a particular implementation. State Change, Return Values and Exceptional return values states the Method's behavior. Detailed demonstration of PEX covered the Axiomatic approach of Test Methods and it’s test coverage scenario together with automated and regression test generation.
  • 149. References
    Evaluating Current Testing Processes of Web-Portal Applications
    Multiple Perspectives on Executable Acceptance Test-Driven Development
    Building Web Apps without Web Forms, Link:
    MVC architecture in ASP. Net using C# and Microsoft Data Access Application block By  MunirShaikh February 28, 2006 , Link:
    Good and Practical Ideas for Testers ,
    Low-Level Web App UI Test Automation, James McCaffrey
    Advanced Concepts : PUT in PEX,
  • 150. Thank You