1. Evaluating Test Driven Development [TDD] and Parameterized Unit Test [PUT] in .NET Web Platform Prepared By: Md. Fazlul Alam Chowdhury, Email: mdfachowdhury@hotmail.com Blog: http://fazlulchowdhury.blogspot.com Date: 14.04.2010 1

3. Automated Web Testing– Selenium Vs. VS Web Testing

4. Unit Testing - NUnit Vs. MS Test/MVC

5. Load Testing – VS Load Testing 2

6. Topics Covered (ii) MVC .NET Test Suite Parameterized Unit Testing (PUT) and Microsoft PEX Strengths and Limitation of Testing Tools Agile Influence in Visual Studio Team Foundation Scope of Future Research and Improvement Conclusion References 3

7. Abstract (i) Web technology has become widely accepted application platform this days. It brings the remote and distributed application integration more successful. It brings parties together from distant places and helps making decisions more collaboratively and effectively. So, to have a reliable and well tested portal application Test Driven Design is obvious. 4

8. Abstract (ii) Application Test Driven Development or simply TDD is an evolutionary approach to produce well tested reliable and shippable code and thus improving the quality of the software through Fault avoidance and Fault detection at early stages and Fault recovery with simpler effort. In Test First Design or TFD approach, codes are being written to fail, executing tests and then updating the functional codes to pass. TDD is basically the combination of TFD and Refactoring. 5

9. Introduction This report is concentrated on Evaluating TDD in.NET environment. Web Application testing comprises of Functionality, Usability, Interface, Compatibility, Performance and Security which is a broad area from testing perspective. Not any of the available toolset performs testing for all the scenarios. So, we tried to narrow down some Web Testing Tools and concepts which contributed to the World Wide acceptance to .NET Applications. 6

10. What is Quality? Quality is a term that stands for wide variety of aspects for a particular product or environment. How well the environment is or how good the product is based on some specific criteria could be regarded as Quality. According to Gerald Weinberg: “Quality is a Value to some person” By Dr. Tom DeMarco “Product's quality is a function of how much it changes the world for the better” Reliability, Scalability, Correctness, Completeness, Maintainability, Extensibility, Fault Tolerance etc. are the key selling points of Software Quality http://ase.cpsc.ucalgary.ca/ase/uploads/SENG515615W2010/Lecture4_Agile_Engineering_Practices.pdf 7

11. Quality Control Vs. Quality Assurance (i) ISO 9001 Definition of Quality Control The operational techniques and activities that are used to fulfill requirements for quality Quality Assurance All those planned and systematic activities implemented to provide adequate confidence that an entity will fulfill requirements for quality http://elsmar.com/pdf_files/QC%20vs%20QA.pdf 8

12. Quality Control Vs. Quality Assurance (ii) QC Vs. QA Product Process Reactive Proactive Line function Staff function Find defects Prevent defects Walkthrough Quality Audit Testing Defining Process Inspection Selection of tools Checkpoint review Training http://elsmar.com/pdf_files/QC%20vs%20QA.pdf 9

13. Test Driven Development The term TDD has been around since NASA’ s Project, Mercury in 1960s. But, when Kent Beck and Ward Cunningham introduced TDD as a eXtreme Programming (XP) practice, it started gaining popularity. So, What is TDD? 10

14. What is Test Driven Development? Ref: A Survey of Evidence for Test Driven Development in Academia, Chetan Desai, David Janzen, Kyle Savage Computer Science California Polytechnic State University San Luis Obispo, California USA {cdesai, djanzen}@calpoly.edu 11

16. Resulting improvement in software qualityEvaluating the Efficacy of Test-Driven Development: Industrial Case Studies, ThirumaleshBhat & NachiappanNagappan 12

17. Windows networking team case study Project A: Context Factor Project A: Product Measure Project A: Outcome Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies, ThirumaleshBhat & NachiappanNagappan 13

18. MSN Team case study Project B: Context Factor Project B: Product Measure Project B: Outcome Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies, ThirumaleshBhat & NachiappanNagappan 14

19. Can someone write my Test? 15

21. MVC .NET Testing Framework

23. MVC .NET with MOCK

24. MS Test for Visual Studio16

28. WCat 6.3 (x86)17

29. Database Testing using DBUnit/DB Pro 18

30. 19

32. DB Functions

33. Triggers

34. Rules

35. DB Assemblies20

37. Feature/Functionality Test

38. Security Test

39. Stock data Test

40. Referential Integrity Constraints21

41. What to Test? Refactoring Databases: Evolutionary Database Design, by Scott W. Ambler and Pramod J. Sadalage, Addison Wesley Professional ISBN#: 0-321-29353-3 http://www.ambysoft.com/books/refactoringDatabases.html 22

42. Database Testing with DBUnit DBUnit is a Open Source Database Testing Framework which was created by Manuel Laflamme DBUnit is a Database Unit Testing Framework to Test Database Driven Classes, Procedures or Functions. NUnit, MVC .NET/Visual Studio Testing platforms can be used to build a standard test suite Following Operations can be performed using DBUnit UPDATE TRUNCATE INSERT REFRESH DELETE CLEAN_INSERT DELETE_ALL NONE Other Operations Composite Operation Transaction Operation Identity Insert Operation 23

43. DBUnit Supported Databases Ref: http://dbunit.wikidot.com/supportedrdbms 24

44. Creating a DBUnit Test Class Create a XML Dataset or Export Data from Databases 25 If the DBUnit Test Class Name is DbUnitTestCase then the Data Set name has to be DbUnitTestCaseDataset.xml and has to be in the same folder Flat XML: <?xml version="1.0" encoding="utf-8" ?> <dataset> <table_name col1="value" /> </dataset>

45. Create a Test Class and Extend DBTestCase using System; using System.Data; using System.Data.Sql; using System.Data.SqlClient; using System.Collections.Generic; using System.Reflection; using System.Xml; using DbUnit.Framework; using DbUnit.Core; using DbUnit.Dataset; using NUnit.Framework; using NMock2; namespace MyDBUnitProject { [TestFixture] public class MyDBUnitTestCase2 : DbUnitTestCase { private SqlConnection connection = null; Mockery mocks; private IDataSet dataset; 26

46. Implement SetUp and TearDown [SetUp] public void SetUpTestCase() { mocks = new Mockery(); dataset = new EmbeddedFlatXmlDataSet(this.GetType()); connection = new SqlConnection("Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=BIteDB;Data Source=localhostsqlexpress"); connection.Open(); } [TearDown] public void TearDownTestCase() { connection.Close(); ; } 27

47. Implement Test Method [Test] public void TestQuerySQLMessage() { string _sql = "SELECT AttachmentLink as ReturnValue from Message WHERE CAST(MsgContent AS varchar(4000)) ='My Test'"; SqlCommand _cmd = new SqlCommand(_sql, connection); SqlDataReader _reader = _cmd.ExecuteReader(); string _AttachmentLink = ""; while (_reader.Read()) { _AttachmentLink = _reader["ReturnValue"].ToString(); } Assert.AreNotEqual("", _AttachmentLink); } [Test] public void TableNameSetCorrectly() { Assert.AreEqual("table_name", dataset.Tables[0].Name); } 28

48. Nunit/DBUnit Test Results 29

49. DBUnit Benefits DBUnit provides a stable framework to create and adopt Database Test cases with minimal effort It can manage the state of database through the test’s life cycle At the end of testing the database can be placed back into its pre-test state A database can be populated with a desired data set before a test DBUnit test can easily automated by controlling major dependencies DBUnit test can be performed utilizing most types of database objects (Tables, Views, Stored Procedures) DBUnit Tests do not require manual intervention between tests DBUnit Test results doesn’t need manual interpretation Code stability increases in a Team environment Easy to Learn and Implement Wide range of support for standard databases DBUnit documentation is pretty well defined DBUnit tests are easily manageable It is easier to generate the seed data using the dump feature 30

50. DBUnit Limitations Primary test data is XML but it is also possible to create test cases for standard databases Test Data generation plan is absent and so have to create and refresh the test dataset when Actual schema changes and so the seed and resulted data could be different Primary support for XML Dataset is Flat XML and need to have all columns defined in the first row DBUnit does not support multiple Resultset 31

51. DBUnit – Short Demo 32

52. Database Testing with MS DB Pro The desire on the part of the database developer to use familiar T-SQL scripts to express the unit-test logic and the desire to take advantage of powerful Microsoft® .NET Framework-targeted languages Encapsulating the test in T-SQL code and then hosting the T-SQL in C# or Visual Basic .NET code. The unit-test designer generates both, enabling either (or both) to be modified to meet individual needs. Database unit testing with Team System provides or supports the features for entire framework and using proper context [Authentication and Security Aspects etc.] Apply Test-Driven Development to your Database Projects - Jamie Laflen http://msdn.microsoft.com/en-us/magazine/cc164243.aspx#S2 33

53. Two sample joined tables 34

54. Stored Preocedure CREATE PROCEDURE [dbo].[uspGetEmployeeManagers] @EmployeeID [int] AS BEGIN SET NOCOUNT ON; -- Use recursive query to list out all Employees required for a particular Manager WITH [EMP_cte]([EmployeeID], [ManagerID], [FirstName], [LastName], [Title], [RecursionLevel]) -- CTE name and columns AS ( SELECT e.[EmployeeID], e.[ManagerID], c.[FirstName], c.[LastName], e.[Title], 0 -- Get the initial Employee FROM [HumanResources].[Employee] e INNER JOIN [Person].[Contact] c ON e.[ContactID] = c.[ContactID] WHERE e.[EmployeeID] = @EmployeeID UNION ALL SELECT e.[EmployeeID], e.[ManagerID], c.[FirstName], c.[LastName], e.[Title], [RecursionLevel] + 1 -- Join recursive member to anchor FROM [HumanResources].[Employee] e INNER JOIN [EMP_cte] ON e.[EmployeeID] = [EMP_cte].[ManagerID] INNER JOIN [Person].[Contact] c ON e.[ContactID] = c.[ContactID] ) -- Join back to Employee to return the manager name SELECT [EMP_cte].[RecursionLevel], [EMP_cte].[EmployeeID], [EMP_cte].[FirstName], [EMP_cte].[LastName], [EMP_cte].[ManagerID], c.[FirstName] AS 'ManagerFirstName', c.[LastName] AS 'ManagerLastName' -- Outer select from the CTE FROM [EMP_cte] INNER JOIN [HumanResources].[Employee] e ON [EMP_cte].[ManagerID] = e.[EmployeeID] INNER JOIN [Person].[Contact] c ON e.[ContactID] = c.[ContactID] ORDER BY [RecursionLevel], [ManagerID], [EmployeeID] OPTION (MAXRECURSION 25) END; 35

55. DB Pro Test Results 36

56. Adding a Data Generation Plan 37

57. Selecting Tables for Test Data 38

58. Data Column Configuration 39

59. Generate Test Data before Unit Test 40

60. DB Pro Benfits Feature Tests, Schema Tests, Security Tests, Stock-Data Tests can be performed using DB Pro Automatic generation of SQL script stubs of unit tests for stored procedures, functions, and triggers DB Pro Provides a set of built-in test conditions to verify results including the rows returned, scalar values, and execution time Automatic Generation of Test Data using Data Generation Tool It sets the database state through it’s data generation feature Same Test Data can be generated repeatedly based on seed value Pre-Test, Test and Post Test results could be written, executed and evaluated 41

61. DB Pro Limitations Only supported database is MS SQL Server Need Visual Studio Team System - Database Edition to get the DB Pro Test Suite Supports C#/VB platforms for unit testing Data generation tool needed to use to generate test data and so database sandbox concept is not in there so that may have to work on real data and could be rolled back the data after testing 42

62. DB Pro – Short Demo 43

63. Web UI Testing using Selenium/Web Test 44

65. Client: Client libraries to Build a Test Case Using NUnit and Mock45

66. Selenium RC (Remote Control) http://seleniumhq.org/projects/remote-control/ 46

67. Selenium Feature Matrix Ref: http://wiki.openqa.org/pages/viewpage.action?pageId=763 47

68. Start with Selenium Project in .NET Download Selenium from Download and Install NUnit or Any other testing suite Create a Visual Studio Test Project Reference the following DLLs: nmock.dll, nunit.core.dll, nunit. framework.dll, ThoughtWorks.Selenium.Core.dll, ThoughtWorks.Selenium. IntegrationTests.dll and ThoughtWorks.Selenium.UnitTests.dll Write a Selenium Test Case 48

69. Selenium Test Class namespace MySeleniumTest { [TestFixture] public class MySeleniumTest1 { private ISelenium selenium; private StringBuilderverificationErrors; [SetUp] public void SetupTest() { selenium = new DefaultSelenium("localhost", 2613, "*firefox", "http://localhost:2613"); selenium.Start(); verificationErrors = new StringBuilder(); } [TearDown] public void TeardownTest() { try { selenium.Stop(); } catch (Exception) { // Ignore errors if unable to close the browser } Assert.AreEqual("", verificationErrors.ToString()); } [Test] public void TheNewTest() { selenium.Open("/Home/"); selenium.Click("link=About"); selenium.WaitForPageToLoad("30000"); } } } 49

70. Selenium Benefits Selenium could be used for the functional, regression, load testing of the web based applications. The automation tool could be implemented for post release validation with continuous integration tools like Hudson or CruiseControl. C#, Java, Perl, PHP, Python, Ruby are being supported as the programming/testing platform Selenium allows simple and powerful Document Object Level (DOM) level testing It supports traditional waterfall or Agile environments Selenium is a pretty good tool for continuous integration Selenium support AJAX 50

71. Selenium Limitations Setting up Selenium Test Suite is complex and step by step documentation is hard to find Only supports web application though it has functional, regression, load testing features Diagnostics process is cumbersome In AJAX implementations, it doesn’t know when AJAX process is completed 51

72. Selenium – Short Demo 52

74. Client Side Scripting/Security Settings/Validation

75. Session Variables and Timeouts53

76. How it works? The test harness system consists of three files (ASP .NET Page, Left.Html and Main.Html) ASP .NET Page is loaded into an HTML frame Left.Html which contains the test automation JScript code is loaded into another HTML frame Main.Html container for the two frames and holds a global variable called timesAppLoaded which tracks how many times the Web application under test has been loaded into its test frame Test scenario script uses the timesAppLoaded value to determine exactly what actions to take on the Web app and then reloads the Web application This Cycle continues until the system reaches a final state that you specify in the test script, and a pass or fail result is determined. Ref: Lightweight UI Test Automation for ASP.NET Web Apps, James McCaffrey, http://msdn.microsoft.com/en-us/magazine/cc163814.aspx#S1 54

77. Web Test Container – Main.Html <html> <head> <script language="JScript"> vartimesAppLoaded = 0; varscenarioID = "001"; </script> </head> <frameset cols="40%,*"> <frame src="left.html" name="left"> <frame src="color.aspx" name="right" onload="left.updateState();"> </frameset> </html> 55

78. Visual Studio Web Test 56

79. Adding Validation Rule 57

80. Visual Studio Web Test Results 58

81. Visual Studio Web UI Test Benefits Creation of Test Cases for Multiple Browser Support and browsers cross-compatibility can be tested using this tool Use different session/data variables for the same test and uncover different feature set according to selected criteria/setting. Let’s say use Admin and Regular user logins Support for AJAX/Silverlight May call and test other test cases(for an example DBUnit/Nunit ) Convert Step to Code option allows testers to capture the outline of the test and dig into the code 59

82. Visual Studio Web UI Test Limitations Need Visual Studio Team System to get the Web UI Testing Platform Supports C#/VB platforms for unit testing Web UI Test is designed for .NET platform 60

83. MS Web Test – Short Demo 61

84. Microsoft MVC .NET Test Suite 62

86. All Business Logics are being hosted in “Model” component

87. And Managed by “Controller” component63

88. MVC Pattern http://msdn.microsoft.com/en-us/library/ms978748.aspx 64

89. Web Portal Testing Constraints Testing results may vary from Dev, Test and Production environment Testing results may vary for users with different security settings or grants Blackbox testing is challenging as less control on the environment and html parsing is slow Applying TDD is difficult in web applications as the test cases are written to fail and then write the test case to pass followed by refactoring. So, needs a web portal mock up to test small changes time and again 65

90. MVC Project Layers Abstract Layer Business Layer Data Layer MVC project 66

91. ASP .NET Framework Modules Globalization Caching Controls Master Pages Pages Roles Profile Membership Handlers Intrinsics Etc. 67

92. ASP .NET MVC Framework ASP.NET Dynamic Data Presentation ASP.NET WebForms ASP.NET MVC ASP.NET Core Runtime http://www.asp.net/(S(d35rmemuuono1wvm1gsp2n45))/mvc/ 68

93. ASP.NET MVC Features (i) Separation of application tasks (input logic, business logic, and UI logic), testability, and test-driven development (TDD) by default All core contracts in the MVC framework are interface-based and can be tested by using mock objects, which are simulated objects that imitate the behavior of actual objects in the application Unit-test applications without having to run the controllers in an ASP.NET process, which makes unit testing fast and flexible Any unit-testing framework can be utilized that is compatible with the .NET Framework. 69

94. ASP.NET MVC Features (ii) An extensible and pluggable framework ASP.NET MVC framework components are easily customizable The ASP.NET MVC framework also supports the use of Dependency Injection (DI) and Inversion of Control (IOC) container models. DI allows you to inject objects into a class, instead of relying on the class to create the object itself Support for existing ASP.NET features. ASP.NET MVC lets you use features such as forms authentication and Windows authentication, URL authorization, membership and roles, output and data caching, session and profile state management, health monitoring, the configuration system, and the provider architecture. 70

95. Creating .NET Web App 71

96. Creating a Unit Test Project 72

97. Model-View-Controller (MVC) 73

98. MVC Top Level Directories 74

99. Typical MVC .NET Project Structure 75

100. Change Password Method [Authorize] [HttpPost] public ActionResultChangePassword(ChangePasswordModel model) { if (ModelState.IsValid) { if (MembershipService.ChangePassword(User.Identity.Name, model.OldPassword, model.NewPassword)) { return RedirectToAction("ChangePasswordSuccess"); } else { ModelState.AddModelError("", "The current password is incorrect or the new password is invalid."); } } // If we got this far, something failed, redisplay form return View(model); } 76

101. Change Password Test Method [TestMethod] public void ChangePassword_Post_ReturnsRedirectOnSuccess() { // Arrange AccountController controller = GetAccountController(); ChangePasswordModel model = new ChangePasswordModel() { OldPassword = "goodOldPassword", NewPassword = "goodNewPassword", ConfirmPassword = "goodNewPassword" }; // Act ActionResult result = controller.ChangePassword(model); // Assert Assert.IsInstanceOfType(result, typeof(RedirectToRouteResult)); RedirectToRouteResultredirectResult = (RedirectToRouteResult)result; Assert.AreEqual("ChangePasswordSuccess", redirectResult.RouteValues["action"]); } 77

102. Right Click and Create Test Method 78

103. Membership Service Mocking private class MockMembershipService : IMembershipService { public intMinPasswordLength { get { return 10; } } public boolValidateUser(string userName, string password) { return (userName == "someUser" && password == "goodPassword"); } public MembershipCreateStatusCreateUser(string userName, string password, string email) { if (userName == "duplicateUser") { return MembershipCreateStatus.DuplicateUserName; } // verify that values are what we expected Assert.AreEqual("goodPassword", password); Assert.AreEqual("goodEmail", email); return MembershipCreateStatus.Success; } public boolChangePassword(string userName, string oldPassword, string newPassword) { return (userName == "someUser" && oldPassword == "goodOldPassword" && newPassword == "goodNewPassword"); } } 79

104. Unit Test Results in .NET 3.5 80

105. ASP.NET MVC Benefits Resolving complexity by dividing an application into the model, the view, and the controller No view state or server-based forms. So, provides full control over the behavior of an application Front Controller pattern processes Web application requests through a single controller Creating application that supports a rich routing infrastructure Provides better support for test-driven development (TDD) Fits into large teams of Developers and Web designers who need a high degree of control over the application behavior. 81

106. ASP.NET MVC Limitations Doesn’t use event model to preserve state over HTTP Doesn’t use view state or server-based forms and so hard to manage states Doesn’t fit well for small teams to implement RAD Complex design for application development, because the components (the Page class, controls, and so on) are tightly integrated and usually require more code than the Traditional Web Application model 82

107. MVC .Net – Short Demo 83

108. Parameterized Unit Testing (PUT) and Microsoft Pex 84

110. Method Sequence

111. Assertionsvoid Add() { int item = 3; var list = new List(); list.Add(item); var count = list.Count; Assert.AreEqual(1, count); } Ref : Parameterized Unit Testing: Principles, Techniques, and Applications in Practice, Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research) Tao Xie (North Carolina State University)http://research.microsoft.com/en-us/projects/pex/pexpublictutorialslides.pptx 85

113. Fixed data values become invalid when product changes

114. Why choose a value which may not make sense?Ref : Parameterized Unit Testing: Principles, Techniques, and Applications in Practice, Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research) Tao Xie (North Carolina State University)http://research.microsoft.com/en-us/projects/pex/pexpublictutorialslides.pptx 86

115. What is Parameterized Unit Test? Parameterized Unit Test is a Unit Test with Parameters Separation of concerns Data is generated by a tool Developer can focus on functional specification void Add(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } Ref : Parameterized Unit Testing: Principles, Techniques, and Applications in Practice, Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research) Tao Xie (North Carolina State University)http://research.microsoft.com/en-us/projects/pex/pexpublictutorialslides.pptx 87

116. PUT Sample Ref: Parameterized Unit Tests, Nikolai Tillmann (nikolait@microsoft.com), Wolfram Schulte (schulte@microsoft.com), Microsoft Research One Microsoft Way, Redmond WA USA 88

117. Parameterized Unit Testing (PUT) Some popular unit testing tools, like Junit, Nunit doesn’t support the automation of creating unit tests In some cases it even takes more lines of code to create the unit tests than more implementation being tested Some Automatic test generation tools may not even track the divide-by-zero type errors rather than specific error message PUT or Parameterized Unit Test accepts behavioral parameters/argument sets for a specific test method Ref: Automating Software Testing Using Program Analysis, Patrice Godefroid, Peli de Halleux, Aditya V. Nori, Sriram K. Rajamani, Wolfram Schulte, and Nikolai Tillmann, Microsoft Research Michael Y. Levin, Microsoft Center for Software Excellence 89

118. PUT Contributions They allow unit tests to play a greater role as specifications of program behavior. In fact, PUTs are axiomatic specifications. They enable automatic case analysis, which avoids writing implementation-specific unit tests. Their generated test cases often result in complete path coverage of the implementation, which amounts to a formal proof of the PUT’s assertions. Ref: Parameterized Unit Tests, Nikolai Tillmann (nikolait@microsoft.com), Wolfram Schulte (schulte@microsoft.com), Microsoft Research One Microsoft Way, Redmond WA USA 90

119. PUT Framework Symbolic State Constraints Symbolic Evaluation Axioms Test Case generation Ref: Parameterized Unit Tests, Nikolai Tillmann (nikolait@microsoft.com), Wolfram Schulte (schulte@microsoft.com), Microsoft Research One Microsoft Way, Redmond WA USA 91

120. Symbolic State Its is basically the state of program execution. It can contain expressions with symbolic variables. Symbolic expressions: E = | o object ids, infinite set of potential object identifiers | v variables, set of symbolic variable identifiers | t types, set of type identifiers | f(E bar) function application, set of function symbols | For All (v bar).E universal quantification, where x to denote lists of items x1, . . . , xn. Function symbols: equals(x, y) denotes whether x and y represent the same value for value Types, type(x) denotes the runtime type of object x, and len(x) the length of array x Heaps:No of Times a method being called in a nested method call Symbolic state: A symbolic state is a 5-tuple S = (O, A, He,Hi, X), where the current set of objects O is a subset of ObjectId, the program stack A is a stack of activation records, He and Hi are expressions denoting the extensional heap and the intentional heap respectively, and finally, X, an object expression, denotes the current exception. Say: O(S) is being written for projection on S Ref: Parameterized Unit Tests, Nikolai Tillmann (nikolait@microsoft.com), Wolfram Schulte (schulte@microsoft.com), Microsoft Research One Microsoft Way, Redmond WA USA 92

121. Symbolic Evaluation Symbolic Evaluation describes the effect of the current instruction from a given constrained state (S,C). Ref: Parameterized Unit Tests, Nikolai Tillmann (nikolait@microsoft.com), Wolfram Schulte (schulte@microsoft.com), Microsoft Research One Microsoft Way, Redmond WA USA 93

122. Axioms Axiom is basically of summary of external behavior [TestAxiom] public void TestQuickSort(int[] a) { Assume.IsTrue(a != null); QuickSort.Sort(a, 0, a.Length - 1); for (int i = 0; i < a.Length - 1; i++) Assert.IsTrue(a[i] <= a[i + 1]); } Ref: Parameterized Unit Tests, Nikolai Tillmann (nikolait@microsoft.com), Wolfram Schulte (schulte@microsoft.com), Microsoft Research One Microsoft Way, Redmond WA USA 94

123. Test Case Generation Each transition sequence (S0,C0) ! (S1,C1) ! · · · represents a unique execution path of the program. [TestMethod] public void TestQuickSort597() { int[] ints = new int[2]; ints[1] = 1; this.TestQuickSort(ints); } [TestMethod] public void TestQuickSort512() { int[] ints = new int[3]; ints[0] = -2147475453; ints[1] = 1073750016; ints[2] = 8194; this.TestQuickSort(ints); } Ref: Parameterized Unit Tests, Nikolai Tillmann (nikolait@microsoft.com), Wolfram Schulte (schulte@microsoft.com), Microsoft Research One Microsoft Way, Redmond WA USA 95

124. Constraints A constraint on a symbolic state is a pair C = (BG, PC), where BG is the static background, which only depends on the program declarations, and PC is the dynamic path condition, which is built up during symbolic evaluation Ref: Parameterized Unit Tests, Nikolai Tillmann (nikolait@microsoft.com), Wolfram Schulte (schulte@microsoft.com), Microsoft Research One Microsoft Way, Redmond WA USA 96

125. What is Pex? Pex stands for Program Exploration which is a white box test generation tool developed at Microsoft Research that helps developers to write PUTs in .NET platform and thus generate automated test cases For each PUT, Pex uses dynamic test-generation techniques to compute a set of input values that exercise all the statements and assertions in the analyzed program which eventually cover all reachable branches Ref: Automating Software Testing Using Program Analysis, Patrice Godefroid, Peli de Halleux, Aditya V. Nori, Sriram K. Rajamani, Wolfram Schulte, and Nikolai Tillmann, Microsoft Research Michael Y. Levin, Microsoft Center for Software Excellence 97

126. Pex Sample Project 98

127. Sample Web Method to Test [ServicePoints.asmx] 99

128. Pex Test Case: Test Axiom [ServicePointsTest.cs] 100

129. Pex Test Method 1: Null Validation 101

130. Pex Test Method 2: Validation with valid string value 102

131. NUnit Test Case [TestFixture] public class UserTest { User _user; [SetUp] public void Setup() { _user = new User(); } [Test] public void EmailTest() { _user.Email = “fazlul@telus.net"; Assert.AreEqual(“fazlul@telus.net", _user.Email); } [Test] public void IsValidTestWhenPasswordsMismatch() { _user.Password = “SENG"; _user.Password = “CPSC"; Assert.IsFalse(_user.IsValid()); } [Test] public void IsValidTestWhenPasswordsMatch() { _user.FirstName = “Fazlul"; _user.LastName = “Chowdhury"; _user.Email = "fazlul@telus.net "; _user.Password = " SENG "; _user.PasswordConfirmation = " SENG "; Assert.IsTrue(_user.IsValid()); } [Test] public void UserNameTest() { _user.Email = "fazlul@telus.net "; Assert.AreEqual(" fazlul@telus.net ", _user.UserName); _user.UserName = "fazlul@telus.net "; Assert.AreEqual("fazlul@telus.net ", _user.Email); } } 103

132. Instantiating PUT in PEX [PexMethod] public string ConvertToUpper([PexAssumeUnderTest]MyPexTest1Class target, string Input) { string result = target.ConvertToUpper(Input); return result; } [PexMethod] public void ConvertToUpperAgain(string Input) { MyPexTest1Class MyP = new MyPexTest1Class(); var First = MyP.ConvertToUpper(Input); var Second = MyP.ConvertToUpper(First); Assert.AreEqual(First, Second); } 104

133. PUT & PEX benefits PUT specifies the external behavior of the methods for all test arguments and so the tests cover all possible test environment and constraints PUTs can be instantiated to re-obtain traditional closed unit tests and to re-verify the result of tests PUT enables automatic case analysis, which avoids writing implementation-specific unit tests Using PUT in PEX, it is fairly simple to auto generate test cases and cover all possible testing scenarios, constraints and values 105

134. PUT & PEX limitations PEX works for deterministic model of test methods and goes in cycle for non deterministic methods until hits exploration bounds PEX doesn’t work in Multi-Thread model PEX can analyze any .NET language but the code generation only supports C# PEX uses an automatic constraint solver to determine which values are relevant for the test and the code-under-test but which is also limited [Say: floating point arithmetic cannot be reasoned precisely] Lack of Tests for Oracle Advanced Concepts : PUT in PEX, http://research.microsoft.com/en-us/projects/pex/pexconcepts.pdf 106

135. Microsoft Pex – Short Demo 107

136. Load Testing 108

138. Identify the workload profile for distributing the entire load among the key scenarios

139. Identify the metrics that we want to collect in order to verify them against your performance objectives.

140. Design tests to simulate the load

141. Use tools to implement the load according to the designed tests, and capture the metrics

142. Analyze the metric data captured during the testBy using an iterative testing process, these steps should help us achieve our performance objectives Performance Testing Guidance for Web Applications, J.D. Meier, Carlos Farre, PrashantBansode, Scott Barber, and Dennis Rea http://msdn.microsoft.com/en-us/library/bb924372.aspx 109

143. Visual Studio Load Testing 110

144. Visual Studio Load Testing Results 111

145. VS Load Test – Short Demo 112

146. Advantages of TDD centric Web Applications design Code will be separated from the Data layer Easy to understand and code transfer It increases the system performance Easy to maintain documentation 113

147. Conclusion (i) This research was done to investigate numerous testing tools in .NET platform. We found that various testing tools are specialized in specific testing contexts. NUnit/Visual Studio Testing Tools/MVC .NET and Microsoft PEX are built for unit testing where as DBUnit/DB Pro are specialized in Database Testing, Selenium and MS Web test are for Web UI Testing and VS Load Test and Wcat are being used for Load/Stress testing. But, Microsoft Team System Developer edition comes with a full blown test suite of which VS Test is for Unit Testing, DB pro is for database testing, Web Test for web UI testing and VS Load Test is for Load testing. External testing tools like DBUnit and Nunit test cases can also be generated from Visual Studio Testing suite. MVC .NET implements the Model, View and Controller pattern which makes TDD easier for a Web Application. MVC resolves the Server based forms which were difficult to test. MVC framework fits into large teams of Developers and Web designers who need a high degree of control over the application behavior. 114

148. Conclusion (ii) This presentation also highlighted the concept of Parameterized Unit Tests or simply PUT which is basically a generalization of established closed unit tests. Parameterized Unit Test or PUT has been implemented in Microsoft PEX to generate automated Unit tests. Behavioral parameters can be injected into PUT to cover a particular implementation. State Change, Return Values and Exceptional return values states the Method's behavior. Detailed demonstration of PEX covered the Axiomatic approach of Test Methods and it’s test coverage scenario together with automated and regression test generation. 115

149. References Evaluating Current Testing Processes of Web-Portal Applications Link: http://ase.cpsc.ucalgary.ca/ase/uploads/Publications/bajwaMaurerXiong2005a.pdf Multiple Perspectives on Executable Acceptance Test-Driven Development Link: http://ase.cpsc.ucalgary.ca/ase/uploads/Publications/XP2007_Melnik_Maurer.pdf http://ase.cpsc.ucalgary.ca/ase/uploads/SENG515615W2010/Lecture4_Agile_Engineering_Practices.pdf http://elsmar.com/pdf_files/QC%20vs%20QA.pdf http://dotnetslackers.com/articles/aspnet/ASPNETMVCFrameworkPart2.aspx http://www.asp.net/learn/mvc http://www.asp.net/(S(pdfrohu0ajmwt445fanvj2r3))/learn/mvc/tutorial-01-cs.aspx http://ase.cpsc.ucalgary.ca/ase/uploads/Publications/XiongMaurerBajwa2005b.pdf Building Web Apps without Web Forms, Link: http://msdn.microsoft.com/en-us/magazine/cc337884.aspx MVC architecture in ASP. Net using C# and Microsoft Data Access Application block By MunirShaikh February 28, 2006 , Link: http://www.c-sharpcorner.com/UploadFile/munnamax/mvcarchitecture02262006091052AM/mvcarchitecture.aspx http://www.agiledata.org/essays/tdd.html http://msdn.microsoft.com/en-us/library/ms978748.aspx http://onjava.com/pub/a/onjava/2004/01/21/dbunit.html http://msdn.microsoft.com/en-us/library/bb381703(VS.80).aspx Good and Practical Ideas for Testers ,http://www.tassq.org/quarterly/docs/tassq_magazine-0609.pdf Low-Level Web App UI Test Automation, James McCaffrey http://msdn.microsoft.com/en-us/magazine/cc163723.aspx Advanced Concepts : PUT in PEX, http://research.microsoft.com/en-us/projects/pex/pexconcepts.pdf 116

150. Thank You 117