MTLM Visual Studio 2010 ALM workshop

BETA  work in progress, I add every training new material and tune currentmaterial. Sogeti Netherlands Microsoft Test and Lab Manager Training Guide A description and short reference guide covering Microsoft Test and Lab Manager training course “Testing Practices with ALM tools”.Clemens Reijnen -- VERSION 0.48/8/2011

The training courseThis guide is an extract from the two and three day course provided by me. It spans thecomplete testing lifecycle and the tool usages. It will look at the infrastructureimplications and testing practices in formal and in agile teams. But, the main focus stayson the usages of the Microsoft Visual Studio testing tools, the knowledge you need to getstarting with it, the practices you must have to work with it in real live and how you canbend the tools, with extensibility and normal use to your team needs.The course and this guide is work in progress. It is not a testing training (I expect youalready have testing knowledge), if you need that test process information I refer to theTMap website from Sogeti where you can find tons of information. This training guidefollows the TMap testing lifecycle.

ContentsThe training course .......................................................... 1Application Lifecycle Management ....................................... 4Test Lifecycle - Planning Phase ........................................... 8 Work items ........................................................................................................ 9 Lab 1: Create team projects. ..............................................................................10 Lab 2: Create user stories and tasks for the employee application. ..........................20 Work Item Queries ............................................................................................21 Reports Basic ....................................................................................................24Microsoft Test Manager .................................................... 28Test Lifecycle – The Control Phase I ................................... 29 Test Plans.........................................................................................................29 Test Configurations ............................................................................................32 Test Suites .......................................................................................................32 Lab 3: Create and configure test plan. .................................................................34 TCM.EXE ..........................................................................................................36Test Lifecycle – Specification Phase .................................... 37 Test Cases ........................................................................................................38 Shared Steps ....................................................................................................41 Test Data Iterations ...........................................................................................42 Lab 4: Create test cases. ....................................................................................43Test Lifecycle – Execution Phase ........................................ 46 Test Run ..........................................................................................................47 Test Runner ......................................................................................................48 Lab 5: Execute the created test cases. .................................................................50 Test Case Execution Results................................................................................51Test Lifecycle – Control Phase II ........................................ 56 File a Bug .........................................................................................................57 Verify a bug ......................................................................................................60 Lab 6: file a bug. ...............................................................................................61 Bug Reports ......................................................................................................67 Test Impact ......................................................................................................67 Lab 7: test impact. ............................................................................................69

Create test from bug ..........................................................................................70 Lab 8: Create an exploratory bug and create a test case from it. .............................71Test Lifecycle - Infrastructure Phase .................................. 72 On Physical Environments with Microsoft Test Manager ..........................................74 Lab 9: Install and configure the test controller and test agent .................................79 Lab 10: Setup Flavor D ......................................................................................79Test Lifecycle - Completion Phase ...................................... 80 Test automation with Visual Studio 2010. .............................................................81 When is test automation the investment worth? ....................................................83 The investment levels explained. ......................................................................84 Lab 13: Create CodedUI .....................................................................................88 The UIMap ........................................................................................................89 Data driven CodedUI tests ..................................................................................90 Lab 14: Execute the CodedUI during the build from Visual Studio ............................91 Setting up Flavor C .........................................................................................91 Lab 16: Execute the codedUI from MTM ...............................................................93 Setting up Flavor D .........................................................................................93 Lab 17: Execute the codedUI during the build from MTM ........................................95 Setting up Flavor E .........................................................................................95 On Virtualized Environments with Lab Management and SCVMM ..............................96Extensibility ................................................................... 99 Tools ...............................................................................................................99 Custome Data Adapter ..................................................................................... 100Azure and test automation ............................................. 101Test Manager and an agile process .................................. 101Intelitrace .................................................................... 101Appendix ..................................................................... 106 proposed training agenda 2 day training............................................................. 106 Student and trainer Environment ....................................................................... 107

Application LifecycleManagementYou must understand ALM otherwise you will interpreter the test tools and connectedtools completely wrong and don’t see and use its full value.IT organizations spend billions of dollars a year on development life-cycle tools that don’tplay well together. The result? For most shops, application life-cycle management (ALM)— the coordination of development life-cycle activities — is still largely a manual process.Today’s ALM suites don’t oﬀer much support for ALM beyond what can be accomplishedthrough brittle tool-to-tool integrations. But tomorrow’s ALM platforms will do muchbetter by providing common services to practitioner tools. These solutions will be easierto implement, maintain, and employ. And at the end of the day, they’ll enabledevelopment organizations to build better software.From Forrester:http://i.i.com.com/cnwk.1d/html/itp/Serena_Life_Cycle_Management.pdf

(tool vendors ALM proposition)http://www-01.ibm.com/software/rational/alm/http://www.microsoft.com/visualstudio/en-us/solutions/managementhttps://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11^37618_4000_100__Software development and all roles involved are supposed to strive for one commongoal: providing added value to the customer. It does not matter what kind of applicationneeds to be implemented. All tasks and roles need to reach for this one single goal. Thesynergy between roles, processes, and tools is also commonly known as ApplicationLifecycle Management [ALM]. Notions like accountability, governance, and complianceare regularly used when talking about ALM. But all these notions refer to the simplenotion of cooperation.There would be a better mutual understanding if developers and testers would have moreinsight in each other’s work and challenges. Annoyances disappear and people worktogether on a common cause. Visual Studio Team System 2010 will better support thiscooperation. By giving the tester aids for performing his/her tasks and making him/her afirst-class-citizen within the Team Foundation Server, he/she has and gives more insightin tasks to be done, progress on these tasks and what steps still need to be performed.Aside from this insight, which often means quite a change of culture, the tester hasbetter insight in and access to the ALM-Artifacts from other roles. For example, use-caseand activity diagrams from business and information analysts and other diagrams fromdesigners and architects are easily accessible through Visual Studio Team ArchitectEdition. Cooperation not only occurs by giving insight in work activities, but also byworking on common products.http://www.clemensreijnen.nl/post/2009/04/21/Testing-in-the-Application-Lifecycle-with-Visual-Studio-2010-Test-Edition.aspxhttp://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx

When talking about Application Lifecycle Management [ALM] terms like accountability,governance and compliance are used. All of them refer back to “working together”, howdo we work together during the application lifecycle. ALM is not about tools, it’s aboutworking together. Working together seamless and flexible while staying in control, beingmeasurable and responsible. All the roles in the application lifecycle have a part in thiscollaboration effort. Tools can help but isn’t’ the core driver.There are lots of examples of wrong interpreted business requirements, misscommunication between development en test [Rob describes a very real example],applications which won’t run in production, operations who don’t understand theapplications. All of them resulting in more work, more faults, more costs or even onlycosts and no application because the project was unplugged. Most of these projectsfaults, these extra costs are a slip in communication.Having a strategy how people have to collaborate within the lifecycle is one importantpiece of Application Lifecycle Management. Most organizational parts have already somekind of process / methodology in place. Having an approach how to share informationand share ideas from their own role point of view through the other roles is a key successfactor for lowering the costs and raise the business value of IT investments.Tools can help with this goal. Having gear in place which supports and stimulatescollaboration is a driver for a successful Application Lifecycle Management. But without aplan, how people should collaborate and communicate, tools are useless.http://www.clemensreijnen.nl/post/2009/02/13/Visual-Studio-Team-System-2010-e28093-Episode-3-The-Lifecycle.aspxApplication Lifecycle Management and a Collaborative Culture.Creating new modes of collaboration supported by technology can only be done byaddressing the human aspect. More specifically, we need to address some of the worriesand obstacles people encounter when collaborating using technology.The three most important concerns are:Trust. Trust is a condition for social interaction. People will only work with people,companies, tools and information they know they can trust. Before we can expectcollaboration to take off online, there must be a way for people to get this “trust.” And atopic closely associated with trust when it refers to people is Identity.Collaborative culture. If one individual is the greatest collaborator in the world, he orshe is probably not getting anywhere. Only when all people involved are part of the samecollaborative culture will new levels of creativity and productivity be reached. Acollaborative culture consists of many things, including:  Collaborative leadership;  Shared goals;

 Shared model of the truth; and  Rules or norms.Reward. Changing the way people work takes effort, so it must be clear for the partiesinvolved what they will gain, at a personal level, from collaborating in a new way.Surprisingly, a “reward” for successful collaboration is most often of a non-financialnature.“The ways that people work together shift over time, which can affect your culture ofcollaboration. More important, the introduction of collaboration technologies can alsochange the culture of collaboration. If handled properly, the tools and the culture willcoevolve.”Dennis Kennedyhttp://www.clemensreijnen.nl/post/2009/02/20/Application-Lifecycle-Management-and-a-Collaborative-Culture.aspxThe TMap life cycle model is used both in the creation of the test plan and in theexecution of the other activities in the test process.Life cycle model The life cycle model is a generic model. It can be applied to all testlevels and test types and used in parallel with the life cycle models for systemdevelopment. In the TMap life cycle model, the test activities are divided across sevenphases: Planning, Control, Setting up and maintaining infrastructure, Preparation,Specification, Execution and Completion. Each phase is split up into a number ofactivities.In the Planning phase, the test manager formulates a coherent approach that issupported by the client to adequately execute the test assignment. This is laid down inthe test plan. In the Control phase the activities in the test plan are executed, monitored,and adjusted if necessary. The Setting up and maintaining infrastructure phase aims toprovide the required test infrastructure that is used in the various TMap phases andactivities. The Preparation phase aims to have access to a test basis, agreed with theclient of the test, of adequate quality to design the test cases. The tests are specified inthe Specification phase and executed in the Execution phase. This provides insight intothe quality of the test object. The test assignment is concluded in the Completion phase.This phase offers the opportunity to learn lessons from experiences gained in the project.Furthermore activities are executed to guarantee reuse of products.http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance_and_system_tests/index.jspBusiness Driven Test Managementhttp://www.clemensreijnen.nl/post/2010/06/28/Business-Driven-Test-Management-with-Microsoft-Test-Manager.aspx

Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecyclemanagement tools for teams to ensure quality results, from design to deployment.Whether youre creating new solutions or enhancing existing applications, Visual Studio2010 Ultimate lets you bring your vision to life targeting an increasing number ofplatforms and technologies—including cloud and parallel computing.(from http://www.microsoft.com/visualstudio/en-us/products/2010-editions/ultimate )Test Lifecycle - Planning PhasePlanning phase of the Testing Lifecycle.The life cycle model is a generic model. It can be applied to all test levels and test typesand used in parallel with the life cycle models for system development. In the TMap lifecycle model, the test activities are divided across seven phases: Planning, Control,Setting up and maintaining infrastructure, Preparation, Specification, Execution andCompletion. Each phase is split up into a number of activities.In the Planning phase, the test manager formulates a coherent approach that issupported by the client to adequately execute the test assignment. This is laid down inthe test plan.This is where the test organization is going to work together and on the same level asthe other ALM roles to plan the effort to realize the business needs with a proven (riskdriven) quality.More reading: http://eng.tmap.net/Home/TMap/The_4_essentials/index.jsp

Work itemsPlanning activities for all the ALM roles are supported by Visual Studio and TeamFoundation Server. In the central repository is the work items collected, so everybodyknows each other’s progress. Every role, every task has its own specific optimized tool,connected with this ‘single point of truth’.The white circle, just below the tools is the process template, which supports, drives apredefined way of working.More reading: http://msdn.microsoft.com/en-us/library/bb385832.aspxThere are many different process templates, which can be found in the MSDN landingpage and every organization has probably made some customizations. In the box thereare the MSF Agile and the MSF CMMi.MSDN page: http://msdn.microsoft.com/en-us/vstudio/aa718795.aspxRead: VSTS 2010 TMap Process Template 1,2 and 3:http://www.clemensreijnen.nl/post/2009/07/03/VSTS-2010-TMap-Process-Template-12-and-3.aspx

Lab 1: Create team projects.1: Open VS2010 and open the Team Explorer2: check if the team explorer is connected to the Default Project Collection.3: add new team project4: give it the name << EmployeeApplication >>5: select msf agile process template, next next finish (an empty source tree will becreated)6: after the team project is created, add the sources to source control.

7: open the application, build and run it as a check.Do the same for the Tailspin application, only after step 6 we have to set some things forthe database access. Another dependency is the availability of MVC template in VS2010,see the USB.Every process template has artifacts, artifacts which support the process, the work of thedifferent roles and provide the team with metrics covering the quality of the process.See:Artifacts (Agile): http://msdn.microsoft.com/en-us/library/dd997580.aspxArtifacts (CMMI): http://msdn.microsoft.com/en-us/library/ee332487.aspxScrum 1.0: http://msdn.microsoft.com/en-us/library/ff731587.aspx

(from MSDN)You can use work items to track and manage your work and information about your teamproject. A work item is a database record that Team Foundation uses to track theassignment and progress of work. You can use different types of work items to trackdifferent types of work, such as customer requirements, product bugs, and developmenttasks.The work item type determines the workflow states and transitions. The work item typesthat are available in a project depend on the project template that the project uses. Also,you can customize the work item types that are available in your team project to trackyour work in the way it makes sense for your development environment.Each work item represents an object that is stored in the Team Foundation database.Each work item is assigned a unique identifier, a work item ID, and type, the work itemtype. Work item IDs are unique across all work item types in all team projects in aproject collection. The work item type determines the work item fields that are availablefor tracking information, defaults defined for each field, and rules and constraintspositioned on these fields and other objects that specify the work item workflow. Everychange made to a work item field is stored in the work item log, which maintains anhistorical record of changes.You can create and modify work items by using Team Explorer, Team System WebAccess, Office Excel, or Office Project. When creating or modifying individual work items,you can work in the work item form, by using Team Explorer or Team System WebAccess. You can make bulk updates to many work items at a time by using Team SystemWeb Access, Office Excel, or Office Project.Work item types in the TMap for MSF Agile process template:User Story, A team creates user stories to define the features, functions, andrequirements to be implemented.TaskTest TaskTest Basis Finding (see: http://tmap.codeplex.com/wikipage?title=Test basis finding)Test CaseBug, You can track a code defect by creating a bug work item. By creating a bug, youcan accurately report the defect in a way that helps other members of the team tounderstand the full impact of the problem.Issue, You can define known or potential problems, impediments, or risks to yourproject by creating issue work items.Shared Step, Shared steps are specific for test cases and are mainly used only inMicrosoft Test and Lab Manager. In shared steps, you define a sequence of action andvalidation steps to run as part of a test case.

A user story communicates functionality that is of value to the end user of the product orsystem.When you define a user story, you must define the Title in the top section of the workitem form. You can leave all other fields blank or accept their default values.Link types: work items can be linked to each other; each link has some specificcharacteristics. You can create work items for bugs, tasks, features, requirements, andvalue propositions for your project and then create relationships among them that meetyour projects business goals.Work item types can have a specific flow, in this example the flow of the user story WIT.Teams can change the flow specific their needs. But be careful, it must support thesystem under development; you can make it as crazy as you can image.The flow can be visualized by using the Web Access Tool or by using the ProcessTemplate Editor.More reading: http://msdn.microsoft.com/en-us/library/dd380634.aspx

An example: the role of the tester and the usages of work items.During the planning phase of the project, also called iteration 0 [first blue piece], userstories are collected / brainstormed / defined /… in VSTS this information is collected inthe work item type ‘user story’During the planning phase of the project, also called iteration 0 [first blue piece], user storiesare collected / brainstormed / defined /… in VS this information is collected in the new workitem type ‘user story’ [image below].During the planning of the iteration developers the team start to breakdown the user stories[which are selected for that iteration] in implementation tasks. Within VSTS this is done in theimplementation tab. The new 2010 functionality of hierarchies between work items is used forthis.Another task also executed during this phase is the creation of test cases. Within the TMapmethodology this is described in the planning phase. Where you create the test plan for thatiteration, based on the risk class [see user story work item] and discussion with the customerthe necessary test techniques are allocated for the user story and functional area. See the

initial work items for iteration 1 in the image below, added during the unfolding of the TMapprocess template.So, not only the implementation tasks needs to be determined also the test tasks needs to beallocated during the planning phase of an iteration. These test tasks are specific to testers like‘create test cases for area …. based on the test technique <… decide based on riskand customer contact>’List of different test techniques…  Data combination test (DCoT)  Data cycle test (DCyT)  Decision table test (DTT)  Elementary comparison test (ECT)  Error guessing (EG)  Error testing (ET)

 Process cycle test (PCT)  Real life test (RLT)  Semantic test (SEM)  Syntactic test (SYN)  Use case test (UCT)So, just like the implementation tasks also the test tasks are a link type, have hierarchy with,the user story. [this is TMap process template specific]We end up with a list of tasks for a user story, implementation tasks and test tasks… testersand developers are going to execute these tasks together in parallel, during the iteration wehave: implemented sources for the user story and test cases which are ready for execution.This user story is finished when: every implementation task is fulfilled, all test cases aresuccessful executed and … the tester hasn’t got any open tasks, so all test cases are created.When giving the tester/ the team a place where they can record their testing tasks, testing isreally going to be a first class citizen of the lifecycle. beside this benefit the connectionbetween test activities, risk, user story and test cases gives a great opportunity for reportsbased on test effort, risks and implementation, later on more on this…With this this lightweight addition to the user story work item we got a closer to several AgileTesting key characteristics, for example: –> just like the developers, testers breakdownthere work discusing it with the customer –> and all tests are created..!

During the planning of the iteration the team starts to breakdown the user stories [whichare selected for that iteration] in implementation tasks. Within VSTS this is done in theimplementation tab of the user story Work item. The new 2010 functionality ofhierarchies between work items is used for this.More reading: http://www.clemensreijnen.nl/post/2009/09/03/Agile-Testing-with-VSTS-2010-and-TMap-Part-01-User-stories.aspxA team creates tasks to track the level of effort that is required to implement a user storyor other areas of work that are identified for the project. Tasks should represent a smallunit of work that can be accomplished within one to two days. You can break larger tasksdown into smaller subtasks.You can create a task to track work to develop code, design and run tests, address bug,and perform regression testing. In addition, you can create tasks to support generic workthat must be performed.By tracking work hours for each task, the team can gain insight into the progress that ithas made on the project.You can use the Remaining Work and Burndown and Burn Rate reports to monitor teamprogress, identify problems in the flow of work, and determine the team burn rate.More reading: http://msdn.microsoft.com/en-us/library/dd380700.aspx

In Visual Studio Team System 2010 all test roles are provided with clear and bettersupport within the Application Lifecycle. Testers do not use their own separate technicaltools anymore, but use integrated tools that are used by architects and developers.Effectively tearing down the wall between developers and testers.But good tools are not enough. Also, a clear separation of roles, tasks, and authorizationsare necessary. Finally and most importantly, a structured approach determines howsuccessful you are with your test strategy.For test tasks look at the activities in the different TMap phases:http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance_and_system_tests/Planning_Phase.jspMore reading: http://www.clemensreijnen.nl/post/2009/04/21/Testing-in-the-Application-Lifecycle-with-Visual-Studio-2010-Test-Edition.aspxA team uses test cases to define tests that will support testing of user stories. You candefine manual test cases that specify a sequence of action and validation steps to run, oryou can specify automated test cases that reference an automation file.

Areas and Iterations are typically artifacts used by development teams, but now test isconnected with them, we should have some knowledge about these topics.Areas are used to breakdown the application in functional pieces, and iterations are usedto breakdown the effort of creating the application in time boxes.Often the testing effort is also broken-down in these parts.More reading:http://msdn.microsoft.com/en-us/library/dd997576.aspxhttp://msdn.microsoft.com/en-us/library/ms181692.aspxSpecific privileges are needed to create them.

Lab 2: Create user stories and tasksfor the employee application.the user stories are used in the other labs.1: open you favorite TFS client (MTM, VS2010, Web access , ... )2: add a user story and fill the necessary fields3: add linked tasks4: repeat several times

Work Item QueriesWith work items in place, several queries and reports can provide valuable informationabout the state of the project.Default there are already some powerful query’s available, to plan and find workassigned to you.To access work item queries, you must have the appropriate permissionsMore reading: http://msdn.microsoft.com/en-us/library/dd380734.aspxShow and run the queries for the previously executed exercise.Beside the default queries it’s also possible to create your own queries. When a query issaved in the ‘my queries’ folder they will show up in all other tools.

More reading: http://msdn.microsoft.com/en-us/library/ms181352.aspxWork Item Query Language (Work Item Query Reference)You can find bugs, tasks, other types of work items, and links between work items byusing one of the WorkItemStore..::.Query methods or a Query object. These queries usethe work item query language (wiql), which resembles T-SQL.More reading:http://msdn.microsoft.com/en-us/library/bb130198(VS.90).aspx .http://msdn.microsoft.com/en-us/library/ms181352.aspxBeside the creation of flat lists queries it’s also possible within VS2010 to createhierarchical list to visualize the relationship of the work items. Two different types ofqueries can be created; one based on link types and one as a tree view.

you can perform the following tasks:Track and check features against value propositions.Review active tasks grouped by feature areaAssess changes that were made to dependent work itemsAssess changes that were made to dependent work items at a point in timeSave changes to work items: You can quickly save any changes that you make tomultiple work items within the list of results from the query.More reading: http://msdn.microsoft.com/en-us/library/dd286632.aspx#FindType of query: You can perform complex queries and view link associations betweenwork items with the addition of these two new types of queries:Work Items and Directed Links: Displays a two-tiered set of work items and theirdependent links based on work-item and link-filter criteria that you specify. You canexpand and collapse leaf nodes and, within Team Explorer, drag work items to changelink associations.Tree of Work Items: Displays multi-tiered hierarchical relationships between work itemsthat are associated with parent-child links. You can expand and collapse leaf nodes and,within Team Explorer, drag work items to change link associations.

Reports BasicNow we can collect the information with queries we need to visualize them. Within TFSthis can be done with SQL Reports and Excel Reports. First the default SQL Reports.Team Foundation Server provides you with many different and powerful ways to get theanswers to your questions. In the 2010 release, we’ve enhanced every SQL ServerReporting Services report. You’ll find and richer experience with better data presentationand detailed documentation to help you use the reports right out of the box. In 2010,we’ve also added new Excel Reports. TFS feeds the reports via Work Item queries for fastand flexible reporting. Finally, rich SharePoint dashboards enable quikc visualization ofproject status. In addition, when used with SharePoint Server products, you can publishreports in a familiar and capable environment.Let’s look at the new Stories Overview report. <Build>First, you see the report contains a description describing what this report is showing.<Build>

Second, you get a list of related reports that might help expanding on the data shownhere. <Build>Finally, you see the work progress on the user stories in context with data about testresults and bugs.Next as you scroll down, you see additional information in the footer. <Build>First, on the left you have questions this report answers with a hyperlink that takes youto the documentation about this report, which has additional details including healthy andunhealthy report examples. <Build>Second, you get the parameter values used to generate the report, something that helpswhen looking over printed copies in meetings for example. <Build>And finally, you get information on how up-to-date is this report. Every report in the2010 release is like this.As you can see the Quality dashboard has four main graphs. <Build>First you can see if your test team is making progress on running test plans. <Build>Second, you can see how build are doing over time. What’s the trend like? Are youhaving lots of success or failure? <Build>Third, what’s your bug trend like. Are you closing out bugs or are you stagnate. Or is thevelocity of your bug filing far exceeding your team’s ability to fix, test, and close outbugs? <Build>Finally, are you seeing a bad trend related to bug reactivations—bugs that were closedreopened by test as not fixed? All of this information is there for you in a quick, heads updashboard format.

More reading: http://msdn.microsoft.com/en-us/library/dd380714.aspxWhile SQL Reports are hard to change are Excel reports much easier.More reading : http://msdn.microsoft.com/en-us/library/dd997876.aspxFor example a report based on the results of a flat list imported inExcel

More sophisticated the creation of excel reports based on a wizard within Excel to createdifferent types of reports.Beside the default SQL reports there are also default Excel reports and workbooks.The default excel reports are:http://msdn.microsoft.com/en-us/library/dd997876.aspxYou can use workbooks to build your product backlog and plan work by assigning theuser stories, tasks, bugs, and issues to specific iterations, also referred to as sprints. Byusing the workbooks that are provided with the process template for MSF for AgileSoftware Development v5.0, you can quickly create work items and modify the rank,priority, state, and assignments of multiple work items at the same time.http://msdn.microsoft.com/en-us/library/dd997891.aspx

Microsoft Test ManagerThe not so standard user interface experience of MTM, what to find where.http://msdn.microsoft.com/en-us/library/ff458183.aspx(from Improving Developer-Tester Collaboration withMicrosoft® Visual Studio ® 2010 http://channel9.msdn.com/shows/VS2010Launch/Doug-Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010/ )Microsoft Test Manager 2010 is for testers what Visual Studio is for developers. That is tosay, where Visual Studio is an IDE – an integrated development environment, TestManager is an ITE – an integrated test environment. This is the interface that a tester willuse to create test cases, organize test plans, track test results, and file bugs whendefects are found. Test Manager is integrated with Team Foundation Server, and isdesigned to improve the productivity of testers. While I am not going to do a deep-diveof all that Test Manager can do, it is important to understand how it integrates with theVisual Studio Agents to make the most of test case execution, and ensure that when atester files a bug, it is actionable with very little work on the tester’s part.

Test Lifecycle – The ControlPhase ITest PlansControlling the testing effort, beside bug triage this also means test case management,what to run when, what has been run, etc.AimProviding the client with sufficient insight into, and the opportunity to influence, thefollowing:The progress of the test processThe quality and risks of the test objectThe quality of the test processTo this end, the test manager manages the test process optimally and reports on it.Within MTM test case can be organized, within test plans, suites and environmentalconfigurations. How the test cases are organized depends on the type of organization.

A test plan organization can be as simple or as complex as you want, based on yourtesting approach. You might want to create a test plan for each sprint if you are using anagile methodology. Then you can add the user stories for each sprint to your test planand create test cases for these user stories. Or, you might create a test plan for eachspecific milestone if you are using another approach. Another way is creating test plansper test type.In Microsoft test manager a test plan holds all the information needed to maintain,organize and execute test cases. It doesn’t hold information about the testing effort, thisis the reason I talk about test execution plan.More reading: http://msdn.microsoft.com/en-us/library/dd286583.aspxWithout a test plan you can’t access test testing center in Microsoft Test Manager.After you created a test plan you can set its properties in

Setting the state property to inactive will make the test plan disappear from the startscreen. But, it still can be accessed from another test plan in the ‘organize’ section.Test plans should have a clean and descriptive title.More reading: http://blogs.msdn.com/b/anutthara/archive/2010/09/22/guidance-for-creating-test-plans-and-test-suites.aspxhttp://www.clemensreijnen.nl/post/2009/09/09/02-Management-preserving-and-organization-of-manual-test-cases-within-VSTS-2010.aspx

Test ConfigurationsYou can assign a set of default configurations to your test plan that you want to cover forquality purposes. You will be able to view which tests have passed or failed for eachconfiguration and how many tests you have left to run.Some proposed configurations are the different environments people work with, test,development etcMore reading: http://blogs.msdn.com/b/vstsqualitytools/archive/2009/10/30/test-configurations-in-a-test-plan.aspxTest SuitesYou can create test suites in your test plan to group your test cases into suites, based onyour needs. There are three different suite types with different characteristics. 1. Requirement bases suite: You can add a requirement to form a suite in your plan that contains all the test cases that are linked to this requirement. Test cases added to this suite are automatically linked to the requirement, and test cases which are linked somewhere else (in visual studio, another suite, in web acces) to the requirement are also visble in this suite. 2. A folder like suite with no additional functionality other than hold the test suites. Folder Test suites can contain child suites (the other two not). 3. A query based suite, which holds a collection of test cases with properties conform the selection criteria.

You can’t add test cases to this suite they are selected by the query.

Lab 3: Create and configure testplan.1: select add test plan in the black start screen and add a test plan.2: open the iteration test plan.3: open the test plan properties and edit the configurations.

3: add a new variable were we can but information about 64 bit and 32 bit machines.6: delete the current configurations and add or own client configuration.  Windows 7 64 bit  Windows 7 32 bit  Windows 2008 R2 64 bit 7: set a default configuration for new test cases  Windows 7 64 bit  Windows 7 32 bit8: save and close the test plan properties dialog.Add an additional end2end test plan.1: go to the ‘organize’ section and add a new test plan2: call it E2E Test plan and set several properties and configurations.Add Test suites to the iteration test plan.1: add requirement based test suite for the userstories2: add query based test suite, for regression candidates… prio 1

3: add query based test suite, for automation … automated status == planned orautomated4: add a ‘normal’ test suite for exploratory tests.TCM.EXEWith the command line tool TCM.exe you can get lists of test plans, configurations andmore. Also you can execute test cases with this command line tool.http://msdn.microsoft.com/en-us/library/dd465192.aspxSee http://msdn.microsoft.com/en-us/library/ff942472.aspx for listing test plans

With the collected ID you also can share links to test artifacts (like test plans). In the‘test plan’ start screen you can find on the top the ‘copy URL for plan’ button.Othere artifacts can be shared the same way.Test Lifecycle – SpecificationPhaseAfter test planning and test control we get to the test specification phase in the testing

lifecycle.AimDuring the Specification phase, the required tests and starting points are specified. Theaim is to have as much as possible prepared, in order to be able to run the test asquickly as possible when the developers deliver the test object.ActivitiesWithin the Specification phase, the following activities are distinguished:Creating test specificationsDefining central starting point(s)Specifying the test object intake.During the test case specification we create test cases with corresponding test steps.Shared steps and test data iterations are test specific items which need special attention.Test CasesTest cases are work items with a specific tab where test steps can be defined. These teststeps can only be edited from within MTM.You can create test cases for your manual tests with both action and validation test stepsby using Visual Studio 2010 Ultimate or Visual Studio Test Professional. You can add testcases to your test plan using Microsoft Test Manager.More information:

http://msdn.microsoft.com/en-us/library/dd380712.aspxhttp://msdn.microsoft.com/en-us/library/dd286729.aspxTest case naming conventions discussion.What is the best way to name your test cases…For quick editing of test cases (only the logical / not the steps) you can use the teamfoundation toolbar in excel to open a test query.A test step within a test case in Test Manager (MTM) can be a normal action step or avalidation action step, and both of them can contain additional information(attachments).See image three steps:1.normal action step2.action step with validation

3.normal action step with attachment (validation action step can also contain anattachment)What kind of action steps do I need to use for my test case and how to write them… youare probably not the one who executes the test case.more reading: http://www.clemensreijnen.nl/post/2010/10/29/Microsoft-Test-Manager-Test-Case-Specification-what-kind-of-action-to-use.aspxAn easy way to share or re-use test steps is by copy paste the steps. It works perfect toand from excel.

Shared StepsYour team can use shared steps to streamline definition and maintenance of manual testcases. Many tests require the same sequence of steps to be performed for multiple testcases. By creating shared steps, you can define a sequence of steps once and insert itinto many test cases. For example, if each test case requires a tester to log on to theapplication, you can create a set of shared steps to perform these actions. You can thenadd the shared steps to each test case and run the steps using Test Runner.Candidates and reasons for shared steps • Reuse – Many steps, reused in many test cases. Benefit: time, retyping, boring work. • Expected changes – for example environment variables change [URL]. Benefit: Time, test cases which need this step are updated automatically, also action recording. • Pre- conditions – bring the system under test to a certain state and execute the different verifications. Benefit: beside time, you are certain the application under test is in the same state for every test case • Back to initial state - after test execution bring the environment back to the initial state for new test cases. Benefit: beside time, you are certain the application under test is always in a clean state for new test cases.

There are somethings to think about and know before using shared steps. - Parameters defined in shared steps must also be defined in the test case, for every test data iteraiton. - Expected result in a shared step limits the re-use of the shared step. Are you testing the same thing over and over again?Test Data IterationsIf you want to run a manual test case multiple times with different data, you do not haveto create multiple copies of the test case. You can add parameters to the actions orexpected results for any test step in your test case. Then you can add multiple sets ofvalues for the parameters that you want to use in your test. Each set of values for theparameters is run as an individual iteration of the test using Test Runner. You can copythe value of each parameter when you run the test and then paste the value into yourinput field in your application under test.More reading: http://msdn.microsoft.com/en-us/library/dd997703.aspx

If you have configured test settings to create an action log for the test being run,selecting the Create action recording option will use the settings that you configured. Ifyou did not configure the actions diagnostic data adapter in your test settings, then thedefault values are used to create your action recordingThe same as with test steps, test data can be copied and re-used in or from excel.Lab 4: Create test cases.For the employee application we are going to create test cases with test data iterationsand shared steps.Specify multiple test cases.1: open the iteration test plan.

2: add a new test case to the requirements based test suite.3: give the test case a useful name.4: add the steps: 1. open Employee application 2. fill ‘Clemens’ 3. fill ‘Reijnen’ 4. push lookup ‘with expected results’ ‘6913’ 5. close application5: check if all the other fields are correct and save the test case.6: create some more test cases, for the ‘show salary’ functionality and employee notfound.Add test data iterations and parameters.1: open the test case which has been created during the first steps.2: change, 6. fill ‘Clemens’  fill @givename 7. fill ‘Reijnen’  fill @surname 8. push lookup with expected results ‘6913’  expected results @phonenumber

3: add the values: 1. Clemens 6913 Reijnen 2. andre andre 1000 3. alf alf 2000Create shared step and reuse it.0: not necessary but will make life easier, create shortcut on the desktop for theemployee application executable. (the executable can be found after compiling thesolution within VS2010 in the bin directory.)1: open the test case which has been created during the first steps.2: select the first step ‘open Employee application’ right click and select the contextmenu item ‘create shared step’.3: give it a name and save the shared step. See that the step is now bold. (optional:save and close the test case)4: go to ‘organize’ – ‘shared steps manager’, select the shared step and click ‘createaction recording’.5: Test runner will open, select ‘create action recording’ and start the application.

6: when finished save and close the Test Runner.7: repeat these steps for the ‘close application’ step.8: use the ‘open’ and ‘close’ shared steps in the other test cases.Test Lifecycle – ExecutionPhaseAim

To obtain insight into the quality of the test object through the execution of the agreedtests.ActivitiesWithin the Execution phase, the following activities are distinguished:Intake of the test objectPreparing the starting pointsExecuting the (re)testsChecking and assessing the test results.Test RunYou can start as manual test run by selecting one or more test cases, this can be doneeither by multiple selecting test cases in the test section or by selecting a complete testsuite, and click run. When the test run starts the test runner will guide the tester to thetest process.From msdn.Running manual tests using Test Runner can help you identify problems in yourapplication. Problems that are typically identified when you run manual tests includeincorrect functionality or bugs, poor performance, and security and connectivity issues.The outcome, resolutions, and comments that you can enter during a manual test aresaved as test results.More reading: http://msdn.microsoft.com/en-us/library/dd286725.aspxSelecting the ‘run with options’ menu will give the testers some more control over thetest settings and environments. Also when the test case is an automated test case youcan use this menu item to run in manually.

More info: http://msdn.microsoft.com/en-us/library/dd286725.aspxTest RunnerThe test runner has a minimalistic user interface, but with a lot of options. - The return to testing center button, opens microsoft test manager again but won’t stop the test. This is great for small edit’s. - In the drop down you can see the ammount of test cases and test data iteration in this test run. - On the bottom you can show hide the recorded steps, will executing the test case. - On the bottom, in the recorded actions pane, you can remove unnecessary recordings for cleaning the action recording. - Steps with action recordings, recognized by the orange bar, can be re-played.

Besides using the test runner it is also possible to execute test runs with the TCM.execommand line tool. Making it easy to automated test execution during builds.http://msdn.microsoft.com/en-us/library/dd465192.aspx

Lab 5: Execute the created testcases.For the employee application we are executing the create test cases.1: go to ‘’Test” – ‘run tests’ and select either a test suite or a single test case and click‘run’.2: test runner will open, check the ‘create action recording’ and start the test. Follow thesteps and mark the steps passed (if succesful).Optional: run them from TCM.exe1: open Visual Studio command prompt, type ‘tcm.exe run /help’ and follow theguidance.

Test Case Execution ResultsThe most important thing after execution of test cases is reporting about the results,what is the quality of my system.In Team Foundation Server there are many ways to get the needed information. All thedata is stored in a data warehouse available for all the different client tools.

Test run results in MTM.In Microsoft test manager the reporting about the test runs is minimal, you can see theresult of the test runs if they failed or passed.After you run tests, you can review your test results to see which tests passed and whichtests failed. For tests that you run from a test plan by using Microsoft Test Manager youcan review your test results, assign a reason for a failed test, and assign a resolution.You can view all these test results for a test plan using pre-defined test reports or bycreating your own custom reports.As you analyze the test results for each test, you can select a failure type and resolutionfor each failed test, or add a comment about your analysis to this point. To help withyour analysis, you can open the test result for each test to see all previous runs of thattest.If you determine that there is an issue, you can create a bug and associate the data fromthe test result for this test with the bug. If you determine that the failed test is caused bya known issue, you can link that test to an existing bug. When this bug is fixed, thetester who verifies the bug can then see all the test cases that have to be rerun to verifythat the bug is fixed.If you want to add data to the test run, you can add attachments or comments to thetest run itself.More readings: http://msdn.microsoft.com/en-us/library/dd553274.aspx

You can view the details of the results of your tests using Microsoft Test Manager basedon each test point in your test suite. A test point is the pairing of a test case with a testconfiguration in a specific test suite. If you have multiple configurations that you want touse when you run your test, you have a test point for each pairing of the test with aconfiguration.More readings: http://msdn.microsoft.com/en-us/library/ee843861.aspxAlso in the test plan properties window you can view the results. But for mostorganizations all this test result information isn’t enough… to say something about thequality of the system.Share the test run with peers by using a URL and the runID

Because Test cases are Work item we can create queries to analyze the work. Thedefaults are…Also default reports and excel reports are available.(show)Browse through the created reports and discus.

There are also pivot capabilities for the test artifacts. Selecting the test fields show them.One important thing to notice is are the fields which don’t belong to a work item, theseare MTM specific artifacts save to the datawarehouse.The organization looks like this. Some pieces for the used test artifacts are stored in theTFS database and some in the TCM server. Not accessible together, only in de olapcubes.Some specific useful fields in the warehouse for reportings.(from msdn)By using the Test perspective in the SQL Server Analysis Services cube for Visual StudioTeam Foundation Server, you can view just the measures, dimensions, and attributesthat pertain to reporting on tests results and test runs. For example, you can use thesemeasures to determine the overall quality of each build, the tests that a particular buildaffected, and the number of test cases that were run. You can also answer questions

about changes to the result outcomes.The Test measure group is based on the Test Results relational table, which enablesreporting on test results as either a property of the tests or an independent outcome. Formore information, see Test Result Tables.More reading: http://msdn.microsoft.com/en-us/library/ms244708.aspxIt is still a bit complicated, or better a bit confusing, how the different parts withinMicrosoft Test Manager are organized. This image is a try to clear things a bit.More reading: http://www.clemensreijnen.nl/post/2010/07/19/Microsoft-Test-Manager-Test-Artifact-organizatione280a6.aspxTest Lifecycle – Control PhaseIIActions a test organization does during the control is not only the management of thetesting effort, but also the management of bugs.AimProviding the client with sufficient insight into, and the opportunity to influence, the

following:The progress of the test processThe quality and risks of the test objectThe quality of the test processFile a BugSubmitting bugs and tracking the status of bugs is an important task that helps youensure that bugs are fixed. This improves the quality of the applications that you aretesting. The more information that you can provide with a bug, the more quickly it can berecreated and fixed. Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimateprovide functionality to collect data that can automatically be added to a bug that youcreate, when you run your tests. You can submit bugs in the following ways by usingVisual Studio Test Professional 2010 or Visual Studio 2010 Ultimate:When you run a test using the Test RunnerFrom Microsoft Test Manager when you view a test resultFrom Microsoft Test Manager when you view your bugsYou can also submit bugs from Microsoft Visual Studio 2010.All these bugs can be viewed, updated and tracked using both Microsoft Test Managerand Microsoft Visual Studio 2010.Important topics for understanding and fully use the capabilities of MTM with bug fillingare:  Diagnostic Data Adapters

 File a Bug  File an Exploratory Bug  Verify a BugCollect Data on a Local Machine by Using Default Test SettingsIf you just want to run manual tests on a local machine and do not want to collect dataon another tier of your application, you can use the default test settingsMore reading:http://msdn.microsoft.com/en-us/library/ee231892.aspxhttp://blogs.msdn.com/b/amit_chatterjee/archive/2009/04/11/diagnostic-data-adapters-changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-collector.aspxFile a bug from Test Runner.When you run a manual test and discover a defect in the application under test, you cansubmit a bug directly from Test Runner without having to switch to Visual Studio.Submitting a bug in this manner automatically adds information about your test andtesting environment to the bug. You can also update an existing bug if you discoveradditional defects while you are running a test.More reading:http://msdn.microsoft.com/en-us/library/dd286746.aspxhttp://msdn.microsoft.com/en-us/library/dd293543.aspx

Submitting Bugs When You Run Your Manual Tests: You can submit a bug immediatelywhen you run a test from Test Runner. Your test steps and any information that youhave collected are automatically added to the new bug that you create.Submitting Bugs from Your Test Result: You can submit a bug from Microsoft TestManager when you view a test result. For example, when you analyze a test result andyou determine that the test failed because of a product issue. You can also submit a bugfrom Microsoft Test Manager when you are viewing your bugs.Submitting Bugs from Microsoft Visual Studio: You can submit bugs using MicrosoftVisual Studio 2010 at any time.

Verify a bugUsing Microsoft Test Manager, you can track the status of both bugs that are assigned toyou and bugs that you created. You can also track the bugs for your whole team projectto determine the overall quality of areas of the application under test. You can triagebugs by viewing the details of the bug and determining whether the bug should be fixedor not. You can assign the bug to another user to provide more information, decide to fixthe bug, or determine whether it is actually a bug.When a bug is fixed, it can be assigned to the tester to verify that the bug has beenfixed. In the Verify Bugs view, you can view bugs that are assigned to you and have tobe retested, as shown in the following illustration. Or you can create a custom query totrack all bugs that have to be retested.

Lab 6: file a bug.Create a buildWe need a build for later use.1: open Visual Studio.2: open ‘Team Explorer’ and select ‘buils’ right mouse click and select the context menu‘new build definition’ ’3: trigger ‘manual’4: build defaults: set the drop folder.

5: process ‘items to build’ the employee app solution.6: try the build, by selecting ‘queue new build’

7: check if successful. (Green bar)When the build doesn’t run, check the build service in the team foundation console.File a bug1: set in the test settings, local test run the wanted data collectors on.

2: add to the first create test case (in lab 4) the test data iteration values 1,1,1.3: run the test case (it will crash at 1.1.1 :-)4: file a bug by selecting ‘create bug’.5: give the bug work item a useful name – assign it to yourself, save and close it.Resolve a bug1: open Visual Studio and the employee application.2: open the Team Explorer and open the team query, ‘my Bugs’, look for the previouslyfiled bug.3: go to the solution explorer, open the ‘employees.cs’ file and look for the sentence with‘else if (firstname.Equals("1") && lastname.Equals("1"))’

Remove the lines: Exception ex = new Exception("Went something wrong"); throw ex;and the // before the two below.Should look like this afterwards:else if (firstname.Equals("1") && lastname.Equals("1")){ Employee emp = new Employee("1", "1", "1@sogeti.nl", "1", 1, "1", "1"); return emp;}4: select the ‘employees.cs’ file, right click and select ‘check in’.5: in the check in dialog, select ‘work items’ and check the bug, select for action‘resolve’.6: queue a new build and wait till it finishes successful.Verify a bug1: open Microsoft test manager.2: go to ‘test’ – ‘verify bugs’, select the resolved bug and click verify.

3: test runner will open. Run the test case… and check if the bug is solved.4: when all steps are finished and marked passed (where necessary) close the testrunner. MTM will ask if you want to close the bug.

Bug ReportsMonitor bug activity, reactivations, and trends. You can use bug reports to track the bugsthat the team is finding and the progress that the team is making toward fixing them.Test ImpactBy using Test Impact Analysis during code development, you can identify the methods ina test project that have been affected by code changes in your managed code solution.At each build of the solution on the local computer, Test Impact Analysis identifies themethods in the code project that have changed and lists the test methods that directly orindirectly call those methods. You can then run the tests from the Test Impact Viewwindow. You can also use the Test Impact View window to identify and run any testmethod that affects a particular code method.

demoBy using test impact analysis, you can identify the tests that you should run, based oncoding changes that were made to the application that you are testing between builds.When you use test impact analysis with Microsoft Test Manager, you are required to useTeam Foundation Build to build your application and Team Foundation version control foryour source code for the application being tested. Test impact information is gatheredonly on tests with a status of passed. It is not collected when you file a bug or when atest that is marked as failed is complete.The requirement for using Team Foundation Build is only applicable to collecting datafrom Microsoft Test Manager, as the test plan must be associated with a build producedby Team Foundation Build for the test impact analysis feature to work properly. To usethe Test Impact View in Microsoft Visual Studio 2010, you do not need to use TeamFoundation Build and the solution does not have to be under any source code controlsystem.

Lab 7: test impact.Attach build to test plan1: open MTM.2: go to ‘plan’ – ‘properties’ and select the build filter created in the previous lab.(no need to set the quality)3: select ‘build in use’, select the latest build and assign it to the plan.Execute tests1: execute several test cases.Make code change and kickoff a build1: as in lab 7 ‘Resolve a bug’ … open the ‘employees.cs’ file/2: remove the line:Employee emp = new Employee("Clemens", "Reijnen", "Clemens.Reijnen@Sogeti.nl", "6913", 12000, "Amstelveen", "Noord Holland");And remove the // from this line.//Employee emp = new Employee("1234", "4567", "1234.4567@Sogeti.nl", "6913", 12000, "Amstelveen", "Noord Holland");

3: as in lab 7 ‘Resolve a bug’ … check in the changes and kickoff a build.Attach new build to test plan and evaluate impacted tests1: go to ‘plan’ – ‘properties’ and assign the new build number to the plan.2: review the ‘Recommended Tests’ and set them back to ‘active’Create test from bugAn often overlooked feature in Microsoft Test Manager is the ‘create test from bug’ optionin the ‘verify bugs’ section. It sounds indeed a bit strange, probably there is already atest case in place which found this bug. But, this create test case feature has someinteresing scenario’s.When use the create test case functionality you will see that it uses the action recordingto define the steps for the tests case.A scenario for which you can use this functionality is when the tester filled an exploratorytest with no predefined steps, you can very easily create a well written tests case fromsuch a action recording.

When you run an exploratory test and find a bug in the application, you can submit anexploratory bug directly from Test Runner. Submitting an exploratory bug in this mannerlets you specify the time range for the actions to include in the bug. The actions arerecorded in an action log which can be used to create a manual test case.More reading:http://msdn.microsoft.com/en-us/library/ee803840.aspxOther scenario’s for this feature is for example an application with no tests defined inMTM, you can let the business guy run his most important process while the test runneris recording. You can use this recoriding to create well written test cases. Anotherinteresting scenario are migration projects, when you want to migrate an applicationfrom VB6 to .NET for example, record the most important test cases and create the testcases.Lab 8: Create an exploratory bugand create a test case from it.1: open MTM.2: create a test case with the name expolritory test.3: add the step ‘start exploratory testing for 5 minutes’, save and close it.4: run the test and select after some clicking around ‘create exploratory bug.’5: go to ‘verify bug’ select the bug, right click and select ‘create test case from bug’ inthe context menu.6: look at the test steps

Test Lifecycle - InfrastructurePhaseBefore moving on to the completion fase we need to setup or infrastrcuture first.AimTo provide the required test infrastructure, which is used in the various TMap phases andactivities.ActivitiesThe Setting up and maintaining infrastructure phase consists of the following activities:Specifying the infrastructureRealizing the infrastructureSpecifying the infrastructure intakeIntake of the infrastructureMaintaining the infrastructurePreserving the infrastructure

The ‘green’ Lab Center is the place to manage your test infrastructure. you can managephysical and vertualized test environments from this center.Where can I run the tests?There are different ways to execute tests, the main difference is between theenvironment types and technology used for it within Visual Studio 2010One major difference is the provisioning of clean environments.

On Physical Environments withMicrosoft Test ManagerThere are multiple ways to execute your test automation effort, you can run them from andon Microsoft Test Manager, from and on Visual Studio, during the build and on the BuildServer or during the build on Test Agents configured with Test Controllers or … many flavors.so, where should you execute your automated tests, and how should you configure your testinfrastructure.Where to run your automated tests, and how to configure the test infrastructure?The different flavors Flavor A: Execution from VS2010…

Purpose: Real developer test / test automation dry run. A valid scenario is when Developers create a codedUI test from a test case which found a bug. To reproduce the bug, developers generate the CodedUI add the assertion and verify the result when they fixed the bug. Triggered by: right mouse click – run tests, in test project, or Test List Editor, run checked tests. Information: http://msdn.microsoft.com/en- us/library/dd286580.aspx Pro Easy setup Debug-able test automation No additional configuration needed* Con No collection of test results in TFS/ Test Manager Run on developer environment * Data Diagnostic Settings in the Local.Testsettings file are configurable.Flavor B: Execution duringbuild with Build controller Purpose: Part of Build Verification Tests (its strange to run UI tests on a build and have a build service running in interactive mode) Triggered by: build Information: Set the Build Service to run in interactive mode. http://blogs.msdn.com/b/mathew_aniyan/archive/ 2009/05/26/coded-ui-test-in-a-team-build.aspx Configure the build to run the tests http://msdn.microsoft.com/en- us/library/ms182465.aspx#CreateBuildType Pro Easy setup Test results in build result Con No collection of test results in Test Manager (no reporting on test points) Build Service needs to run in interactive mode (what happens after a restart) Tests are executed on build environmentFlavor C: Execution duringbuild with Test controller

Purpose: Part of Build Verification Tests, distribute tests over multiple test agents. (Preferred configuration above flavor B) Triggered by: build Information: Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en- us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en- us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B) Pro Tests run distributed over multiple environments Test Results in Build result Con No collection of test results in Test Manager Harder to configure Need for specific test client environments Test Settings from VSFlavor D: Execution fromMicrosoft Test Manager Purpose: Part of Regression Tests (other type of test than BVT). Triggered by: MTM user, right mouse click on test case, run Information: Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode) Configure Lab Center in MTM to use test controller and create test ‘agent’ physical environment. http://msdn.microsoft.com/en- us/library/ee390842.aspx http://msdn.microsoft.com/en- us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-touse-codedui-tests-watin-and-mtm.html Pro Test run distributed over test environments Test Result in MTM Test Settings from MTM Full control by the tester

Con Test Controller needs to be configured with a Project Collection (one controller per collection) Manually triggered by Tester (or pro) Hard to configure Hard to see which test case is automated Flavor E: Execution from MTM during Build… Purpose: Part of BVT. Preferred configuration above flavor C. Flavor D and E can be configured together. Triggered by: Build Information: Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run TCM or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/201 0/10/30/how-to-run-coded-ui-tests-from- command-line.aspx How to: Run Test Cases with Automation from the Command Line Using Tcm http://msdn.microsoft.com/en- us/library/dd465192.aspx Pro Test run distributed over test environments. Tests can be configured to run on different configured environments Test Result in MTM and TFS Triggered by build Test Settings from MTM Con Hard to configure maintenance of TCM commands in buildEnough flavors to choose from… the pro and cons I wrote down can differ per situation. Mypreferred flavors are E and D configured together. The reason, test settings and test resultsare where the belong, in the testers hands. And, the TCM command line gives much moreflexibility what to run where during the build.Lab Management for virtual environments has different configurations, pros and cons…will make a post for this later…An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solutionfrom Colin Dembovsky it’s a XAML build template which lets you easily configure theBuild-Deploy-Test workflow for physical environments. one drawback from flavor E will be

minimized when using this template, the maintenance of the TCM commands.

Lab 9: Install and configure the testcontroller and test agenthttp://msdn.microsoft.com/en-us/library/dd648127.aspxhttp://msdn.microsoft.com/en-us/library/ff469838.aspxLab 10: Setup Flavor DAutomated Test Execution from Microsoft Test Manager Configure the test controller that it is associated with the team project collection. Create a physical environment. Verify in the controllers menu that there is an environment connected with a test agent.

In the test plan properties configure the automated test settings. To add an automation script to a test case we must open visual studio and add a test script to the test case. (see CodedUI test in this document for the details) Now your test infrastructure is ready to execute automated tests from MTM. Select run on a test case with an automation script.Test Lifecycle - CompletionPhase

AimTo learn from experience gained during this test and to preserve test ware for reuse in afuture test.ActivitiesThe Completion phase consists of the following activities:Evaluating the test processPreserving the testware.Test automation with Visual Studio2010.Different test levels, in which phase should we use test automation?What is what level and what do you do at that level

VS2010 has many technologies and capabilities for test automation.What is what:….………Within the different levels different technologies are interesting.Why? …In what phase of the project and by whom is the technology used?

Who en when………Why do you want to automate it with that technology at that phase, what do you want toaccomplish.In this guid we focus on the UI automation capabilities of VS2010. How to use it, why bywhom and how does the infrastructure looks like.When is test automation theinvestment worth?With VS2010 ALM you can very easily take a manual test case and turn it in an automatedtest case. But this isn’t without investment. There are still some additional things to do whichtake time and money before we get the benefit of automated execution of a test case.

These are some very basic test automation investment levels. Just to keep in mind whenworking with Microsoft Test Manager and CodedUI.VS2010 ALM has several automation capabilities. The most noticeable is CodedUI, C# codegenerated from a MTM action recording or recorded within VS2010 another automationcapability is the Fast Forward functionality in Microsoft Test Runner, Fast Forward till avalidation step for manual validation. This Fast Forward functionality is also valuable in theshared steps and for test data iterations.The investment levels explained. Zero time investment: completely no investment in any kind of automation,no Fast Forward no CodedUI. Although the FF is recorded for every test it isn’t used. MTM isused in this situation not for its automation capabilities, but more for its test casemanagement, bug filling and reporting capabilities.Which test cases don’t need any kind of automation depends. The most logic reason would bethat the test case only is going to be executed once. the other reason can be that automatingthe test case would be a to big investment and to complex, and there are more reasons.This post is a useful source “effort-estimation-for-test-automation” to make your decision toautomate or not, other good reading is this pdf “TEST AUTOMATION EFFORT ESTIMATION -Best practices”. Little time investment: executing the same test case on multipleenvironments, or test cases executed several times during the sprint are candidates forautomation, but sometimes the investment of making it CodedUI is too big. Making use of the

Microsoft Test Runner’s Fast Forward capabilities will speed up test execution. but, gettingthat additional benefit of it you have to tune the action recording to make the Fast Forwardfunctionality in the Test Runner more smooth and less error prone.When executing a manual test case, you never do it correct the first time. You have to searcha bit, maybe make a wrong step, go back in the application etc… execute a test scriptcomplete correct with the optimal amount of clicks the first time is hard. And when you doclick everything correct the first time the action recording will collect a bit too muchinformation, in some situation (see image, opening a browser window with Bing as homepageand than move your mouse around to browse for the website under test). Cleaning the actionrecording will make the FF more efficient and less error prone. this cleaning of the actionrecording needs to be done during the execution of the test case in Microsoft Test Runner.Note: The little investment of cleaning the action recording, also is for the shared step andtest cases with test data iteration. Test case which use test data iterations are candidate for abit more investment for a clean action recording, shared steps are for sure candidates. Some more investment with Basic CodedUI functionality: Although it isvery easy to create CodedUI C# files for already run manual test cases by using their actionrecordings (hopeful a clean action recording from the little investment step). It is still a biggerin time investment to create them. Beside the time it takes to create the CodedUI C# it alsoasks an investment in test infrastructure to execute them, see this post “Running AutomatedTests on Physical Environments, the different flavors…”.

See image, you also have to create the assertion for the automated manual test case. 1) themanual validation in the Test Case. 2) Add the assertion by using the UI Test Builder 3) dragthe crosshair on the application under test and select the property to validate 4) the assertionis added to the CodedUI test.The main idea for this investment level is that only the basic generated code is used (notcustomized) and an assertion is manual added to it. Serious investment in time with advanced CodedUI, customization ofthe CodedUI and UIMap: Customizing the generated CodedUI is a real trap. You can makethe most sophisticated UI tests in the world. Resulting in C# code which is even morecomplicated as the functionality it tests with probably even as many or more bugs, who teststhe tests trap. So, this investment should be done careful, but sometimes it’s a worthyinvestment. For example for test cases which are going to be run the whole life of the system,or maybe are boring and have many steps. (see the links in the zero investment level.)Customizations of this level can be started from the ‘UIMap builder’ (see image), change thesearch properties, move steps out of the code generation and customize them, add fancy testdata iteration to the steps, etc etc…

Lab 13: Create CodedUI Create CodedUI from Action Recording. Add a CodedUI test to the test project in the solution. Select ‘use excisting action recording’. In the work item picker dialog select a test case with a valid action recoding. Generate the C#. Look at the generated methods and compare them with the test steps in the test case. Run the CodedUI test. Add Assertion. All steps are generated except the assertion, we need to create / generate this to make the CodedUI test complete. Right click in the CodedUI code where you want to add the assertion and select ‘Use CodedUI test Builder’. In the lower right corner of your screen a little dialog appears, you can use the crosshair to select the control you want to validate. Generate the assertion and run the CodedUI again.Create a CodedUI from scratch by using the recording option. Record action for CodedUI

The UIMap The UIMap file in created and used during the generation of the CodedUI can be edited for optimization purpose. Methods can be moved, controls renamed and its search criteria tuned. The search configuration can be edited from the properties menu. http://msdn.microsoft.com/en- us/library/ff398062.aspx

Data driven CodedUI tests Just as unit tests can CodedUI test be data craven, run them more as once with different datasets. These datasets can come from a test case or other data source like excel and databases. http://msdn.microsoft.com/en- us/library/ee624082.aspx

Lab 14: Execute the CodedUI duringthe build from Visual StudioExecute codedUI tests from Visual Studio within a build.Setting up Flavor C For this lab the test controller isn’t connected to a team project collection, we are going to execute the test case from Visual Studio. In the test settings file we must configure the roles where the test will be execute. http://msdn.microsoft.com/en- us/library/ee256991.aspx there has to be a test controller agent infrastructure in pace.

Define a build and set the test properties to usethe test settings file.Queue the build and watch the test agent.

Lab 16: Execute the codedUI fromMTMSetting up Flavor D The test controller must be associated with the team project collection. Create in the Lab Center in MTM a physical environment from the environment with the test agent installed and configured.

Lab 17: Execute the codedUI duringthe build from MTMSetting up Flavor EYou need two environments with test agents for this lab, I use a vpc and its host for it. With the test configuration property in the test plan properties we can configure for which configurations the test cases need to be executed. In this lab we will configure the build so test cases are executed on these specific environments. Default the configuration of a test controller is for load, run automated test cases on all agents so we execute test cases as fast as possible. When you want to run a test case on a specific environment we need to do some additional things.

We will execute the test cases from the command line tool tcm.exe. Get the necessary information. And compose the command lines with parameters. Look at the query in step 2 where we select the test cases. Open the build defeinition workflow xaml and add two ‘invokeprocess’ actions where we call the tcm.exe tool with its parameters (note: the build workflow definitely need more tweaks as mentioned in this step when you want to use this in production, this is only for demo purpose.) Queue the build and watch the test agentsOn Virtualized Environments withLab Management and SCVMM

For a complete guid about Virtulized Test environments see the Rangers Lab Guidehttp://ralabman.codeplex.com/

From http://ralabman.codeplex.com/

ExtensibilityTools

Custome Data AdapterSee lab guid rangers

Azure and test automationhttp://www.clemensreijnen.nl/post/2011/03/29/VS2010-ALM-MTLM-usages-patterns-for-Windows-Azure-hosted-application-development.aspxTest Manager and an agileprocesshttp://www.clemensreijnen.nl/post/2010/09/21/Agile-Test-practices-with-Microsoft-Visual-Studio-2010.aspxIntelitrace

AppendixProposed training agenda 2 daytraining

Student and trainer Environment

http://www.clemensreijnen.nl	1224
http://clemensreijnen.nl	125
http://feeds.feedburner.com	39
http://paper.li	10

MTLM Visual Studio 2010 ALM workshop

by Clemens Reijnen on Sep 08, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 1,398

Statistics

MTLM Visual Studio 2010 ALM workshop — Document Transcript