MTLM Visual Studio 2010 ALM - day2

Start Topics
9:00 Recap
9:15 Create Basic CodedUI, from Test Case and Manual
9:45 UIMap and customize the code for optimization
10:15 Data driven Tests and assertions
10:45 break
11:00 Troubleshoot CodedUI, Common Practices and questions
11:30 Configure the build to execute the CodedUI tests, test settings from VS2010, Execute
from VS2010 with test agents and analyze the results
11:30 Associate test automation with test case, execute automated test case from MTM,
configure the build to run from the build within MTM and analyze the results
12:30 break
13:15 MTLM, Scrum and other project Methodologies
13:45 Lab Management - Virtualized Test Environment and execution Demo
14:15 Test info in the OLAP Cube
14:30 Extensibility - test Scribe, TCMIMport
14:45 Create your own Data Diagnostic Adapter
15:15 MTLM usages with Azure projects
16:00 Closing

RECAP
Test Activities Planning…

Test Case Management
RECAP

Test Environments

RECAP

1
Open the employee application
Set the intellitrace options
Set a breakpoint at …

2
When on the breakpoint, save the trace file on your desktop

3
Close Visual Studio and double click the intellitrace file

4
In the intellitrace summary, open the main threat …

5 … and search for
The values that where entered
during the previous run.

6
And now from Microsoft Test Manager

7
Configure the build… it needs a drop location for the symbols (pdb files)

.
Make test case with the parameter 1 and 1, crash again…
See lab 7 yesterday

8
Run the test case and see it crash… file a bug.

9
See the itrace file in the bug report...

10

Close MTM, open VS (with or without the employee application).
Got to the bug and click the itrace file…

See the exception and
start debugging…

11

Go back in time … and find the entered values

Regression Test Candidates

See also on www.tmap.net: Completion Phase

Test Automation

Test automation
Where? When? How? Who? Why? What?

Where? When? How? Who? Why? What?

VS2010 Test Tools
UI Test Automation Capabilities

Fast Forward
CodedUI
Test Infrastructure

Test Automation Investment Levels
0 No investment in any kind of automation. Just click and test.

1 Clean up of the Action log maybe re-execute, for more smooth Fast Forward.

2 Basic CodedUI. Only use the default generation, add your own assertion.

3 Advanced CodedUI. Customize the CodedUI and UIMap for optimization.

2 Basic CodedUI

VS2010 ALM MTLM usages patterns for
Windows Azure hosted application
development

Cleaning the action recording
When: executing the same test case on
multiple environments or when using test data
iteration or … and don’t forget the shared
steps

CodedUI
from existing action recording from Manual Tester

CodedUI
from existing action recording from Manual Tester

2

1

4

3

CodedUI
Create a new action recording by Developer

Lab 12
Create a CodedUI from Test Case
Create a CodedUI by using the CodedUI Builder

Timebox: 10 minuts

Lab 13
Tune the CodedUI with the UIMap…

Timebox: 10 minuts

Lab 14
Make CodedUI parameterized from an Excel sheet

Timebox: 10 minuts

Start Topics
9:00 Recap
10:45 break
12:30 break BREAK TILL:
13:45 11:00
Lab Management - Virtualized Test Environment and execution Demo
(extra time to finish the labs :)
16:00 Closing

Start Topics
9:00 Recap
10:45 break
12:30 break
16:00 Closing

CodedUI Common Questions

• In what version what functionality?

• How to debug?

• Where can I find test run results for a test run?

• Where do I see these assertions are passed or failed?

• For which properties can I make an assertion?

• …

CodedUI Advanced Questions

• How does CodedUI work?

• Can I use CodedUI for performance testing?

• How to maintain CodedUI tests?

• How to execute in the build?

• …

CodedUI Challenges

• The playback failed to find the control with the given search
properties.

• Identifying controls with no name search criteria or changing
ID’s

• Selecting correct Search Configurations for a control

• How to select the particular value from combo box

• …find controls

Start Topics
9:00 Recap
10:45 break
12:30 Lunch
16:00 Closing

Execute CodedUI
During the build with Visual Studio 2010

For the next 1,5 hour, you must have:
- A successful running codedui
- Test agent and controller
- A build
- Testplan and a test case

Configurations for automated test execution:

C

Flavor C: Execution during build with Test controller… B
Preferred configuration above flavor
Part of Build Verification Tests on multiple test agents
(triggered by: build)

Configure Test Controller (don’t register it with a project collection )
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers
Configure Test Agents on clients (interactive mode)
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents
Configure *.Testsettings file in solution to use Test Controller and Test Agents
Configure the build to run the tests (see B)

Pro: Con:
Test run on test environments No collection of test results in TFS
Tests run on multiple environments Harder to configure
Test Results in Build result Need for specific test client environments
Test Settings from VS

2
Add new Test Settings file
Set remote execution and
test controller.

3
Create a new build definition
Set the Automated Test Properties

4
Queue a new Build

Watch the Test Agent and
Test Environment

Lab 15
Run the automated test from the build.
Look for the test results

Timebox: 10 minuts

Execute CodedUI
Within a test case as an associated automation manualy


D

Flavor D: Execution from Microsoft Test Manager…than BVT
Other type of test
Part of Regression Tests
(triggered by: MTM user, right mouse click on test case, run)

Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode, can be the same as MTM)
Configure Lab Center in MTM to use test controller and create test ‘agent’ environment.
http://msdn.microsoft.com/en-us/library/ee390842.aspx
http://msdn.microsoft.com/en-us/library/dd293551.aspx
Associate CodedUI test with WI Test Case from VS.
http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html

Pro: Con:
Test run on test environments Manually triggered by Tester
Tests run on multiple environments Hard to configure
Test Result in MTM and TFS Need for specific test client (same as MTM?)
Test Settings from MTM

2
Associate the CodedUI
with a Test Case.

4
Associate the CodedUI
with a Test Case.

5
Run Automated Tests from Test Manager

Lab 16
Run the automated test from MTM.
Look for the test results

Timebox: 10 minuts

Execute CodedUI
Within a test case as an associated automation from the build


E

Flavor E: Execution from MTM duringPreferred configuration above flavor C
Build…
Part of BVT Flavor D and E can be configured together
(triggered by: Build)

Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode, can be the same as MTM)
Configure Lab Center in MTM to use test controller and create test ‘agent’ environment.
Associate CodedUI test with WI Test Case from VS.
Create Build task to run MTC or MSTEST task for Test Plan
http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx

Pro: Con:
Test run on test environments Hard to configure
Tests run on multiple environments Need for specific test client (same as MTM?)
Test Result in MTM and TFS
Triggered by build
Test Settings from MTM

Execute automated test cases on
different environments

1
Configure additional environments in Test Plan properties.
Win 7 IE9 configuration

TC MTLM Client
Physical
Test Client
TFS 2010

Physical
Test Client

1
Get Test Plan ID
Get Test Suite
Get Test Configurations
with TCM.exe

2
Create command line for Test Run for the different Test Configurations

"run /create /title:""Run on Host for specific Test Point"" /planid:16
/querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=13""
/settingsname:""Automated Test on Windows 7 Environment IE9""
/testenvironment:""Win 7 IE9"" /collection:http://clemenes-
pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"

"run /create /title:""Run on test env for specific Test Point"" /planid:16
/querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=12""
/settingsname:""Automated Test on Windows 7 Environment IE8""
/testenvironment:""Win 7 IE8"" /collection:http://clemenes-
pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"

6
Queue a new Build

Watch the Test Agent and
Test Environment

Start Topics
9:00 Recap
10:45 break
12:30 Lunch LUNCH TILL:
13:45 13:15
Lab Management - Virtualized Test Environment and execution Demo
(extra time to finish the labs :)
16:00 Closing

Start Topics
9:00 Recap
10:45 break
12:30 Lunch
16:00 Closing

TMap, ALM and process templates

# Release planning
meeting
Estimating and prioritizing the Product Backlog

# Product Backlog

Evolving
User Needs
Owned by Product Owner
Prio by Business Value

Test during the Release Planning meeting.

‘only execute tasks who make an addition to
the product are worthy to execute’

# Sprint planning
meeting
Add PBI to sprint
Decomposition
Commitment

# Sprint Backlog

Tasks
Related to PBI’s
Hours remaining

Test during the Sprint Planning meeting

# Daily scrum
Track

# Sprint Review
# Sprint
Demo the value
Retrospective
Agile Practices – Improvement
Test early and often
Build and deploy continuously
Acceptance Test Driven Development
Emergent Architectures
Refactor
Test Driven Development
Agile Database Development
Pair Programming

Undone
End to End
Testing

http://www.scrum.org/scrumguides/

Build
application

Provision
Run tests
‘clean’ env

Deploy
application

VS2010 Lab Management
• Reproduce the exact conditions of a bug or other development
issue.

• Build, deploy, and test applications automatically in a clean
environment.

• Reduce the time required to create and configure machines for
testing an application.

• Run multiple copies of a test or development at the same time.

• Enable members of a team to create and manage virtual
environments without requiring system administrator privileges.

MTLM Client

TFS 2010 System Center Virtual Machine
Manager (SCVMM)

Hyper-V Library
Host Shares

VM share

VM share

TEAM FOUNDATION SERVER INTERNET LAB INFRASTRUCTURE Prerequisites ?
SAN

Domain Controller 1. TFS Application Tier (AT)
• Windows Server 2003 SP2, 2003 R2, 2003 R2, 2008 SP2, 2008 R2
• Windows Vista SP2, 7
• Ports used by services 8443 team web access
8080 server

Internet LAB DOMAIN (could be same as CORP DOMAIN) 2. TFS Data Tier (DT)
• Windows Server 2003 SP2, 2003 R2, 2003 R2, 2008 SP2, 2008 R2
NLB External Users • Windows Vista SP2, 7
• Ports used by services 1433 SQL

Walkthrough VHD
Lab Management
3. TFS Build Controller
• Windows Server 2003 SP2, 2003 R2, 2003 R2 SP2, 2008 SP2, 2008 R2
• Windows XP SP3, Vista SP2, 7
TFS 1 VS/MTM Clients • Ports used by services 9191 controller
Application Tier (AT)
4 4. TFS Clients
• Windows Server 2003 SP2, 2003 R2, 2008 SP2, 2008 R2

VMM Server 5. VM Factory
6 • Windows Server 2003 SP2, 2003 R2, 2008 SP2, 2008 R2
• Windows XP SP2, Vista, 7
• Ports used by services 3389 RDP
137 UDP
9 139 TCP
VMM Library • For more information, refer to the quick reference poster:
8 Visual Studio 2010 LAB Management Guide – VM Factory Checklist
3
Test Controller 6. VMM Server
• Windows Server 2008 or 2008 R2. R2 is recommended.
• Ports used by services 8100 VMM Agent
TFS 135 WMI (49152-65535)
Build Create LabMan template 3389 RDP
Servers from golden image 7. VMM Hyper-V Host
• Windows Server 2008, 2008R2. R2 is recommended.
Build
VS/MTM 8. Test Controller
Deploy and • Windows Server 2003 SP2, 2003 R2, 2003 R2 SP2, 2008 SP2, 2008 R2
Client Build Golden Images
Machines Host VMs
• Ports used by services 6901 .NET Remoting
9050 Lab Agent
9. VMM Library
Primary Factory • Windows Server 2008 or 2008 R2. R2 is recommended.
2
TFS 5 • Ports used by services 443 BITS
Data Tier (DT) 80 WinRm
VMM Hyper-V Host
7

Secondary Factory
VM FACTORIES LAB MANAGEMENT
References
• Visual Studio 2010 Lab Management Guide
CORP DOMAIN • http://go.microsoft.com/fwlink/?LinkID=206935&clcid=0x409
• Visual Studio ALM Rangers VM Factory Guidance
• http://rangersvsvmfactory.codeplex.com/
Domain Controller • Capacity Planning Quick Reference Poster
Network Storage
Team Project Collection Planning Quick Reference Poster
CORP INFRASTRUCTURE • http://vs2010quickref.codeplex.com/

Visual Studio 2010 LAB Management Guide – Infrastructure
Visual Studio ALM Rangers
2011-03-23

Visual Studio 2010 Test Architecture

Also known as user The final integration
acceptance testing, testing the solution.
the validation testing In the world of NASA  Test Cases
validates the solution System this could be the  Testing
against test final countdown,
requirements, which doing a complete
are based on business system and

Common Test Phases
requirements. environment test  Test
Validation before the launch. results
Design validation
 Cannot reproduce
focused testing,
which validates
Building upon each Integration solution deliverables
other and with requirements. It
supporting the is especially  Suspicions,
testing strategy. important when add more tests
testing interfaces
which are shared
and are the basis of
some or all of the
Unit of work focused
Unit building blocks.
 Corrections  Debugging
testing, at the core of the
solution, typically done by
the developer and
automated testing. Unit
No more “No Repro”!
tests focus on topics such
as initialization, logic and
error handling at a code
level.

Visual Studio Products

Test Lab
Ultimate Premium Test Load Agent
Professional Management

Included in Ultimate Manage virtual environments for
• Generalist Tester
use in application development,
• Minimal Footprint
deployment, and testing.

References Visual Studio ALM Rangers, Copyright © 2009 Microsoft Corporation
Visual Studio MSDN Site http://msdn.microsoft.com Visual Studio ALM Rangers http://msdn.microsoft.com/en-us/teamsystem/ee358786.aspx

Start Topics
9:00 Recap
9:15
9:45
BREAK TILL:
Create Basic CodedUI, from Test Case and Manual
UIMap and customize the code for optimization
10:15 Data driven Tests and assertions 14:30
10:45 break
11:30 Configure the build to execute the CodedUI(extra time settings fromlabs :) Execute
tests, test to finish the VS2010,
12:30 Lunch
14:15 Test info in the OLAP Cube Break
14:30 Test Scribe, TCMImport
16:00 Closing

Lab 17
Install Test Scribe and create a report

Timebox: 10 minuts

Start Topics
9:00 Recap
10:45 break
12:30 Lunch
14:15 Test info in the OLAP Cube Break
14:30 Test Scribe, TCMImport
16:00 Closing

ALM 4 Azure
Automated ALM for Azure Cloud Development

127

Start Topics
9:00 Recap
10:45 break
12:30 Lunch
14:15 Test info in the OLAP Cube Extensibility - test Scribe, TCMIMport
14:30 break
16:00 Closing

Testing Practices with ALM tools

GOAL:
Know the value of MTM and know how to use its capabilities

MTLM Visual Studio 2010 ALM - day2

More Related Content

What's hot

Similar to MTLM Visual Studio 2010 ALM - day2

More from Clemens Reijnen

Recently uploaded

MTLM Visual Studio 2010 ALM - day2