VS2010 ALM Tools Meeting: Test Automation

Technical Meeting
Automated Testing with
VS2010

Suite of Application Lifecycle Management tools

http://msdn.microsoft.com/en-us/library/fda2bad5.aspx

application lifecycle management
http://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx

work together
__with work packages
__on alm artifacts

Testing Lifecycle

See also on www.tmap.net:
- The Test Lifecycle Model
- The 4 essentials

http://msdn.microsoft.com/en-us/library/bb385832.aspx

Test Case Management

See also on www.tmap.net: Control Phase

Test Plans
Test Suites
Test Configurations
Organize the testing effort in a Test Execution Plan
Create dynamic test suites for changing selection criteria
Define various test configurations and share automation across those

Test Plan Organization
Naming of test plans for agile environments:
Iteration number and name and one extra test plan for end
to end tests.

Test Suites Organization
Naming of test suites for agile environments:
Suite per user story (requirement based) and one additional
test suite for regression tests.(query based?)

(test cases are copied to the regression test suite)

Share MTM Artifacts

mtm://clemenes-pc:8080/tfs/defaultcollection/p:MSF/Testing/testplan/connect?id=19

mtm://<server name>:<port>/<tfs vdir>/<Collection name>/p:<project name>/<center
group>/<group specific>
Open Test results
Open Test Cases (or any work item)
Open Test Runs
Open Test Plans
Connect to test plans
mtm://server:8080/tfs/ProjectCollection/p:Project/testing/testcase/open?id=67

http://blogs.msdn.com/b/vstsqualitytools/archive/2010/02/09/sharing-content-in-microsoft-
test-manager.aspx

Test Case Specification

See also on www.tmap.net: Specification Phase

Test Case
Shared Steps
Test Data Iterations
Link test cases and requirements to provide requirement traceability
Data drive test cases with different parameters
Refactor common shared steps in test cases

Test steps
Make them descriptive
Not every step is a validation step
Use the step attachment

Test Data Iterations
Data drive test cases with different parameters

Shared Steps
Refactor common shared steps in test cases
Reduce test maintenance by sharing test steps across test cases

Shared Steps
Shared Step candidates?

• REUSE – Many steps, reused for many test cases . Benefit: time, retyping, boring work.
• EXPECTED CHANGES – for example environment variables change [URL]. Benefit: Time, test cases
which need this step are updated automatically, also action recording.

• PRE- CONDITIONS – bring the system under test to a certain state and execute the different
verifications. Benefit: beside time, you are certain the application under test is in the same state for every test case

• BACK TO INITIAL STATE - after test execution bring the environment back to the initial state for
new test cases. Benefit: beside time, you are certain the application under test is always in a clean state for new test
cases.

Test Execution
Test Runner

A test that you will want to run multiple times.
Different manual tests that contain common steps.
Verifying a bug is fixed by using a manual test.

Test Case Execution

See also on www.tmap.net: Execution Phase

From TCM
run /export id:239 /resultsfile:”c:runresult”

Share with mtm://

mtm://clemens-pc:8080/tfs/defaultcollection/p:MSF/Testing/testrun/open?id=292

Controlling the testing effort
BUG’s

Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug

File a Bug
Submit a Bug using Test Runner

Ways to file a Bug

Submit a Bug from the Test Runner
Submit a Bug from the Verify Bugs View
Submit a Bug from Test Results for a Test

Verify a Bug
Verify a Bug Fix Using Microsoft Test Manager

The Bug Flow
Link build 2 test
plan. T Execute File Bug Verify fix Close or
Configure Data E Manual by using reopen
collectors. S Tests playback Bug
T

Build
definition is
the
intergration

Setup build and
deploy D Resolve Checkin
E Bug Build
V Deploy

Test Impact
Assign build 2
test plan. T Execute Assign Evaluated and reset
Execute
Execute
Configure Data E ‘passed’ ‘NEW’ build tests impacted by
‘passed’
‘passed’
collectors. S Tests 2 Test Plan code churn
Tests
Tests
T

Build
definition is
the
intergration

Setup build and
deploy D Make Checkin
E code Build
V changes Deploy

File an Exploratory Bug
Submit an Exploratory Bug using Test Runner

PRO:
- Reuse scenario knowledge business user
- No typing detailed test steps
- Easy to parameterize
1 Record scenario by business user with - Writing standards for test steps
MTM on application under test. Action - MTM capabilities in step 3. (bug, codedUI,
recommended test etc …)
recording is captured. Most benefit during step 2 (test case writing),
benefit in step 1: run once in common situation.
Benefit in step 3: common MTM benefits (record
playback, fast forward, parameters etc) and
benefit of well written steps.

2 Create test steps (with parameters) in
MTM by using the action recording.
CON:
- Manual actions and tests.
- Test steps text are technical names
- Action recording (step 1) must be clean.
- Unknown what is validated, during step 2, need
additional knowledge.
- The business user must think of the important
scenarios
3 Save test case and execute it by a tester
The action recording during step 1 can have
on application under test, by using MTM. unnecessary steps, these can be removed during the
execution (1) or during the specification (2).

RECAP
Test Activities Planning…

Test Case Management
RECAP

1
Open the employee application
Set the intellitrace options
Set a breakpoint at …

2
When on the breakpoint, save the trace file on your desktop

3
Close Visual Studio and double click the intellitrace file

4
In the intellitrace summary, open the main threat …

5 … and search for
The values that where entered
during the previous run.

6
And now from Microsoft Test Manager

7
Configure the build… it needs a drop location for the symbols (pdb files)

8
Run the test case and see it crash… file a bug.

9
See the itrace file in the bug report...

10

Close MTM, open VS (with or without the employee application).
Got to the bug and click the itrace file…

See the exception and
start debugging…

11

Go back in time … and find the entered values

Regression Test Candidates

See also on www.tmap.net: Completion Phase

Test Automation
met Visual Studio 2010

Test automation
Where? When? How? Who? Why? What?

Where? When? How? Who? Why? What?

VS2010 Test Tools
UI Test Automation Capabilities

Fast Forward
CodedUI
Test Infrastructure

Test Automation Investment Levels
0 No investment in any kind of automation. Just click and test.

1 Clean up of the Action log maybe re-execute, for more smooth Fast Forward.

2 Basic CodedUI. Only use the default generation, add your own assertion.

3 Advanced CodedUI. Customize the CodedUI and UIMap for optimization.

2 Basic CodedUI

VS2010 ALM MTLM usages patterns for
Windows Azure hosted application
development

Cleaning the action recording
When: executing the same test case on
multiple environments or when using test data
iteration or … and don’t forget the shared
steps

CodedUI
from existing action recording from Manual Tester

CodedUI
from existing action recording from Manual Tester

2

1

4

3

CodedUI
Create a new action recording by Developer

Test Infrastructure

Where and how to run the automated test?

Test Environments
See also on www.TMap.net:
- Checklist "Intake Test Environment“
- Checklist "Test Environment“
- Infrastructure Phase

Test lab
Visual Studio 2010 Lab Management

Where and how to run the automated test?
physical test environments
virtualized test environments

physical test virtualized test
environments environments
MTLM Client MTLM Client

Physical
Test Client TFS 2010 System Center Virtual
TFS 2010
Machine Manager
(SCVMM)

Hyper-V Library
Physical Host Shares
Test Client VM
VM
share

share

Build
application
physical test environments
versus
virtualized test environments

Provision
Run tests
‘clean’ env

Deploy
application

Lab for Physical
Environments, the
different flavors…
Visual Studio Team
Foundation
Server

Test Manager
Build Server Test Agent
Test Agent
Test Agent

Configurations for automated test execution:

A

Flavor A: Execution from VS2010…
Real developer test / test automation dry run
(Triggered by: right mouse click – run tests, in test project)
http://msdn.microsoft.com/en-us/library/dd286580.aspx

No additional configuration needed*
* Data Diagnostic Settings in the Local.Testsettings file are configurable.

Pro: Con:
Easy setup No collection of test results in TFS
Debug-able test automation Run on developer environment


B

Flavor B: Execution during build with Build controller…
not recommended, strange to run UI tests on build
Part of Build Verification Tests
(triggered by: build)

Set the Build Service to run in interactive mode
http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx
Configure the build to run the tests
http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType
* Data Diagnostic Settings in the *.Testsettings file are configurable.

Pro: Con:
Easy setup No collection of test results in TFS
Test results in build result Build Controller needs to run in interactive mode
Tests are executed on build environment
Run on build environment


C

Flavor C: Execution during build with Test controller… B
Preferred configuration above flavor
Part of Build Verification Tests on multiple test agents
(triggered by: build)

Configure Test Controller (don’t register it with a project collection )
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers
Configure Test Agents on clients (interactive mode)
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents
Configure *.Testsettings file in solution to use Test Controller and Test Agents
Configure the build to run the tests (see B)

Pro: Con:
Test run on test environments No collection of test results in TFS
Tests run on multiple environments Harder to configure
Test Results in Build result Need for specific test client environments
Test Settings from VS


D

Flavor D: Execution from Microsoft Test Manager…than BVT
Other type of test
Part of Regression Tests
(triggered by: MTM user, right mouse click on test case, run)

Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode, can be the same as MTM)
Configure Lab Center in MTM to use test controller and create test ‘agent’ environment.
http://msdn.microsoft.com/en-us/library/ee390842.aspx
http://msdn.microsoft.com/en-us/library/dd293551.aspx
Associate CodedUI test with WI Test Case from VS.
http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html

Pro: Con:
Test run on test environments Manually triggered by Tester
Tests run on multiple environments Hard to configure
Test Result in MTM and TFS Need for specific test client (same as MTM?)
Test Settings from MTM


E

Flavor E: Execution from MTM duringPreferred configuration above flavor C
Build…
Part of BVT Flavor D and E can be configured together
(triggered by: Build)

Create Build task to run TCM or MSTEST task for Test Plan
http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx

Pro: Con:
Test run on test environments Hard to configure
Tests run on multiple environments Need for specific test client (same as MTM?)
Test Result in MTM and TFS
Triggered by build

Execute CodedUI
During the build with Visual Studio 2010

2
Add new Test Settings file
Set remote execution and
test controller.

3
Create a new build definition
Set the Automated Test Properties

4
Queue a new Build

Watch the Test Agent and
Test Environment

Execute CodedUI
Within a test case as an associated automation manualy

2
Associate the CodedUI
with a Test Case.

4
Associate the CodedUI
with a Test Case.

5
Run Automated Tests from Test Manager

Execute CodedUI
Within a test case as an associated automation from the build


E

Flavor E: Execution from MTM duringPreferred configuration above flavor C
Build…
Part of BVT Flavor D and E can be configured together
(triggered by: Build)

Create Build task to run MTC or MSTEST task for Test Plan
http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx

Pro: Con:
Test run on test environments Hard to configure
Tests run on multiple environments Need for specific test client (same as MTM?)
Test Result in MTM and TFS
Triggered by build
Test Settings from MTM

Execute automated test cases on
different environments

1
Configure additional environments in Test Plan properties.
Win 7 IE9 configuration

TC MTLM Client
Physical
Test Client
TFS 2010

Physical
Test Client

1
Get Test Plan ID
Get Test Suite
Get Test Configurations
with TCM.exe

2
Create command line for Test Run for the different Test Configurations

"run /create /title:""Run on Host for specific Test Point"" /planid:16
/querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=13""
/settingsname:""Automated Test on Windows 7 Environment IE9""
/testenvironment:""Win 7 IE9"" /collection:http://clemenes-
pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"

"run /create /title:""Run on test env for specific Test Point"" /planid:16
/querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=12""
/settingsname:""Automated Test on Windows 7 Environment IE8""
/testenvironment:""Win 7 IE8"" /collection:http://clemenes-
pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"

6
Queue a new Build

Watch the Test Agent and
Test Environment

Build
application

Provision
Run tests
‘clean’ env

Deploy
application

VS2010 Lab Management
• Reproduce the exact conditions of a bug or other development
issue.

• Build, deploy, and test applications automatically in a clean
environment.

• Reduce the time required to create and configure machines for
testing an application.

• Run multiple copies of a test or development at the same time.

• Enable members of a team to create and manage virtual
environments without requiring system administrator privileges.

MTLM Client

TFS 2010 System Center Virtual Machine
Manager (SCVMM)

Hyper-V Library
Host Shares

VM share

VM share

VS2010 ALM Tools Meeting: Test Automation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to VS2010 ALM Tools Meeting: Test Automation

Similar to VS2010 ALM Tools Meeting: Test Automation (20)

More from Clemens Reijnen

More from Clemens Reijnen (18)

Recently uploaded

Recently uploaded (20)

VS2010 ALM Tools Meeting: Test Automation