“Of course, we’ve had Visual Studio for a long time…” [click] “In 2005, we introduced Team Foundation Server, which allowed us to provide a central repository of data around application development…” [click] “This central server also became our integration point for those not working within Visual Studio…” [click] “With 11, we are incorporating the DevOps story and other tools for managing an application outside of its relatively short time in developer’s hands…” [click] “And, as with most Microsoft products, the real strength of this platform is the ability to extend it, integrate it, and make it your own!”
Considering how IT has evolved over the past decades. Why was Agile a relatively newcomer to the process?
Setting aside that debate for a second. The cost of transitioning states was and still can be an expensive endeavor. For instance, if the cost to deploy your application to the QA team is expensive, I would try to do that as few times as I could. The same holds – if the cost of deploying the application is high, I’d want to do that as few times as possible. So – the longer that I could stay in that state – that I just transitioned to – I get more value. My team (it was thought) was more productive because they are doing the same thing in a repeatable fashion. Now what I mean by that was – after I’m in my test/stabilization phase, I knew I could have my testers execute tests, and execute tests repeatable. I can have my developers fix bugs repeatedly. I wasn’t trying to mix things up by having people work on different functions on the project at the same time.
Setting aside agile v waterfall at the outset, what if there was this world, where at least we reduced or almost removed those hurdles to giving teams the way to decide what is most efficient for them without the artificial challenges. What if we automate the build/deploy/test cycle for one. What if we could actually have tools that could make us more efficient in doing our real jobs – in this case – the testing of our environment.
Practices to continuously test software against the defined acceptance criteria as it is being developed, using the appropriate tools, and by the appropriate stakeholder functions (product owners, developers, testers, users). Continuous acceptance testing helps prevent late detection of unmet requirements and reduces rework costs.
An ALM solution that enables continuous acceptance testing should include tools and services to:
Enable a mix of testing practices that engage cross-functional stakeholders in continuously validating the fulfillment of acceptance criteria (product owners, developers, testers, users, and operations). Engaging stakeholder functions early and often in the lifecycle to validate the fulfillment of acceptance criteria will help prevent the late detection of unmet requirements and reduce rework costs Build -> deploy -> test automation services to reduce cycle times in continuously integrating and validating the quality of team builds Test lab management automation services to reduce the cycle times in standing up and releasing test environments needed to continuously execute tests of varying intents (build verification testing, systems integration testing, user acceptance testing, stress testing)
Talk Track: A lab management automation solution reduces the cycle times to manage and provision the test environments needed to enable continuous acceptance testing.
A lab management automation solution reduces the cycle times to manage and provision the test environments needed to enable continuous acceptance testing. A lab management automation solution should include capabilities to:
Store and manage baseline templates for the needed test environments Provision environments on demand using the environment templates Provision environments to on-premise bare metal, private cloud, and public cloud infrastructures Snapshot environments when defects are encountered and needed to be reported Provision environments from snapshots to enable defect reproduction and resolution
Environments supported include:
On-premise self management environments On-premise virtualized environments managed using SCCM
Talk Track: Lab Center lets you create and manage your testing environments from a single location. Environments can be stored on local metal, or virtualized on site or in the cloud. Maintain multiple realistic environments so that your application can be quickly deployed and tested under the configurations you expect your users to use.
Environments can be based on self-managed infrastructure or virtualized infrastructure managed using SCCM (System Center Configuration Manager)
Talk Track: With Lab Manager, your build and deployment cycle times can be reduced in step with your development and testing cycle times. Automating your build and deployment pipeline will reduce manual errors, decrease cycle times, and increase quality.
Talk Track: Whether it is bare metal machines, or multi-part SCVMM environments, Lab Manager helps you manage your test environments from a single location. Testers can quickly find the right environment to test on, reducing hand-off times and decreasing manual deployment defects. The ability to take environment snapshots when bugs are found significantly reduces the likelihood of “no-repro” errors.
Talk Track: Enable data collectors and generate detailed test results to ensure any bugs that are found are actionable. Team Foundation Server keeps track of all the pieces, keeping your entire team on the same page and reducing time to repair.
Talk Track: Historically, the build-deploy-test pipeline is the most time expensive and manually-driven portion of the software development lifecycle. Introducing continuous deployment with Lab Manager ensures that testers know which builds to test and have the correct virtual environments to test on; it eliminates the hand-off wait time between developers and QA. Quality is increased, cycle times are reduced, and value is delivered to your customers.
Overview of Lab Management and TFS
Setting Up Dev & Test
With TFS and Visual Studio
Chris Kadel - Principal
Testing Maturity / Dynamic Environment
Ad Hoc Planned Virtual Lab
Characteristics of Efficient Testing
Bugs are Reproduced
Devs Fix on First Time
We Have Environments On Demand
Why did Waterfall dominate approaches
throughout most of IT “time”?
Economy of scale…
Test / Stabilize
Cost of Setup
Avoid Context Switch
Continuous acceptance testing
Business Rules and Logic
Ad-hoc exploratory testing
Planned manual testing
UI automation testing
Build | Deploy | Test
Role tailored tools
Why Lab Environment Automation?
Can I get another QA environment to isolate some testing
without slowing down my team?
We need to isolate that environment like it were a retail
location (network isolated)?
We just need a QA environment for a few weeks and then
Remember those data collectors for my client-server
application? I want those.
I want to do all that above, but I want it up in Windows Azure.
Installed on the “client” machines
Three types of agents
Test Agent-Enables automated testing
Build Agent-Enables automated
Lab Agent-Enables network isolation
Installed Automatically by Lab
Create new multi-machine
Create a new
environment using virtual
machine templates in
machines running on
Quickly choose multiple
test suites to run after
Choose your environment,
and build, deploy and test in
that environment from a
during manual testing to
Spin up virtual test environments
from within Test Manager, or attach a
virtual environment to a manual
taken and attached as a
link with bug
Developers can quickly debug
issues on the exact same
Test run results are attached to
specific builds and environments
Collect all the data required
to recreate bugs from virtual
Test Controllers and Agents
Collect Test Impact
Register with Team
Test agent controller
manages test agents
Test agents run tests and collect
data or impact the system
About the Demo
Define Tests Define Environments Define Data Collection Analyze Results
Development / Release Team
Author Build Author Deployment Run
On Demand in the Cloud
Gallery or VHD
Manage in Lab Center
Snapshot in Azure
Labs and Walkthroughs
Create and store VMs - MSDN doc
Prepare a domain controller VM
Manage virtual environments
Creating virtual envs - MSDN doc
Creating and working with virtual envs
Concepts and guidelines
Creating network isolated environments