This document provides an overview of a presentation titled "Large-Scale Agile Test Automation Strategies in Practice" given by Geoff Meyer from Dell, Inc. The presentation covered key agile testing concepts, real-world examples from large-scale agile projects at Dell, organizational strategies for testing at scale, test automation strategies and tooling considerations, and lessons learned from Dell's experience with large agile projects. The document includes an agenda, background on Geoff Meyer, and details on test automation approaches for three example projects at Dell.

1. AT8
Concurrent Session
11/12/15 11:30am
“Large-Scale Agile Test Automation Strategies
in Practice”
Presented by:
Geoff Meyer
Dell, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 · 904-278-0524 · info@techwell.com · www.techwell.com

2. Geoff Meyer
Dell, Inc.
As director of SW Test and DevOps for Dell.com, Geoff Meyer leads the definition and
implementation of the Agile Scrum Software Test processes, test automation strategy, and the
adoption of continuous delivery practices for Dell’s eCommerce site. A seventeen-year veteran
of Dell, Geoff’s roles have spanned the software development spectrum including test architect,
program manager, software manager, and developer. Geoff has been an active member of the
Agile Austin community since 2010.

3. Agile Test Strategies in Practice:
Tales of Large-scale Agile projects
12 November 2015
Geoff Meyer, geoffrey_meyer@dell.com
SW Engineering Director, Dell Inc
• Introduction to Key Agile Testing concepts
• Real-world examples − from multiple, large-scale
projects
• Organizational strategies & considerations
• Test automation strategies and tooling
considerations
• Test Data Management
• An In-depth Tools Discussion
• Focused on how to develop Test Automation
IS
IS
NOT
Session Objectives

4. Geoff Meyer, SW Director
Dell Commerce Services Group
2015
2010
2005
2000
1998
1995
1990
1984
Personal
Family is paramount; my
wife is a Professional
Networking queen and
my kids make me proud!
Lives
Austin, TX
Lived
San Diego, Twin Cities
Hobbies
Travel, Avid fan of NFL,
NHL, EPL, MLB, Golfer,
Monday Morning
Quarterback
Business
Specialty
Change Agent and
champion for Agile
transformations
Passion
Solving problems with
great people and
enjoying every workday
Operating Model
Embrace Change and
Continuous Learning
Roles
Director, Architect,
Program Manager,
Manager, Developer
My Role @ Dell
Leads
Test and DevOps
functional areas for
Dell.com
Mission
Implement Continuous
Testing, Integration and
Delivery practices
Goal
Transform Dell.com into
a SW Development
engine characterized by
Quality, Predictability
and Continuous Delivery
Agenda
• Agile Testing Concepts
– Organization and
Architecture
– Agile Testing
– Test Automation
– Continuous Testing
– Large-Scale Agile and SAFE
• Agile Projects in Action

5. Agile Roles/Responsibilities
5
Program
Product Owner
Product Owner
Scrum Master
Development
Test
• Small teams
• Co-located
• Multi-disciplined
• Feature-based
…
Program Manager
SW Architect
Test Architect
DevOps
• Distributed Projects
• Scrum of Scrums
Week 1 Week 2
Agile Sprint Testing Cycle
Manual Regression
Automated
Regression
Automated
Regression
Automated
Regression
Sprint
Planning
Automated
Regression
Automated
Regression
Sprint
Review
Retro
spective
Demo
Accept
User
Story
Demo
Accept
User
Story
Demo
Accept
User
Story
Demo
Accept
User
Story
Demo
Accept
User
Story
Product Owner
Scrum Master
Development
Test

6. Automation Pyramid
7
CAUTION: Architecture (and Organization!) can be an enabler or inhibitor
“Automate Within the
Sprint. Automation on a
Scrum project is not
optional.” - Mike Cohn
3
Agile Testing Quadrant
8
More Agile Testing: Learning Journeys for the Whole Team. Gregory, Janet; Crispin, Lisa (2014-09-30).
Feature
Team
System
Team(s)
UAT
E2E
Performance

7. Automation Tooling
9
• IDE
• UI Tools
• Scripting Languages
• Services
• Unit
• Frameworks
MS-Test
Scaled Agile Framework™ Big Picture
© 2008- 2013 Scaled Agile, Inc and Leffingwell, All rights reserved

8. Agile Projects in Action @ Scale
Project Profiles
Application Architecture
Presentation Layer
Business/Service Layer
Data Access Layer
Moe
Systems Mgmt
Application
Curley
eCommerce UI
Application
Larry
eCommerce
Services

9. Organizing
Team Roles/Responsibilities
13
Program
Product Owner
Product Owner
Scrum Master
Development
Test
Agile Test Strategy: Moe
All Agile Testing
managed within
the program

10. Agile Test Strategy: Curley
UAT and ‘ility’
testing conducted
by external teams
UAT Regression
tests automated by
the program team
Agile Test Strategy: Larry
‘ility’ testing
conducted by
external teams

11. Automation Strategy: Moe
UI Automation limited to
Scenario-based workflows
In-Sprint Automation at the
Unit and Service Levels
Confidential18
Functional
Validation
Performance-
Characterization and
non-Functional Validation
Longevity
11 M2 1.0
SST
Baseline
M1
Scale
Concurrency
Simulation
L
a
r
g
e
-
S
c
a
l
e
L
a
r
g
e
-
S
c
a
l
e
12 13 17 18 19…
Sprint 15
‘ility Testing’; Moe
14 16
Sprint 21
Key Dependency:
• 3rd party HW available
• 3rd party user stories
completed

12. Automation Strategy: Curley
UI Automation to Scenario-
based workflows+UI functional
tests
Functional Test Automation
done In-Sprint at the Unit and
UI Levels
Services Virtualization to support
Continuous Testing
Automation Strategy: Larry
Functional Test Automation
done in-Sprint at the Unit and
Service Levels
MS-Test
Services Virtualization to support
Continuous Testing

13. Continuous Integration &
Automated Testing: Moe
21
Automated TestingContinuous BVT
Team #1 Feature Test
• Exploratory Testing
• Acceptance Testing
Team #2 Feature Test
• Exploratory Testing
• Acceptance Testing
Team #N Feature Test
• Exploratory Testing
• Acceptance Testing
CI/Build
TFS TeamBuild
To Factory
Continuous Integration &
Continuous Testing: Curley
22
Continuous TestingContinuous BVT
SIT
App Test VM
Team #1 Feature Test
• Exploratory Testing
• Acceptance Testing
App Test VM
Team #2 Feature Test
• Exploratory Testing
• Acceptance Testing
App Test VM
Team #N Feature Test
• Exploratory Testing
• Acceptance Testing
…
CI/Build
TFS TeamBuild
To Staging &
Production

14. Continuous Delivery
23
CI/Build
BVT
Tests
BVT Automated
Smoke
DVT
UAT
Production
Baseline
Load
Stress
Performance
E2E Tests
UAT regression
Staging
UAT
Func. regression
UAT regression
E2E
SIT
Unit test
Integration test
Dev Test
Unit test
Integration test
Dev Test
Unit test
Integration test
Dev Test
Unit test
Integration test
Dev Test
Acceptance Tests
Exploratory Tests
UI Tests
Func. Regression
UAT regression
App Test
Acceptance Tests
Exploratory Tests
UI Tests
Func. Regression
UAT regression
App Test
Acceptance Tests
Exploratory Tests
UI Tests
Func. Regression
UAT regression
App Test
Unit test
Integration test
Dev Test
Static Analysis
Auto. Unit Test
Gated Check-ins
Minutes Minutes Accept
Auto
Depoy
Auto
Deploy
Zero-Touch
Deploy
Auto
Deploy
• Automated Smoke, DVT, BVT & Regression Suites
• Continuous Integration–Fast feedback/Green Builds
• On-demand Environment Provisioning & Deployment
• Local Control of Environments: Mocks & Data
The Lessons of ‘Large’
• There is no cookie-cutter Test
Strategy
• Strive for co-located Teams
• Establish Culture of
Automation
• Create Cross-team roles
• Inspect and Adapt

15. Dell - Restricted - Confidential
Recommended Readings
25
Questions?