SlideShare a Scribd company logo
© Utopia Solutions
Building a Strategy for Test Automation Success
Lee Barnes, CTO
Utopia Solutions
© Utopia Solutions
Agenda
•What is the end goal?
•Why do we need a strategy?
•What do we need to consider?
•What should a strategy look like?
Test Automation Strategy Overview
Interactive Case Study
© Utopia Solutions
SECTION 1
What is Our End Goal?
© Utopia Solutions4
Test Automation Success Criteria
• Issue detection and recovery
• Accurate verification
• Unattended execution
Reliable
• Minimum sensitivity to application and test
case changes
• Test case definition separate from
automation code
Maintainable
• Test coverage expanded efficiently
• Automated test cases created by non-
automation resources
Scalable
© Utopia Solutions5
Test Automation Approach
Test Automation Framework
Automated Test Cases Test Results
Test Definition
Test Execution
Non-technical / Test Analysts & Subject Matter Experts
Technical / Automation Engineers
© Utopia Solutions
SECTION 2
Why Do We Need a Strategy?
© Utopia Solutions8
Why Test Automation Fails
Organizational
• Lack of support
• Unrealistic expectations
• Disjointed org structure
• Missing skill sets
Process
• Immature testing process
• Segregated processes
• Lack of test environment / data
control
Technical
• Poor tool support for AUT
• Under or over design automation
approach
© Utopia Solutions9
Are We Ready?
© Utopia Solutions
SECTION 3
What Do We Need to Know First?
© Utopia Solutions11
Test Automation Readiness Assessment
Org
• Structure
• Support
• Goals and
Expectations
• Resources and Skills
Process
• Test Maturity
• Test Assets
• Process Integration
• Test Environment /
Data
Tech
• AUT Technology
• AUT / Tool
Integration
Assessment
Topics
© Utopia Solutions
Where Do We Get the Information?
Assessment
Input
Stakeholders
Existing
Assets
AUTs
© Utopia Solutions
Ideal Organizational Attributes
Structure
• Centralized
testing
function
• Testing is a
peer of
development
• Automation
roles are
defined
Goals and
Expectations
• Automation is
an investment
• 100%
automation is
not likely
• Automation
requires
specialized
skills
• Automation
will free up
resources to
work on high-
value tasks
Support
• Executive
management
has visibility to
automation
• Automation
has a
dedicated
budget
Resources and
Skills
• Resources are
dedicated to
automation
• Resources are
skilled in
automation as
a process AND
technology
© Utopia Solutions
Ideal Process Attributes
Test Process
• Test objectives
are clear and
measurable
• Test cases are
defined and
documented
• Expected
results are
predictable
SDLC Integration
• AUT in a
frequent
release state
with visibility to
future roadmap
• Visibility to
system changes
as early as
possible in
release cycle
Test Environment
• Test
environment
dedicated to
automation
• Test data
controlled
© Utopia Solutions
Ideal Technology Attributes
AUT Technology
• Multiple AUTs built on
common technology
and/or development
framework
• Consistent use of UI
objects
• Consistent navigation
• Consistent presentation
of user feedback
(warnings, errors, etc.)
Tool / AUT Integration
• All UI objects supported
by tool
• Any alternative interfaces
(web services, API, DB,
etc.) supported by tool
• All target end user
environments supported
by tool
© Utopia Solutions
SECTION 4
Common Assessment Findings
© Utopia Solutions17
Organizational
Distributed
Testing /
Automation
Efforts
App
App
App
App
• Uneven automation
resource demand
• Inconsistent approach
• Duplicated efforts
• Restricted skill growth
and knowledge sharing
Leads to…
© Utopia Solutions18
Organizational
AUT A
AUT B AUT C
AUT D
Centralized
Test Automation
Team
Common Resources,
Processes &
Technology
• Flatten demand curve
• Accelerate skills
development
• Leverage reusable
knowledge and technology
Benefits
Centralize Automation
Activities
© Utopia Solutions19
Process
Analyze Design Build
Test Deploy
No ability to identify impact of app changes on
automated test suite
Unexpected /
Unpredictable
Maintenance
© Utopia Solutions20
Process
Analyze Design Build Test Deploy
Capture potential impacts as early as possible to enable
proactive maintenance
Capture App Changes
Analyze
Impact
Maintain Execute
Automation Activities
Application SDLC
© Utopia Solutions21
Environment
Shared Test Environment
Uncontrolled Test Data
Integration
Functional
Performance
Integration
Test
• Schedule conflicts
• Unreliable execution
• Unnecessary “hardening”
of the test suite
Leads to…
Automation
© Utopia Solutions22
Environment
Shared Test Environment
Controlled Test Data
Integration
Functional
Performance
Integration
Test
Dedicated Environment for
Automation
Test Data Controls
Dedicated
Environment
Controlled
Test Data
Automation
• Restore to known state
• Partition data related
to specific test
activities
• Create required data
each execution
© Utopia Solutions
SECTION 5
What Does a Strategy Look Like?
© Utopia Solutions24
Mobile Test Strategy Topics
Test
Automation
Strategy
Process &
Organization
Environment
TechnicalResources
Scope &
Roadmap
© Utopia Solutions25
Automation Strategy – Process and Organization
• Political and financial support
• Realistic expectations
• Centralized automation function
Key Organizational Components
• Test objectives and coverage goals
• Integration with SDLC → proactive
maintenance
Key Process Components
© Utopia Solutions26
Automation Strategy – Environment
• Dedicated environment for execution
• Controlled test data
• Controlled code migration
Key Environment Components
© Utopia Solutions27
Automation Strategy – Resources
• Roles
• Skills
• Time
Key People Components
• AUT test environment(s)
• Test suite development
• Test suite execution
Key Hardware Components
• Test tool
• Supporting utilities
Key Software Components
© Utopia Solutions28
Automation Strategy – Scope
• Test data setup
• AUT / environment preparation
• Complex / tedious results verification
Test Support Tasks
• Applications built on common technology
• Applications with frequent releases & friendly UI’s
Application Selection
• Frequently tested
• High test case to UI ratio
• Predictable results
• Easy to automate
Test Case Selection
© Utopia Solutions29
Automation Strategy – Roadmap
• Begin with required organizational and process changes
• Overlap with base framework and minimal coverage
• Enhance and add test coverage over small iterations
Create Implementation Timeline
Phase nPhase 1
Continuous Process & Organizational Improvements
Base Framework
Initial Coverage
AUT 1
Framework Enhancements
Expand Coverage
Initial Coverage
Phase 2
AUT 2
Expand Coverage
Expand Coverage
Initial Coverage
Phase 3
AUT 3
© Utopia Solutions
SECTION 6
Case Study
© Utopia Solutions31
Key Assessment Findings
Organization
• All enterprise testing activities
currently performed centrally
• Management believes automation
is a tool implementation
Process
• Manual tests are well structured,
but not maintained consistently
• Rely on SME’s to execute tests
Environment
• Manual regression test performed
in a isolated environment
• Test data refreshed in all
environments once per quarter
Technology
• All AUTs have browser based UI
• Developed on disparate
technologies
• Test tool supports all AUT
components
Resources
• Test organization consists of non-
technical SME’s
• System performance (mobile)
• Device performance
© Utopia Solutions32
Key Summary Points
• DO NOT automate without a clear vision and plan
• Understand your readiness from these perspectives
• Organization
• Process & Environment
• Technical
• Resources
• Implement in small phases
© Utopia Solutions33
Questions…
…. and Answers!
Direct future questions to:
Lee Barnes
Founder and CTO
Utopia Solutions, Inc.
Email: lee.barnes@utopiasolutions.com
Twitter: twitter.com/USI_LeeBarnes
LinkedIn: linkedin.com/in/leebarnes
Blog: www.RaiseYourMQ.com

More Related Content

What's hot

Test Automation Framework Designs
Test Automation Framework DesignsTest Automation Framework Designs
Test Automation Framework DesignsSauce Labs
 
Test Automation Framework Design | www.idexcel.com
Test Automation Framework Design | www.idexcel.comTest Automation Framework Design | www.idexcel.com
Test Automation Framework Design | www.idexcel.comIdexcel Technologies
 
Test automation
Test automationTest automation
Test automationXavier Yin
 
Test Automation
Test AutomationTest Automation
Test Automationrockoder
 
Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)Leonard Fingerman
 
Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8a34sharm
 
Software Testing Process, Testing Automation and Software Testing Trends
Software Testing Process, Testing Automation and Software Testing TrendsSoftware Testing Process, Testing Automation and Software Testing Trends
Software Testing Process, Testing Automation and Software Testing TrendsKMS Technology
 
Agile Testing Strategy
Agile Testing StrategyAgile Testing Strategy
Agile Testing Strategytharindakasun
 
Test Automation Frameworks: Assumptions, Concepts & Tools
Test Automation Frameworks: Assumptions, Concepts & ToolsTest Automation Frameworks: Assumptions, Concepts & Tools
Test Automation Frameworks: Assumptions, Concepts & ToolsAmit Rawat
 
Introduction to Automation Testing
Introduction to Automation TestingIntroduction to Automation Testing
Introduction to Automation TestingArchana Krushnan
 
Agile QA and Testing process
Agile QA and Testing processAgile QA and Testing process
Agile QA and Testing processGloria Stoilova
 
6 Traits of a Successful Test Automation Architecture
6 Traits of a Successful Test Automation Architecture6 Traits of a Successful Test Automation Architecture
6 Traits of a Successful Test Automation ArchitectureErdem YILDIRIM
 
Introduction to Automation Testing
Introduction to Automation TestingIntroduction to Automation Testing
Introduction to Automation TestingArchana Krushnan
 
Automation testing introduction for FujiNet
Automation testing introduction for FujiNetAutomation testing introduction for FujiNet
Automation testing introduction for FujiNetHai Tran Son
 

What's hot (20)

Test Automation Framework Designs
Test Automation Framework DesignsTest Automation Framework Designs
Test Automation Framework Designs
 
Test Automation - Keytorc Approach
Test Automation - Keytorc Approach Test Automation - Keytorc Approach
Test Automation - Keytorc Approach
 
Test Automation Framework Design | www.idexcel.com
Test Automation Framework Design | www.idexcel.comTest Automation Framework Design | www.idexcel.com
Test Automation Framework Design | www.idexcel.com
 
Automation testing
Automation testingAutomation testing
Automation testing
 
Test automation
Test automationTest automation
Test automation
 
Automation Testing
Automation TestingAutomation Testing
Automation Testing
 
Introduction to Agile Testing
Introduction to Agile TestingIntroduction to Agile Testing
Introduction to Agile Testing
 
Test Automation
Test AutomationTest Automation
Test Automation
 
Introduction to Software Test Automation
Introduction to Software Test AutomationIntroduction to Software Test Automation
Introduction to Software Test Automation
 
Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)
 
Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8
 
QA Best Practices in Agile World_new
QA Best Practices in Agile World_newQA Best Practices in Agile World_new
QA Best Practices in Agile World_new
 
Software Testing Process, Testing Automation and Software Testing Trends
Software Testing Process, Testing Automation and Software Testing TrendsSoftware Testing Process, Testing Automation and Software Testing Trends
Software Testing Process, Testing Automation and Software Testing Trends
 
Agile Testing Strategy
Agile Testing StrategyAgile Testing Strategy
Agile Testing Strategy
 
Test Automation Frameworks: Assumptions, Concepts & Tools
Test Automation Frameworks: Assumptions, Concepts & ToolsTest Automation Frameworks: Assumptions, Concepts & Tools
Test Automation Frameworks: Assumptions, Concepts & Tools
 
Introduction to Automation Testing
Introduction to Automation TestingIntroduction to Automation Testing
Introduction to Automation Testing
 
Agile QA and Testing process
Agile QA and Testing processAgile QA and Testing process
Agile QA and Testing process
 
6 Traits of a Successful Test Automation Architecture
6 Traits of a Successful Test Automation Architecture6 Traits of a Successful Test Automation Architecture
6 Traits of a Successful Test Automation Architecture
 
Introduction to Automation Testing
Introduction to Automation TestingIntroduction to Automation Testing
Introduction to Automation Testing
 
Automation testing introduction for FujiNet
Automation testing introduction for FujiNetAutomation testing introduction for FujiNet
Automation testing introduction for FujiNet
 

Viewers also liked

Test Automation: A Roadmap For Sucesss
Test Automation: A Roadmap For SucesssTest Automation: A Roadmap For Sucesss
Test Automation: A Roadmap For SucesssDavid O'Dowd
 
The Three Pillars Approach to Your Agile Test Strategy
The Three Pillars Approach to Your Agile Test Strategy The Three Pillars Approach to Your Agile Test Strategy
The Three Pillars Approach to Your Agile Test Strategy Brian Estep
 
Agile 2012 the 0-page agile test plan - paul carvalho
Agile 2012   the 0-page agile test plan - paul carvalhoAgile 2012   the 0-page agile test plan - paul carvalho
Agile 2012 the 0-page agile test plan - paul carvalhodrewz lin
 
How to brew a tasty agile test strategy
How to brew a tasty agile test strategyHow to brew a tasty agile test strategy
How to brew a tasty agile test strategyDr. Alexander Schwartz
 
Designing Agile Test Strategy for Mobile Apps By Parimala Hariprasad
Designing Agile Test Strategy for Mobile Apps By Parimala HariprasadDesigning Agile Test Strategy for Mobile Apps By Parimala Hariprasad
Designing Agile Test Strategy for Mobile Apps By Parimala HariprasadAgile Testing Alliance
 
Jarian van de Laar - Test Policy - Test Strategy
Jarian van de Laar - Test Policy - Test Strategy Jarian van de Laar - Test Policy - Test Strategy
Jarian van de Laar - Test Policy - Test Strategy TEST Huddle
 
AJRA Test Strategy Discussion
AJRA Test Strategy DiscussionAJRA Test Strategy Discussion
AJRA Test Strategy Discussionajrhem
 
Introduction to Test Automation - Technology and Tools
Introduction to Test Automation - Technology and ToolsIntroduction to Test Automation - Technology and Tools
Introduction to Test Automation - Technology and ToolsKMS Technology
 
Web Test Automation with Selenium
Web Test Automation with SeleniumWeb Test Automation with Selenium
Web Test Automation with Seleniumvivek_prahlad
 
Automation Testing using Selenium
Automation Testing using SeleniumAutomation Testing using Selenium
Automation Testing using SeleniumNaresh Chintalcheru
 
Enough is not enough - Test Strategy for Mobile
Enough is not enough - Test Strategy for MobileEnough is not enough - Test Strategy for Mobile
Enough is not enough - Test Strategy for MobilevodQA
 
Release Management for Large Enterprises
Release Management for Large EnterprisesRelease Management for Large Enterprises
Release Management for Large EnterprisesSalesforce Developers
 
Centralized test automation framework implementation
Centralized test automation framework implementationCentralized test automation framework implementation
Centralized test automation framework implementationBharathi Krishnamurthi
 
Drive Marketing Automation Success - Cross-Functional Alignment
Drive Marketing Automation Success - Cross-Functional AlignmentDrive Marketing Automation Success - Cross-Functional Alignment
Drive Marketing Automation Success - Cross-Functional AlignmentMarketoEnterpriseContent
 
360logica_Testing_center_of_excellence
360logica_Testing_center_of_excellence360logica_Testing_center_of_excellence
360logica_Testing_center_of_excellenceguestf2e1db
 

Viewers also liked (20)

Test Automation: A Roadmap For Sucesss
Test Automation: A Roadmap For SucesssTest Automation: A Roadmap For Sucesss
Test Automation: A Roadmap For Sucesss
 
The Three Pillars Approach to Your Agile Test Strategy
The Three Pillars Approach to Your Agile Test Strategy The Three Pillars Approach to Your Agile Test Strategy
The Three Pillars Approach to Your Agile Test Strategy
 
Agile 2012 the 0-page agile test plan - paul carvalho
Agile 2012   the 0-page agile test plan - paul carvalhoAgile 2012   the 0-page agile test plan - paul carvalho
Agile 2012 the 0-page agile test plan - paul carvalho
 
How to brew a tasty agile test strategy
How to brew a tasty agile test strategyHow to brew a tasty agile test strategy
How to brew a tasty agile test strategy
 
Test Strategy
Test StrategyTest Strategy
Test Strategy
 
Designing Agile Test Strategy for Mobile Apps By Parimala Hariprasad
Designing Agile Test Strategy for Mobile Apps By Parimala HariprasadDesigning Agile Test Strategy for Mobile Apps By Parimala Hariprasad
Designing Agile Test Strategy for Mobile Apps By Parimala Hariprasad
 
Jarian van de Laar - Test Policy - Test Strategy
Jarian van de Laar - Test Policy - Test Strategy Jarian van de Laar - Test Policy - Test Strategy
Jarian van de Laar - Test Policy - Test Strategy
 
AJRA Test Strategy Discussion
AJRA Test Strategy DiscussionAJRA Test Strategy Discussion
AJRA Test Strategy Discussion
 
Introduction to Test Automation - Technology and Tools
Introduction to Test Automation - Technology and ToolsIntroduction to Test Automation - Technology and Tools
Introduction to Test Automation - Technology and Tools
 
Web Test Automation with Selenium
Web Test Automation with SeleniumWeb Test Automation with Selenium
Web Test Automation with Selenium
 
Why a Mobile Test Strategy is just Test Strategy
Why a Mobile Test Strategy is just Test StrategyWhy a Mobile Test Strategy is just Test Strategy
Why a Mobile Test Strategy is just Test Strategy
 
Automation Testing by Selenium Web Driver
Automation Testing by Selenium Web DriverAutomation Testing by Selenium Web Driver
Automation Testing by Selenium Web Driver
 
Automation Testing using Selenium
Automation Testing using SeleniumAutomation Testing using Selenium
Automation Testing using Selenium
 
Selenium ppt
Selenium pptSelenium ppt
Selenium ppt
 
Enough is not enough - Test Strategy for Mobile
Enough is not enough - Test Strategy for MobileEnough is not enough - Test Strategy for Mobile
Enough is not enough - Test Strategy for Mobile
 
Release Management for Large Enterprises
Release Management for Large EnterprisesRelease Management for Large Enterprises
Release Management for Large Enterprises
 
Centralized test automation framework implementation
Centralized test automation framework implementationCentralized test automation framework implementation
Centralized test automation framework implementation
 
Drive Marketing Automation Success - Cross-Functional Alignment
Drive Marketing Automation Success - Cross-Functional AlignmentDrive Marketing Automation Success - Cross-Functional Alignment
Drive Marketing Automation Success - Cross-Functional Alignment
 
A380
A380A380
A380
 
360logica_Testing_center_of_excellence
360logica_Testing_center_of_excellence360logica_Testing_center_of_excellence
360logica_Testing_center_of_excellence
 

Similar to Building a Test Automation Strategy for Success

Don’t Be Another Statistic! Develop a Long-Term Test Automation Strategy
Don’t Be Another Statistic! Develop a Long-Term Test Automation StrategyDon’t Be Another Statistic! Develop a Long-Term Test Automation Strategy
Don’t Be Another Statistic! Develop a Long-Term Test Automation StrategyJosiah Renaudin
 
Questions for successful test automation projects
Questions for successful test automation projectsQuestions for successful test automation projects
Questions for successful test automation projectsDaniel Ionita
 
Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Kari Kakkonen
 
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".Emerasoft, solutions to collaborate
 
7 Tips from Siemens Energy for Success with Automation
7 Tips from Siemens Energy for Success with Automation7 Tips from Siemens Energy for Success with Automation
7 Tips from Siemens Energy for Success with AutomationWorksoft
 
Automation Culture: Essential to Agile Success
Automation Culture: Essential to Agile SuccessAutomation Culture: Essential to Agile Success
Automation Culture: Essential to Agile SuccessTechWell
 
Exploratory Testing Basics and Future Kari Kakkonen Moscow 2018
Exploratory Testing Basics and Future Kari Kakkonen Moscow 2018Exploratory Testing Basics and Future Kari Kakkonen Moscow 2018
Exploratory Testing Basics and Future Kari Kakkonen Moscow 2018Kari Kakkonen
 
Test planning and software's engineering
Test planning and software's engineeringTest planning and software's engineering
Test planning and software's engineeringMansiganeshJawale
 
Presentation1
Presentation1Presentation1
Presentation1anuvip
 
Exploratory Testing Kari Kakkonen BTD 2017
Exploratory Testing Kari Kakkonen BTD 2017Exploratory Testing Kari Kakkonen BTD 2017
Exploratory Testing Kari Kakkonen BTD 2017Kari Kakkonen
 
What_is_Software_Testing.pdf
What_is_Software_Testing.pdfWhat_is_Software_Testing.pdf
What_is_Software_Testing.pdfVuongPhm
 
Role of Test Automation in Modern Software Delivery Pipelines
Role of Test Automation in Modern Software Delivery PipelinesRole of Test Automation in Modern Software Delivery Pipelines
Role of Test Automation in Modern Software Delivery PipelinesKasun Kodagoda
 
Lean Solutions – Agile Transformation at the United States Postal Service
Lean Solutions  – Agile Transformation at the United States Postal ServiceLean Solutions  – Agile Transformation at the United States Postal Service
Lean Solutions – Agile Transformation at the United States Postal ServiceITSM Academy, Inc.
 
Setting up Center of Excellence for QA in Healthcare
Setting up Center of Excellence for QA in HealthcareSetting up Center of Excellence for QA in Healthcare
Setting up Center of Excellence for QA in HealthcareCitiusTech
 
Automation in the world of project
Automation  in the world of projectAutomation  in the world of project
Automation in the world of projectZbyszek Mockun
 

Similar to Building a Test Automation Strategy for Success (20)

Don’t Be Another Statistic! Develop a Long-Term Test Automation Strategy
Don’t Be Another Statistic! Develop a Long-Term Test Automation StrategyDon’t Be Another Statistic! Develop a Long-Term Test Automation Strategy
Don’t Be Another Statistic! Develop a Long-Term Test Automation Strategy
 
Questions for successful test automation projects
Questions for successful test automation projectsQuestions for successful test automation projects
Questions for successful test automation projects
 
Software Testing
Software Testing Software Testing
Software Testing
 
Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle
 
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".
 
7 Tips from Siemens Energy for Success with Automation
7 Tips from Siemens Energy for Success with Automation7 Tips from Siemens Energy for Success with Automation
7 Tips from Siemens Energy for Success with Automation
 
Automation Culture: Essential to Agile Success
Automation Culture: Essential to Agile SuccessAutomation Culture: Essential to Agile Success
Automation Culture: Essential to Agile Success
 
Exploratory Testing Basics and Future Kari Kakkonen Moscow 2018
Exploratory Testing Basics and Future Kari Kakkonen Moscow 2018Exploratory Testing Basics and Future Kari Kakkonen Moscow 2018
Exploratory Testing Basics and Future Kari Kakkonen Moscow 2018
 
Test planning and software's engineering
Test planning and software's engineeringTest planning and software's engineering
Test planning and software's engineering
 
Presentation1
Presentation1Presentation1
Presentation1
 
Exploratory Testing Kari Kakkonen BTD 2017
Exploratory Testing Kari Kakkonen BTD 2017Exploratory Testing Kari Kakkonen BTD 2017
Exploratory Testing Kari Kakkonen BTD 2017
 
7 steps to Software test automation success
7 steps to Software test automation success7 steps to Software test automation success
7 steps to Software test automation success
 
What_is_Software_Testing.pdf
What_is_Software_Testing.pdfWhat_is_Software_Testing.pdf
What_is_Software_Testing.pdf
 
Role of Test Automation in Modern Software Delivery Pipelines
Role of Test Automation in Modern Software Delivery PipelinesRole of Test Automation in Modern Software Delivery Pipelines
Role of Test Automation in Modern Software Delivery Pipelines
 
Lean Solutions – Agile Transformation at the United States Postal Service
Lean Solutions  – Agile Transformation at the United States Postal ServiceLean Solutions  – Agile Transformation at the United States Postal Service
Lean Solutions – Agile Transformation at the United States Postal Service
 
Setting up Center of Excellence for QA in Healthcare
Setting up Center of Excellence for QA in HealthcareSetting up Center of Excellence for QA in Healthcare
Setting up Center of Excellence for QA in Healthcare
 
Journey to the center of DevOps - v6
Journey to the center of DevOps - v6Journey to the center of DevOps - v6
Journey to the center of DevOps - v6
 
[HCMC STC Jan 2015] Practical Experiences In Test Automation
[HCMC STC Jan 2015] Practical Experiences In Test Automation[HCMC STC Jan 2015] Practical Experiences In Test Automation
[HCMC STC Jan 2015] Practical Experiences In Test Automation
 
Embedded world 2017
Embedded world 2017Embedded world 2017
Embedded world 2017
 
Automation in the world of project
Automation  in the world of projectAutomation  in the world of project
Automation in the world of project
 

Recently uploaded

Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka DoktorováCzechDreamin
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2DianaGray10
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...CzechDreamin
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyUXDXConf
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCzechDreamin
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekCzechDreamin
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfFIDO Alliance
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Julian Hyde
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIES VE
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxAbida Shariff
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoTAnalytics
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyJohn Staveley
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxDavid Michel
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastUXDXConf
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...CzechDreamin
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsStefano
 

Recently uploaded (20)

Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 

Building a Test Automation Strategy for Success

  • 1. © Utopia Solutions Building a Strategy for Test Automation Success Lee Barnes, CTO Utopia Solutions
  • 2. © Utopia Solutions Agenda •What is the end goal? •Why do we need a strategy? •What do we need to consider? •What should a strategy look like? Test Automation Strategy Overview Interactive Case Study
  • 3. © Utopia Solutions SECTION 1 What is Our End Goal?
  • 4. © Utopia Solutions4 Test Automation Success Criteria • Issue detection and recovery • Accurate verification • Unattended execution Reliable • Minimum sensitivity to application and test case changes • Test case definition separate from automation code Maintainable • Test coverage expanded efficiently • Automated test cases created by non- automation resources Scalable
  • 5. © Utopia Solutions5 Test Automation Approach Test Automation Framework Automated Test Cases Test Results Test Definition Test Execution Non-technical / Test Analysts & Subject Matter Experts Technical / Automation Engineers
  • 6. © Utopia Solutions SECTION 2 Why Do We Need a Strategy?
  • 7. © Utopia Solutions8 Why Test Automation Fails Organizational • Lack of support • Unrealistic expectations • Disjointed org structure • Missing skill sets Process • Immature testing process • Segregated processes • Lack of test environment / data control Technical • Poor tool support for AUT • Under or over design automation approach
  • 9. © Utopia Solutions SECTION 3 What Do We Need to Know First?
  • 10. © Utopia Solutions11 Test Automation Readiness Assessment Org • Structure • Support • Goals and Expectations • Resources and Skills Process • Test Maturity • Test Assets • Process Integration • Test Environment / Data Tech • AUT Technology • AUT / Tool Integration Assessment Topics
  • 11. © Utopia Solutions Where Do We Get the Information? Assessment Input Stakeholders Existing Assets AUTs
  • 12. © Utopia Solutions Ideal Organizational Attributes Structure • Centralized testing function • Testing is a peer of development • Automation roles are defined Goals and Expectations • Automation is an investment • 100% automation is not likely • Automation requires specialized skills • Automation will free up resources to work on high- value tasks Support • Executive management has visibility to automation • Automation has a dedicated budget Resources and Skills • Resources are dedicated to automation • Resources are skilled in automation as a process AND technology
  • 13. © Utopia Solutions Ideal Process Attributes Test Process • Test objectives are clear and measurable • Test cases are defined and documented • Expected results are predictable SDLC Integration • AUT in a frequent release state with visibility to future roadmap • Visibility to system changes as early as possible in release cycle Test Environment • Test environment dedicated to automation • Test data controlled
  • 14. © Utopia Solutions Ideal Technology Attributes AUT Technology • Multiple AUTs built on common technology and/or development framework • Consistent use of UI objects • Consistent navigation • Consistent presentation of user feedback (warnings, errors, etc.) Tool / AUT Integration • All UI objects supported by tool • Any alternative interfaces (web services, API, DB, etc.) supported by tool • All target end user environments supported by tool
  • 15. © Utopia Solutions SECTION 4 Common Assessment Findings
  • 16. © Utopia Solutions17 Organizational Distributed Testing / Automation Efforts App App App App • Uneven automation resource demand • Inconsistent approach • Duplicated efforts • Restricted skill growth and knowledge sharing Leads to…
  • 17. © Utopia Solutions18 Organizational AUT A AUT B AUT C AUT D Centralized Test Automation Team Common Resources, Processes & Technology • Flatten demand curve • Accelerate skills development • Leverage reusable knowledge and technology Benefits Centralize Automation Activities
  • 18. © Utopia Solutions19 Process Analyze Design Build Test Deploy No ability to identify impact of app changes on automated test suite Unexpected / Unpredictable Maintenance
  • 19. © Utopia Solutions20 Process Analyze Design Build Test Deploy Capture potential impacts as early as possible to enable proactive maintenance Capture App Changes Analyze Impact Maintain Execute Automation Activities Application SDLC
  • 20. © Utopia Solutions21 Environment Shared Test Environment Uncontrolled Test Data Integration Functional Performance Integration Test • Schedule conflicts • Unreliable execution • Unnecessary “hardening” of the test suite Leads to… Automation
  • 21. © Utopia Solutions22 Environment Shared Test Environment Controlled Test Data Integration Functional Performance Integration Test Dedicated Environment for Automation Test Data Controls Dedicated Environment Controlled Test Data Automation • Restore to known state • Partition data related to specific test activities • Create required data each execution
  • 22. © Utopia Solutions SECTION 5 What Does a Strategy Look Like?
  • 23. © Utopia Solutions24 Mobile Test Strategy Topics Test Automation Strategy Process & Organization Environment TechnicalResources Scope & Roadmap
  • 24. © Utopia Solutions25 Automation Strategy – Process and Organization • Political and financial support • Realistic expectations • Centralized automation function Key Organizational Components • Test objectives and coverage goals • Integration with SDLC → proactive maintenance Key Process Components
  • 25. © Utopia Solutions26 Automation Strategy – Environment • Dedicated environment for execution • Controlled test data • Controlled code migration Key Environment Components
  • 26. © Utopia Solutions27 Automation Strategy – Resources • Roles • Skills • Time Key People Components • AUT test environment(s) • Test suite development • Test suite execution Key Hardware Components • Test tool • Supporting utilities Key Software Components
  • 27. © Utopia Solutions28 Automation Strategy – Scope • Test data setup • AUT / environment preparation • Complex / tedious results verification Test Support Tasks • Applications built on common technology • Applications with frequent releases & friendly UI’s Application Selection • Frequently tested • High test case to UI ratio • Predictable results • Easy to automate Test Case Selection
  • 28. © Utopia Solutions29 Automation Strategy – Roadmap • Begin with required organizational and process changes • Overlap with base framework and minimal coverage • Enhance and add test coverage over small iterations Create Implementation Timeline Phase nPhase 1 Continuous Process & Organizational Improvements Base Framework Initial Coverage AUT 1 Framework Enhancements Expand Coverage Initial Coverage Phase 2 AUT 2 Expand Coverage Expand Coverage Initial Coverage Phase 3 AUT 3
  • 30. © Utopia Solutions31 Key Assessment Findings Organization • All enterprise testing activities currently performed centrally • Management believes automation is a tool implementation Process • Manual tests are well structured, but not maintained consistently • Rely on SME’s to execute tests Environment • Manual regression test performed in a isolated environment • Test data refreshed in all environments once per quarter Technology • All AUTs have browser based UI • Developed on disparate technologies • Test tool supports all AUT components Resources • Test organization consists of non- technical SME’s • System performance (mobile) • Device performance
  • 31. © Utopia Solutions32 Key Summary Points • DO NOT automate without a clear vision and plan • Understand your readiness from these perspectives • Organization • Process & Environment • Technical • Resources • Implement in small phases
  • 32. © Utopia Solutions33 Questions… …. and Answers! Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: lee.barnes@utopiasolutions.com Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com