The document discusses the importance of test automation and continuous testing within DevOps to address bottlenecks in the software development process. It highlights the need for better integration of testing practices, automation strategies, and collaboration across teams to enhance quality and speed of releases. Key recommendations include adopting test-driven development, utilizing cloud services, and shifting testing earlier in the development lifecycle.

1. | Create Better Software © 2018 QASymphony – Proprietary
Developing a Testing Strategy
for DevOps Success
Ryan Yackel | Director of Product Dev

2. © 2018 QASymphony – Proprietary
Who Is This Guy?
Ryan Yackel
Insert Pic HERE
Director of Product Dev
Hobbies
• Specialty Coffee
• Superheroes
Testing Experience
• CGI Group | Test Automation Engineer
• Macy’s Systems & Tech | Enterprise Test Lead

3. Agenda
• What’s going on in the Industry
• What should we do?
• What steps can we take?
• Q&A

4. © 2018 QASymphony – Proprietary
The World Has Changed
Every business is a
software business
Software is eating
the world
Agile and DevOps
are the standard
Open source is
everywhere

5. © 2018 QASymphony – Proprietary
Organizations
practicing agile
development
Organizations have or
are planning a DevOps
initiative in their
company in the next 12
months
The difference in the
number of code
deployments between
high performing and
low performing IT
organizations
94% 71% 46x
The Pace of Software Development is
Moving Faster Than Ever
Source: V1 2016 State of Agile, Puppet 2016 State of DevOps

6. © 2018 QASymphony – Proprietary
An Accelerating World
Source: 2017, Fortune Magazine Survey of all Fortune 500 CEOs.
Technology was chosen as
either the greatest, or one of the
three or four greatest, challenges
facing their companies by 73% of
CEOs

7. © 2018 QASymphony – Proprietary
Explosion of Modern Technology

8. INSERT PHOTO ON TOP
Increased Automation
Design Code Test Deploy
Design Code Test Deploy
Waterfall
Agile
DevOps – CI/CD
1-4 releases a year
6-24 releases a year
100’s releases a year
Design Code Test Deploy
Faster cycle times =
increased output,
reduced waste

9. Customer Centric Hybrid Governance Oriented
Retail eCommerce
Software
Tech
Gaming
Telecom
Financial
Retail Traditional
Financial Services
Telecom
Healthcare
Insurance
Manufacturing
Automotive
Dev Ops
Industry Adoption is Not Equal

10. © 2018 QASymphony – Proprietary
Speed. Yes.
Quality…maybe.

11. © 2018 QASymphony – Proprietary
What’s Going On?

12. © 2018 QASymphony – Proprietary
Testing Remains the Bottleneck
Source: GitLab – 2018 Global Developer Report
90% of managers, compared to 84% of
developers, agree that practicing CI alleviates
blockers in the development process.
Developers said that they encountered the
most delays in the testing phase, whereas
among management, planning topped the
list as the number one cause of delay.

13. © 2018 QASymphony – Proprietary
Testing Siloed From Tool Stack
Source: VersionOne – 12th
State of Agile Report

14. © 2018 QASymphony – Proprietary
Lower Test Automation = Slower
Releases
Source: Sauce Labs, Testing Trends for 2018
The number of respondents with high levels
of test automation dropped from
32% (2017)
28% (2018)

15. © 2018 QASymphony – Proprietary
Source: QASymphony – 2018 Evolution of Test Automation

16. INSERT PHOTO ON TOP
Summing In Up
• Delays in test and planning
• Testing is disconnected with
pipeline and agile tools
• Low levels of test automation
• Confusion around test
automation and optimization
CODE
BUILD
T
E
S
T
PLAN

17. © 2018 QASymphony – Proprietary
What Should We Do?

19. What’s Missing?

20. © 2018 QASymphony – Proprietary
In real world DevOps, the question is not
“Do you have test automation?”…
…rather it is “How are you approaching
your test automation?”

21. © 2018 QASymphony – Proprietary
DevOps
Development Testin
g
Staging Production
Continuous
Integration
Continuous
Monitoring
Continuous
Testin
g
Continuous
Delivery
Continuous
Deployment
Quality is Now Continuous

22. Continuous Testing
Continuous Testing Buzz
“Continuous Testing is
required for extreme
automation”
voke
“Continuous Delivery
Needs Continuous
Testing”
Forrester
“DevOps initiatives will
cause enterprises to
implement continuous
testing”
Gartner
”Continuous Testing
has obvious application
in agile environments.”
Bloor

23. © 2018 QASymphony – Proprietary
What Steps Should
We Take?

24. © 2018 QASymphony – Proprietary
Testing in DevOps Stages
PLAN
RELEASE

25. © 2018 QASymphony – Proprietary
02Code
03Build
01Plan
04Test
PLAN
RELEASE

26. Manual &
Exploratory
Automat
ed
GUI
TestsAcceptance/
Integration/
Component
Unit
Tests
Uni
t
Tes
ts
Acceptance/
Integration/
Component
Automat
ed
GUI
Tests
Automation “Ice Cream Cone”
Anti-Pattern
Ideal Test
Automation Pyramid
Manual &
Exploratory
Higher
ROI
More Time &
Effort

27. © 2018 QASymphony – Proprietary
What Can We Automate?

28. © 2018 QASymphony – Proprietary
The Old “Silo-ed” Approach

29. © 2018 QASymphony – Proprietary
Capabilities Needed for Better Planning
Capability Why it’s Needed?
Visibility into Testing For all Teams – Dev, BA,
Operations
Promote organizational quality culture and allow
entire team to remove testing roadblocks
Real Time Status Updates Between Jira and Other
Systems
Make sure teams are all working on the same
version of the truth, tougher when doing many
releases/day
Consolidated and Customizable Dashboards –
Across Projects
Consolidate the various team level activities and
processes for executive buy-in and visibility

30. © 2018 QASymphony – Proprietary
02Code
03Build
01Plan
04Test
PLAN
RELEASE

31. © 2018 QASymphony – Proprietary
TDD
AI Cloud
Shifting testing earlier in the
development lifecycle through
open source automation
frameworks
Test Driven Development
Leveraging cloud services to
scale test automation across
multiple environments and
configurations
Cloud
Optimizing test planning
through predictive analytics
and code analysis
AI
Trends That Influence Testing in DevOps

32. © 2018 QASymphony – Proprietary
Moving Testing Up Front
Traditional Development Timeline
Moving Testing Up Front removes the risk of having to make compromises at
the end of the cycle on quality or on-time delivery:
There is risk in this process that any step, typically Code, will run over and either squeeze development,
or push release dates. TDD removes it!
Ends on: Day 1 Day 3 Day 14 Day 20 Day 21

33. © 2018 QASymphony – Proprietary
Moving Testing Up Front
Traditional Approach
Test-first Approach

34. © 2018 QASymphony – Proprietary
What’s the Catch?
Test-First methodologies were coined “Test Driven Development”. Less technically
focused versions called Acceptance Test Driven Development (ATDD) and Behavior
Driven Development (BDD) also emerged:

35. © 2018 QASymphony – Proprietary
Capabilities Needed to Drive Test First
Adoption
Capability Why it’s Needed?
Structured Syntax Support for Automation Reuse
Structured Syntaxes like Gherkin allow for
automation to be reused, reducing effort and
maintenance
Support for Feature & Scenario Creation in ALM
Specifications replace acceptance criteria and need
to be visible and defined in Jira or other ALMs
Integration of tests into Version Control Systems
Determine which tests should fail and which should
not when using red-green—refactor process

36. © 2018 QASymphony – Proprietary
02Code
03Build
01Plan
04Test
PLAN
RELEASE

37. © 2018 QASymphony – Proprietary
Typical CI Workflow
Stop the line when tests fail
Feedback – test results, monitoring data, etc.
Code - Dev Build - Unit
Test
Test - Staging Prod
Commit
Code
Build and
Test
Exploratory
Automated
Test
Deploy to
Prod and
Test
Release
Automated
Trigger
Manual
Trigger
Plan -
Team
User
Stories

39. © 2018 QASymphony – Proprietary
Capabilities Needed to Integrate
Feedback Early
Capability Why it’s Needed?
Exploratory Testing Support
Quick feedback on new features created as
automated tests may be under development
CI/Build Tool Integration
Ingest results of build tests in real time, link to user
stories and defects in Jira, consolidate with ad hoc
automated and manual tests
Customizable Workflows and Pipeline Automation
Reduce manual work and process risk – repeatable
and real-time rules to connect activities in the
pipeline

40. © 2018 QASymphony – Proprietary
02Code
03Build
01Plan
04Test
PLAN
RELEASE

41. © 2018 QASymphony – Proprietary
Continuous Integration Has Limits
PLAN
RELEASE
Unit
API
Functional Automation
E2E / UI
Compatibility
Load Performance

42. © 2018 QASymphony – Proprietary
Building a Test Umbrella…
Usability Testing
E2E Testing
Compatibility Testing
Load Testing
Manual UI Testing
Functional Automation
Enterprise
Com
m
ercial + Open
Source
Open
Source Only
Com
m
ercial Only
Customer
Facing
Less
Technica
l
Internal
Facing
More
Technical
Source: QASymphony – 2018 Evolution of Test Automation
Other / None

43. © 2018 QASymphony – Proprietary
…Supported by Automation Freeway
Source: Bryan Osterkamp – What to do When Everyone Wants a Different Automation Tool
•
•
•
•

44. © 2018 QASymphony – Proprietary
Wrap It Up: Quality Tactics for Success
Use software version control practices
and tools for change control
Industry Tactic Benefit Test Strategy
Facilitate better collaboration between
product, dev, test, operations in DevOps
Test Automation Strategy matching
people, process, tools
Build test automation strategy into a
continuous testing process
Shift quality and test automation to
front of software development
Form Automation Freeway to Facilitate
Adoption and Standardization
Version tests with code to be
incorporated with CI builds
Incorporate open source automation
like Cucumber for shift left success
Integrate CI into testing process along
with scheduled functional automation
Integrate ALM systems of record for
continuous developer feedback
Brest of breed tools for automation
Catch software bugs and automate
tests earlier
Testing the quality of development
becomes integral part in DevOps
Quality becomes everyone’s job rather
than a siloed approach to development
Source code = single record of truth for
automated deployments for audit

45. © 2018 QASymphony – Proprietary
Additional Webinars | QASymphony.com
•
•
•
•

46. © 2018 QASymphony – Proprietary
Thank You | Q&A
Ryan Yackel
Insert Pic HERE
Director of Product Dev