The document discusses building confidence in software release cycles, emphasizing the importance of software quality and deployment strategies to ensure successful releases. It outlines challenges in test automation and best practices, such as integrating automated testing into workflows to enhance efficiency and customer experience. The document also highlights key metrics for evaluating release performance and various rollout strategies to mitigate risks.

1. How to Build Confidence in
your Release Cycle
Discussion & Drinks Event | Thursday 4 May

2. Proprietary & Confidential
2
2
Ben Clare
Senior Consultant,
Software Engineer
DiUS
Meet our Speakers
Christof Sunthorn
Solutions Engineer
SmartBear

3. Proprietary & Confidential
3
Today’s Agenda
• Importance of release cycle confidence and
software quality
• Test automation challenges and best
practices
• Deployment and release strategies
• Panel discussion

4. Building Confidence Through
Software Quality

5. Proprietary & Confidential
5
5
What is Release Cycle
Confidence?
“The software release life cycle is the processof
developing, testing, and distributing a software
product.”
Software release life cycle– Wikipedia
“Release cycle confidence refers to the level of
certaintythat a software release will be
successful”
Agile Metrics : Agile Health Metricsfor Predictability– LeadingAgile
What does “success”look like for your team?
• You haven’t broken the application/service
• You haven’t introduced vulnerabilities
• You haven’t negatively impacted the business and
revenue
This PhotobyUnknown Authoris licensedunder CCBY-SA
This PhotobyUnknown Authoris licensedunder CCBY

6. Proprietary & Confidential
6
Why is Release Cycle Confidence Important?
• Visibility and information about your release cycle processes allows you proactively manage
issues in your release cycle for better outcomes.
• Focus on driving efficiency throughout the development cycle and product innovation
• Focus on providing the best customer experiences.
• Not being occupied with worrying and resolving problems with your release.
Visibility
for better outcomes
Focus
that drives efficiency
Confidence
to accelerate

7. Proprietary & Confidential
7
7
How Do We Achieve Release
Cycle Confidence?
The level of certainty that a software
release will be successful has two main
have contributing factors:
• Software Quality
▪ ISO 9001 standardsdefine as "capability
of a software product to conform to
requirements.“
• Deployment Strategy
This PhotobyUnknown Authoris licensedunder CCBY-SA
Software
Quality
Deployment
Strategy
Release Cycle
Confidence

8. Proprietary & Confidential
8
8
Why does Software Quality
matter?
Risks with low quality releases:
• Security vulnerabilities
▪ Public unauthenticatedaccess to data
▪ Malware
▪ Ransomware
▪ IP loss
• Fines and penalties
• Reputation loss
• Poor customer experience leading to
customer retention problems
• Loss of competitive advantage
• Revenue loss

9. Proprietary & Confidential
9
9
State of Software Quality
Report
• Survey conducted annually since 2017
• Report is designed to capture
benchmarks for software testing
professionals.
• It reaches across industries and
considers:
• The methodologies
• Practices
• Tools used to build, validate, and
deliver software
• It reveals emerging trends year after
year and provides insights into what
teams across the world think the future
will hold.

10. Proprietary & Confidential
10
How often do you release/deploy?
Organizations
continue to
release more
frequently

11. Proprietary & Confidential
11
The biggest challenge for test automation is the frequency the
application changes

12. Proprietary & Confidential
12
Lack of time is a challenge to achieving goals
Most challenging when scaling testinginitiatives

13. Proprietary & Confidential
13
Writing test case requirements is a cross-functional effort

14. Quality Through
Software Testing

15. Proprietary & Confidential
15
15
Software Testing (Manual)
Tester/QA Engineer manually performs a
set of actions (a test case) to verify/validate
a particular feature/functionality.
Purpose
▪ Find bugs, issues and defects in the
application.
▪ “See what the customer/end user sees”
▪ Manualtesting is still necessary
Limitations
▪ Requires human resources (time
intensive, error-prone and tedious)

16. Proprietary & Confidential
16
Why Test Automation Is Needed
This is supposed to happen fast!
Check-In Build Unit Tests
Integration
Tests
E2E, UAT,
Load Testing
Integrate
Release
Mgmt
Deploy to
Production
Check-In Build Unit Tests
Integration
Tests
E2E, UAT,
Load Testing
Integrate
Release
Mgmt
Deploy to
Production
Testing is slow!

17. Proprietary & Confidential
17
Automation Benefits
Faster DevelopmentCycles
Increase Test Coverage
Faster TimeTo Resolution
Reduce the time spent in the “testing” phase of development drastically.
Becomes a part of the workflow and not a bottleneck.
Build testing into your entire application to increase quality while still not
increasing test time when it comes time to deploy.
Get instant feedback from test results back to the development team for faster
bug fixes and reduced time triaging.

18. Proprietary & Confidential
18
What does it really mean?
• Visibility/traceability
• Less time dealing with bugs in
production
• Less worry about impact to
customerexperiences
• Can focus on development
ideas and innovation
• More time (more productive/
meaningful)
• Less stress
• Revenue for business

19. Test Automation
Solutions

20. Proprietary & Confidential
20
Triquestra | New Zealand
Testing across their 150+ products had become a bottleneck. To alleviate this, Triquestra
uses TestComplete to test their windows desktop applications, and web applications. Not
only have they shortenedtesting cycles from 3 days to 3 hours,but automated testing with
TestComplete mitigated the team’s dependency on individual knowledge, ensuring
consistenttesting across all features, every time.
“Manual regression testing for one product takes around
three days. And by automating it [with TestComplete],
we're able to test it in three hours. That's much bigger than
just the time savings. It's the savings on the sanity of a test
analyst and allows them to focus on more valuable tasks.”

21. Proprietary & Confidential
21
“The time it took for someone to learn how to write an automated test in
TestComplete was much lower than everything else we were looking at. It
wasn't just that it was good. It was that the entry-level for the technical
expertise was lower as well.”
“At the end of the day, if I'm delivering quality software and the customer
sees the effect, and they’re happy, that means they renew their support
subscriptions. You've got this direct tie between the results of making quality
software and increased revenue for the company.”
Aras | USA
Aras improved product quality and releasespeed using TestComplete to automate their
desktop application. Testingcycles decreased fromfivemonths to two weeks, and they
achieved 88% test automation. This drastically reduced the time it took to implement
customer requestsand add newfunctionality.

22. Proprietary & Confidential
22
Evaluating Solutions
• Unit tests are usually scripted by
developers/software engineers.
• Can be partially automatedwith AI
LLMs (Large LanguageModels) such
as ChatGPT, GitHubCopilot now
• Integration and E2E tests can be automated
through various frameworks and tools.
• Open source - scripting requires expertise
• COTS (Commercial off-the-shelf) - no/low
code allows contributionand participation
from non-technicalteams like business
units.
• Usefulness of AI still a developing story
and we need to consider the impact of
“fake confidence”
• Remote working and off-shore teams
means there will continue to be a
technological skills and knowledge gap.
• No/low code test automation tools give
users the confidence and understanding
to contribute to testing efforts

23. Proprietary & Confidential
23
23
Trends in the UI Test
Automation space
• Moving away from older frameworks
like Selenium
• Cypress and Playwright
• Visual Testing tools
• Leveraging AI LLM
• GitHubCopilot
• Cross-functional involvement
• Developers involvedwith QA team
• Non-technical business/admin users
needing to adapt

24. Best Practices

25. Proprietary & Confidential
25
• Isolate your tests from dependencies where possible.
How to Make UI Tests more Robust and Scalable
• Avoid any trivial UI tests that aren't important. Every test
should be either an important user workflow for the business
or a highly trafficked user workflow.
• Identify the most accurate object recognition approach for your
application.
• Improve maintainability by modularizing your test cases over
time.
• Mock backend servers to isolate front-end code and eliminate
any issues with unreliable API calls or other backend requests
that can throw errors.
Isolate
Maintain
Critical
Reuse
Recognition
Mock

26. Proprietary & Confidential
26
• Build tests that are easy to refactor by using don't-repeat-
yourself (DRY) principals, sensible variable names, and other
techniques.
• Choose a good testing framework.
• Include any edge cases and negative cases for each test to
ensure complete coverage and avoid unexpected errors
despite passing tests.
• Improve reliability by adding wait times for async processes,
data loading, or other cases where you're asserting a state
change.
Isolate
Maintain
Critical
Reuse
Recognition
Mock
How to Make UI Tests more Robust and Scalable

27. Proprietary & Confidential
27
Best Practices - Planning
• Starts with planning and mindset - can never automate100%.
• Ratio/split to aim for is dependent on your application and complexity of test cases.
• In general, 80 automated/20 manual is a good target.
• Don't go in blind.
• Ensure test cases are clearly defined - know what requirements need to be validated and what actions need
to be performed.
• (Manual) garbage in, (Automated) garbage out.
• Start with test cases that are mostly executed/repeatedto havethat quick win in the team.
• Don't expect the first script recordings to be perfect.
• It's normal to refine test case flows over time to be more efficient, accurate and modular.
• Avoid complex test cases until the team has built skills and confidence.
• Assistance from Dev teams can be helpful.

28. Proprietary & Confidential
28
Best Practices - Execution
• Prioritise platforms, devices and OS’s relevant to your user base.
• Leverage CI/CD Tools to automate your pipeline where possible.
• Ensure DevOps team is involved to support the test executions as part of CI/CD
• Havethe infrastructure and systems ready.
• Know the application, system and network limitationsearly so you can plan for it.

29. Proprietary & Confidential
29
29
How Do We Achieve Release
Cycle Confidence?
The level of certainty that a software
release will be successful has two main
contributing factors:
• Software Quality
▪ ISO 9001 standardsdefine as "capability
of a software product to conform to
requirements.“
• Deployment Strategy
This PhotobyUnknown Authoris licensedunder CCBY-SA
Software
Quality
Deployment
Strategy
Release Cycle
Confidence

30. How to Build Confidence in
your Release Cycle
Discussion & Drinks Event | Thursday 4 May

31. Releasing with Confidence

32. Proprietary & Confidential
32
32
But how do we release this?
• So, you've got a tested artifact
• It's probably safe to release?
• But we've missed things before?
How can we be sure?

33. Proprietary & Confidential
33
33
Well, how good are our releases?
• How often do they go wrong?
• What's the impact when it goes wrong?
• How fast can we recover?
What metrics could we measure?

34. 34
DORA Metrics
Metric Elite High Medium Low
Deployment Frequency
How often you deploy
On-Demand
(>1 per day)
1 per week
to 1 per month
1 per month
to 1 per 6 months
Longer than 1 per 6
months
LeadTime for Changes
Time fromcodecommit -> release
<1 hour 1 day
to 1 week
1 month
to 6 months
>6 months
Time to Restore
Time to go frombroken -> fixed
<1 hour <1 day 1 day to 1 week >1 week
Change Failure Rate
% of releases requiring remediation (rollback, hotfix, etc)
0%-15% 16%-30% 30%-60% >60%

35. Proprietary & Confidential
35
35
Break-Glass Pipeline
• Given standard CI/CD may take >1hr
• Common to have a "break glass" option
• Comes with its own problems
• What if we can't roll back?
• What if multiple services need to roll
back?
• Etc.
Most importantly: still takes time

36. 36
Blue-Green Deploys

37. Proprietary & Confidential
37
37
Blue-Green Deploys
• Deploy can be slow, as its not tied to
release
• Rollback can be near instant
However, still swapping 100% of customers
to new version
Can we limit the risk further?

38. Proprietary & Confidential
38
38
Internal-First Rollout
• Why not test Green, before we show it
to customers?
• Why not release Green to internal users
for a few hours?
• See if they spot any issues, before we
roll it to prod?
We now have control of our rollout!
Can rollback before issues are in the wild!

39. Proprietary & Confidential
39
39
Percentage Based Rollout
• Rollout the new version to a percentage
of customers
• Gradually increase the percentage, as
your confidence increases
• Can be done by
• Traffic
• Login sessions
• Device
• Anything you can differentiateon

40. 40
Region Based Rollout
Kangarooicons createdbyIcongeek26 - Flaticon
Kiwi fruit icons created byantonia_again – icons-icons

41. Proprietary & Confidential
41
41
Phased Rollout
• Combo of a few techniques
• Extremely mature, extremely rare
• Typically, only used by massive multi-
national companies
• Facebook
• Google
• Netflix
• Tailored to the business

42. 42
Pick what you need
• If you’resmall
• Simple release strategy may be
fine!
• Easiest to manage by far
• Growing? Wanta bit more safety?
• Consider break-glass pipelines!
• Maturing your change management?
• Consider blue-green!

43. Panel Discussion
Discussion & Drinks Event

44. Proprietary & Confidential
44
44
Ben Clare
Snr Consultant,
Software Engineer
DiUS
Meet our Panelists
Christof Sunthorn
Solutions Engineer
SmartBear
Ilia Mogilevsky
Engineering Manager
SmartBear
James Wolstenholme
Tech Lead
Independent Consultant
cumulus lab

45. We'd love to know
what you thought of
today's event
https://www.surveymonkey.com/r
/smartbeardiusevent
Thank you for coming!