Quality Engineering and Testing with TMAP in DevOps IT delivery

1
© 2021 Sogeti. All rights reserved.
Quality Engineering & Testing
with TMAP
in DevOps IT delivery
TMAP: the body of knowledge for
quality engineering in IT delivery
Rik Marselis
27 October 2021

2
Rik Marselis
Principal Quality Consultant
2007 2012 2012
2008 2014
2009
2018
1980 2020

3
Quality Engineering and testing in DevOps…

4
Challenges of today’s high-performance IT delivery
The business demands:
• Deliver business value
• Deliver quality at speed
The team challenges are:
• Quality engineering is everyone's responsibility
• QA & testing is integrated in process and people
The focus is:
• Organize high-performing cross-functional teams (you build it, you run it!)
• Automate everything (as long as it is useful)
Thus  implement the DevOps culture

5
Take BUSINESS VALUE as the starting point

6
Supply information
based on
measuring indicators
Our “VOICE model” for delivering business value
Measuring indicators
=
(mainly) testing
Figure 3.1 from the book Quality for DevOps teams

7
Measure indicators to supply information about quality and risks

8
Measure indicators to establish confidence in business value
Examples for the four groups of indicators
Business value related IT delivery related
Team related Problem related
Indicators
Quality
Product
People
Process

9
Waterfall V-model Scrum DevOps Demand-Supply SAFe®
Figure 7.1 from the book Quality for DevOps teams Focus book Focus Agile mindset

10
DevOps – highlights
The six DevOps principles:
(source: the DevOps handbook)
1. Customer-centric action
2. Create with the end in mind
3. End-to-end responsibility
4. Cross-functional autonomous teams
5. Continuous improvement
6. Automate everything you can
DevOps is a cross-functional systems engineering culture that aims at unifying
systems development (Dev) and systems operations (Ops) with the ability to create and
deliver fast, cheap, flexible and with adequate quality, whereby the team as a whole is
responsible for the quality. Other areas of expertise, such as business analysis and quality
assurance (including testing) are usually integrated in the team. A DevOps culture has an
Agile mindset that can be supported/implemented by e.g. the Scrum framework.
Fundamental DevOps activities

11
TEAM:
Together
Everyone
Achieves
More
Example of a cross-functional
high-performance IT delivery team
The starting point for DevOps:
A cross-functional team
Aligned with the DevOps culture of IT delivery – where roles are important and functions
are not – keep in mind that if in TMAP you see a term such as developer or tester, we
refer to the role of a team member at a specific moment in time, not to a function.

12
Quality Engineering is about team members and their stakeholders
taking joint responsibility to continuously deliver IT systems
with the right quality at the right moment
to the businesspeople and their customers.
It is a principle of software engineering concerned with
applying quality measures to assure built-in quality.

13
Implement quality engineering  part of high-performance IT delivery
High-performance IT delivery is an approach that enables
cross-functional teams
to continuously improve
products, processes and people
that are required to deliver value to the end users.
Quality
Product People
Process

14
To achieve “Quality @ Speed” you need a CI/CD pipeline (don’t you?)
CI = Continuous Integration CD = Continuous Delivery/Deployment

15
CI is team-focus (one build pipeline per team)
Figure 6.1 from the book
Team 1
Team 2
Team 3
CD is multiple team focus (one release pipeline per business solution)

16
Automate everything (as long as it is useful)
An automated pipeline enables: • Elimination of errors due to manual task execution
• Provisioning of standardized development feedback loops
• Enabling of fast product iterations
The spheres
of testing

17
Automate everything (as long as it is useful)
By continuous testing and continuous monitoring gather information on various
indicators from both test environment and live environment.
Mainly automated testing but
some manual exploration too

18
Quality Engineering & testing in DevOps…

19
Quality engineering topics
Two groups of topics:
• Organizing
Aimed at: orchestrating, arranging, planning,
preparing and controlling
• Performing
Aimed at the operational QA & testing activities.
Note: it is not black-and-white, some topics may be relevant
for both, but the emphasize is on one or the other.
These topics are always relevant for quality
engineering, regardless of the IT development,
operations and maintenance approach.
For effective and efficient IT delivery all of these
topics need to be addressed in one way or another.
Team scope
Multi-team scope

20
The QA & testing topics related to the DevOps activities
Organizing
topics
Performing
topics
DevOps
activities

21
Examples of quality measures that are described by TMAP:
• Orchestration
• Specification and Example
• Quality risk analysis & test strategy
• Test design approaches & techniques
• unit testing & mutation testing
• Test automation
• Feature toggles
• Monitoring
• and many many more…
Quality measures
Quality
measures for
every role
Download the Quality Engineering Strategy template with an overview of quality measures at:
www.tmap.net/page/download-templates

22
Specification and Example: guiding the team
To understand what "it" is that should be built and try to build "it" right the first time,
the team(s) can use Specification and Example mapping approaches.
These are collaborative approaches to define requirements and business-oriented functional tests
for software products, based on capturing and illustrating requirements using realistic examples
instead of abstract statements.
Some commonly used approaches are:
• Specification by Example (SbE)
• Example-driven development (EDD)
• Executable requirements
• Acceptance test-driven development (ATDD)
• Behavior-driven development (BDD)
• Agile acceptance testing
• Test-driven requirements
Keywords:
• Common understanding of stories/features
• Test-first
• Exploring ideas
BA

23
Quality Engineering & Testing support delivering business value
Quality Engineering
Testing
Focus:
“Detective”
quality measures
Focus:
“Preventive”
quality measures
Build quality in,
right first time
Provide information
about quality & risks

24
Definition of testing as part of quality engineering
Testing consists of verification, validation and exploration activities
that provide information about the quality and the related risks,
to establish the level of confidence
that a test object will be able to deliver the pursued business value.

25
Test approaches and techniques
Experience-based
testing
Error-guessing
Exploratory
testing
Crowd
testing
Checklists
Test

26
16 test design techniques defined in TMAP
https://tmap.net/page/test-design and https://tmap.net/building-blocks/coverage
Figures 45.2, 45.3, 45.4 and 45.5 from the book Quality for DevOps teams

27
Unit testing
Keep in mind that the statement “we have reached
100% code coverage” in itself doesn’t give useful
information.
The type of code coverage achieved is what matters
Suppose we have the following code:
IF A > 10 PRINT “YES” ENDIF
Code coverage demonstrates the percentage of program code that is
covered by tests. Different test design techniques guarantee less or
more coverage.
DEV

28
Mutation testing tests the test
Mutation testing is a type of testing where certain statements in the source code are changed
(mutated) to check if test cases will identify the fault that was introduced this way.
This is a manner to verify the quality of the test set (instead of the object under test).
Mutation testing usually is supported by tools.
Mutation testing focuses on conditional statements.
Suppose we have the following code and test cases:
IF A > 10 PRINT “YES” ENDIF
Testcase1: A := 10, expected result no print
Testcase2: A := 11, expected result “YES”
What mutation in this code would not make any of these tests fail?
IF A <> 10 PRINT “YES” ENDIF  Testcase3: A := 9, expected result no print
IF A = 11 PRINT “YES” ENDIF  Testcase4: A := 12, expected result “YES”

29
Examples of quality measures that are described in TMAP:
• Specification and Example (SaE)
• unit testing & mutation testing
• Test design techniques
• Feature toggles
• Monitoring
• and many many more…
Quality measures
OPS

30
Quality & Test policy  for example:
Risk-based development & testing
High quality risk – high test intensity
Medium quality risk – medium test intensity
Low quality risk – low test intensity
No quality risk – no testing & no development !!!

31
Conclusion: The change that quality engineering achieves
Testing = Messenger
of bad news
Quality engineering =
Enabler of an
adequate quality level
that delivers
Business Value
No more Instead

32
So this is why and how…
Want to know more?
Quality Engineering & testing in DevOps…

33
TMAP training courses and certification scheme of
TMAP: Quality
for cross-
functional teams
TMAP: High-
performance
quality
engineering
TMAP:
Organizing built-
in quality
at scale
Performing
people
3-day training course
exam of 30 questions
3-day training courses
exams of 40 questions
All people
involved in
cross-
functional
teams
Organizing
people
Exams provided by
Info about TMAP certification at:
www.TMAPcert.com
https://academy.sogeti.nl/tmap https://tmap.net/page/tmap-training-and-certification

34
Audience per training/certification
www.TMAPcert.com
https://academy.sogeti.nl/tmap
https://tmap.net/page/tmap-training-and-certification

35
www.TMAPcert.com

36
& Test managers,
orchestrators,
Projectleaders,
Release train engineers,
etc.
www.TMAPcert.com

37
Stay up-to-date with the latest TMAP developments
Follow TMAP on LinkedIn:
https://www.linkedin.com/company/2020-tmap-quality-
engineering-for-high-performance-it-delivery/
(use the link or just search
for TMAP on LinkedIn)

38
Questions & Answers

39
About Sogeti
Learn more about us at
www.sogeti.com
Sogeti is a leading provider of technology and engineering services. Sogeti delivers
solutions that enable digital transformation and offers cutting-edge expertise in Cloud,
Cybersecurity, Digital Manufacturing, Digital Assurance & Testing, and emerging
technologies. Sogeti combines agility and speed of implementation with strong technology
supplier partnerships, world class methodologies and its global delivery model,
Rightshore®. Sogeti brings together more than 25,000 professionals in 15 countries,
based in over 100 locations in Europe, USA and India. Sogeti is a wholly-owned subsidiary
of Capgemini SE, listed on the Paris Stock Exchange.
TMAP website:
www.tmap.net
Sogeti academy:
academy.sogeti.nl
TMAP:
body of knowledge for
quality engineering
Feel free to contact me on:
Rik.Marselis@Sogeti.com

Quality Engineering and Testing with TMAP in DevOps IT delivery

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Quality Engineering and Testing with TMAP in DevOps IT delivery

Similar to Quality Engineering and Testing with TMAP in DevOps IT delivery (20)

More from Rik Marselis

More from Rik Marselis (20)

Recently uploaded

Recently uploaded (20)

Quality Engineering and Testing with TMAP in DevOps IT delivery