Foundations of Software Testing Lecture 4

1 Build Software to Test Software exactpro.com
Lecture 4
Foundations of Software Testing
Software Testing for
Complex Intelligent Systems
and Autonomous Vehicles
Online Course

Lecture structure
Sources of information about testing
What is testing?
What is quality, errors, defects, failures?
What testing is not: QA, unit testing, BDD, bad testing
Principles and types of testing
Testing objectives; what constitutes good testing
Testing Components: processes, platforms, people
The testing process: phases and results
Is it the testers' fault that failures occur in production

Exactpro
The company was founded on
Tester Day, September 9,
2009
The company’s
headquarters are in
London
Moscow Kostroma Saratov Obninsk
Tbilisi London New York

What We Do
Our company specializes in development and implementation of high
load market infrastructures as well as systems that work with ﬁnancial
tools/platforms. Our solutions are used in more than 20 countries all
over the world.

Sources of information
- Absence-of-errors fallacy
and other principles of
software testing.
- Types and levels of
software testing.
- Understanding of test
objectives and
methodologies in
different schools of
thought in the software
testing domain.

Sources of Information
https://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/

https://youtu.be/jZitXMQaXvE
Watch
Yegor Bugayenko

https://ingophilipp.medium.com/excellent-software-testing-6e0fd8f3e11e

http://cartoontester.blogspot.com/
Andy Glover

Software Testing: Deﬁnition
● Testing is about extracting information
about software quality
● Testing is pain
● Testing is relentless learning
● Testing is about processes, platforms,
people

West World
TEST WORLD: REBOOT

Ray Dalio
PAIN + REFLECTION = PROGRESS

Ray Dalio

How We Get Information

https://blogs.scientificamerican.com/observations/the-line-between-science-and-journali
sm-is-getting-blurry-again/
● Journalism is communication of 'what's new'. A journalist is
anyone who can say "I’m there, you’re not, let me tell you about
it."
● Science is communication of 'how the world works'. A scientist is
anyone who can say "I understand something about the world,
you don't, let me explain it to you".
● Neither deﬁnition necessitates that what they say is True, just
what they know to the best of their ability and understanding.

Quality
Quality is value to some people
Quality is the degree to which a component or system satisfies
the stated and implied needs of its various stakeholders.
Quality is inherently subjective.
Different stakeholders will perceive the same product as
having different levels of quality.

What is a Bug?
Bug is a discrepancy between expectations
and reality
An attribute of a software product that reduces its value to a
stakeholder or increases its value to a competitor.
May or may not be a coding error or a functional error
Design errors are bugs too.

What is a Bug?
Error – a human action that produces an
incorrect result
Defect (bug) – an imperfection or deficiency
in a work product where it does not meet its
requirements or specifications
Failure – an event in which a component or
system does not perform a required function
within specified limits

What is a Bug?
65,586 m/ h (~65.6 km/h)
error defect/bug failure

What is a Bug?
Breaking path

What is a Bug?
Breaking path
https://cordis.europa.eu/arti
cle/id/19509-ariane-5-explo
sion-caused-by-fault-in-mai
n-engine-cooling-system

https://www.snopes.com/fact-check/cone-of-silence
PASS
FAIL
PASS
+ =
=
=
+
+
What is a Bug?

What is Software Testing?
Software Testing is an
● empirical
● technical
● investigation
● conducted to provide stakeholders
● with information
● about the quality
● of the product or service under test
Software Testing is exploring software
with the intent of ﬁnding bugs

Watch
“Software is Eating the World.”

“Software is Eating the World.”
http://www3.weforum.org/docs/WEF_
The_Global_Risks_Report_2021.pdf

Software Testing is not Quality Assurance
Testers:
Get Out of the Quality Assurance Business
Quality
Assurance
Quality Control
Testing
Code
review
Unit
testing
Audit
Process
Improvements
Change
management
Early reviews
and testing

Software Testing is an Information Service

Testing is not Quality Assurance
Testers:
Get Out of the
Quality Assurance
Business
https://www.developsense.com/presentations/2010-06-TestersGetO
utOfTheQABusiness.pdf

Unit Testing
Development of unit tests allows you
to isolate individual parts of the
program and test their performance -
creating tests, executing them and
obtaining reports using open source
frameworks Pytest and PyUnit.

BDD
Studying the Behavior Driven
Development (BDD) software
development methodology
Using the Gherkin in the requirements
deﬁnition process, use cases, and
acceptance testing

Software Testing Not Software Testing

Examples of Obtaining Information

Examples of obtaining and disseminating
information

Bad Testing
https://youtu.be/2S--Ge3c-yA

Software Testing is not...
https://mrslavchev.com/2016/03/28/software-testing-not/

Testing Principles
Principle 1: Testing shows the presence of defects, not their absence
Principle 2: Exhaustive testing is impossible
Principle 3: Early testing saves time and money
Principle 4: Defects cluster together
Principle 5: Beware of the pesticide paradox
Principle 6: Testing is context dependent
Principle 7: Absence-of-errors is a fallacy

Testing shows the presence of defects,
not their absence

Exhaustive testing is impossible

Autonomous Vehicle Testing

LGSVL Road Simulator
- Introduction to the multi-agent
road simulator developed by LG's
Research and Development
division based on the
cross-platform environment for
creating video games - Unity.
- Overview of installation, launch,
and user interface.

Pareto Principle

Beware of the Pesticide Paradox

Context Driven Testing
Consider:
● Software development lifecycle model and project
methodologies being used
● Test levels and test types being considered
● Product and project risks
● Business domain
● Operational constraints, including but not limited to:
a. Budgets and resources
b. Timescales
c. Complexity
d. Contractual and regulatory requirements
● Organizational policies and practices
● Required internal and external standards

Context Driven Testing
The Seven Basic Principles of the
Context-Driven School
1. The value of any practice depends on its context.
2. There are good practices in context, but there are no best practices.
3. People, working together, are the most important part of any project’s
context.
4. Projects unfold over time in ways that are often not predictable.
5. The product is a solution. If the problem isn’t solved, the product doesn’t
work.
6. Good software testing is a challenging intellectual process.
7. Only through judgment and skill, exercised cooperatively throughout the
entire project, are we able to do the right things at the right times to
eﬀectively test our products.
https://context-driven-testing.com/

Absence-of-errors
is a fallacy

Verification vs. Validation
Verification – checks whether the system meets the
specified requirements
Validation – checks whether the system will meet user and
other stakeholder needs
● https://techcrunch.com/2018/06/22/uber-safety-driver-of-fatal-self-driving-crash-was-watchin
g-hulu-not-the-road
● https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf

https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf

Types of Software Testing
● Functional testing — WHAT
● Non-functional testing — HOW
● Testing with access to the source code —
White Box

Software Testing at
the conﬂuence
Functional Testing Nonfunctional Testing

Black Box Testing vs. White Box Testing

Testing Components

● Test planning – involves deﬁning the objectives of testing and the
approach for meeting those objectives within project constraints
and contexts
● Test monitoring and control – compares actual progress against
the plan, takes actions which are necessary to meet the mission
and objectives of the project, and/or adjust the plan
● Test analysis – determines ‘what to test’
● Test design – addresses the question ‘how to test’
● Test implementation – asks ‘do we now have everything in place
to run the tests?
● Test execution – runs tests
● Test completion – summarizes the results of testing
Testing Components

Test Objectives
1. To prevent defects by evaluating work products such as requirements,
user stories, design, and code
2. To verify whether all specified requirements have been fulfilled
3. To check whether the test object is complete and validate if it works as
the users and other stakeholders expect
4. To build confidence in the level of quality of the test object
5. To find defects and failures thus reduce the level of risk of inadequate
software quality
6. To provide sufficient information to stakeholders to allow them to
make informed decisions, especially regarding the level of quality of
the test object
7. To comply with contractual, legal, or regulatory requirements or
standards, and/or to verify the test object’s compliance with such
requirements or standards

Bad Testing
Doctor Fauci

Testers vs. Activists

Test Objectives
● To prevent defects by evaluate work products such as requirements,
user stories, design, and code
● To verify whether all specified requirements have been fulfilled
● To check whether the test object is complete and validate if it works as
the users and other stakeholders expect
● To build confidence in the level of quality of the test object
● To find defects and failures thus reduce the level of risk of inadequate
software quality
● To provide sufficient information to stakeholders to allow them to
make informed decisions, especially regarding the level of quality of
the test object
● To comply with contractual, legal, or regulatory requirements or
standards, and/or to verify the test object’s compliance with such
requirements or standards

2021 IEEE Autonomous Driving AI Test Challenge
http://av-test-challenge.org/

Good Testing

Test Objectives

Testing is not Quality Assurance
Testers:
Get Out of the Quality Assurance Business
Quality
Assurance
Quality Control
Testing
Code
review
Unit
testing
Audit
Process
Improvements
Change
management
Early reviews
and testing

Good Testing

Good Testing
● Ownership – taking care of your company. Treat others the way you
would like to be treated. Take responsibility for company's needs.
Remember: responsibility is power. It is not granted. It is taken. Extreme
ownership is the way to lead and win.
● Deep Impact – exitus acta probat. The outcome justifies the effort. Make
sure that your work results in a valuable contribution. If you want to go
fast, go alone. If you want to go far, go together. Involve others to achieve
scalability and increase the impact of our work.
● Hard Work – no pain, no gain. Sometimes putting in long hours does not
help achieve any result. We do not measure time spent in the office.
Having said that, we've never observed extraordinary results without hard
work and deliberate effort.

● Learning – software testing is relentless learning. Change is the end
result of all true learning. The beautiful thing about learning is that
nobody can take it away from you. The Mistakes and problems are
inevitable. However, it is unacceptable not to learn from them. The
best way to learn something is to teach it.
● Agility – the only constant is change. We embrace the values
reﬂected in the Manifesto for Agile Software Development. Agility
allows us to turn learning into action and action into change. From its
inception, Exactpro has been able to adapt to change. It is important
to keep this ability as we scale up our business.
● Communication – the medium is the message. A relationship is
based on communication. We need to make sure others understand
us and we understand them.
● Objectivity – you have nothing to fear from knowing the truth. It is at
the core of our business. Our main service is to build software to test
software and provide objective information about it to our clients.
Our business beneﬁts from openness. Everyone in our company has
the right to understand what makes sense, and no one has the right
to hold a critical opinion without speaking up.
Good Testing

Thank you!
Join our Telegram chat, meet
your peers, ask questions
and get new software testing
skills with Exactpro!
Subscribe to our YouTube
channel to see all the new
videos!

Foundations of Software Testing Lecture 4

Recommended

Recommended

More Related Content

Similar to Foundations of Software Testing Lecture 4

Similar to Foundations of Software Testing Lecture 4 (20)

More from Iosif Itkin

More from Iosif Itkin (20)

Recently uploaded

Recently uploaded (20)

Foundations of Software Testing Lecture 4