This document provides an overview of software testing. It begins by introducing the topic and noting the large economic costs of software failures. The document then discusses what software is, why we test software, and defines what software testing is. It outlines the importance of software testing and some basic concepts like the differences between errors, faults and failures. The document also discusses test design methodologies, levels of testing from unit to system, and common challenges in software testing. The overall purpose is to provide an introduction to software testing concepts and practices.

1. Software Testing
Future and Challenges
Presented By
Bakr Salim Ba-Quttayyan
Master of Software Engineering (UTM)
PhD. Candidate (UUM) – Software Quality

2. About this Symposium
Bakr
Salim
29/5/2021 2
• Who should attend this Talk?
• This international talk is specifically relevant to academicians and non-
academicians who are interested to learn basics and importance of
software testing as an integral part of the software development life
cycle (SDLC).
• Learning Outcomes:
• Upon completion of this talk, participants will be able to understand
the basic software testing concepts and differentiate between various
kinds, process, techniques, methodologies, and activities of testing.
• They also will be aware of importance of testing software intended to be
released to the real world as well as the challenges facing applying
testing in practice.

3. Agenda
Bakr
Salim
29/5/2021 3
• Introduction
• What is software?
• Why do we test software?
• What is software testing?
• Importance of software testing
• Basic Concepts
• Test Design Methodologies
• Future and Challenges

4. Introduction
Bakr
Salim
29/5/2021 4
• Krasner (2021) reported that the United States has been lost
nearly $2.08 trillion due to the poor quality of software in 2020.
• Computerworld (2020) identified the Top software failures in
recent history (around 33 issues), for example:
• February 2020: Heathrow Airport (London) Disruption:

5. Introduction ..
Bakr
Salim
29/5/2021 5
• Top software failures in recent history:
• July 2019: Social media (Facebook, Instagram and
WhatsApp) outage:
• April 2018: TSP bank outage:
Millions of TSB customers were locked out of their
accounts after an IT upgrade led to an online
banking outage.
And others are clear examples of the negative
impact of software errors in the public life.

6. CHAOS 2020: Beyond Infinity – The
Standish Group Report
Bakr
Salim
29/5/2021 6

7. What is Software?
Bakr
Salim
29/5/2021 7

8. Why Do We Test Software?
Bakr
Salim
29/5/2021 8

9. What is Software Testing?
Bakr
Salim
29/5/2021 9
• Testing is the process of executing a program with the intent of finding errors.
Myers’s definition – 1979
• Software testing is a process that detects important bugs with the objective of having
better quality software.
Chauhan’s definition – 2010
• The process consisting of all lifecycle activities, both static and dynamic, concerned
with planning, preparation and evaluation of a component or system and related work
products to determine that they satisfy specified requirements, to demonstrate that
they are fit for purpose and to detect defects.
ISTQB definition – 2018

10. Importance of Software Testing
Bakr
Salim
29/5/2021 10
• To gain customer confidence.
• To check software adoptability.
• To identify errors/issues.
• To avoid extra costs.
• To accelerate software development.
• To avoid risks.
• To optimize business.

11. Basic Concepts
1. Error, Fault, and Failure
2. Reasons for Errors Occurring
3. Cost of Finding and Removing Faults
4. A Test Case: the Smallest Item
5. Software Testing Activities
6. Testing Levels
7. Testing Types

12. Error, Fault, and Failure
Bakr
Salim
29/5/2021 12
• Error
• A human mistake in any phase of SDLC.
• Usually, static.
• Also, called: mistake.
• Fault
• A condition that causes a system to produce failure.
• A fault is the result of an error.
• Not all faults/defects lead to a Failure.
• Also, called: defect or bug.
• Failure
• Incorrect behavior of the system caused by fault.
• Describes the problems in a system on the output side.

13. Error, Fault, and Failure ..
Bakr
Salim
29/5/2021 13

14. Error, Fault, and Failure ..
Bakr
Salim
29/5/2021 14
• Software failures, then, can lead to:
• Loss of money.
• Loss of time.
• Loss of data and information.
• Loss of business reputation.
• Loss of stakeholders (e.g., investors, customers, employees, etc.).
• Loss of competition.
• Breach of privacy.
• Human Injury.
• Damage.
• Death.

15. Reasons for Errors Occurring
Bakr
Salim
29/5/2021 15
• Errors may occur for many reasons, such as:
• Time pressure.
• Human fallibility.
• Inexperienced or insufficiently skilled project participants.
• Miscommunication between project participants, including
miscommunication about requirements and design.
• Complexity of the code, design, architecture, the underlying problem to
be solved, and/or the technologies used.
• Misunderstandings about intra-system and inter-system interfaces,
especially when such intra-system and inter-system interactions are
large in number.
• New, unfamiliar technologies.

16. Four Typical Scenarios of Faults Occurring
in SDLC
Bakr
Salim
29/5/2021 16

17. Cost of Finding and Removing Faults
Bakr
Salim
29/5/2021 17

18. A Test Case: the Smallest Item
Bakr
Salim
29/5/2021 18
• Test condition
• An item or event of a component or system that can be verified by one
or more test cases, such as, for example, a function, transaction,
feature, quality attribute, or structural element.
Test Case – is a set of preconditions, inputs, actions (where
applicable), expected results and postconditions, developed
based on test conditions.
The primary purpose of designing a test case is to find
errors in the system.

19. A Test Case: the Smallest Item ..
Bakr
Salim
29/5/2021 19
• Every test case has:
• Test case ID
• The identification number given to each test case.
• Purpose / Description
• Defines why the test case is being designed.
• Preconditions (optional)
• Environmental and state conditions that must be fulfilled before the component or system can be executed with a
particular test or test procedure.
• Inputs
• Specific inputs must be provided, instead of general inputs.
• For example, if two integer numbers have to be provided as input, then specifically mention them as 23 and 56.
• Actions (procedures)
• Sequence of steps for the execution of a test case.
• Expected results
• The predicted or expected output or behavior of a system or its components, as defined by the specification or
another source (for every test case).
• Postconditions
• Environmental and state conditions that must be fulfilled after the execution of a test or test procedure.

20. Example: Log in (Sign in)
Bakr
Salim
29/5/2021 20
For this system, use Ahmed
as username and 123 as
password.

21. Test Cases: An Example
Bakr
Salim
29/5/2021 21
ID
Purpose /
Description
Preconditions Inputs Actions (procedures)
Expected
results
Postconditions
011 Checking sign in
with true values
System must be
launched
UN: Ahmed
PW: 123
Step 1: enter your username.
Step 2: enter your password.
Step 3: press “Sign in” button.
Sign in
successful
The main window
appears.
012 Checking sign in
with false values
System must be
launched
UN: Ali
PW: 456
Step 1: enter your username.
Step 2: enter your password.
Step 3: press “Sign in” button.
Sign in fail Error message appears.
013 Checking sign in
with false second
value
System must be
launched
UN: Ahmed
PW: 456
Step 1: enter your username.
Step 2: enter your password.
Step 3: press “Sign in” button.
Sign in fail Error message appears.
014 Checking sign in
with false first
value
System must be
launched
UN: Ali
PW: 123
Step 1: enter your username.
Step 2: enter your password.
Step 3: press “Sign in” button.
Sign in fail Error message appears.

22. A Test Case: Template
Bakr
Salim
29/5/2021 22

23. A Test Case: Template 2
Bakr
Salim
29/5/2021 23

24. A Test Case: Template 3
Bakr
Salim
29/5/2021 24

25. Software Testing Activities
Bakr
Salim
29/5/2021 25

26. Testing Levels
Bakr
Salim
29/5/2021 26
• Test levels: are groups of test activities that are organized and
managed together.
• The testing levels are:
• Component/Unit testing.
• Integration testing.
• System testing.
• Acceptance testing.

27. Test Levels: Component/Unit Testing
Bakr
Salim
29/5/2021 27
• Focuses on components that are separately testable.
• Design tests based on work products (artifacts):
• Detailed design.
• Code.
• Data model.
• Component specification (if available).
• Objects to be tested:
• Components, units, or modules.
• Code and data structures.
• Classes.
• Database models.
• Typical defects and failures:
• Incorrect functionality.
• Data flow problems.
• Incorrect logic.
Tools:
• Unit test frameworks.
• Debugging tools (e.g., compilers).
Responsibility:
• Developer.
Benefits:
• Fixing errors.
• Root cause analysis.
• Code refactoring.

28. Test Levels: Integration Testing
Bakr
Salim
29/5/2021 28
• Integration testing focuses on interactions between components or
systems.
• Two types:
• Component integration testing
• Focuses on the interactions and interfaces between integrated components.
• System integration testing
• Focuses on the interactions and interfaces between systems, packages, and microservices.
• Design tests based on work products (artifacts):
• Software and system design.
• Sequence diagrams.
• Interface and communication protocol specifications.
• Use cases.
• Architecture at component or system level.
• Workflows.
• External interface definitions.

29. Test Levels: Integration Testing ..
Bakr
Salim
29/5/2021 29
• Objects to be tested:
• Subsystems.
• Databases.
• Infrastructure.
• Interfaces.
• APIs (Application Programming Interfaces).
• Microservices.
• Typical defects and failures:
• Incorrect data, missing data, or incorrect data encoding.
• Incorrect sequencing or timing of interface calls.
• Interface mismatch.
• Failures in communication between components.
• Unhandled or improperly handled communication failures between components.
• Incorrect assumptions about the meaning, units, or boundaries of the data being
passed between components.
Tools:
• Risk analysis.
Responsibility:
• Developers.
• Testers.
Benefits:
• Fixing errors.
• Reducing risk.
• System integration.

30. Test Levels: System Testing
Bakr
Salim
29/5/2021 30
• System testing focuses on the behavior and capabilities of a whole
system or product (e.g., end-to-end tasks and the non-functional
behaviors).
• Design tests based on work products (artifacts):
• System and software requirement specifications (functional and non-
functional).
• Risk analysis reports.
• Use cases.
• Epics and user stories.
• Models of system behavior.
• State diagrams.
• System and user manuals.
Tools:
• Specifications.
• Modeling.
Responsibility:
• Independent testers.
• Test team.
• Third-party team.
Benefits:
• Fixing errors.
• Functionality confidence.

31. Test Levels: System Testing ..
Bakr
Salim
29/5/2021 31
• Objects to be tested:
• Applications.
• Hardware/software systems.
• Operating systems.
• System under test (SUT).
• System configuration and configuration data.
• Typical defects and failures:
• Incorrect calculations.
• Incorrect or unexpected system functional or non-functional behavior.
• Incorrect control and/or data flows within the system.
• Failure to properly and completely carry out end-to-end functional tasks.
• Failure of the system to work properly in the system environment(s).
• Failure of the system to work as described in system and user manuals.

32. Test Levels: Acceptance Testing
Bakr
Salim
29/5/2021 32
• Acceptance testing focuses on the behavior and capabilities of a whole system or
product.
• Four common types:
• User acceptance testing
• Focused on validating the fitness for use of the system by intended users in a real or simulated operational
environment.
• Operational acceptance testing.
• The acceptance testing of the system by operations or systems administration staff is usually
performed in a (simulated) production environment.
• Focuses on operational aspects:
• Backup and restore.
• Installing, uninstalling and upgrading.
• Disaster recovery.
• User management.
• Maintenance tasks.
• Data load and migration tasks.
• Checks for security vulnerabilities.
• Performance testing.

33. Test Levels: Acceptance Testing ..
Bakr
Salim
29/5/2021 33
• Contractual and regulatory acceptance testing
• Performed against a contract’s acceptance criteria for producing custom-developed
software.
• To build confidence that contractual or regulatory compliance has been achieved.
• Often performed by users or by independent testers.
• Alpha and beta testing
• Alpha and beta testing are typically used by developers of COTS software who want to
get feedback from potential or existing users, customers, and/or operators before the
software product is put on the market.
• Alpha Testing – is performed at the developing organization’s site, not by the
development team, but by potential or existing customers, and/or operators or an
independent test team.
• Beta Testing – is performed by potential or existing customers, and/or operators at
their own locations.
• Beta testing may come after alpha testing, or may occur without any preceding alpha testing
having occurred.

34. Test Levels: Acceptance Testing ..
Bakr
Salim
29/5/2021 34
• Alpha and Beta testing
• Design tests based on work products (artifacts):
• Business processes.
• User or business requirements.
• Regulations, legal contracts and standards.
• Use cases and/or user stories.
• System requirements.
• System or user documentation.
• Installation procedures.
• Risk analysis reports.
• Objects to be tested:
• System under test.
• System configuration and configuration data.
• Business processes for a fully integrated system.
• Operational and maintenance processes.
• Forms.
• Reports.
• Existing and converted production data.
• Recovery systems and hot sites (for business continuity and disaster recovery testing).
Tools:
• Contract’s.
• Manuals.
Responsibility:
• Customers.
• Business users.
• Product owners.
• Operators of a system.
• Other stakeholders.
Benefits:
• Fixing errors.
• Accepting the system.
• System release.

35. Test Levels: Acceptance Testing ..
Bakr
Salim
29/5/2021 35
• Alpha and Beta testing
• Typical defects and failures:
• System workflows do not meet business or user requirements.
• Business rules are not implemented correctly.
• System does not satisfy contractual or regulatory requirements.
• Non-functional failures such as security vulnerabilities, inadequate performance
efficiency under high loads, or improper operation on a supported platform.

36. Testing Types
Bakr
Salim
29/5/2021 36

37. Test Types ..
Bakr
Salim
29/5/2021 37

38. Test Design Methodologies
1. Test Design Methods
2. Black-box Testing (Specification-based)
3. White-box Testing (Structural-based)
4. Experience-based Testing
5. How to Perform Testing?
6. Software Testing Tools
7. Documenting the Testing Process

39. Test Design Methods
Bakr
Salim
29/5/2021 39
• Test conditions, test cases, and test data can be designed based
on three methods:
• Black-box.
• White-box.
• Experience-based.

40. Black-box Testing (Specification-based)
Bakr
Salim
29/5/2021 40
• Black-box testing
• Also known as: behavioral testing, specification-based testing.
• Black-box test techniques concentrate on the inputs and outputs of the test object without reference
to its internal structure.
• Tests derived based on system behavior, which include:
• Requirements.
• Specifications.
• use cases, and
• User stories.
• Applicable to both functional and non-functional testing.
• Black-box techniques
• Boundary Value Analysis (BVA).
• Equivalence Class Testing.
• State Table-Based Testing.
• Decision Table-Based Testing.
• Cause-Effect Graphing Based Testing.
• Error Guessing.

41. White-box Testing (Structural-based)
Bakr
Salim
29/5/2021 41
• White-box testing
• Also known as: structural testing, code-based testing.
• Tests derived based on the system’s internal structure or implementation.
• Internal structure may include:
• Code.
• Architecture.
• Workflows, and/or
• data flows within the system.
• White-box techniques
• Basis path testing.
• Graph matrices.
• Loop testing.
• Data flow testing.
• Mutation testing.

42. Experience-based Testing
Bakr
Salim
29/5/2021 42
• Test derived based on knowledge and experience of:
• Testers.
• Developers.
• Users and
• Other stakeholders.
• This knowledge and experience includes:
• Expected use of the software.
• Its environment.
• Likely defects, and
• The distribution of those defects.
• Experience-based techniques:
• Error Guessing.
• Exploratory Testing.
• Checklist-based Testing.

43. How to Perform Testing?
Bakr
Salim
29/5/2021 43
Two ways
Test
conducted
in:
Test
execution
Manual
Automated
(Tools)

44. Manual Testing
Bakr
Salim
29/5/2021 44
• Performing test activities manually.
• In this process, the software testers execute the test cases and generate
the test reports without the help of any automation software testing
tools.
• Benefits
• Testing engineer has total control over executing every action.
• Manual testing does not require any tools.
• More effective for detecting visual and UX faults and.
• Drawbacks
• It is time-consuming testing task.
• Prone to human mistakes.

45. Test Automation
Bakr
Salim
29/5/2021 45
• The use of software to perform or support test activities is called test
automation.
• The process includes implementing tests into executable test scripts.
• Testers use appropriate automation tools to develop the test scripts and
validate the software.
• Benefits
• Reduction of testing effort – especially for huge amount of test cases and
repetitive testing.
• Reduces the testers’ involvement in executing tests – then testers have more
time for test planning and thinking up new tests.
• Facilitates regression testing – as a repetitive activity, regression testing is the
most time-consuming process.
• Avoids human mistakes - as human is fallible.
• Reduces overall cost of the software – by decreasing time and cost.

46. Test Automation ..
Bakr
Salim
29/5/2021 46
• Drawbacks
• There is no single tool that cover the whole testing process.
• Testing tools cannot replace the whole testing activity.
• Automated testing can not replace manual testing.
• Cost of tools implementation ad training.
• When to automate tests
• Not all test activities can be automated, only repetitive which are
frequently used.
• Design your tests first, before deciding which to automate.
• Design automated tests differently from manual tests.
• Select the tools according to organizational needs.

47. Software Testing Tools
Bakr
Salim
29/5/2021 47
• Test tools can be used to support one or more testing activities.
• Benefits of using tools:
• Reduction in repetitive manual work (saving time).
• Greater consistency and repeatability.
• More objective assessment (e.g., static measures, coverage).
• Easier access to information about testing (e.g., statistics and graphs about
test progress, defect rates and performance).
• Drawbacks of using tools:
• Expectations for the tool may be unrealistic (including functionality and ease
of use).
• The time, cost and effort for the initial introduction of a tool may be under-
estimated (including training and external expertise).
• There may be no clear ownership of the tool (e.g., for mentoring, updates,
etc.).

48. Famous Testing Tools
Bakr
Salim
29/5/2021 48
• Test automation
• Kobiton – functional testing.
• Selenium – web applications.
• Katalon Studio – API, web and
mobile testing.
• Unified Functional Testing
(formerly QTP).
• TestComplete – GUI testing.
• IBM Rational Functional Tester.
• Test management
• TestRail.
• Testpad.
• SpiraTest.
• TestMonitor.
• Defect management
• JIRA.
• Mantishub.
• FogBugz.
• Bugzilla.
• Mobile Testing
• Appium.
• Espresso.
• Perfecto.

49. Documenting the Testing Process
Bakr
Salim
29/5/2021 49
• Test documentation
• is the documentation of artifacts created before or during the testing
process of the software.
Test
Documentation
Standards
MIL-STD-
498
IEEE 859

50. MIL-STD-498 (formerly known as: DOD-
STD-2167A)
Bakr
Salim
29/5/2021 50
• MIL-STD-498
• is a United States military standard whose purpose was to "establish uniform
requirements for software development and documentation.“
• It provides templates and guidelines for developing most of software
development documents through different stages of SDLC.
• Testing documentations:
• Software Test Plan (STP)
• A plan for conducting qualification testing.
• Software Test Description (STD)
• Test cases/procedures for qualification testing.
• Software Test Report (STR)
• Test results of qualification testing.
• For more information, visit:
• https://en.wikipedia.org/wiki/MIL-STD-498.

51. IEEE 829 – Software and System Test
Documentation standard
Bakr
Salim
29/5/2021 51
• Also, known as: IEEE 829-2008.
• is a standard for software testing that specifies all the stages of software
testing and documentation at each stage.
• The stages involved in software testing and reporting are:
• Test plan (TP).
• Test design specification (TDS).
• Test case specification (TCS).
• Test procedure specification (TPS).
• Test item transmittal report (TITR).
• Test log (TL).
• Test incident report (TIR).
• Test summary report (TSR).

52. Software Testing Roles and Responsibilities
Bakr
Salim
29/5/2021 52
Role Responsibilities
Test Lead/ Test Manager • Defining the test strategy (including testing activities and documentations).
• Planning the testing process.
• Monitoring and controlling the testing process and team members.
• Reporting to Project Manager (and top management).
Test Engineer / Test Analyst • Identifying test conditions and features.
• Designing high-level test cases.
• Developing scripts to run automated tests.
• Developing test scenarios and documentation.
Senior Tester • Preparing test cases (test case description).
• Reviewing test cases with test lead.
• Coordinating with junior testers for defect tracking.
Junior Tester • Test execution – execute test cases.
• Reporting defects.
• Reporting test status to senior tester and test Lead.

53. Different between Testing and Debugging
Bakr
Salim
29/5/2021 53
Testing Debugging
• Finding and locating defects. • Fixing that defect.
• Performed by testing team. • Performed by development team.
• Purpose is to find many defects as
possible.
• Purpose is to remove the detected defects.
• Can be done manually or automatically. • Done only manually.
• Can be done in the early stages of the
software development.
• Can begin only after the program is coded.

54. Future and Challenges
The Future of Software Testing
Challenges Facing Software Testing

55. The Future of Software Testing
Bakr
Salim
29/5/2021 55
• Role of test automation to accelerate digital transformation
• Tools like Selenium, Katalon, and TestComplete will evolve with new features to ease up automation.
• Advanced tools and practices are expected to evolve in the coming years to make test automation
utilization in business projects better and smarter.
• Security and cybersecurity will become a priority
• QA teams or cybersecurity testing will uncover the weaknesses within the applications and security
will be the top priority in the coming years.
• Power of automation and agile teams in testing
• Agile testing stresses continuous testing, continuous feedback, resolving bugs in real time, and to
lessen the redundant tasks in test cases. Ultimately it aims to reduce the time to market.
• Artificial intelligence and machine learning for smarter testing
• In 2021 and beyond, AI and ML applications will address issues related to quality prediction, test
case generation, prioritization of test cases, fault classification, and assignment to optimize testing
processes, and so on.
• Mobile testing inceasing
• According to the Kobiton report, about seventy-nine percent (79%) of enterprises would invest more
in mobile app testing in 2021.

56. Challenges Facing Software Testing
Bakr
Salim
29/5/2021 56
• Lack of Communication
• Testers often tend to work in complete isolation, and very rarely indulge in
communicating with other team members.
• Ensuring great communication across the team and associated stakeholders
means testers have access to all the information they need to start testing
quickly and efficiently.
• Undefined Quality Standards
• Software products tend to comply with quality requirements when the
standards are clearly defined. Unfortunately, perfectly defined quality
standards is not always the case which makes testing a real challenge.
• Testing the complete application
• Impossible!. There are millions of test combinations. It’s not possible to test
each and every combination both in the Manual as well as in Automation
Testing.

57. Challenges Facing Software Testing ..
Bakr
Salim
29/5/2021 57
• Relationship with Developers
• Requires very skilled tester to handle this relation positively and even
by completing the work in testers way.
• Testing always under time constraint
• Lack of Skilled Testers
• Automation Testing
• Till what level automation should be done?
• Test automation has not reached 100% in the industry.

58. References
Bakr
Salim
29/5/2021 58
• Jorgensen, P. C. (2014). Software Testing: A Craftsman’s
Approach. CRC press.
• Graham, D.; Black, R. and Veenendaal, E. (2020). Foundations
of Software Testing: ISTQB Certification. 4th Edition, Cengage
Learning EMEA.
• Chauhan, N. (2010). Software Testing: Principles and
Practices. Oxford university press.

59. Bakr
Salim
29/5/2021 59

60. Bakr
Salim
29/5/2021 60