An insights to the general best practice used in the giant companies with overview of basic concepts of software testing

1. How Companies Test their
Software before Released to the
Digital Market
Prepared by
Eng. Bakr Salim Ba-Quttayyan
Ph.D. Candidate (UUM)

2. Agenda
• Software Development Lifecycle
‒ What is Software Testing?
‒ Software Testing Goals
‒ Software Testing History
‒ Software Testing Principles
• Testing in the Industry
• Testing with The Big Giants
• Software Testing Approaches
• Defect Management
• Software Testing and Generative AI
21 Oct. 2023 2
Eng.
Bakr
Salim

3. “I am Human, and I make mistakes”
21 Oct. 2023 3
Eng.
Bakr
Salim
Fixing
Programmer
Mistake

4. Software Development Lifecycle
Traditional (Waterfall) Agile Methodologies
21 Oct. 2023 4
Eng.
Bakr
Salim

5. What is Software Testing?
• Testing
‒ is the process of executing a program with the intent of finding errors [Myers1979].
• People make errors. A good synonym is mistake.
‒ This may be a syntax error or misunderstanding of specifications.
‒ Sometimes, there are logical errors.
• When developers make mistakes while coding, we call these mistakes
“bugs”.
• A fault is the representation of an error,
‒ where representation is the mode of expression, such as narrative text, data flow
diagrams, ER diagrams, source code etc.
‒ Defect is a good synonym for fault.
• A failure occurs when a fault executes.
‒ A particular fault may cause different failures, depending on how it has been
exercised.
21 Oct. 2023 5
Eng.
Bakr
Salim

6. What is Software Testing? ..
21 Oct. 2023 6
Eng.
Bakr
Salim

7. What is Software Testing? ..
• Test Strategy
‒ The test strategy is an organizational-level document that describes the general test
approach i.e., what needs to be achieved and how to achieve it.
 This document does not specify testing requirements for a specific project.
 Instead, it establishes the common testing principles for all the organization's projects.
‒ It consists of a set of guidelines that determine test design and provide organized
documentation of testing techniques specific to a software product.
 A test strategy acts as a summary of essential information and helps developers, testers, and
stakeholders in maintaining consistent testing practices.
• Test Plan
‒ It is a comprehensive document that entails test strategies, estimation, deadline,
scope, objective, and resources to complete the testing process.
 It includes key components like test objectives, test environments, test cases (high-level), roles
and responsibilities, special requirements, deliverables, exist criteria, and schedules.
 The purpose of a test plan is to ensure systematic and effective testing, identify defects, and improve
software quality.
21 Oct. 2023 7
Eng.
Bakr
Salim

8. What is Software Testing? ..
• Test Data
‒ Test data is a production-like set of data used by test cases to determine
whether an application is working correctly.
 While executing test cases, test engineers need to input some data into the
application to get the expected output.
‒ Test data have the following types:
 Valid Test Data (positive data), Invalid Test Data (negative), No Data, Boundary Data.
• Test Environment
‒ A test environment is a controlled setup where software or systems undergo
evaluation to ensure functionality, compatibility, and reliability.
 The test environment is used by the testing teams to check the quality and impact of
the application before handing it to the user.
21 Oct. 2023 8
Eng.
Bakr
Salim

9. What is Software Testing? ..
Test
Environment
Database
Automation
testers
Manual testers
Network Protocols
Operating system
(Testing server)
21 Oct. 2023 9
Eng.
Bakr
Salim

10. Software Testing Goals
21 Oct. 2023 10
Eng.
Bakr
Salim

11. Software Testing History
21 Oct. 2023 11
Eng.
Bakr
Salim

12. Software Testing History …
• Debugging-oriented Era (Early 1950s):
‒ There was no distinction between testing and debugging.
‒ There was no concept of testers, the developers write and debug their codes.
• Demonstration-oriented Era (1957 – 1978):
‒ Testing is a separate activity in software development.
‒ A clear distinction was there between debugging and software testing.
‒ The major goal was to make sure that the software requirements are satisfied,
and that the software meets the customer’s expectations.
• Destruction-oriented Era (1979 – 1982):
‒ The focus is on breaking the code and detecting undiscovered errors.
‒ This phase failed because fixing bug leads to a new one.
21 Oct. 2023 12
Eng.
Bakr
Salim

13. Software Testing History …
• Evaluation-oriented Era (1983 – 1987):
‒ The focus here is to evaluate and measure the quality of the software.
‒ Testers test the software up to an acceptable point until they reach a
point where the number of detected bugs was reduced.
• Prevention-oriented Era (1988 – 2000):
‒ This focuses on conducting tests to show that the software meets its
specification.
‒ Code here is divided into testable and non-testable where testable
codes had fewer bugs than the non-testable code.
• The current Era(2001 – now):
‒ Focused on using AI tools, cross-browser testing tools, etc.
21 Oct. 2023 13
Eng.
Bakr
Salim

14. Test cases (ISTQB)
• A test case is a set of input values, execution preconditions, expected results, and
execution postconditions, developed for a particular objective or test condition, such as to
exercise a particular program path or to verify compliance with a specific requirement.
• Contents
‒ The test case ID
‒ Test case description
‒ Prerequisites
‒ Test steps
‒ Test data
‒ Expected result
‒ Actual result
‒ Status
‒ Created by
‒ Date of creation
‒ Executed by
‒ Date of execution
21 Oct. 2023 14
Eng.
Bakr
Salim
Test case example

15. Software Testing Principles
• Principle 1: Testing shows the Presence
of Defects
‒ Testing cannot prove that there are no
defects.
• Principle 2: Exhaustive Testing is
Impossible
‒ All 𝑪𝒐𝒎𝒃𝒊𝒏𝒂𝒕𝒊𝒐𝒏𝒔 =
𝑺𝒄𝒆𝒏𝒂𝒓𝒊𝒐𝒔(𝑭𝒊𝒆𝒍𝒅𝒔/𝑰𝒕𝒆𝒎𝒔)
.
 Example: Facebook login Screen = 𝟑𝟐= 9 tests.
• Principle 3: Early Testing
‒ Start testing as soon as possible, even
since requirements/initial phase.
21 Oct. 2023 15
Eng.
Bakr
Salim

16. Software Testing Principles ..
21 Oct. 2023 16
Eng.
Bakr
Salim

17. Software Testing Principles ..
• Principle 4: Defect Clustering
‒ Pareto Principle: software testing is
approximately 80:20.
 which means 80% of the problems are found
due to 20% of the modules.
• Principle 5: Pesticide Paradox
‒ Running the same tests frequently have less
chance of getting new bugs that are
discovered by these test cases.
• Principle 6: Testing is Context Dependent
‒ Different types of applications can be tested
differently, and the same test cases cannot be
applied for different projects.
21 Oct. 2023 17
Eng.
Bakr
Salim

18. Software Testing Principles ..
• Principle 7: Absence of Errors Fallacy
‒ If there are no bugs in the software, then
you should not consider that software is
ready to be used; because testing should
be conducted along with the right
requirements.
‒ Maybe program is unusable because it
does not meet the user’s expectations.
21 Oct. 2023 18
Eng.
Bakr
Salim

19. Testing in the Industry
• Practice #1. Automation at the heart.
‒ Automation through tools.
• Practice #2. Special methodologies in testing
‒ Using canary, dogfooding, and testing in the toilet.
 Canary – testing your software only on a part of your customers.
 Amazon.
 Dogfooding – using your product for work or personal purposes.
 Microsoft, Apple, Facebook, … etc.
 Testing in the toilet
 Google.
• Practice #3. Beta testing and communities
‒ Real users will be able to test the application better.
 Apple, Google, and Microsoft.
21 Oct. 2023 19
Eng.
Bakr
Salim

20. Testing in the Industry
• Practice #4. Suitable testing tools.
‒ Tech companies strive to have their tools for successful automation.
 Wayfair, Apple, Expedia Group, Facebook, and Amazon.
• Practice #5. QA engineers perform complex types of testing.
‒ QA engineers are responsible for product quality at a higher level.
 Amazon, Microsoft, Google, Facebook.
21 Oct. 2023 20
Eng.
Bakr
Salim

21. Testing with The Big Giants
Google: Searching for Best Practices
Facebook: Developer-driven Testing
Amazon: Deployment Comes First
Spotify: Squads, Tribes and Chapters
Microsoft: Engineers and Testers are One
Lessons Learnt from the Big Giants

22. Google: Searching for Best Practices
• Testing efforts depend on the team and the product.
‒ Team
 Maintains a large and rigorous testing framework.
‒ Product
• Google employs a four-stage testing process consisting of:
‒ Testing by dedicated, internal testers (Google employees).
 Further testing on a crowd testing platform => Global App Testing.
‒ "Dogfooding," which involves having Google employees use the product in
their daily work.
‒ Beta testing, which involves releasing the product to a small group of Google
product end users
• Google takes testing seriously.
‒ In fact, testers' and developers' salaries are equal.
21 Oct. 2023 22
Eng.
Bakr
Salim

23. Facebook: Developer-driven Testing
• Facebook does not employ any dedicated testers at all.
‒ Relies on its developers to test their own (as well as one another's) work.
• Automated testing solutions.
‒ PHPUnit for back-end unit testing.
‒ Jest (a JavaScript test tool developed internally at Facebook).
‒ Watir for end-to-end testing efforts.
‒ Others.
• Use "canary" releases and an incremental rollout strategy to test fixes,
updates, and new features in production.
‒ For example, a new feature might first be made available only to a small percentage
of the total number of users.
• By tracking the usage of the feature and the feedback received, the
company decides either to increase the rollout or to disable the feature,
either improving it or discarding it altogether.
21 Oct. 2023 23
Eng.
Bakr
Salim

24. Amazon: Deployment Comes First
• Amazon does not have a large QA infrastructure in place (Like
Facebook).
‒ One test engineer to every seven developers.
‒ Testing is not considered an essential activity at Amazon.
• Amazon increases its testing efforts when:
‒ It notices that revenue is decreasing, or
‒ Customers are moving away due to anomalies on the website.
• Focus on development and deployment processes
‒ The company famously deploys software every 11.6 seconds!.
‒ No need for elaborate and extensive testing efforts.
‒ It is all about making software easy to deploy, and, equally if not more
important, easy to roll back in case of a failure.
21 Oct. 2023 24
Eng.
Bakr
Salim

25. Spotify: Squads, Tribes and Chapters
• Spotify does employ dedicated testers.
‒ Part of cross-functional teams, each with a specific mission.
• At Spotify, employees are organized as the Spotify model:
‒ Squads.
 A Scrum team, with less focus on practices and more on principles.
 Motto “Rules are a good start but break them when needed”.
 Some squads might have one or more testers, and others might have no testers
at all, depending on the mission.
‒ Tribes.
 Groups of squads that belong together based on their business domain.
 Any tester that's part of a squad automatically belongs to the overarching tribe of
that squad.
21 Oct. 2023 25
Eng.
Bakr
Salim

26. Spotify: Squads, Tribes and Chapters..
‒ Chapters.
 A group of people that have the same skillset.
 To promote learning and sharing experiences.
 For example, all testers from different squads are grouped together in a testing chapter.
• Testing at Spotify is taken very seriously.
‒ Testing is considered a creative process, and something that cannot be (fully)
automated.
• Spotify heavily relies on dedicated testers .
‒ To explore and evaluate the product.
 Instead of trying to automate as much as possible.
• Kristian Karl
‒ Test manager and creator of the model-based testing tool GraphWalker, said:
 “I think we will spend as much time on testing tomorrow as we do today, but the tools and the
information that we will have and get from our automation will make our testing job different”.
21 Oct. 2023 26
Eng.
Bakr
Salim

27. Microsoft: Engineers and Testers are One
• The ratio of testers to developers is currently around 2:3.
‒ Microsoft pays testers and developers equally (like Google).
 They aren't called testers; they're software development engineers in test (or
SDETs).
• High ratio of testers to developers at Microsoft
‒ A very large chunk of the company's revenue comes from shippable
products that are installed on client computers, rather than websites
and online services.
• Harder (or at least much more annoying) to update these
products in case of bugs or new features.
‒ Microsoft invests a lot of time, effort, and money in making sure that
the quality of its products is of a high standard before shipping.
21 Oct. 2023 27
Eng.
Bakr
Salim

28. Lessons Learnt from the Big Giants
• Testing responsibility spectrum
‒ Testers are primarily responsible for executing tests.
‒ Everybody is responsible for performing testing activities.
• Testing importance spectrum
‒ Nothing goes to production untested.
‒ We put everything in production, and then we test there, if at all.
• Test automation has a significant presence in all five companies.
‒ All five companies employ tools to optimize their testing efforts.
Best fits the skillset
Risk, roll back and fix problems
21 Oct. 2023 28
Eng.
Bakr
Salim

29. Mismatch in Needs and Goals
• Industry wants testing to be simple and easy.
‒ Testers with no background in computing or math.
• Universities are graduating scientists
‒ Industry needs engineers.
• Testing needs to be done more rigorously (strictly, carefully).
• Agile processes put lots of demands on testing
‒ Programmers must unit test – with no training, education or tools!
‒ Tests are key components of functional requirements – but who builds
those tests ?
21 Oct. 2023 29
Eng.
Bakr
Salim

30. Relationship between Developers and Testers
• The relationship between a Developer and a Tester is not
normally an easy one because:
‒ Testers point out problems with software.
‒ Developers like to think their software is perfect.
‒ Testers are perceived as delaying the project by finding faults in the
system.
‒ When the development slips, testers normally have to work long hours
to test the product, which in turn can cause anger.
‒ It is important that they work together.
‒ It is also important that they have mutual respect.
‒ Collaboration is the right approach – we work to a common goal!
‒ Communicate findings objectively, not subjectively.
21 Oct. 2023 30
Eng.
Bakr
Salim

31. Independent Testing
• The right approach could enable Developers to test the code
‒ However, passing this responsibility to trained and professional testing
resources has many benefits (such as higher defect find rates)
• Authors tend to bring across assumptions they have made when
developing the software.
‒ They are less likely to write tests to show faults in their own software
(human nature).
• With testing performed by independent testers, testing effort is
focused and not compromised by development effort and bias.
• It is generally believed that objective independent testing is
more effective.
21 Oct. 2023 31
Eng.
Bakr
Salim

32. Independent Testing ..
• There are several levels of Independence (from Low to High):
‒ Tests designed by the person(s) who wrote the software under test.
‒ Tests designed by another person(s) (e.g., from the development team).
‒ Tests designed by a person(s) from a different organizational group
(e.g., an independent test team).
‒ Tests designed by a person(s) from a different organization or company
(e.g., outsourcing to an in-house or external test specialist
organization).
21 Oct. 2023 32
Eng.
Bakr
Salim

33. Testing Roles
21 Oct. 2023 33
Eng.
Bakr
Salim

34. Software Testing Approaches
Static vs Dynamic Testing Approaches
Static Testing Techniques
Dynamic Testing Techniques

35. Static vs Dynamic Testing Approaches
• Static Testing
‒ Assessing the structural characteristics of source code, design specifications or any
notational representation that conforms to well-defined syntactic rules.
‒ It is called as static because we never execute the code in this technique.
 For example, the structure of code is examined by the teams, but the code is not executed.
‒ Static testing involves verification.
• Static testing techniques
‒ Inspections
‒ Walkthroughs
‒ Technical reviews
• Dynamic testing
‒ All the methods that execute the code to test a software.
‒ Dynamic testing involves validation.
21 Oct. 2023 35
Eng.
Bakr
Salim

36. Software Testing Approaches
21 Oct. 2023 36
Eng.
Bakr
Salim

37. Software Inspections
• First introduced at IBM by Fagan in the early 1970s.
• Commonly inspected work products include software requirements
specifications and test plans.
• Inspection team
‒ Author/Owner/Producer
 A programmer or designer responsible for producing the program or document.
‒ Inspector
 A peer member of the team who finds errors, omissions, and inconsistencies in
programs and documents.
‒ Moderator
 A team member who manages the whole inspection process.
 He schedules, leads, and controls the inspection session.
‒ Recorder
 One who records all the results of the inspection meeting.
21 Oct. 2023 37
Eng.
Bakr
Salim

38. Structured Walkthroughs
• Walkthrough was proposed by Yourdon in 1979.
‒ Walkthrough is less formal and less steps than inspection.
• A typical structured walkthrough team consists of:
‒ Coordinator
 Organizes, moderates, and follows up the walkthrough activities.
‒ Presenter/Developer
 Introduces the item to be inspected. This member is optional.
‒ Scribe/Recorder
 Notes down the defects found and suggestion proposed by the members.
‒ Reviewer/Tester
 Finds the defects in the item.
‒ Maintenance Oracle
 Focuses on long-term implications and future maintenance of the project.
‒ Standards Bearer
 Assesses adherence to standards.
‒ User Representative/Accreditation
 Agent Reflects the needs and concerns of the user.
21 Oct. 2023 38
Eng.
Bakr
Salim

39. Technical Reviews
• Evaluate the software according to development standards,
guidelines, and specifications.
‒ A higher-level technique as compared to inspection or walkthrough.
• Technical review team includes (same as walkthrough)
‒ Management-level representatives.
‒ Project management representatives.
• Things to be reviewed:
‒ Problem definition and requirements.
‒ Adherence to standards.
‒ Consistency.
‒ Completeness.
‒ Documentation.
21 Oct. 2023 39
Eng.
Bakr
Salim

40. Dynamic Testing
• Dynamic testing
‒ Dynamic testing involves validation.
• Techniques:
‒ Structural Testing (White-box).
 Logic coverage.
 Data flow.
 Graph matrices.
‒ Behavioral/Functional Testing (Black-box)
 Boundary value analysis.
 Decision table.
 Equivalence class.
21 Oct. 2023 40
Eng.
Bakr
Salim

41. White-box testing example
Program Code Flow Graph
21 Oct. 2023 41
Eng.
Bakr
Salim

42. Big Picture
21 Oct. 2023 42
Eng.
Bakr
Salim

43. Defect Management
Defect Management Process
Defect Life Cycle
Defect Log
Defect Management Tools
Test Management Tools

44. Defect Management Process
21 Oct. 2023 44
Eng.
Bakr
Salim

45. Defect Life Cycle
21 Oct. 2023 45
Eng.
Bakr
Salim

46. Defect Log
• Defect ID number
• Descriptive defect name and type
• Source of defect – test case or other source
• Defect severity (5/7 levels – Bugzilla) => (determines by testers and
developers)
‒ (Critical, Major, Normal, Minor, Trivial, and Enhancement).
• Defect Priority (determined by business analyst/users)
‒ High Priority.
‒ Medium Priority.
‒ Low Priority.
• Defect status (e.g., New, open, assigned, fixed, closed, reopen, reject).
21 Oct. 2023 46
Eng.
Bakr
Salim

47. Defect Log ..
Defect
ID
Description Source Created on Creator
Assigned
to
Severity Priority Status
Linked
issues
D0001 Unable to add items in
the cart
TC_025 1-Dec-2021 Omar Ahmed Major High Open
D0002 Unable to login using g-
mail
TC_030 2-Dec-2021 Omar Ali Critical High Open D0001
D0003 There is no space between
username and welcome
on the dashboard
TC_045 3-Dec-2021 Omar Ahmed Trivial Low Open
21 Oct. 2023 47
Eng.
Bakr
Salim

48. Defect Management Tools
21 Oct. 2023 48
Eng.
Bakr
Salim

49. Test Management Tools
21 Oct. 2023 49
Eng.
Bakr
Salim

50. Software Testing and Generative
AI
Overview
AI-based Testing
The Role of AI in Software Testing
Using ChatGPT in Software Testing
ChatGPT Generates Test Script

51. Overview
• Is ChatGPT coming to take automation engineers' jobs?
‒ The short answer is “maybe, but probably not”.
• ChatGPT is capable of amazing things that may feel like a threat
to traditional engineering jobs.
‒ But there are also many things it can't do – or, at least, do well.
• Automation engineers who want to use generative AI to do their
job better – instead of being replaced by AI – must understand
the difference.
21 Oct. 2023 51
Eng.
Bakr
Salim

52. AI-based Testing
• The idea of applying AI to software testing is not new.
‒ Robo on Google Firebase (Android-based) is one popular examples.
• These bots can help automation engineers to define test cases
and execute tests.
‒ But they have clear shortcomings.
• So, automation engineers have historically thought of AI as a
tool that can reduce some of the toil and tedium required to
carry out basic tests.
‒ but not as anything approaching a replacement for human testing
experts.
21 Oct. 2023 52
Eng.
Bakr
Salim

53. The Role of AI in Software Testing
• Automated Test Creation
‒ Generate and automate test cases using AI algorithms.
• Test Maintenance
‒ Manage and update test suites by analyzing code, then automatically adjust
test cases accordingly.
• Enhanced Test Coverage
‒ Identify and prioritize test scenarios based on historical data and code
changes.
• Efficient Bug Detection
‒ AI algorithms can be trained to identify patterns associated with common
regression issues.
• Automated Reporting
‒ AI can automate the generation of detailed test reports.
21 Oct. 2023 53
Eng.
Bakr
Salim

54. Using ChatGPT in Software Testing
• ChatGPT can be used to generate automation scripts with
limitations in:
‒ Web application testing.
‒ Mobile testing.
21 Oct. 2023 54
Eng.
Bakr
Salim

55. ChatGPT Generates Test Script
• You could use ChatGPT to help generate the automation script
with a prompt such as the following:
Generate a Selenium script that visits
kobiton.com, chooses the “Platform” menu
option, and then clicks on “Real Device Testing”.
21 Oct. 2023 55
Eng.
Bakr
Salim

56. ChatGPT Generates Test Script ..
ChatGPT
would
respond
with
a
script
such
as
this
one
I
got
from
it:
21 Oct. 2023 56
Eng.
Bakr
Salim

57. In conclusion
Automation engineers who want to excel in
an AI-centric world should use generative AI
to automate the simpler, tedious aspects of
test automation – while focusing their
energies on doing the things that AI can't.
21 Oct. 2023 57
Eng.
Bakr
Salim

58. References
• 5 effective and powerful ways to test like tech giants, by:
‒ Bas Dijkstra (Test automation speaker and writer).
• Software Testing: What Generative AI Can—and Can't—Do, by:
‒ Frank Moyer (CTO, Kobiton), May 16, 2023.
• QA insights from Fortune 500 companies: Agile, QA management &
processes optimization, by:
‒ zebrunner.com/.
• Stages of Defect Management Process, by:
‒ Madhuri Hammad, Last Updated : 08 Sep 2020.
• Wikipedia website
‒ Software testing
• SOFTWARE TESTING: Principles and Practices (2010), by:
‒ Naresh Chauhan, OXFORD UNIVERSITY PRESS.
21 Oct. 2023 58
Eng.
Bakr
Salim

59. References ..
• Certified Tester: Foundation Level Syllabus, v4.0, by:
‒ International Software Testing Qualifications Board (hereinafter
called ISTQB®), 2023.
• ISTQB Glossary, by:
‒ ISTQB.
• Software Engineering at Google, by:
‒ Titus Winters, Tom Manshreck, and Hyrum Wright, 2020.
• How Google Tests Software, by:
‒ James Whittaker, Jason Arbon, and Jeff Carollo, 2012.
21 Oct. 2023 59
Eng.
Bakr
Salim

60. 21 Oct. 2023
Eng.
Bakr
Salim
60