SlideShare a Scribd company logo
1 of 18
ATestAnalysisMethodfor
BlackBoxTesting
UsingAUTandFaultKnowledge
11th Sep. 2013
KES2013
Tsuyoshi Yumoto, Toru Matsuodani, Kazuhiko Tsuda
2
Agenda
1. Introduction
2. Issues of Test Analysis in Black Box Testing
3. An Approach of Test Analysis Based on Test Categories
4. Comparison of Test Condition Derivation Result by Test Categories
Practical Use Existence
5. conclusion
3
1. Introduction
With a rapid increase in size and complexity of software today,
the scope of software testing is also expanding.
Testing
increaseinsize
increasecomplexity
Testing
Development
・Requirement
・Specification
・Design
・CodeDevelopment
・Requirement
・Specification
・Design
・Code
The efficiency of software testing
needs to be improved in order to ensure
the appropriate delivery deadline and
cost of software development.
4
2.Issues of Test Analysis in Black Box Testing
Development Lifecycle and Test Process
• Software testing is performed in multiple verification levels during the
development life-cycle. These verification levels are depicted in the V model.
• This proposal will focus on the black box testing performed at the system-level
verification, enclosed in a red frame.
V model (Forsberg,1995 et.al)
Testing performed at
each level depicted in
the V model has a
process similar to the
development process.
5
2.Issues of Test Analysis in Black Box Testing(Cont.)
Test development process and Test analysis
• Three activities, test analysis, test design, and test implementation in the
test process are also called Test development process.
Test analysis
An activity of selecting
and organizing items to be
covered by the test .
6
2.Issues of Test Analysis in Black Box Testing(Cont.)
Issues of Test Analysis
ソフトウェア
テスト
・要件定義
・基本設計
・詳細設計
・コーディング
・Preventing defects in software
development
・Test needs to be designed in a
way that the number of test is
sufficient and appropriate in
quantity
・Test execution should be
automated
For improving efficiency of
software testing
.
.
.
White box testing
Analyzing the
structure of AUT
to cover by test
Test analysis for
White box
testing is
consistent in its
refinement
Black box testing
Analyzing the
spec of AUT to
cover by test
Test analysis for
black box testing
is often not
consistent in its
refinement
Some of the test cases are
often lacking or overlapping
Types of test desgin
7
2.Issues of Test Analysis in Black Box Testing(Cont.)
Issues of Test Analysis in Black Box testing
• The test condition, which is an output of test analysis, is a generic term for
elements such as functions, transactions, quality characteristics, and structure
elements.
• Therefore it is necessary to structure in order to sort out relationships between
each element. That is also called test suite architecture.
• However, in research and practice, structuring of test conditions in the test
analysis stays just experiences and heuristics.
Examples of Table 1 contain the following issues.
• The category “Setting” appears both in Level 1 and Level 2.
• The number of levels is not constant, thus there is a variance in meaning of each level.
• The expected result is not indicated in the upper row.
• The same specification is written on both upper and lower rows.
8
Structure of Test Condition
• Test condition can be considered as spec item from the testing point of view.
• Since black box testing is the method of the test design by external observation,
selection should be start from the feature is contained in test basis
• the test category, which is the classification defined using knowledge of AUT
and fault, is added to the structure.
The test analysis in alignment with a clear rule is attained by such
a classification and arrangement.
3.An Approach of Test Analysis Based on Test Categories
9
3.An Approach of Test Analysis Based on Test Categories
(Cont.)
Test Category(1)
• A set of test categories is the classification that multilaterally captures a feature
to be tested without omission.
An logical structure of feature (Sakuhei Omura,2005 )
An logical structure of feature is
an abstract concept of a set of test categories.
10
3.An Approach of Test Analysis Based on Test Categories
(Cont.)
Test Category(2)
• The test category needs to have a specific naming and a meaning specialized in
the scope of testing against an abstract concept.
• The knowledge of AUT and the fault experienced in the past is used for defining
the test categories.
Knowledge of AUT
Knowledge of faults
experienced in the past
11
3.An Approach of Test Analysis Based on Test Categories
(Cont.)
Listing and Organizing Test Conditions Using Test Categories
• Spec items, expected results, and test parameters for testing the feature to be
tested, these are selected from test basis, are listed as shown in Table below.
Test categories can be used as a guide at the time of selecting spec
items for covering feature to be tested.
Same test categories are used
for all features to be tested.
12
4. Comparison of Test Condition Derivation Result by Test
Categories Practical Use Existence
Execution Steps of Test Analysis
• In order to sequentially derive
test conditions, test analysis
activities are divided into
steps.
Comparison of test condition
derivation ware executed on
Step3
13
4. Comparison of Test Condition Derivation Result by Test
Categories Practical Use Existence(Cont.)
Exercise Overview
Feature to be
tested
Test
categories
Spec items Expected
results
Test
Parameters
Feature to be
tested
Spec Item Expected
results
Test
Parameters
Test Basis
Results
were
classified
to test
categories
Analyze
Comparison
Spec items Expected results Test parameters
The team that use test categories
The team that didn’t use test categories
14
4. Comparison of Test Condition Derivation Result by Test
Categories Practical Use Existence(Cont.)
Comparison of Spec Items Selection Percentage
• The result showed the rate of selecting spec items was higher by 22% for the
team that used test categories than for the team that did not use test
categories.
Inputadjustment
Management
InputA
ConvA
ConvB
SuppA
SuppB
OutputA
OutputB
StrageA
StrageB
ManagA
Func A 0 2 1 0 2 1 0 1 0 2 Suggested answers 9 100%
0 2 0 0 0 1 0 1 0 1 NOT using test categories 5 56%
0 2 0 0 2 1 0 1 0 1 Using test categories 7 78%
Using test categories-NOT using test categories = 22%
Persentage
SpecitemTotal
Conversion
Support
Strage
Output
adjustment
Features tobe tested
Test categories
15
4. Comparison of Test Condition Derivation Result by Test
Categories Practical Use Existence(Cont.)
Results of the selection of the number of expected results and test
parameters
• Although the ratio of the number of expect results without test categories to the
number of them with test categories is 43%, the number of test parameters is
1.5 times higher.
<the team that did NOT use testing categories> <the team that use testing categories>
Features
to be tested Test categories Spec items
Number of
Expected Results
Number of
Test parameters
Features
to be tested Test categories Spec items
Number of
Expected Results
Number of
Test parameters
Func A InputA N/A N/A N/A Func A InputA N/A N/A N/A
ConvA Spec item1 1 5(3) ConvA Spec item1 1 N/A
Spec item2 N/A 1 Spec item2 1 N/A
ConvB Spec item3 N/A N/A ConvB Spec item3 N/A N/A
SuppA N/A N/A N/A SuppA N/A N/A N/A
SuppB Spec item4 N/A N/A SuppB Spec item4 1 2
Spec item5 N/A N/A Spec item5 1 2
OutputA Spec item6 1 N/A OutputA Spec item6 1 2
OutputB N/A N/A N/A OutputB N/A N/A N/A
StrageA Spec item7 1 3(1) StrageA Spec item7 1 2
StrageB N/A N/A N/A StrageB N/A N/A N/A
ManagA Spec item8 N/A 1 ManagA Spec item8 1 1
Spec item9 N/A N/A Spec item9 N/A N/A
Total 3 10(4) Total 7 9(0)
※() in the coloum of test parametrs is the number of test parametors for one spec item are written dispersedly throughout the test condition list.
Ratio by Number of expected results : NOT using test categories / Using test categories = 43%
Ratio by Number of test parameters : NOT using test categories / Using test categories = 156%
16
4. Comparison of Test Condition Derivation Result by Test
Categories Practical Use Existence(Cont.)
Sum up
Feature to be
tested
Test
categories
Spec items Expected
results
Test
Parameters
Feature to be
tested
Spec Item Expected
results
Test
Parameters
Test Basis
Results
were
classified
to test
categories
exercise
Comparison
Spec items Expected results Test parameters
The team that didn’t use test
categories had a higher possibility
of designing duplicated test cases
at the time of a test design.
• 4 of all test parameters that can
possibly become test parameters
for the same spec item, had
addressed as another spec item,
when it was 0/9 for the team that
used test categories.
The team that
didn’t use
test categories
did not clearly
specify the
expected results
The team that didn’t use
test categories had a
higher possibility of
designing lacked test
cases at the time of a
test design.
• The rate of selecting
spec items is different
more than 20%.
17
Conclusion
• Test analysis method based on the rule by organizing classifications of test
conditions, and knowledge for it is using the concept of test categories.
• To avoid lacks or duplications of test cases, which often results from a test
analysis and classifications of test conditions based on an individual’s own
judgments.
• In case of performing a test analysis using test categories…
• the rate of selecting spec items had increased more than 20%.
• the rate of spec items was improved when the expected results were also
selected.
• the variance in descriptions was alleviated.
18
References
[1] IPA, Report of industry actual survey for Embedded Software in 2009: METI; 2009 (In Japanese).
[2] Myers, Glenford J., Corey Sandler, and Tom Badgett. The art of software testing.: Wiley; 2011.
[3] Forsberg, K., and H. Mooz. "The relationship of systems engineering to the project cycle. " Engineering
Management Journal 4.3 ;1992 p.36-43.
[4] Uetsuki Keiji, Tohru Matsuodani, and Kazuhiko Tsuda. "An efficient software testing method by decision
table verification." International Journal of Computer Applications in Technology 46.1 ; 2013, p.54-64.
[5] ISTQB FLWG. "Foundation Level Syllabus Version 2011. ": International Software Testing Qualifications
Board ; 2011.
[6] Nishi, Yasuharu. "Viewpoint-based Test Architecture Design." Software Security and Reliability Companion
(SERE-C), 2012 IEEE Sixth International Conference on. IEEE; 2012, p.194- 197.
[7] Kang, Kyo C., et al. "Feature-oriented domain analysis (FODA) feasibility study. No. CMU/SEI-90-TR-21. ":
CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERING INST; 1990.
[8] IEEE ."IEEE standard for software test documentation ", IEEE829. 2008.; 2008.
[9] Tsuyoshi Yumoto,Tohru Matsuodani,Kazuhiko Tsuda. "A practical Using Method for Efficent Design of
Functionai Testing": 75th National Convention of IPSJ; 2013 , paper#5B-4 (In Japanese).
[10] Sakuhei Omura. Phenomenology of general system: gohodoshoppan; 2005 (In Japanese).
[11] Rasiel, E. M. The McKinsey Way. McGraw-Hill; 1999.
[12] Haimes, Yacov Y., Stan Kaplan, and James H. Lambert. "Risk filtering, ranking, and management
framework using hierarchical holographic modeling." : Risk Analysis 22.2 ; 2002, p.383-397.

More Related Content

What's hot

<p>Software Testing</p>
<p>Software Testing</p><p>Software Testing</p>
<p>Software Testing</p>Atul Mishra
 
Fundamental Test Design Techniques
Fundamental Test Design TechniquesFundamental Test Design Techniques
Fundamental Test Design TechniquesTechWell
 
Test Case Design and Technique
Test Case Design and TechniqueTest Case Design and Technique
Test Case Design and TechniqueSachin-QA
 
Introduction to specification based test design techniques
Introduction to specification based test design techniquesIntroduction to specification based test design techniques
Introduction to specification based test design techniquesYogindernath Gupta
 
Code coverage based test case selection and prioritization
Code coverage based test case selection and prioritizationCode coverage based test case selection and prioritization
Code coverage based test case selection and prioritizationijseajournal
 
Software Testing Techniques
Software Testing TechniquesSoftware Testing Techniques
Software Testing TechniquesKiran Kumar
 
Software Testing Techniques
Software Testing TechniquesSoftware Testing Techniques
Software Testing TechniquesKiran Kumar
 
Testing survey by_directions
Testing survey by_directionsTesting survey by_directions
Testing survey by_directionsTao He
 
Test Case Design
Test Case DesignTest Case Design
Test Case Designacatalin
 
Software Quality Testing
Software Quality TestingSoftware Quality Testing
Software Quality TestingKiran Kumar
 
Black boxtestingmethodsforsoftwarecomponents
Black boxtestingmethodsforsoftwarecomponentsBlack boxtestingmethodsforsoftwarecomponents
Black boxtestingmethodsforsoftwarecomponentsAstrid yolanda
 
EXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODEL
EXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODELEXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODEL
EXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODELijaia
 
Boundary and equivalnce systematic test design
Boundary and equivalnce   systematic test designBoundary and equivalnce   systematic test design
Boundary and equivalnce systematic test designIan McDonald
 
White Box Testing
White Box TestingWhite Box Testing
White Box TestingAlisha Roy
 
Software Testing Foundations Part 6 - Intuitive and Experience-based testing
Software Testing Foundations Part 6 - Intuitive and Experience-based testingSoftware Testing Foundations Part 6 - Intuitive and Experience-based testing
Software Testing Foundations Part 6 - Intuitive and Experience-based testingNikita Knysh
 

What's hot (20)

<p>Software Testing</p>
<p>Software Testing</p><p>Software Testing</p>
<p>Software Testing</p>
 
@#$@#$@#$"""@#$@#$"""
@#$@#$@#$"""@#$@#$"""@#$@#$@#$"""@#$@#$"""
@#$@#$@#$"""@#$@#$"""
 
Fundamental Test Design Techniques
Fundamental Test Design TechniquesFundamental Test Design Techniques
Fundamental Test Design Techniques
 
Test Case Design and Technique
Test Case Design and TechniqueTest Case Design and Technique
Test Case Design and Technique
 
Introduction to specification based test design techniques
Introduction to specification based test design techniquesIntroduction to specification based test design techniques
Introduction to specification based test design techniques
 
Code coverage based test case selection and prioritization
Code coverage based test case selection and prioritizationCode coverage based test case selection and prioritization
Code coverage based test case selection and prioritization
 
Software Testing Techniques
Software Testing TechniquesSoftware Testing Techniques
Software Testing Techniques
 
Software Testing Techniques
Software Testing TechniquesSoftware Testing Techniques
Software Testing Techniques
 
Testing survey by_directions
Testing survey by_directionsTesting survey by_directions
Testing survey by_directions
 
Black & White Box testing
Black & White Box testingBlack & White Box testing
Black & White Box testing
 
Test Case Design
Test Case DesignTest Case Design
Test Case Design
 
Software Quality Testing
Software Quality TestingSoftware Quality Testing
Software Quality Testing
 
IP 200 Introduction
IP 200 IntroductionIP 200 Introduction
IP 200 Introduction
 
Black boxtestingmethodsforsoftwarecomponents
Black boxtestingmethodsforsoftwarecomponentsBlack boxtestingmethodsforsoftwarecomponents
Black boxtestingmethodsforsoftwarecomponents
 
EXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODEL
EXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODELEXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODEL
EXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODEL
 
Boundary and equivalnce systematic test design
Boundary and equivalnce   systematic test designBoundary and equivalnce   systematic test design
Boundary and equivalnce systematic test design
 
White Box Testing
White Box TestingWhite Box Testing
White Box Testing
 
Sta unit 3(abimanyu)
Sta unit 3(abimanyu)Sta unit 3(abimanyu)
Sta unit 3(abimanyu)
 
Software Testing Foundations Part 6 - Intuitive and Experience-based testing
Software Testing Foundations Part 6 - Intuitive and Experience-based testingSoftware Testing Foundations Part 6 - Intuitive and Experience-based testing
Software Testing Foundations Part 6 - Intuitive and Experience-based testing
 
Dynamic Testing
Dynamic TestingDynamic Testing
Dynamic Testing
 

Viewers also liked

とてか03「「いかす!」のために大事だと思う4つのこと」
とてか03「「いかす!」のために大事だと思う4つのこと」とてか03「「いかす!」のために大事だと思う4つのこと」
とてか03「「いかす!」のために大事だと思う4つのこと」Tsuyoshi Yumoto
 
WACATE 2010夏 ゆもつよ講演スライド
WACATE 2010夏 ゆもつよ講演スライドWACATE 2010夏 ゆもつよ講演スライド
WACATE 2010夏 ゆもつよ講演スライドTsuyoshi Yumoto
 
テスト分析についての説明資料公開用
テスト分析についての説明資料公開用テスト分析についての説明資料公開用
テスト分析についての説明資料公開用Tsuyoshi Yumoto
 
20150424 jasst新潟基調講演
20150424 jasst新潟基調講演20150424 jasst新潟基調講演
20150424 jasst新潟基調講演Kouichi Akiyama
 
20170203 test analysisdesign
20170203 test analysisdesign20170203 test analysisdesign
20170203 test analysisdesignKouichi Akiyama
 
テスト自動化とアーキテクチャ
テスト自動化とアーキテクチャテスト自動化とアーキテクチャ
テスト自動化とアーキテクチャToru Koido
 
Иркутск - середина Земли
Иркутск - середина ЗемлиИркутск - середина Земли
Иркутск - середина ЗемлиNatalya Dyrda
 
DTADA: Distributed Trusted Agent Based Detection Approach For Doline And Sen...
DTADA: Distributed Trusted Agent Based Detection Approach  For Doline And Sen...DTADA: Distributed Trusted Agent Based Detection Approach  For Doline And Sen...
DTADA: Distributed Trusted Agent Based Detection Approach For Doline And Sen...IOSR Journals
 
An Overview of TRIZ Problem-Solving Methodology and its Applications
An Overview of TRIZ Problem-Solving Methodology and its ApplicationsAn Overview of TRIZ Problem-Solving Methodology and its Applications
An Overview of TRIZ Problem-Solving Methodology and its ApplicationsIOSR Journals
 
The Wind Powered Generator
The Wind Powered GeneratorThe Wind Powered Generator
The Wind Powered GeneratorIOSR Journals
 
Modeling Of Carbon Deposit From Methane Gas On Zeolite Y Catalyst Activity In...
Modeling Of Carbon Deposit From Methane Gas On Zeolite Y Catalyst Activity In...Modeling Of Carbon Deposit From Methane Gas On Zeolite Y Catalyst Activity In...
Modeling Of Carbon Deposit From Methane Gas On Zeolite Y Catalyst Activity In...IOSR Journals
 
Two Factor Authentication Using Smartphone Generated One Time Password
Two Factor Authentication Using Smartphone Generated One Time PasswordTwo Factor Authentication Using Smartphone Generated One Time Password
Two Factor Authentication Using Smartphone Generated One Time PasswordIOSR Journals
 
Evaluation of Uncertain Location
Evaluation of Uncertain Location Evaluation of Uncertain Location
Evaluation of Uncertain Location IOSR Journals
 

Viewers also liked (20)

とてか03「「いかす!」のために大事だと思う4つのこと」
とてか03「「いかす!」のために大事だと思う4つのこと」とてか03「「いかす!」のために大事だと思う4つのこと」
とてか03「「いかす!」のために大事だと思う4つのこと」
 
WACATE 2010夏 ゆもつよ講演スライド
WACATE 2010夏 ゆもつよ講演スライドWACATE 2010夏 ゆもつよ講演スライド
WACATE 2010夏 ゆもつよ講演スライド
 
テスト分析についての説明資料公開用
テスト分析についての説明資料公開用テスト分析についての説明資料公開用
テスト分析についての説明資料公開用
 
20150424 jasst新潟基調講演
20150424 jasst新潟基調講演20150424 jasst新潟基調講演
20150424 jasst新潟基調講演
 
20160619 wacate 解答
20160619 wacate   解答20160619 wacate   解答
20160619 wacate 解答
 
20170203 test analysisdesign
20170203 test analysisdesign20170203 test analysisdesign
20170203 test analysisdesign
 
Toteka 04
Toteka 04Toteka 04
Toteka 04
 
テスト自動化とアーキテクチャ
テスト自動化とアーキテクチャテスト自動化とアーキテクチャ
テスト自動化とアーキテクチャ
 
Иркутск - середина Земли
Иркутск - середина ЗемлиИркутск - середина Земли
Иркутск - середина Земли
 
DTADA: Distributed Trusted Agent Based Detection Approach For Doline And Sen...
DTADA: Distributed Trusted Agent Based Detection Approach  For Doline And Sen...DTADA: Distributed Trusted Agent Based Detection Approach  For Doline And Sen...
DTADA: Distributed Trusted Agent Based Detection Approach For Doline And Sen...
 
Thanhham
ThanhhamThanhham
Thanhham
 
An Overview of TRIZ Problem-Solving Methodology and its Applications
An Overview of TRIZ Problem-Solving Methodology and its ApplicationsAn Overview of TRIZ Problem-Solving Methodology and its Applications
An Overview of TRIZ Problem-Solving Methodology and its Applications
 
The Wind Powered Generator
The Wind Powered GeneratorThe Wind Powered Generator
The Wind Powered Generator
 
Modeling Of Carbon Deposit From Methane Gas On Zeolite Y Catalyst Activity In...
Modeling Of Carbon Deposit From Methane Gas On Zeolite Y Catalyst Activity In...Modeling Of Carbon Deposit From Methane Gas On Zeolite Y Catalyst Activity In...
Modeling Of Carbon Deposit From Methane Gas On Zeolite Y Catalyst Activity In...
 
D0551821
D0551821D0551821
D0551821
 
J0535964
J0535964J0535964
J0535964
 
H0613644
H0613644H0613644
H0613644
 
Two Factor Authentication Using Smartphone Generated One Time Password
Two Factor Authentication Using Smartphone Generated One Time PasswordTwo Factor Authentication Using Smartphone Generated One Time Password
Two Factor Authentication Using Smartphone Generated One Time Password
 
Evaluation of Uncertain Location
Evaluation of Uncertain Location Evaluation of Uncertain Location
Evaluation of Uncertain Location
 
C0541529
C0541529C0541529
C0541529
 

Similar to A Test Analysis Method for Black Box Testing Using AUT and Fault Knowledge.

Software engineering Testing technique,test case,test suit design
Software engineering Testing technique,test case,test suit designSoftware engineering Testing technique,test case,test suit design
Software engineering Testing technique,test case,test suit designMaitree Patel
 
Test Case Design
Test Case DesignTest Case Design
Test Case DesignVidya-QA
 
Test Case Design and Technique
Test Case Design and TechniqueTest Case Design and Technique
Test Case Design and TechniqueFayis-QA
 
Test Case Design and Technique
Test Case Design and TechniqueTest Case Design and Technique
Test Case Design and TechniqueANKUR-BA
 
Test Case Design & Technique
Test Case Design & TechniqueTest Case Design & Technique
Test Case Design & TechniqueRajesh-QA
 
Basic Engineering Design (Part 6): Test and Evaluate
Basic Engineering Design (Part 6): Test and EvaluateBasic Engineering Design (Part 6): Test and Evaluate
Basic Engineering Design (Part 6): Test and EvaluateDenise Wilson
 
КАТЕРИНА АБЗЯТОВА - Certify with confidence: ISTQB Foundation 4.0. Common err...
КАТЕРИНА АБЗЯТОВА - Certify with confidence: ISTQB Foundation 4.0. Common err...КАТЕРИНА АБЗЯТОВА - Certify with confidence: ISTQB Foundation 4.0. Common err...
КАТЕРИНА АБЗЯТОВА - Certify with confidence: ISTQB Foundation 4.0. Common err...GoQA
 
CS8494 SOFTWARE ENGINEERING Unit-4
CS8494 SOFTWARE ENGINEERING Unit-4CS8494 SOFTWARE ENGINEERING Unit-4
CS8494 SOFTWARE ENGINEERING Unit-4SIMONTHOMAS S
 
software testing unit 3 notes anna university 2017
software testing unit 3 notes anna university 2017software testing unit 3 notes anna university 2017
software testing unit 3 notes anna university 2017SathyaP56
 
Dynamic analysis in Software Testing
Dynamic analysis in Software TestingDynamic analysis in Software Testing
Dynamic analysis in Software TestingSagar Pednekar
 
Software Testing
Software TestingSoftware Testing
Software TestingSKumar11384
 
Software Testing Foundations Part 4 - Black Box Testing
Software Testing Foundations Part 4 - Black Box TestingSoftware Testing Foundations Part 4 - Black Box Testing
Software Testing Foundations Part 4 - Black Box TestingNikita Knysh
 
Orthogonal array approach a case study
Orthogonal array approach   a case studyOrthogonal array approach   a case study
Orthogonal array approach a case studyKarthikeyan Rajendran
 

Similar to A Test Analysis Method for Black Box Testing Using AUT and Fault Knowledge. (20)

Software engineering Testing technique,test case,test suit design
Software engineering Testing technique,test case,test suit designSoftware engineering Testing technique,test case,test suit design
Software engineering Testing technique,test case,test suit design
 
Qa documentation pp
Qa documentation ppQa documentation pp
Qa documentation pp
 
Unit2 for st
Unit2 for stUnit2 for st
Unit2 for st
 
Test Case Design
Test Case DesignTest Case Design
Test Case Design
 
Test Case Design and Technique
Test Case Design and TechniqueTest Case Design and Technique
Test Case Design and Technique
 
Test Case Design and Technique
Test Case Design and TechniqueTest Case Design and Technique
Test Case Design and Technique
 
Test Case Design & Technique
Test Case Design & TechniqueTest Case Design & Technique
Test Case Design & Technique
 
Basic Engineering Design (Part 6): Test and Evaluate
Basic Engineering Design (Part 6): Test and EvaluateBasic Engineering Design (Part 6): Test and Evaluate
Basic Engineering Design (Part 6): Test and Evaluate
 
КАТЕРИНА АБЗЯТОВА - Certify with confidence: ISTQB Foundation 4.0. Common err...
КАТЕРИНА АБЗЯТОВА - Certify with confidence: ISTQB Foundation 4.0. Common err...КАТЕРИНА АБЗЯТОВА - Certify with confidence: ISTQB Foundation 4.0. Common err...
КАТЕРИНА АБЗЯТОВА - Certify with confidence: ISTQB Foundation 4.0. Common err...
 
Black box testing
Black box testingBlack box testing
Black box testing
 
L software testing
L   software testingL   software testing
L software testing
 
CS8494 SOFTWARE ENGINEERING Unit-4
CS8494 SOFTWARE ENGINEERING Unit-4CS8494 SOFTWARE ENGINEERING Unit-4
CS8494 SOFTWARE ENGINEERING Unit-4
 
software testing unit 3 notes anna university 2017
software testing unit 3 notes anna university 2017software testing unit 3 notes anna university 2017
software testing unit 3 notes anna university 2017
 
Dynamic analysis in Software Testing
Dynamic analysis in Software TestingDynamic analysis in Software Testing
Dynamic analysis in Software Testing
 
Software Testing
Software TestingSoftware Testing
Software Testing
 
Unit 3 for st
Unit 3 for stUnit 3 for st
Unit 3 for st
 
Software Testing Foundations Part 4 - Black Box Testing
Software Testing Foundations Part 4 - Black Box TestingSoftware Testing Foundations Part 4 - Black Box Testing
Software Testing Foundations Part 4 - Black Box Testing
 
Block 1 ms-034 unit-1
Block 1 ms-034 unit-1Block 1 ms-034 unit-1
Block 1 ms-034 unit-1
 
Orthogonal array approach a case study
Orthogonal array approach   a case studyOrthogonal array approach   a case study
Orthogonal array approach a case study
 
software testing
software testingsoftware testing
software testing
 

More from Tsuyoshi Yumoto

20170704Wモデル導入の基礎-公開.pdf
20170704Wモデル導入の基礎-公開.pdf20170704Wモデル導入の基礎-公開.pdf
20170704Wモデル導入の基礎-公開.pdfTsuyoshi Yumoto
 
Gui自動テストツール基本
Gui自動テストツール基本Gui自動テストツール基本
Gui自動テストツール基本Tsuyoshi Yumoto
 
Useful software testing in the latest development – short version
Useful software testing in the latest development – short versionUseful software testing in the latest development – short version
Useful software testing in the latest development – short versionTsuyoshi Yumoto
 
Ja sst19tokyojstqb ctfl2018
Ja sst19tokyojstqb ctfl2018Ja sst19tokyojstqb ctfl2018
Ja sst19tokyojstqb ctfl2018Tsuyoshi Yumoto
 
HP_almqc_concepts20150701
HP_almqc_concepts20150701HP_almqc_concepts20150701
HP_almqc_concepts20150701Tsuyoshi Yumoto
 
ゆもつよ博士論文説明資料公開
ゆもつよ博士論文説明資料公開ゆもつよ博士論文説明資料公開
ゆもつよ博士論文説明資料公開Tsuyoshi Yumoto
 

More from Tsuyoshi Yumoto (6)

20170704Wモデル導入の基礎-公開.pdf
20170704Wモデル導入の基礎-公開.pdf20170704Wモデル導入の基礎-公開.pdf
20170704Wモデル導入の基礎-公開.pdf
 
Gui自動テストツール基本
Gui自動テストツール基本Gui自動テストツール基本
Gui自動テストツール基本
 
Useful software testing in the latest development – short version
Useful software testing in the latest development – short versionUseful software testing in the latest development – short version
Useful software testing in the latest development – short version
 
Ja sst19tokyojstqb ctfl2018
Ja sst19tokyojstqb ctfl2018Ja sst19tokyojstqb ctfl2018
Ja sst19tokyojstqb ctfl2018
 
HP_almqc_concepts20150701
HP_almqc_concepts20150701HP_almqc_concepts20150701
HP_almqc_concepts20150701
 
ゆもつよ博士論文説明資料公開
ゆもつよ博士論文説明資料公開ゆもつよ博士論文説明資料公開
ゆもつよ博士論文説明資料公開
 

Recently uploaded

The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)Roberto Bettazzoni
 
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
^Clinic ^%[+27788225528*Abortion Pills For Sale In hararekasambamuno
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024SimonedeGijt
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaNeo4j
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfICS
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Henry Schreiner
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfWSO2
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdftimtebeek1
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Andreas Granig
 
Microsoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMicrosoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMarkus Moeller
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIInflectra
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Lisi Hocke
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanNeo4j
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Andrea Goulet
 
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Flutter Agency
 
What is a Recruitment Management Software?
What is a Recruitment Management Software?What is a Recruitment Management Software?
What is a Recruitment Management Software?NYGGS Automation Suite
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AIAGATSoftware
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...naitiksharma1124
 

Recently uploaded (20)

The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)
 
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdf
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
 
Microsoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMicrosoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdf
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST API
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
 
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
 
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
 
What is a Recruitment Management Software?
What is a Recruitment Management Software?What is a Recruitment Management Software?
What is a Recruitment Management Software?
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 

A Test Analysis Method for Black Box Testing Using AUT and Fault Knowledge.

  • 2. 2 Agenda 1. Introduction 2. Issues of Test Analysis in Black Box Testing 3. An Approach of Test Analysis Based on Test Categories 4. Comparison of Test Condition Derivation Result by Test Categories Practical Use Existence 5. conclusion
  • 3. 3 1. Introduction With a rapid increase in size and complexity of software today, the scope of software testing is also expanding. Testing increaseinsize increasecomplexity Testing Development ・Requirement ・Specification ・Design ・CodeDevelopment ・Requirement ・Specification ・Design ・Code The efficiency of software testing needs to be improved in order to ensure the appropriate delivery deadline and cost of software development.
  • 4. 4 2.Issues of Test Analysis in Black Box Testing Development Lifecycle and Test Process • Software testing is performed in multiple verification levels during the development life-cycle. These verification levels are depicted in the V model. • This proposal will focus on the black box testing performed at the system-level verification, enclosed in a red frame. V model (Forsberg,1995 et.al) Testing performed at each level depicted in the V model has a process similar to the development process.
  • 5. 5 2.Issues of Test Analysis in Black Box Testing(Cont.) Test development process and Test analysis • Three activities, test analysis, test design, and test implementation in the test process are also called Test development process. Test analysis An activity of selecting and organizing items to be covered by the test .
  • 6. 6 2.Issues of Test Analysis in Black Box Testing(Cont.) Issues of Test Analysis ソフトウェア テスト ・要件定義 ・基本設計 ・詳細設計 ・コーディング ・Preventing defects in software development ・Test needs to be designed in a way that the number of test is sufficient and appropriate in quantity ・Test execution should be automated For improving efficiency of software testing . . . White box testing Analyzing the structure of AUT to cover by test Test analysis for White box testing is consistent in its refinement Black box testing Analyzing the spec of AUT to cover by test Test analysis for black box testing is often not consistent in its refinement Some of the test cases are often lacking or overlapping Types of test desgin
  • 7. 7 2.Issues of Test Analysis in Black Box Testing(Cont.) Issues of Test Analysis in Black Box testing • The test condition, which is an output of test analysis, is a generic term for elements such as functions, transactions, quality characteristics, and structure elements. • Therefore it is necessary to structure in order to sort out relationships between each element. That is also called test suite architecture. • However, in research and practice, structuring of test conditions in the test analysis stays just experiences and heuristics. Examples of Table 1 contain the following issues. • The category “Setting” appears both in Level 1 and Level 2. • The number of levels is not constant, thus there is a variance in meaning of each level. • The expected result is not indicated in the upper row. • The same specification is written on both upper and lower rows.
  • 8. 8 Structure of Test Condition • Test condition can be considered as spec item from the testing point of view. • Since black box testing is the method of the test design by external observation, selection should be start from the feature is contained in test basis • the test category, which is the classification defined using knowledge of AUT and fault, is added to the structure. The test analysis in alignment with a clear rule is attained by such a classification and arrangement. 3.An Approach of Test Analysis Based on Test Categories
  • 9. 9 3.An Approach of Test Analysis Based on Test Categories (Cont.) Test Category(1) • A set of test categories is the classification that multilaterally captures a feature to be tested without omission. An logical structure of feature (Sakuhei Omura,2005 ) An logical structure of feature is an abstract concept of a set of test categories.
  • 10. 10 3.An Approach of Test Analysis Based on Test Categories (Cont.) Test Category(2) • The test category needs to have a specific naming and a meaning specialized in the scope of testing against an abstract concept. • The knowledge of AUT and the fault experienced in the past is used for defining the test categories. Knowledge of AUT Knowledge of faults experienced in the past
  • 11. 11 3.An Approach of Test Analysis Based on Test Categories (Cont.) Listing and Organizing Test Conditions Using Test Categories • Spec items, expected results, and test parameters for testing the feature to be tested, these are selected from test basis, are listed as shown in Table below. Test categories can be used as a guide at the time of selecting spec items for covering feature to be tested. Same test categories are used for all features to be tested.
  • 12. 12 4. Comparison of Test Condition Derivation Result by Test Categories Practical Use Existence Execution Steps of Test Analysis • In order to sequentially derive test conditions, test analysis activities are divided into steps. Comparison of test condition derivation ware executed on Step3
  • 13. 13 4. Comparison of Test Condition Derivation Result by Test Categories Practical Use Existence(Cont.) Exercise Overview Feature to be tested Test categories Spec items Expected results Test Parameters Feature to be tested Spec Item Expected results Test Parameters Test Basis Results were classified to test categories Analyze Comparison Spec items Expected results Test parameters The team that use test categories The team that didn’t use test categories
  • 14. 14 4. Comparison of Test Condition Derivation Result by Test Categories Practical Use Existence(Cont.) Comparison of Spec Items Selection Percentage • The result showed the rate of selecting spec items was higher by 22% for the team that used test categories than for the team that did not use test categories. Inputadjustment Management InputA ConvA ConvB SuppA SuppB OutputA OutputB StrageA StrageB ManagA Func A 0 2 1 0 2 1 0 1 0 2 Suggested answers 9 100% 0 2 0 0 0 1 0 1 0 1 NOT using test categories 5 56% 0 2 0 0 2 1 0 1 0 1 Using test categories 7 78% Using test categories-NOT using test categories = 22% Persentage SpecitemTotal Conversion Support Strage Output adjustment Features tobe tested Test categories
  • 15. 15 4. Comparison of Test Condition Derivation Result by Test Categories Practical Use Existence(Cont.) Results of the selection of the number of expected results and test parameters • Although the ratio of the number of expect results without test categories to the number of them with test categories is 43%, the number of test parameters is 1.5 times higher. <the team that did NOT use testing categories> <the team that use testing categories> Features to be tested Test categories Spec items Number of Expected Results Number of Test parameters Features to be tested Test categories Spec items Number of Expected Results Number of Test parameters Func A InputA N/A N/A N/A Func A InputA N/A N/A N/A ConvA Spec item1 1 5(3) ConvA Spec item1 1 N/A Spec item2 N/A 1 Spec item2 1 N/A ConvB Spec item3 N/A N/A ConvB Spec item3 N/A N/A SuppA N/A N/A N/A SuppA N/A N/A N/A SuppB Spec item4 N/A N/A SuppB Spec item4 1 2 Spec item5 N/A N/A Spec item5 1 2 OutputA Spec item6 1 N/A OutputA Spec item6 1 2 OutputB N/A N/A N/A OutputB N/A N/A N/A StrageA Spec item7 1 3(1) StrageA Spec item7 1 2 StrageB N/A N/A N/A StrageB N/A N/A N/A ManagA Spec item8 N/A 1 ManagA Spec item8 1 1 Spec item9 N/A N/A Spec item9 N/A N/A Total 3 10(4) Total 7 9(0) ※() in the coloum of test parametrs is the number of test parametors for one spec item are written dispersedly throughout the test condition list. Ratio by Number of expected results : NOT using test categories / Using test categories = 43% Ratio by Number of test parameters : NOT using test categories / Using test categories = 156%
  • 16. 16 4. Comparison of Test Condition Derivation Result by Test Categories Practical Use Existence(Cont.) Sum up Feature to be tested Test categories Spec items Expected results Test Parameters Feature to be tested Spec Item Expected results Test Parameters Test Basis Results were classified to test categories exercise Comparison Spec items Expected results Test parameters The team that didn’t use test categories had a higher possibility of designing duplicated test cases at the time of a test design. • 4 of all test parameters that can possibly become test parameters for the same spec item, had addressed as another spec item, when it was 0/9 for the team that used test categories. The team that didn’t use test categories did not clearly specify the expected results The team that didn’t use test categories had a higher possibility of designing lacked test cases at the time of a test design. • The rate of selecting spec items is different more than 20%.
  • 17. 17 Conclusion • Test analysis method based on the rule by organizing classifications of test conditions, and knowledge for it is using the concept of test categories. • To avoid lacks or duplications of test cases, which often results from a test analysis and classifications of test conditions based on an individual’s own judgments. • In case of performing a test analysis using test categories… • the rate of selecting spec items had increased more than 20%. • the rate of spec items was improved when the expected results were also selected. • the variance in descriptions was alleviated.
  • 18. 18 References [1] IPA, Report of industry actual survey for Embedded Software in 2009: METI; 2009 (In Japanese). [2] Myers, Glenford J., Corey Sandler, and Tom Badgett. The art of software testing.: Wiley; 2011. [3] Forsberg, K., and H. Mooz. "The relationship of systems engineering to the project cycle. " Engineering Management Journal 4.3 ;1992 p.36-43. [4] Uetsuki Keiji, Tohru Matsuodani, and Kazuhiko Tsuda. "An efficient software testing method by decision table verification." International Journal of Computer Applications in Technology 46.1 ; 2013, p.54-64. [5] ISTQB FLWG. "Foundation Level Syllabus Version 2011. ": International Software Testing Qualifications Board ; 2011. [6] Nishi, Yasuharu. "Viewpoint-based Test Architecture Design." Software Security and Reliability Companion (SERE-C), 2012 IEEE Sixth International Conference on. IEEE; 2012, p.194- 197. [7] Kang, Kyo C., et al. "Feature-oriented domain analysis (FODA) feasibility study. No. CMU/SEI-90-TR-21. ": CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERING INST; 1990. [8] IEEE ."IEEE standard for software test documentation ", IEEE829. 2008.; 2008. [9] Tsuyoshi Yumoto,Tohru Matsuodani,Kazuhiko Tsuda. "A practical Using Method for Efficent Design of Functionai Testing": 75th National Convention of IPSJ; 2013 , paper#5B-4 (In Japanese). [10] Sakuhei Omura. Phenomenology of general system: gohodoshoppan; 2005 (In Japanese). [11] Rasiel, E. M. The McKinsey Way. McGraw-Hill; 1999. [12] Haimes, Yacov Y., Stan Kaplan, and James H. Lambert. "Risk filtering, ranking, and management framework using hierarchical holographic modeling." : Risk Analysis 22.2 ; 2002, p.383-397.

Editor's Notes

  1. 9月11日 10時30分~10時50分 18分、質問2分   10枚で1枚2分 Good morning everyone. My name is Tsuyoshi Yumoto. Today, I am going to talk about [A Test analysis Method for black box testing] And the key point of this method is using Knowledge of AUT, that means application under test, and Software Faults.
  2. This is an agenda of this presentation. First of all, I will explain the background, in short, I will talk about testing is the big challenge for software development. Second of all, I will explain about position of testing in software development process, and test process as underlying premise(パーミス). And then, I will explain about what is issues of Test Analysis in Black Box Testing. After explanation of issues, I will propose an Approach of Test Analysis Based on Test Categories. And I will talk about what is test categories and how to use it. Next, I will introduce one survey about Comparison(コンパリスン) of Test Condition Derivation(デリベイション) Result by Test Categories Practical Use Existence(イグジスタンス). And efficiency of this proposal can be confirmable from this survey result.
  3. This slide, I will tell you the situation of software development today. As I wrote in this slide, With a rapid increase in size and complexity of software today, the scope of software testing is also expanding. Actually, There is a survey that indicates 45% of development cost is spent on software testing. Therefore, the efficiency of software testing needs to be improved in order to ensure the appropriate delivery deadline and cost of software development. For improving efficiency of software testing, the test needs to be designed in a way that the number of test cases is sufficient and appropriate in quantity.
  4. Before talking about issues of software testing, I would like to explain about position of testing activity in software development process. V-model is really famous to depict it. As V model is shown, software testing is performed in multiple verification levels during the development life-cycle. This proposal will focus on the black box testing performed at the system-level verification, enclosed in a red frame. Because the system-level is where the effects caused by the increase in size and complexity of software are greatest. Testing performed at each level depicted in V model has a process similar to the development process, respectively . In the testing process, the test planning is performed in parallel with the activities on left side of the V model. Then, another activities of test process are performed in chronological(クロノロジカル) order.
  5. In addition, three activities, test analysis, test design, and test implementation in the testing process are also called test development process. Because these are main activities to derive tests for AUT. Underlying specifications to test design for black box testing are artifacts on left side of the V model. These are called test basis. When performing test design, it is necessary to refine the size of AUT . In test development process, test basis is first input, and after analyzing the test basis, test condition is going to be output. This activity is the test analysis.
  6. As everybody know, the test design can be divided into a white box testing and a black box testing. White box testing is based on structure of AUT such as the source code. On the other hand, black box testing is based on a specification that describes the behavior and operating conditions of AUT. But, test analysis for black box testing is often not consistent in its refinement. Because, the specification documents are normally written for subsequent processes. Therefore, the specification of one item that test design techniques are applied to, can be written dispersedly(ディスパーストゥリ) throughout the document. Test analysis in black box testing is an activity of selecting and organizing items from the test basis that I described. That’s why some of the test cases are often lacking or overlapping.
  7. Please look at the example table of typical test analysis. It is almost impossible to determine if there are lacks or duplications of test cases from the example. Because test condition is a generic term for elements such as functions, transactions, quality characteristics, and structure elements. So, if there is no structure in order to sort out relationships between each element, it is easy to be disorganized. In fact, in research and practice, structuring of test conditions in the test analysis stays just experiences and heuristics.
  8. In the approach of the test analysis method proposed in this paper, test condition ,as the ambiguity of the term, is eliminated(エリミネイテッド) by layering elements based on the test case structure. Using this approach means that the test analysis in alignment with a clear rule is attained by such a classification and arrangement. There are 3 key points in this structure of test condition. 1)layering elements based on the test case structure. 2)starting from the feature that is contained in test basis. 3)the test category, which is the classification defined using knowledge of AUT and fault, is added to the structure. This aims to help selecting spec items without a lack or duplication by using the test category as a guide for test analysis.
  9. In this slide, I want to explain about test category. A test category is the classification that multilaterally(モーティラレラリィ) captures a feature to be tested without omission. A feature is a conversion equipment that changes a certain input, and it adds value to the input and produces a new output. Therefore, it is supposed that it has structure logical certainly as shown in this slide. The spec items belonging to the feature for a test can be arranged to MECE based on the logical structure. Actually, the spec items of feature to be tested often distribute to whole test basis. Therefore, since it is also the cause of the omission in selection and duplication by test analysis, guiding by test categories become effective.
  10. The test category needs to have a specific naming and a meaning specialized in the scope of testing against an abstract concept. The knowledge of AUT and the fault experienced in the past is used for defining the test categories. In this slide, UI input , operation, UI display and so on in a column “test categories” are specific names.
  11. When you analyzing test basis, spec items, expected results, and test parameters, these are selected from test basis, are listed as shown in the table like this. The column headings in this table are same as structure elements shown in structure of Test Condition. Multiplicity(モーティプリシティ) is according to structure of Test Condition. This is the way to avoid disorganized listing. And, same test categories are used for all features to be tested. It is the practical way that multilaterally(モーティラレラリィ) captures a feature to be tested without omission.
  12. From now on, I would like to introduce an example to use test categories. In order to sequentially derive test conditions, test analysis activities are divided into steps shown like this. And Step3 is the target for an example. I organized the workshop on test analysis. And its participants carry out an exercise on test analysis step 3. This is for Comparison of test condition derivation between test categories in practical use and not in use.
  13. This is the Exercise overview. the participants were divided into 2 teams to carry out an exercise on test analysis step 3. One team had listed spec items, expected results, and test parameters of AUT without using test categories, and another team had listed the same with using test categories. All participants have knowledge in the theme of this exercise through past testing experiences. The team that carried out an exercise with test categories used the same set of test categories as the workshop facilitator’s. On the other hand, another team that undertook an exercise without using test categories. Then Results from this team classified to the test categories after the workshop by facilitator And then Exercise results are compared.
  14. This table in the slide shows the comparison result of the number of spec items being selected. The numbers on the first row are suggested answers provided by the workshop facilitator. The numbers on 2nd and 3rd rows are answers in the exercise by the participants during the workshop. The result showed the rate of selecting spec items was higher by 22% for the team that used test categories than for the team that did not use test categories.
  15. And then, this table in the slides shows the summary of the numbers of expected results and test parameters. The ratio of the number of expect results without test categories to the number of them with test categories is 43%, although the number of test parameters is 1.5 times higher.
  16. From these results, it is safe to sum up 3 points 1) The team that didn’t use test categories had a higher possibility of designing lacked test cases at the time of a test design. Because, the rate of selecting spec items was different more than 20%. 2) The team also did not clearly specify the expected results. Because the ratio of the number of expect results is 43% less. 3) The team also had a higher possibility of designing duplicated test cases at the time of a test design. Because 4 test parameters had addressed as another spec items, Although that can possibly become test parameters for the same spec item.
  17. So, I would like to conclude this presentation. This proposal is test analysis method based on the rule by organizing classifications of test conditions and knowledge for it is using the concept of test categories. Generally , classifications of test conditions often based on an individual’s own judgments. Disorganized classifications cause lacks or duplications of test cases. so this proposal is challenge to avoid lacks or duplications of test cases. And through the survey in the workshop, efficiency is conformed between test analysis using test categories and not using test categories.
  18. This is the end of my presentation. Thank you very much.