SlideShare a Scribd company logo
1 of 19
A Study on the Efficiency of a Test 
Analysis Method Utilizing Test-Categories 
Based on AUT and Fault Knowledge. 
15th Aug. 2014 
ICACTCM’2014 
Tsuyoshi Yumoto, Keiji Uetsuki, Tohru Matsuodani, 
Kazuhiko Tsuda
Agenda 
1. INTRODUCTION 
2. VARIABILITY OF TEST ANALYSIS RESULTS 
3. AN APPROACH OF TEST CATEGORIES BASED TESTING 
4. REMARKS FROM THE VERIFICATION EXPERIMENT 
5. CONCLUSION 
2
1. Introduction 
Along with a rapid increase in the size and complexity of software 
today, the number of required test cases is also increasing. 
3 
Testing 
Development 
・Requirement 
・Specification 
・Design 
・Code Development 
・Requirement 
・Specification 
・Design 
・Code 
increase in size 
increase complexity 
Testing 
10 times 
for 30 years 
Quantity 
FP Test cases 
Quantity 
FP Test cases 
Test cases is equals the FP(function 
point) sum to the power of 1.3 to 1.5 
(by Capers Jones
2. VARIABILITY OF TEST ANALYSIS RESULTS 
Generally, a significant number of testers should be assigned to a project in 
order to manage this increase of test cases. 
Therefore, there are no clearly defined general rules for test development 
process (see next page). 
• They are developed according to the individual’s own judgment. 
• This has the potential to cause the lacking or duplication of test cases. 
Test analysis results when rules are not defined and interpretation is left to the individual 
4
2. VARIABILITY OF TEST ANALYSIS RESULTS(Cont.) 
Definition of Test development process and Test analysis 
• Testing performed at each level depicted in the V model has a process similar to 
the development process, and three activities, test analysis, test design, and 
test implementation in the test process are called Test development process. 
5 
A part of V-model 
Test analysis 
An activity of selecting 
and organizing items to be 
covered by the test .
3. AN APPROACH OF TEST CATEGORIES BASED TESTING 
A set of rules for a test analysis method for black box testing utilizing Test- 
Categories based on the Application Under Test (AUT) and fault knowledge 
have been proposed. 
Overview of the method 
The logical structure of a feature 
Test –Category 
The procedure and document format 
6
3. AN APPROACH OF TEST CATEGORIES BASED TESTING 
The logical structure of a feature 
• It can be used to test the feature in a MECE way . Each box in the logical 
structure can be a useful guide to determine the required test conditions. 
7 
Logical Structure… a MECE way to find test conditions from features.
3. AN APPROACH OF TEST CATEGORIES BASED TESTING 
Test-Category 
• In order to have a consistent interpretation of determining test conditions, a 
8 
name specialized for the AUT is put in each box of the logical structure. 
• In order to ensure for clearly understanding the meaning of Test-Category , 
potential failures and/or faults which may arise for Test-Category are 
discussed. 
Knowledge of AUT 
Knowledge of faults 
experienced in the past 
Building consensus 
on the decided 
Test-Categories
3. AN APPROACH OF TEST CATEGORIES BASED TESTING 
The procedure and document format 
• Things used to determine test conditions 
from a test basis are defined. 
9 
The outputs are eliminated by layering elements 
based on the test case structure.
3. AN APPROACH OF TEST CATEGORIES BASED TESTING 
(Cont.) 
The Main benefit of this method 
• Higher test coverage overall, delivering higher quality testing 
– By implementing the set of rules it will be easier to determine the necessary 
10 
test conditions. 
Our hypothesis 
The following issues currently make determining the necessary test conditions difficult: 
1. Certain aspects of specification are not written if they are thought to be obvious. 
2. Specification is not completely written within single target section in a document. (for 
example: a behavior about a combination of functions) 
– When many testers are involved in test development and proceed according to 
the procedure and document format, all of the testers can carry out their work 
according to the same set of rules. 
The developed suite of test conditions are more 
comprehensive and do not contain duplicates
3. AN APPROACH OF TEST CATEGORIES BASED TESTING 
(Cont.) 
Another benefits of this method 
• High Readability 
– The procedure is based on the structure of the test conditions defined in the proposal of 
11 
this method . Elements which are included in the test basis are classified into 
specification-items, expected results, and test parameters. One-by-one, these are 
determined and selected through this procedure. Thus, the end result readability is 
improved. 
• Variation in the interpretation is minimized 
– Once a consensus regarding the building of Test-Categories is met, team members can 
more easily determine and select specification-items. 
• Repeatable 
– Since this method is structured, standardized and easy to walk though, testers will come 
up with consistent and repeatable test analysis.
4. REMARKS FROM THE VERIFICATION EXPERIMENT 
The verification experiment Overview 
Explain about 
Test category 
12 
Results of the exercise conducted 
Feature to be 
tested 
according to Test-Categories 
Test 
categories 
Spec items Expected 
results 
Test Basis 
Test 
Parameters 
Feature to be 
tested 
Spec Item Expected 
results 
Test 
Parameters 
Results 
were 
classified 
to test 
categories 
Results of the exercise conducted 
according to Test-Categories 
1st exercise 
2nd exercise 
Comparison and Evaluation for Spec items 
B - A A+
4. REMARKS FROM THE VERIFICATION EXPERIMENT 
(Cont.) 
Evaluation results of the verification experiment 
• Eight comparison results table were taken from the two verification experiments. 
• There was a measurable improvement resulting from implementing the Test- 
Categories method for seven out of eight teams. 
• However, there was no indication as to exactly which categories received the 
greatest benefit from implementing the method. 
Music reproduction equipment Flight Booking Application 
13 
Logical Team 
Structure 
Logical 
Structure 
Team 
TM1 TM2 TM3 TM4 TM5 TM6 TM1 TM2 
Conv B A B B B B Conv A A 
Input Input A B 
Output - - - - - A+ Output A A 
Storage - A+ - A+ A+ - Storage A A 
Support B B B B B B Support B A 
Mngt B A A A+ A A+ Mngt B A
4. REMARKS FROM THE VERIFICATION EXPERIMENT 
(Cont.) 
Evaluation using patterns of input and output data of AUT 
• In order to classify and evaluate the results of the verification experiments, patterns of 
inputting and outputting data into/from the AUT (I/O data patterns) is focused as a 
comprehensive AUT model. 
The total combined patterns of 
inputting and outputting data 
into/from the AUT can be 
summarized into the nine patterns 
14
4. REMARKS FROM THE VERIFICATION EXPERIMENT 
(Cont.) 
Evaluation results by I/O data patterns 
Music reproduction equipment Flight Booking Application 
15 
Logical Team Logical 
Team 
Structure 
Structure 
TM1 TM2 TM3 TM4 TM5 TM6 TM1 TM2 
Conv P1 - - - - - - Input P1 B B 
Conv P1 B A+ B B B - Support P1 - A+ 
Support P1 B B B B B B Support P1 B B 
Mngt P1 B A+ A+ A+ A+ A+ Storage P2 A+ A+ 
Storage P2 - A+ - A+ A+ - Input P4 - - 
Conv P4 B B B B - B Mngt P4 B A 
Mngt P4 B B B A+ B A+ Output P4 B B 
Output P7 - - - - - A+ Output P4 A A 
Input P7 A - 
Conv P7 A A 
Output P7 B B 
Summarizing results in the I/O data pattern 
P2 is 100% for both the 
experiments. It can be said 
that P2 has had the most 
significant result. 
There were no detailed 
description about all spec items 
for P2 in the test basis. 
These results may suggest that 
the results are aligned with our 
assumption in Section 3. 
Music reproduction equipment Flight Application 
Evaluation level I/O Evaluation level 
pattern 
A+ A - B A+ A - B 
I/O 
pattern 
P1 6 0 7 11 35.3% P1 1 0 1 4 20.0% 
P2 3 0 3 0 100.0% P2 2 0 0 0 100.0% 
P4 2 0 1 9 18.2% P4 0 3 2 3 50.0% 
P7 1 0 5 0 100.0% P7 0 3 1 2 60.0% 
the persentage = (A+ + A) /(A+ + A + B)
Conclusion 
• Through these verification experiments it has been observed that after briefly 
16 
explaining this proposed method to participants there was a measurable 
improvement in quantity and consistency of spec-item which they were able to 
determine. 
• Moreover, by analyzing I/O data patterns, the patterns of inputting and outputting 
data into/from the AUT, a part of the evaluation results from the verification 
experiments aligned with the assumption in the proposed method. 
• Further verification experiments are necessary in order to carry out trend analysis 
with higher accuracy. 
• Conducting further experiments and deepening our understanding of the 
tendencies and factors relating to effectiveness of the proposed method, rules for 
creating Test-Categories based on the AUT knowledge and fault knowledge can be 
more refined.
Thanks
References 
[1] C.Jones,"Estimating Software Costs 2nd Edition": McGraw-Hill, 2007. 
[2] D.Longstreet, "Productivity of Software from 1970 to Present", 
http://www.softwaremetrics.com/Articles/history.htm, 2000 
[3] C.Ebert, and C.Jones. "Embedded Software: Facts, Figures, and Future." IEEE Computer 
42.4 ,2009, pp 42-52. 
[4] N.Owada"System integration 'straightforward way'",Nikkei Business Publications, 
Inc,2009 (In Japanese). 
[5] S. Eldh, H.Hansson, and S.Punnekkat, “Analysis of mistakes as a method to improve 
test case design.” In Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth 
International Conference on.IEEE,2011,pp. 70-79 
[6] T.Yumoto, T.Matsuodani, and K.Tsuda. "A Test Analysis Method for Black Box Testing 
Using AUT and Fault Knowledge.": Procedia Computer Science 22 ,2013, pp.551-560. 
[7] ISTQB FLWG, "Foundation Level Syllabus Version 2011": International Software Testing 
Qualifications Board , 2011. 
[8] T.J.Ostrand, and M.J.Balcer, "The category-partition method for specifying and 
generating fuctional tests.":Communications of the ACM 31.6 ,1988,pp. 676-686. 
[9] M.Grindal, and J.Offutt, "Input parameter modeling for combination strategies.": 
IASTED International Conference on Software Engineering (SE 2007), 2007. 
[10] G.J.Myers, C.Sandler, and T.Badgett, "The art of software testing ": Wiley, 2011. 
18
References (cont.) 
[11] B.Beizer, "Software testing techniques ": Dreamtech Press, 2003. 
[12] Y.Nishi, "Viewpoint-based Test Architecture Design.": Software Security and Reliability 
Companion (SERE-C), 2012 IEEE Sixth International Conference on. IEEE (2012), pp.194- 
197. 
[13] K.Akiyama, T.Takagi, and Z.Furukawa, "Development and Evaluation of HAYST Method 
Tool (Software Testing)." : SoMeT. ,2010, pp.398-414 
[14] S.Omura, "Phenomenology of general system": gohodoshoppan, 2005 (In Japanese). 
[15] "IEEE standard for software test documentation ": IEEE829. 2002,2002 
[16] Rasiel, E. M. The McKinsey Way. McGraw-Hill; 1999. 
[17] Tsuyoshi Yumoto,Tohru Matsuodani,Kazuhiko Tsuda. "A practical Using Method for 
Efficent Design of Functionai Testing": 75th National Convention of IPSJ; 2013 , paper#5B-4 
(In Japanese). 
[18] J.A.Whittaker, "How to break software ": Addison Wesley, 2003. 
19

More Related Content

What's hot

Input Space Partitioning
Input Space PartitioningInput Space Partitioning
Input Space Partitioning
Riyad Parvez
 
Introduction to specification based test design techniques
Introduction to specification based test design techniquesIntroduction to specification based test design techniques
Introduction to specification based test design techniques
Yogindernath Gupta
 
<p>Software Testing</p>
<p>Software Testing</p><p>Software Testing</p>
<p>Software Testing</p>
Atul Mishra
 
Testing survey by_directions
Testing survey by_directionsTesting survey by_directions
Testing survey by_directions
Tao He
 
Test Case Design
Test Case DesignTest Case Design
Test Case Design
acatalin
 
Software Testing Foundations Part 6 - Intuitive and Experience-based testing
Software Testing Foundations Part 6 - Intuitive and Experience-based testingSoftware Testing Foundations Part 6 - Intuitive and Experience-based testing
Software Testing Foundations Part 6 - Intuitive and Experience-based testing
Nikita Knysh
 

What's hot (20)

Input Space Partitioning
Input Space PartitioningInput Space Partitioning
Input Space Partitioning
 
Fundamental Test Design Techniques
Fundamental Test Design TechniquesFundamental Test Design Techniques
Fundamental Test Design Techniques
 
Introduction to specification based test design techniques
Introduction to specification based test design techniquesIntroduction to specification based test design techniques
Introduction to specification based test design techniques
 
<p>Software Testing</p>
<p>Software Testing</p><p>Software Testing</p>
<p>Software Testing</p>
 
Test Case Design and Technique
Test Case Design and TechniqueTest Case Design and Technique
Test Case Design and Technique
 
Software Testing Techniques
Software Testing TechniquesSoftware Testing Techniques
Software Testing Techniques
 
Code coverage based test case selection and prioritization
Code coverage based test case selection and prioritizationCode coverage based test case selection and prioritization
Code coverage based test case selection and prioritization
 
Software Testing Techniques
Software Testing TechniquesSoftware Testing Techniques
Software Testing Techniques
 
EXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODEL
EXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODELEXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODEL
EXTRACTING THE MINIMIZED TEST SUITE FOR REVISED SIMULINK/STATEFLOW MODEL
 
Testing survey by_directions
Testing survey by_directionsTesting survey by_directions
Testing survey by_directions
 
Black & White Box testing
Black & White Box testingBlack & White Box testing
Black & White Box testing
 
Software Quality Testing
Software Quality TestingSoftware Quality Testing
Software Quality Testing
 
IP 200 Introduction
IP 200 IntroductionIP 200 Introduction
IP 200 Introduction
 
Test Case Design
Test Case DesignTest Case Design
Test Case Design
 
Boundary and equivalnce systematic test design
Boundary and equivalnce   systematic test designBoundary and equivalnce   systematic test design
Boundary and equivalnce systematic test design
 
Black boxtestingmethodsforsoftwarecomponents
Black boxtestingmethodsforsoftwarecomponentsBlack boxtestingmethodsforsoftwarecomponents
Black boxtestingmethodsforsoftwarecomponents
 
White Box Testing
White Box TestingWhite Box Testing
White Box Testing
 
Sta unit 3(abimanyu)
Sta unit 3(abimanyu)Sta unit 3(abimanyu)
Sta unit 3(abimanyu)
 
Software Testing Foundations Part 6 - Intuitive and Experience-based testing
Software Testing Foundations Part 6 - Intuitive and Experience-based testingSoftware Testing Foundations Part 6 - Intuitive and Experience-based testing
Software Testing Foundations Part 6 - Intuitive and Experience-based testing
 
Sta unit 4(abimanyu)
Sta unit 4(abimanyu)Sta unit 4(abimanyu)
Sta unit 4(abimanyu)
 

Viewers also liked

WACATE 2010夏 ゆもつよ講演スライド
WACATE 2010夏 ゆもつよ講演スライドWACATE 2010夏 ゆもつよ講演スライド
WACATE 2010夏 ゆもつよ講演スライド
Tsuyoshi Yumoto
 
20150424 jasst新潟基調講演
20150424 jasst新潟基調講演20150424 jasst新潟基調講演
20150424 jasst新潟基調講演
Kouichi Akiyama
 
英語ⅳ Jiro
英語ⅳ Jiro英語ⅳ Jiro
英語ⅳ Jiro
c1411046
 
Perfect Holiday Gift for the Vinyl Lover
Perfect Holiday Gift for the Vinyl LoverPerfect Holiday Gift for the Vinyl Lover
Perfect Holiday Gift for the Vinyl Lover
C.r. Louisville
 
Religion and warfare
Religion and warfareReligion and warfare
Religion and warfare
achandravc
 
Almentariak hodei ehiztari
Almentariak hodei ehiztariAlmentariak hodei ehiztari
Almentariak hodei ehiztari
Almentariak
 

Viewers also liked (20)

とてか03「「いかす!」のために大事だと思う4つのこと」
とてか03「「いかす!」のために大事だと思う4つのこと」とてか03「「いかす!」のために大事だと思う4つのこと」
とてか03「「いかす!」のために大事だと思う4つのこと」
 
WACATE 2010夏 ゆもつよ講演スライド
WACATE 2010夏 ゆもつよ講演スライドWACATE 2010夏 ゆもつよ講演スライド
WACATE 2010夏 ゆもつよ講演スライド
 
テスト分析についての説明資料公開用
テスト分析についての説明資料公開用テスト分析についての説明資料公開用
テスト分析についての説明資料公開用
 
20150424 jasst新潟基調講演
20150424 jasst新潟基調講演20150424 jasst新潟基調講演
20150424 jasst新潟基調講演
 
20160619 wacate 解答
20160619 wacate   解答20160619 wacate   解答
20160619 wacate 解答
 
20170203 test analysisdesign
20170203 test analysisdesign20170203 test analysisdesign
20170203 test analysisdesign
 
Toteka 04
Toteka 04Toteka 04
Toteka 04
 
テスト自動化とアーキテクチャ
テスト自動化とアーキテクチャテスト自動化とアーキテクチャ
テスト自動化とアーキテクチャ
 
英語ⅳ Jiro
英語ⅳ Jiro英語ⅳ Jiro
英語ⅳ Jiro
 
H0425066
H0425066H0425066
H0425066
 
How To Create Your Own Info Product
How To Create Your Own Info ProductHow To Create Your Own Info Product
How To Create Your Own Info Product
 
Perfect Holiday Gift for the Vinyl Lover
Perfect Holiday Gift for the Vinyl LoverPerfect Holiday Gift for the Vinyl Lover
Perfect Holiday Gift for the Vinyl Lover
 
K0216571
K0216571K0216571
K0216571
 
Abu
AbuAbu
Abu
 
A0940111
A0940111A0940111
A0940111
 
Proposal Technique for an Static Var Compensator
Proposal Technique for an Static Var CompensatorProposal Technique for an Static Var Compensator
Proposal Technique for an Static Var Compensator
 
Focused Exploration of Geospatial Context on Linked Open Data
Focused Exploration of Geospatial Context on Linked Open DataFocused Exploration of Geospatial Context on Linked Open Data
Focused Exploration of Geospatial Context on Linked Open Data
 
History platform
History platformHistory platform
History platform
 
Religion and warfare
Religion and warfareReligion and warfare
Religion and warfare
 
Almentariak hodei ehiztari
Almentariak hodei ehiztariAlmentariak hodei ehiztari
Almentariak hodei ehiztari
 

Similar to A study on the efficiency of a test analysis method utilizing test-categories based on AUT and fault knowledge

Test Cases Maintaining & Documenting
Test Cases Maintaining & DocumentingTest Cases Maintaining & Documenting
Test Cases Maintaining & Documenting
Seyed Ali Marjaie
 
ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2
Yogindernath Gupta
 
Testing documents
Testing documentsTesting documents
Testing documents
suhasreddy1
 

Similar to A study on the efficiency of a test analysis method utilizing test-categories based on AUT and fault knowledge (20)

Qa documentation pp
Qa documentation ppQa documentation pp
Qa documentation pp
 
Introduction to testing.
Introduction to testing.Introduction to testing.
Introduction to testing.
 
ISTQB, ISEB Lecture Notes- 2
ISTQB, ISEB Lecture Notes- 2ISTQB, ISEB Lecture Notes- 2
ISTQB, ISEB Lecture Notes- 2
 
Chapter 1 Fundamental of Testing
Chapter 1 Fundamental of TestingChapter 1 Fundamental of Testing
Chapter 1 Fundamental of Testing
 
A NOVEL APPROACH FOR TEST CASEPRIORITIZATION
A NOVEL APPROACH FOR TEST CASEPRIORITIZATIONA NOVEL APPROACH FOR TEST CASEPRIORITIZATION
A NOVEL APPROACH FOR TEST CASEPRIORITIZATION
 
Test Cases Maintaining & Documenting
Test Cases Maintaining & DocumentingTest Cases Maintaining & Documenting
Test Cases Maintaining & Documenting
 
Bab i fundamental of testing
Bab i fundamental of testingBab i fundamental of testing
Bab i fundamental of testing
 
Bab i fundamental of testing
Bab i fundamental of testingBab i fundamental of testing
Bab i fundamental of testing
 
Bab i fundamental of testing (yoga)
Bab i fundamental of testing (yoga)Bab i fundamental of testing (yoga)
Bab i fundamental of testing (yoga)
 
Software testing life cycle
Software testing life cycleSoftware testing life cycle
Software testing life cycle
 
Software Testing Process
Software Testing ProcessSoftware Testing Process
Software Testing Process
 
Testing process
Testing processTesting process
Testing process
 
software testing unit 3 notes anna university 2017
software testing unit 3 notes anna university 2017software testing unit 3 notes anna university 2017
software testing unit 3 notes anna university 2017
 
ProjectReport_SPCinAM
ProjectReport_SPCinAMProjectReport_SPCinAM
ProjectReport_SPCinAM
 
ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2
 
Calibration and Validation of Micro-Simulation Models
Calibration and Validation of Micro-Simulation ModelsCalibration and Validation of Micro-Simulation Models
Calibration and Validation of Micro-Simulation Models
 
Hydraulics Team Full-Technical Lab Report
Hydraulics Team Full-Technical Lab ReportHydraulics Team Full-Technical Lab Report
Hydraulics Team Full-Technical Lab Report
 
Testing documents
Testing documentsTesting documents
Testing documents
 
Software engineering Testing technique,test case,test suit design
Software engineering Testing technique,test case,test suit designSoftware engineering Testing technique,test case,test suit design
Software engineering Testing technique,test case,test suit design
 
ISTQB Foundation - Chapter 2
ISTQB Foundation - Chapter 2ISTQB Foundation - Chapter 2
ISTQB Foundation - Chapter 2
 

More from Tsuyoshi Yumoto (6)

20170704Wモデル導入の基礎-公開.pdf
20170704Wモデル導入の基礎-公開.pdf20170704Wモデル導入の基礎-公開.pdf
20170704Wモデル導入の基礎-公開.pdf
 
Gui自動テストツール基本
Gui自動テストツール基本Gui自動テストツール基本
Gui自動テストツール基本
 
Useful software testing in the latest development – short version
Useful software testing in the latest development – short versionUseful software testing in the latest development – short version
Useful software testing in the latest development – short version
 
Ja sst19tokyojstqb ctfl2018
Ja sst19tokyojstqb ctfl2018Ja sst19tokyojstqb ctfl2018
Ja sst19tokyojstqb ctfl2018
 
HP_almqc_concepts20150701
HP_almqc_concepts20150701HP_almqc_concepts20150701
HP_almqc_concepts20150701
 
ゆもつよ博士論文説明資料公開
ゆもつよ博士論文説明資料公開ゆもつよ博士論文説明資料公開
ゆもつよ博士論文説明資料公開
 

Recently uploaded

Final DBMS Manual (2).pdf final lab manual
Final DBMS Manual (2).pdf final lab manualFinal DBMS Manual (2).pdf final lab manual
Final DBMS Manual (2).pdf final lab manual
BalamuruganV28
 
Maher Othman Interior Design Portfolio..
Maher Othman Interior Design Portfolio..Maher Othman Interior Design Portfolio..
Maher Othman Interior Design Portfolio..
MaherOthman7
 
Online crime reporting system project.pdf
Online crime reporting system project.pdfOnline crime reporting system project.pdf
Online crime reporting system project.pdf
Kamal Acharya
 

Recently uploaded (20)

Final DBMS Manual (2).pdf final lab manual
Final DBMS Manual (2).pdf final lab manualFinal DBMS Manual (2).pdf final lab manual
Final DBMS Manual (2).pdf final lab manual
 
Operating System chapter 9 (Virtual Memory)
Operating System chapter 9 (Virtual Memory)Operating System chapter 9 (Virtual Memory)
Operating System chapter 9 (Virtual Memory)
 
Fabrication Of Automatic Star Delta Starter Using Relay And GSM Module By Utk...
Fabrication Of Automatic Star Delta Starter Using Relay And GSM Module By Utk...Fabrication Of Automatic Star Delta Starter Using Relay And GSM Module By Utk...
Fabrication Of Automatic Star Delta Starter Using Relay And GSM Module By Utk...
 
RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5
RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5
RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5
 
Supermarket billing system project report..pdf
Supermarket billing system project report..pdfSupermarket billing system project report..pdf
Supermarket billing system project report..pdf
 
Lab Manual Arduino UNO Microcontrollar.docx
Lab Manual Arduino UNO Microcontrollar.docxLab Manual Arduino UNO Microcontrollar.docx
Lab Manual Arduino UNO Microcontrollar.docx
 
Seismic Hazard Assessment Software in Python by Prof. Dr. Costas Sachpazis
Seismic Hazard Assessment Software in Python by Prof. Dr. Costas SachpazisSeismic Hazard Assessment Software in Python by Prof. Dr. Costas Sachpazis
Seismic Hazard Assessment Software in Python by Prof. Dr. Costas Sachpazis
 
The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...
The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...
The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...
 
SLIDESHARE PPT-DECISION MAKING METHODS.pptx
SLIDESHARE PPT-DECISION MAKING METHODS.pptxSLIDESHARE PPT-DECISION MAKING METHODS.pptx
SLIDESHARE PPT-DECISION MAKING METHODS.pptx
 
Research Methodolgy & Intellectual Property Rights Series 2
Research Methodolgy & Intellectual Property Rights Series 2Research Methodolgy & Intellectual Property Rights Series 2
Research Methodolgy & Intellectual Property Rights Series 2
 
Intelligent Agents, A discovery on How A Rational Agent Acts
Intelligent Agents, A discovery on How A Rational Agent ActsIntelligent Agents, A discovery on How A Rational Agent Acts
Intelligent Agents, A discovery on How A Rational Agent Acts
 
Maher Othman Interior Design Portfolio..
Maher Othman Interior Design Portfolio..Maher Othman Interior Design Portfolio..
Maher Othman Interior Design Portfolio..
 
Electrical shop management system project report.pdf
Electrical shop management system project report.pdfElectrical shop management system project report.pdf
Electrical shop management system project report.pdf
 
"United Nations Park" Site Visit Report.
"United Nations Park" Site  Visit Report."United Nations Park" Site  Visit Report.
"United Nations Park" Site Visit Report.
 
ChatGPT Prompt Engineering for project managers.pdf
ChatGPT Prompt Engineering for project managers.pdfChatGPT Prompt Engineering for project managers.pdf
ChatGPT Prompt Engineering for project managers.pdf
 
Introduction to Arduino Programming: Features of Arduino
Introduction to Arduino Programming: Features of ArduinoIntroduction to Arduino Programming: Features of Arduino
Introduction to Arduino Programming: Features of Arduino
 
Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1
 
Lesson no16 application of Induction Generator in Wind.ppsx
Lesson no16 application of Induction Generator in Wind.ppsxLesson no16 application of Induction Generator in Wind.ppsx
Lesson no16 application of Induction Generator in Wind.ppsx
 
Online crime reporting system project.pdf
Online crime reporting system project.pdfOnline crime reporting system project.pdf
Online crime reporting system project.pdf
 
BURGER ORDERING SYSYTEM PROJECT REPORT..pdf
BURGER ORDERING SYSYTEM PROJECT REPORT..pdfBURGER ORDERING SYSYTEM PROJECT REPORT..pdf
BURGER ORDERING SYSYTEM PROJECT REPORT..pdf
 

A study on the efficiency of a test analysis method utilizing test-categories based on AUT and fault knowledge

  • 1. A Study on the Efficiency of a Test Analysis Method Utilizing Test-Categories Based on AUT and Fault Knowledge. 15th Aug. 2014 ICACTCM’2014 Tsuyoshi Yumoto, Keiji Uetsuki, Tohru Matsuodani, Kazuhiko Tsuda
  • 2. Agenda 1. INTRODUCTION 2. VARIABILITY OF TEST ANALYSIS RESULTS 3. AN APPROACH OF TEST CATEGORIES BASED TESTING 4. REMARKS FROM THE VERIFICATION EXPERIMENT 5. CONCLUSION 2
  • 3. 1. Introduction Along with a rapid increase in the size and complexity of software today, the number of required test cases is also increasing. 3 Testing Development ・Requirement ・Specification ・Design ・Code Development ・Requirement ・Specification ・Design ・Code increase in size increase complexity Testing 10 times for 30 years Quantity FP Test cases Quantity FP Test cases Test cases is equals the FP(function point) sum to the power of 1.3 to 1.5 (by Capers Jones
  • 4. 2. VARIABILITY OF TEST ANALYSIS RESULTS Generally, a significant number of testers should be assigned to a project in order to manage this increase of test cases. Therefore, there are no clearly defined general rules for test development process (see next page). • They are developed according to the individual’s own judgment. • This has the potential to cause the lacking or duplication of test cases. Test analysis results when rules are not defined and interpretation is left to the individual 4
  • 5. 2. VARIABILITY OF TEST ANALYSIS RESULTS(Cont.) Definition of Test development process and Test analysis • Testing performed at each level depicted in the V model has a process similar to the development process, and three activities, test analysis, test design, and test implementation in the test process are called Test development process. 5 A part of V-model Test analysis An activity of selecting and organizing items to be covered by the test .
  • 6. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING A set of rules for a test analysis method for black box testing utilizing Test- Categories based on the Application Under Test (AUT) and fault knowledge have been proposed. Overview of the method The logical structure of a feature Test –Category The procedure and document format 6
  • 7. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING The logical structure of a feature • It can be used to test the feature in a MECE way . Each box in the logical structure can be a useful guide to determine the required test conditions. 7 Logical Structure… a MECE way to find test conditions from features.
  • 8. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING Test-Category • In order to have a consistent interpretation of determining test conditions, a 8 name specialized for the AUT is put in each box of the logical structure. • In order to ensure for clearly understanding the meaning of Test-Category , potential failures and/or faults which may arise for Test-Category are discussed. Knowledge of AUT Knowledge of faults experienced in the past Building consensus on the decided Test-Categories
  • 9. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING The procedure and document format • Things used to determine test conditions from a test basis are defined. 9 The outputs are eliminated by layering elements based on the test case structure.
  • 10. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING (Cont.) The Main benefit of this method • Higher test coverage overall, delivering higher quality testing – By implementing the set of rules it will be easier to determine the necessary 10 test conditions. Our hypothesis The following issues currently make determining the necessary test conditions difficult: 1. Certain aspects of specification are not written if they are thought to be obvious. 2. Specification is not completely written within single target section in a document. (for example: a behavior about a combination of functions) – When many testers are involved in test development and proceed according to the procedure and document format, all of the testers can carry out their work according to the same set of rules. The developed suite of test conditions are more comprehensive and do not contain duplicates
  • 11. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING (Cont.) Another benefits of this method • High Readability – The procedure is based on the structure of the test conditions defined in the proposal of 11 this method . Elements which are included in the test basis are classified into specification-items, expected results, and test parameters. One-by-one, these are determined and selected through this procedure. Thus, the end result readability is improved. • Variation in the interpretation is minimized – Once a consensus regarding the building of Test-Categories is met, team members can more easily determine and select specification-items. • Repeatable – Since this method is structured, standardized and easy to walk though, testers will come up with consistent and repeatable test analysis.
  • 12. 4. REMARKS FROM THE VERIFICATION EXPERIMENT The verification experiment Overview Explain about Test category 12 Results of the exercise conducted Feature to be tested according to Test-Categories Test categories Spec items Expected results Test Basis Test Parameters Feature to be tested Spec Item Expected results Test Parameters Results were classified to test categories Results of the exercise conducted according to Test-Categories 1st exercise 2nd exercise Comparison and Evaluation for Spec items B - A A+
  • 13. 4. REMARKS FROM THE VERIFICATION EXPERIMENT (Cont.) Evaluation results of the verification experiment • Eight comparison results table were taken from the two verification experiments. • There was a measurable improvement resulting from implementing the Test- Categories method for seven out of eight teams. • However, there was no indication as to exactly which categories received the greatest benefit from implementing the method. Music reproduction equipment Flight Booking Application 13 Logical Team Structure Logical Structure Team TM1 TM2 TM3 TM4 TM5 TM6 TM1 TM2 Conv B A B B B B Conv A A Input Input A B Output - - - - - A+ Output A A Storage - A+ - A+ A+ - Storage A A Support B B B B B B Support B A Mngt B A A A+ A A+ Mngt B A
  • 14. 4. REMARKS FROM THE VERIFICATION EXPERIMENT (Cont.) Evaluation using patterns of input and output data of AUT • In order to classify and evaluate the results of the verification experiments, patterns of inputting and outputting data into/from the AUT (I/O data patterns) is focused as a comprehensive AUT model. The total combined patterns of inputting and outputting data into/from the AUT can be summarized into the nine patterns 14
  • 15. 4. REMARKS FROM THE VERIFICATION EXPERIMENT (Cont.) Evaluation results by I/O data patterns Music reproduction equipment Flight Booking Application 15 Logical Team Logical Team Structure Structure TM1 TM2 TM3 TM4 TM5 TM6 TM1 TM2 Conv P1 - - - - - - Input P1 B B Conv P1 B A+ B B B - Support P1 - A+ Support P1 B B B B B B Support P1 B B Mngt P1 B A+ A+ A+ A+ A+ Storage P2 A+ A+ Storage P2 - A+ - A+ A+ - Input P4 - - Conv P4 B B B B - B Mngt P4 B A Mngt P4 B B B A+ B A+ Output P4 B B Output P7 - - - - - A+ Output P4 A A Input P7 A - Conv P7 A A Output P7 B B Summarizing results in the I/O data pattern P2 is 100% for both the experiments. It can be said that P2 has had the most significant result. There were no detailed description about all spec items for P2 in the test basis. These results may suggest that the results are aligned with our assumption in Section 3. Music reproduction equipment Flight Application Evaluation level I/O Evaluation level pattern A+ A - B A+ A - B I/O pattern P1 6 0 7 11 35.3% P1 1 0 1 4 20.0% P2 3 0 3 0 100.0% P2 2 0 0 0 100.0% P4 2 0 1 9 18.2% P4 0 3 2 3 50.0% P7 1 0 5 0 100.0% P7 0 3 1 2 60.0% the persentage = (A+ + A) /(A+ + A + B)
  • 16. Conclusion • Through these verification experiments it has been observed that after briefly 16 explaining this proposed method to participants there was a measurable improvement in quantity and consistency of spec-item which they were able to determine. • Moreover, by analyzing I/O data patterns, the patterns of inputting and outputting data into/from the AUT, a part of the evaluation results from the verification experiments aligned with the assumption in the proposed method. • Further verification experiments are necessary in order to carry out trend analysis with higher accuracy. • Conducting further experiments and deepening our understanding of the tendencies and factors relating to effectiveness of the proposed method, rules for creating Test-Categories based on the AUT knowledge and fault knowledge can be more refined.
  • 18. References [1] C.Jones,"Estimating Software Costs 2nd Edition": McGraw-Hill, 2007. [2] D.Longstreet, "Productivity of Software from 1970 to Present", http://www.softwaremetrics.com/Articles/history.htm, 2000 [3] C.Ebert, and C.Jones. "Embedded Software: Facts, Figures, and Future." IEEE Computer 42.4 ,2009, pp 42-52. [4] N.Owada"System integration 'straightforward way'",Nikkei Business Publications, Inc,2009 (In Japanese). [5] S. Eldh, H.Hansson, and S.Punnekkat, “Analysis of mistakes as a method to improve test case design.” In Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference on.IEEE,2011,pp. 70-79 [6] T.Yumoto, T.Matsuodani, and K.Tsuda. "A Test Analysis Method for Black Box Testing Using AUT and Fault Knowledge.": Procedia Computer Science 22 ,2013, pp.551-560. [7] ISTQB FLWG, "Foundation Level Syllabus Version 2011": International Software Testing Qualifications Board , 2011. [8] T.J.Ostrand, and M.J.Balcer, "The category-partition method for specifying and generating fuctional tests.":Communications of the ACM 31.6 ,1988,pp. 676-686. [9] M.Grindal, and J.Offutt, "Input parameter modeling for combination strategies.": IASTED International Conference on Software Engineering (SE 2007), 2007. [10] G.J.Myers, C.Sandler, and T.Badgett, "The art of software testing ": Wiley, 2011. 18
  • 19. References (cont.) [11] B.Beizer, "Software testing techniques ": Dreamtech Press, 2003. [12] Y.Nishi, "Viewpoint-based Test Architecture Design.": Software Security and Reliability Companion (SERE-C), 2012 IEEE Sixth International Conference on. IEEE (2012), pp.194- 197. [13] K.Akiyama, T.Takagi, and Z.Furukawa, "Development and Evaluation of HAYST Method Tool (Software Testing)." : SoMeT. ,2010, pp.398-414 [14] S.Omura, "Phenomenology of general system": gohodoshoppan, 2005 (In Japanese). [15] "IEEE standard for software test documentation ": IEEE829. 2002,2002 [16] Rasiel, E. M. The McKinsey Way. McGraw-Hill; 1999. [17] Tsuyoshi Yumoto,Tohru Matsuodani,Kazuhiko Tsuda. "A practical Using Method for Efficent Design of Functionai Testing": 75th National Convention of IPSJ; 2013 , paper#5B-4 (In Japanese). [18] J.A.Whittaker, "How to break software ": Addison Wesley, 2003. 19

Editor's Notes

  1. 8月15日 11時15分~11時30分 12分、質問2分   12枚1分づつ Good morning everyone. My name is Tsuyoshi Yumoto. Today, I am going to talk about [A Study on the Efficiency of a Test Analysis Method that I have proposed. ]
  2. This is an agenda of this presentation. First of all, I will explain the background in section 1 and 2. In short, I will talk about testing is the big challenge for software development Actually, to solve these issues, I have proposed a method of Test Analysis Based on Test Categories. So, I will talk about what is test categories and how to use it in section 3 And then, this is the main topic of this paper, I will introduce remarks from the verification experiments for confirming the efficiency of this test analysis method.
  3. This slide, I will tell you the situation of software development today. As I wrote in this slide, a set of test cases that required for software development is really increasing. Actually, there is a survey that test cases is equals the function point sum to the power of 1.3 to 1.5. That is why, the increase and complexity of software gives big impact to the quantity of test cases, and a significant number of testers should be assigned to a project in order to manage this increase of test cases.
  4. Although the majority of testers are involved in the test development activity , we can see many software development projects that there are no clearly defined general rules for test development. In these cases, testers are developed test cases according to the individual’s own judgment. This has the potential to cause the lacking or duplication of test cases.
  5. By the way, I would like to tell you a definition of test development process and test analysis Normally, testing has a process similar to the development process. In the testing process, the test planning is performed in parallel with the activities on left side of the V model. Then, another activities of test process are performed in chronological(クロノロジカル) order. In addition, three activities, test analysis, test design, and test implementation in the testing process are also called test development process. Because these are main activities to derive tests for AUT. Underlying specifications to test design are artifacts on left side of the V model. These are called test basis. When performing test development, it is necessary to refine the size of AUT at first. In test development process, test basis is first input, and after analyzing and refining the test basis, and the determine what we should test. That is called test condition. test condition is going to be output of the activity. This activity is the test analysis.
  6. To help solving this issue, I have proposed a set of rules for a test analysis method for black box testing utilizing Test-Categories based on the Application Under Test (AUT) and fault knowledge. An overview of this test analysis method can be summarize three points.
  7. One is, it is used the logical structure of a feature for determine test conditions. This is a reference model to see a feature in a MECE way . Each box in the logical structure can be a useful guide to determine required test conditions
  8. Two is test category. In order to have a consistent interpretation of determining test conditions using the logical structure as a guide, a name specialized for the AUT is put in each box of the logical structure of a feature. This is a test category. And there is an activity in the procedure that potential failures and/or faults which may arise for Test-Category are discussed for building consensus among the testers of the project.
  9. Three is, it is defined the procedure and document formats for test analysis activity.
  10. We assume 2 changes by implementing the set of rules from the method. One is that it will be easier to determine the necessary test conditions. Two is that all of the testers can carry out their work according to the same set of rules. As a result, the developed suite of test conditions are more comprehensive and do not contain duplicates. This ensures higher test coverage overall, delivering higher quality testing. This is the main benefit of this method.
  11. There are also another benefits to use this method. One is High Readability. Two is minimized variation in the interpretation for test analyzing. three is improving to the repeatable process. Every benefits are good for a project that has a lot of testers .
  12. We have done two verification experiments to know the efficiency of test-category-based testing. This is the exercise overview in the verification experiment.
  13. We have got 8 comparison results from the two verification experiments. And there was a measurable improvement from implementation the test category method for seven out of eight teams. But, there was no indication as to exactly which categories received the greatest benefit from implementing the test category method.
  14. So, in order to classify and evaluate the results of the verification experiments as another way, We try to focus patterns of inputting and outputting data into/from the AUT as a comprehensive AUT model. Because testing is the activity to do input data to AUT and get output data from AUT. It can be describe testing itself.
  15. From this way , it is safe to sum up a few points. P2 of patterns of inputting and outputting data into/from the AUT is 100% improving for both the experiments. It can be said that P2 has had the most significant result. And there were no detailed description about all spec items for P2 in the test basis. These results may suggest that the results are aligned with our assumption in Section 3.
  16. So, I would like to conclude this presentation. We can observe a measurable improvement through the verification experiments. Besides this, the patterns of inputting and outputting data into or from the AUT, a part of the evaluation results from the verification experiments aligned with the assumption in the method that I proposed. I think conducting further experiments and deepening our understanding of the tendencies and factors relating to effectiveness of the proposed method, rules for creating Test-Categories can be more refined.
  17. This is the end of my presentation. Thank you very much.
  18. This is the end of my presentation. Thank you very much.
  19. This is the end of my presentation. Thank you very much.