Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to Learn The History of Software Testing


Published on

Published in: Software
  • Thanks for a very interesting presentation,
    Now we need to add recent times changes of Context Driven, Exploratory, 'Test is Dead' era, etc. :-)
    And the big question of our times is: Does SW history teach us that the separation of development from testing is necessary, or are the new profits of Agile right and same person should do it all...?
    @halperinko - Kobi Halperin
    Are you sure you want to  Yes  No
    Your message goes here

How to Learn The History of Software Testing

  1. 1. HISTORYOFSOFTWARETESTING How to Learn The History of Software Testing Keizo Tatsumi 2010-12-19 Translated into English on July 2014 If it wasn't useful, it wouldn't be learned. If it couldn't ever be interesting, it wouldn't deserve to be learned. WACATE 2010 Winter
  2. 2. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Agenda  Prehistory  The History of Computers and Software Engineering  The Growth of Software Testing  The History of Testing Techniques  The History of Testing in Japan  The Forefront of Software Testing Research  Concluding Remarks (C) K. Tatsumi 20142
  3. 3. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Prehistory  The First Computer  Charles Babbage’s Analytical Engine • Conceived in 1837 (unfinished) • The input via punched cards • The output via a printer, a curve plotter and a bell  The First Programmer  Ada Byron, Lady Lovelace • Worked on the Analytical Engine (1843) – Ada’s notes on the Analytical Engine were recognized as a description of a computer and software (1/2) (C) K. Tatsumi 20143
  4. 4. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Prehistory  Who was The First Tester ? (2/2) The first programmer, Lady Lovelace, who coded for Charles Babbage's wonderful but unfinished computer in the nineteenth century, I'm sure often said, "Just one more week, Mr. Babbage, and it'll be done." Lucky for her the hardware was never finished, so she never did have to go beyond desk checking. (B. Beizer, Software system testing and quality assurance, 1984, p.277)  Boris Beizer wrote: • “The first discussion of testing and debugging predates computers by almost a century, in the memoirs of lady Ada Lovelace who wrote software for Babbage’s (never completed) mechanical computer.” (posted to on Sep. 18 2007) (C) K. Tatsumi 20144
  5. 5. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Agenda  Prehistory  The History of Computers and Software Engineering  The Growth of Software Testing  The History of Testing Techniques  The History of Testing in Japan  The Forefront of Software Testing Research  Concluding Remarks (C) K. Tatsumi 20145
  6. 6. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The History of Computers and Software Engineering  Timeline (C) K. Tatsumi 20146
  7. 7. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING  Victor Basili and John Musa, The Future Engineering of Software: A Management Perspective, 1991  1960s: The Functional Era • IT penetrated institutions • How to exploit IT to meet institutional needs  1970s: The Schedule Era • Software crisis NATO Software Engineering Conference, 1968 • How to develop software in a timely, planned, and controlled fashion • Life-cycle models and schedule tracking ex. Royce waterfall model, Quality models 1950 1980 19901960 1970 2000 2010 Schedule Era Cost Era Quality EraFunctional Era Historical Division by Management Perspective (1/2) (C) K. Tatsumi 20147
  8. 8. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING  1980s: The Cost Era • Hardware costs decreased, PC created a mass market • The importance of productivity in software development increased • Various cost models ex. Function point, Putnam model, COCOMO  1990s: The Quality Era • The increased dependence of institutions on information processing • The consumer mass market increases the demands on quality 1950 1980 19901960 1970 2000 2010 Schedule Era Cost Era Quality EraFunctional Era Historical Division by Management Perspective (2/2) (C) K. Tatsumi 20148
  9. 9. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Software Engineering Evolution  Barry Boehm, A Hegelian View of Software Engineering Evolution, 2006 Autonomy; Bio- Computing 1990’s 2010’s2000’s1970’s 1980’s1960’s1950’s COTS Software as Craft Theses Syntheses Antitheses Formality, Waterfall Productivity; Reuse; Objects; Peopleware Plan- Driven Software Maturity Models Agile Methods Engineer Software like Hardware Risk-Based Agile/Plan- Driven Hybrids; Model-Driven Development Integrated Sw-systems Engineering Value-Based Methods; Collaboration; Global Development; Enterprise Architectures Software Differences, Engineer Shortages Many defects Scalability, Risk Mgmt. Prototyping Time to Market, Rapid Change Scalability Domain Engr. Risk Mgmt. Compliance Process Overhead Software Value-Add Soft SysE Global Systems of Systems B. Boehm, A View of 20th and 21st Century Software Engineering, 2006 (C) K. Tatsumi 20149
  10. 10. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Agenda  Prehistory  The History of Computers and Software Engineering  The Growth of Software Testing  The History of Testing Techniques  The History of Testing in Japan  The Forefront of Software Testing Research  Concluding Remarks (C) K. Tatsumi 201410
  11. 11. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The Growth of Software Testing  D. Gelperin and W. Hetzel, The Growth of Software Testing, 1988  Five evolutionary periods divided by landmark literatures 1. The Debugging-Oriented Period ( -1956) 2. The Demonstration-Oriented Period (1957-1978) 3. The Destruction-Oriented Period (1979-1982) 4. The Evaluation-Oriented Period (1983-1987) 5. The Prevention-Oriented Period (1988- ) (C) K. Tatsumi 201411
  12. 12. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The Debugging-Oriented Period  Testing was not differentiated from debugging Evalua tion Demonstration Destru ction Debugging Prevention 1950 1980 19901960 1970 2000 2010 S. Gill, The diagnosis of mistakes in programmes on the EDSAC, p.539, 1951  " Software problems were submerged in the concern for hardware reliability."  "The difficulty lies not in detecting the presence of a mistake, but in diagnosing it."  You "wrote" a program and then you "checked it out." • Program checkout, debugging, and testing were not clearly differentiated.  McCracken, Digital Computer Programming, 1957 (The earliest programming text) • Techniques referred to as today‘s debugging and testing are described in "Chapter 13 Program Checkout". (C) K. Tatsumi 201412
  13. 13. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The Demonstration-Oriented Period  Testing to make sure that the software satisfies its requirements  C. Baker, Review of "Digital Computer Programming", 1957 • "McCracken fails to distinguish the two phases of program checkout." – Debugging: The process of making sure that the program does what the coder meant it to do. – Testing: The process of making sure that the program solves the problem it is intended to solve.  Big Projects (late 1950s-1960s) • SAGE (US air defense control system), SABRE (Computer reservation system), NASA's Mercury, Gemini, and Apollo projects • IBM OS/360 ref. F. Brooks, "Mythical Man-Month"  "Software Engineering" • NATO Software Engineering Conferences (1968, 1969) – "Testing shows the presence, not the absence of bugs." (Dijkstra,1969) (1/2) DemonstrationDebugging Evalua tion Destru ction Prevention 1950 1980 19901960 1970 2000 2010 (C) K. Tatsumi 201413
  14. 14. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The Demonstration-Oriented Period  Growth of Professionalism • Debugging techniques – Debugging Techniques in Large Systems Symposium, 1970 • The first formal conference on software testing – The Computer Program Test Methods Symposium, 1972 • The first book on software testing – Hetzel (Ed.), "Program Test Methods," 1973  Growth of Software Testing Research • The fundamental theorem of software testing – Goodenough and Gerhart, "Toward a Theory of Test Data Selection," 1975 • The first workshop on software testing – Software Testing and Test Documentation Workshop, 1978 The origin of ISSTA (International Symposium on Software Testing and Analysis) cf. W. Wulf, Concise definitions of software quality attributes, 1973 B. Boehm, et al., Software quality characteristics, 1973 DemonstrationDebugging Evalua tion Destru ction Prevention 1950 1980 19901960 1970 2000 2010 (2/2) (C) K. Tatsumi 201414
  15. 15. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The Destruction-Oriented Period  Testing to detect implementation faults  G. Myers, The Art of Software Testing, 1979 • "Testing is the process of executing a program with the intent of finding errors" • "Since exhaustive testing is out of the question, the objective should be to maximize the yield of the testing investment by maximizing the number of errors found by a finite number of test cases." • "Test-case design is so important because complete testing is impossible; a test of any program must be necessarily incomplete. The obvious strategy, then, is to try to make tests as complete as possible." 1950 1980 19901960 1970 2000 2010 DemonstrationDebugging Evalua tion Destru ction Prevention (C) K. Tatsumi 201415
  16. 16. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The Evaluation-Oriented Period  Testing to evaluate products during the software lifecycle  FIPS 101, Guideline for Lifecycle Validation, Verification, and Testing of Computer Software, 1983 1950 1980 19901960 1970 2000 2010 DemonstrationDebugging Evalua tion Destru ction Prevention • "A VV&T methodology is a procedure of review, analysis, and testing employed throughout the software lifecycle from software planning through the end of software use to ensure the production and maintenance of quality software." • "No single VV&T technique can guarantee correct, error-free software. However, a carefully chosen set of techniques for a specific project can help to ensure the development and maintenance of quality software for that project." (C) K. Tatsumi 201416
  17. 17. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The Prevention-Oriented Period  Testing to prevent faults in requirements, design, and implementation  W. Hetzel, The Complete Guide to Software Testing (2nd Ed.), 1988 1950 1980 19901960 1970 2000 2010 DemonstrationDebugging Evalua tion Destru ction Prevention • STEP (Systematic Test and Evaluation Process) A life cycle prevention model that sees testing parallel to development with an activity sequence containing planning, analysis (setting test requirements or objectives), design (specifying an architecture for the set of tests and details of individual cases and procedures), implementation (acquiring or developing test data, procedures, and test support software), execution (running and rerunning tests and determining the results), and maintenance (saving and updating the tests as the software changes). cf. W-Model (C) K. Tatsumi 201417
  18. 18. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Phases in a Tester‘s Mental Life Phase 0 - There‘s no difference between testing and debugging. Other than in support of debugging, testing has no purpose. Phase 1 - The purpose of testing is to show that the software works. Phase 2 - The purpose of testing is to show that the software doesn't work. Phase 3 - The purpose of testing is not to prove anything, but to reduce the perceived risk of not working to an acceptable value. Phase 4 - Testing is not an act. It is a mental discipline that results in low-risk software without much testing effort. B. Beizer, Software Testing Techniques, 2nd Ed., 1990 (C) K. Tatsumi 201418
  19. 19. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Agenda  Prehistory  The History of Computers and Software Engineering  The Growth of Software Testing  The History of Testing Techniques  The History of Testing in Japan  The Forefront of Software Testing Research  Concluding Remarks (C) K. Tatsumi 201419
  20. 20. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Testing Techniques : Beginnings  The state of the art in software testing before 1970s < The early 1960s >  Published papers (cumulative) • -1969: 57 papers (*1) cf. -1973: 200 papers (*2), -1977: 400 papers+ (*3) • There are few papers on software testing techniques. • Many papers only address testing in a peripheral manner such as test procedures or automation. < The late 1960s >  IBM OS/360 Project (F. Brooks, "Mythical Man-Month") • Released in 1966. MVT, multi tasking OS, was released in 1967. • A lot of software testing work must have been carried out. • Much know-how about test process, test techniques, and test management must have been accumulated. (1/2) (*1) W. Elmendorf, "Program Testing – A Bibliography of Published Literature, 1962–1968", 1969 (*2) W. Hetzel, Number of references in "Program Test Methods", 1973 (*3) E. Miller, Number of references in "Tutorial: Program Testing Techniques", COMPSAC '77, 1977 (C) K. Tatsumi 201420
  21. 21. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Testing Techniques : Beginnings  The test control process, IBM  W. Elmendorf, "Controlling the functional testing of an operating system," 1969 (2/2) • Elmendorf started to work on software testing for OS/360 in 1965. • A disciplined test control process - From the "laissez-faire approach" to a disciplined approach - From the "testing-is-an-art" approach to a scientific approach A pioneering work of a preventive test approach like W-Model (C) K. Tatsumi 201421
  22. 22. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Equivalence Partitioning / Boundary-Value Analysis <History>  1979: The names and concepts of EP and BVA were first introduced by G. Myers of IBM in his "The Art of Software Testing."  Similar concepts had already appeared in Elmendorf‘s paper in 1967. • External Interaction Variations "... If the actions taken by the control program are predictable, acceptable ranges of field values, field sizes, field repetitions, etc., will be identified as variations at, and just beyond, the extremes of each range. There are external limits which, if exceeded, cause unpredictable actions." W. Elmendorf, Evaluation of the Functional Testing of Control Programs, 1967 (C) K. Tatsumi 201422
  23. 23. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Cause-Effect Graphing <History>  1970: W. Elmendorf developed Cause-Effect Graphing by adopting a hardware logic testing method. • Extending the concepts to cover software specific issues such as constraints on the inputs • Development of a test case generation tool – TELDAP (TEst Library Design Automation Program) was developed for generating test cases from a cause-effect graph, based on the techniques for generating test patterns for the logic circuits in hardware.  1979: Cause-Effect Graphing became widely known after publication of Myers‘ book which introduced this technique. W. Elmendorf, Automated Design of Program Test Libraries, IBM-TR-00.2089, 1970 (C) K. Tatsumi 201423
  24. 24. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Decision Table Testing <History>  Circa 1958: Decision tables were developed by General Electric and Sutherland corp. individually • To express the logic of product design, operation planning, management decision rules, etc. .  1960: GE developed TABSOL (Tabular systems oriented language) which generated program codes from decision tables.  1965: The early paper to apply DT to software testing • Test categories and test parameters are entered to condition fields. Consequences and test actions are entered to action fields.  1975: Condition table method by Goodenough & Gerhart • A technique derived from decision table techniques for developing and describing test predicates B. Grad, Tabular form in decision logic, DATAMATION, July 1961 B. Scheff, An application of decision tables as the source language for automatic testing, 1965 J. Goodenough and S. Gerhart, Toward a theory of test data selection, 1975 T. Kavanagh, TABSOL A fundamental concept for system-oriented languages, 1960 (C) K. Tatsumi 201424
  25. 25. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Orthogonal Array / Pairwise Testing <History>  1920s-30s: Design of Experiments (DoE), R. Fisher  1940s: Concepts of orthogonal arrays as DoE, C. Rao  1950s: Quality engineering (Taguchi Methods)  1980s: Applying DoE to software testing • 1984: Operating system testing, Fujitsu Ltd., Japan • 1985: Ada compiler testing, R. Mandl, US  1990s: Spreading in the US from the mid 1990s • 1992: OATS, AT&T Bell Lab. • 1994: CATS, AT&T • 1994: AETG, commercial tool, Belcore S. Sato and H. Shimokawa, Methods for setting software test parameters using the DoE, 1984 R. Mandl, Orthogonal Latin squares: an application of experiment design to compiler testing, 1985 R. Brownlie, J. Prowse, M. Phadke, Robust testing of AT&T PMX/Starmail using OATS, 1992 G. Sherwood, Effective testing of factor combinations, 1994 D. Cohen et al., The automatic efficient test generator (AETG) system, 1994 (C) K. Tatsumi 201425
  26. 26. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Timeline of Combinatorial Testing (1957: Decision table) ~1967: Equivalence partitioning, Boundary value analysis 1970: Cause Effect Graph [Elmendorf] (1983~)1984: Orthogonal array / Combination Table [Satoh, Shimokawa] 1987: Test Case Design Support System [Tatsumi] (198x~)1992: OATS [Brownlie, Prowse, Phadke] (1990~)1994: CATS [Sherwood] (1992~)1994: AETG [Cohen, et. al] 1998: IPO [Lei, Tai] 2000: Covering arrays [Williams] 2000: CTE XL [Daimler Chrystler] (2000~)2004: PICT [Microsoft] 2007: FireEye 2009:ACTS (IPOG) [Lei, Kuhn] AT&T, Bellcore 1988: Category-partition method [Ostrand, Balcer] 1993: Classification-tree method [Grochtmann, Grimm] (1976: Test Factor Analysis Method) [Fujitsu] (199x~)2004: Orthogonal Array (HAYST method) [Akiyama(Fuji Xerox)] 1980 1990 2000 20101950 1960 1970 ‘85 ‘95 ‘05 Combinatorial techniques Input condition analysis techniques (1983~)1985: Orthogonal Latin Squares [R. Mandl] Fujitsu (C) K. Tatsumi 201426
  27. 27. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Control Flow Testing (1/2) <History>  1960s: Applying graph theory to software development • 1960: Compiler development, Karp • 1963: Applying to software test design, Miller & Maloney • 1976: Cyclomatic complexity and Basis path testing, McCabe  1960s: Measuring test coverage • Code coverage tool development in IBM Poughkeepsie – C. Warner Jr., Evaluation of program testing, 1964 Earliest known use of a hardware instruction coverage monitor: COBOL and FORTRAN source. (from Beizer) – I. Hirsh, MEMMAP/360, 1967 Earliest known description of a software statement and branch coverage analyzer. (from Beizer) R. Karp, A note on the application of graph theory to digital computer programming, 1960 J. Miller and C. Maloney, Systematic mistake analysis of digital computer programs, 1963 T. McCabe, A Complexity Measure, 1976 (C) K. Tatsumi 201427
  28. 28. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Control Flow Testing (2/2)  Early 1970s: Dynamic analysis systems (Coverage measurement tools) • 1972: PACE (Product Assurance Confidence Evaluator), TRW • 1972: PET (Program Evaluator and Tester), McDonnell Douglas • 1974: RXVP (Requirement Evaluation & Verification Package), General Research J. Brown, et al., Automated Software Quality Assurance: A Case Study of Three Systems, 1972 L. Stucki, Automatic Generation of Self Metric Software, 1972 E. Miller, et al., Structurally based automatic program testing, 1974  Early 1970s: Coverage criteria • 1972: TER (Test Effectiveness Ratio) [Brown] • 1975: C0, C1, C2, … [Miller] (1975-1977) C0 : Programmer‘s intuition, C1 : Every statement in a program exercised at least once, C2 : Every program predicate outcome exercised at least once, ... (1977-) C0 : Every statement executed at least once, C1 : Every segment executed at least once, C1 p : Every predicate term executed for each outcome, C2 : C1 + interior and boundary tests for each iteration, ... J. Brown, Practical applications of automated software tools, 1972 E. Miller, The Art and the Theory of Program Testing, 1975 E. Miller, Coverage levels, in "Infotech State of the Art Report: Software Testing," Vol. 1, 1979 (C) K. Tatsumi 201428
  29. 29. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Data Flow Testing <History>  Late 1960s: Data flow analysis techniques were developed in the field of optimizing compilers. F. Allen( IBM)  1974: Applying data flow analysis techniques to software testing, Osterweil & Fosdick (Univ. of Colorado)  1982-: Data flow criteria, Rapps & Weyuker Coverage criteria subsumption including control flow criteria and data flow criteria F. Allen and J. Cocke, A program data flow analysis procedure, 1976 L. Osterweil and L. Fosdick, Data Flow Analysis as an Aid in Documentation, Assertion Generation, Validation and Error Detection, 1974 S. Rapps and E. Weyuker, Data Flow Analysis Techniques for Test Data Selection, 1982 (C) K. Tatsumi 201429
  30. 30. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING State Transition Testing <History>  Originally, state transition diagrams and state transition tables were developed to represent finite state machines (FSM).  1956: Moore‘s "Gedanken-experiments on sequential machines" • The origin of FSM model based testing  1978: Coverage criteria "n-switch cover" proposed by Chow E. Moore, Gedanken-experiments on sequential machines, 1956 T. Chow, Testing software designs modeled by finite-state machines, 1978 (C) K. Tatsumi 201430
  31. 31. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Agenda  Prehistory  The History of Computers and Software Engineering  The Growth of Software Testing  The History of Testing Techniques  The History of Testing in Japan  The Forefront of Software Testing Research  Concluding Remarks (C) K. Tatsumi 201431
  32. 32. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The History of Testing in Japan (1/3)  1958, The Earliest paper on software testing  Fujiwara, A report on program testing on the IBM 704 large scale electronic computer (in Japanese), 1958 • Preliminary program testing for a numerical weather prediction system of the Japan Meteorological Agency  1964, Discussion on the program inspection  Information Processing Society of Japan, Special features : Software ,IPSJ Magazine, 1964 • Necessity of independent section for software inspection • Role of software quality assurance section • Process improvement through the feedback from QA section  1969, Hitachi’s Software Works (Factory)  Establishing the function of software quality assurance in the software factory system  1971: Also, Fujitsu created a formal product inspection procedures for release to the customer. (C) K. Tatsumi 201432
  33. 33. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The History of Testing in Japan (2/3)  1972, An Approach for Software Inspection  A. Kanno, An Approach for Software Inspection, 1972 • Concepts, activities and techniques in software quality assurance department in Hitachi software factory  1974, Development of QA techniques  K. Sakata(Hitachi), Formulation for predictive methods in software production control, 1974 • Static prediction and failure rate transition model Quality prediction by reliability growth curve or Gompertz curve • Dynamic prediction: quality probe Quality prediction by "quality probe" that comprises a small percentage of the complete regular test (C) K. Tatsumi 201433
  34. 34. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The History of Testing in Japan (3/3)  1980-, R&D of testing techniques and tools  1980: AGENT(Automated GENeration system for Test cases), Hitachi • Test case generation tool based on cause-effect graphing  1984: AGENT Function Diagram, Hitachi • Representation of functional specification (dynamic part -> state transition diagram, static part -> decision table or cause-effect graph)  1984: Application of DoE to software testing, Fujitsu • Test case generation technique and tool for application of DoE  1988: CFD(Case Flow Diagram or Cause Flow Diagram), NEC • External and internal specifications are organized into Cause Flow Diagram, then decision table is generated from the diagram.  1991, Japan's Software Factories  M. Cusumano, Japan's Software Factories: A Challenge to U.S. Management, 1991 • A management research work about Japanese software development and a survey of the making of Japan‘s computer industry, Hitachi, Toshiba, NEC, and Fujitsu etc. (C) K. Tatsumi 201434
  35. 35. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Agenda  Prehistory  The History of Computers and Software Engineering  The Growth of Software Testing  The History of Testing Techniques  The History of Testing in Japan  The Forefront of Software Testing Research  Concluding Remarks (C) K. Tatsumi 201435
  36. 36. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The Forefront of Software Testing  A. Bertolino, "Software Testing Research: Achievements, Challenges, Dreams," 2007 •29th ICSE, Future of Software Engineering track •Bertolino is the KA co-leader for Software Testing in the IEEE Guide to the SWEBOK  The software testing research roadmap • Achievements The most notable achievements from past research • Dreams The desired destination • Challenges The challenges faced by current and future testing research (C) K. Tatsumi 201436
  37. 37. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING The Many Faces of Software Testing  Software testing:  Observing a sample of executions, and giving a verdict over them  6 aspects of software testing  WHY: test objective looking for faults?, the product can be released?, evaluate the usability?  HOW: test selection ad hoc, at random, or systematic way  HOW MUCH: test adequacy coverage analysis, reliability measures  WHAT: levels of testing unit test, component/subsystem test, integration test  WHERE in house, simulated environment, the target final context  WHEN when is it in the product lifecycle that we perform the observations? (C) K. Tatsumi 201437
  38. 38. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Software Testing Research Roadmap A. Bertolino, “Software Testing Research: Achievements, Challenges, Dreams,” FOSE’07, p.85-103, 2007, Figure 1 Testing process Reliability testing WHY How How much What Where When Protocol testing Test criteria Comparison among test criteria Component-basedtesting Object-orientedtesting Education of software testers Testing patterns Controlling evolution Leveraging user population and resources Understanding the costs of testing Test input generation On-line testing Test oracles Model-based testing Anti-model-based testing Explicit test hypotheses Test effectiveness Empirical body of evidence Compositional Testing Domain-specific test approaches Achievements Challenges Dreams Efficacy- maximized test engineering 100% automatic testing Test-based modeling Universal test theory (C) K. Tatsumi 201438
  39. 39. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Achievements  Testing process  Test criteria  Comparison among test criteria  Object-oriented testing  Component-based testing  Protocol testing  Reliability testing (C) K. Tatsumi 201439
  40. 40. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Dreams (1) Universal test theory  A comprehensive theory which is useful to backup and nourish test technology (2) Test-based modeling  Test-based modeling is closely related to the old idea of “Design-for-testability.” (3) 100% automatic testing  Advanced techniques for generating the test inputs, innovative support procedures to automate the testing process (4) Efficacy-maximized test engineering  Practical testing methods, tools and processes for development of high quality software (C) K. Tatsumi 201440
  41. 41. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Dream: Universal test theory <Challenges>  Explicit test hypotheses • Making explicit for each technique which are its underlying assumptions  Test effectiveness • Assessing the effectiveness of existing and novel test criteria  Compositional testing • Reusing the test results observed in the separate testing of the individual levels • A foundational theory for component-based software reliability  Empirical body of evidence • An empirical body of knowledge which is at the basis for building and evolving the theory for testing (C) K. Tatsumi 201441
  42. 42. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Dreams: Test-based Modeling <Challenges>  Model-based testing • Combining different styles of modeling (transition-based, pre/post condition-based and scenario-based) • Integrating model-based testing practice into current software processes  Anti-model-based testing • A model is derived a posteriori via testing for the cases in which the models do not exist or are not accessible, such as for COTS or legacy components.  Test oracles (C) K. Tatsumi 201442
  43. 43. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Dream: 100% automatic testing <Challenges>  Test input generation • Model-based test generation, random test generation, search- based test generation  Domain-specific test approaches  On-line testing • Monitoring a system‘s behavior in real life operation using dynamic analysis and self-test techniques (C) K. Tatsumi 201443
  44. 44. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Dream: Efficacy-maximized test engineering <Challenges>  Controlling evolution  Leveraging user population and resources  Testing patterns  Understanding the costs of testing  Education of software testers (C) K. Tatsumi 201444
  45. 45. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Transversal challenges <Challenges>  Testing within the emerging development paradigm • Testing of service-oriented applications in the Service-oriented Computing • A special importance for testing of services (monitoring the real- world execution is the only way to observe the application behavior)  Coherent testing of functional and extra- functional properties • Conventional functionality testing does not provide for any notion of time nor tackle resource usage and workloads. • For the model-based approach, we need effective ways to enhance models with desired extra-functional constraints. (C) K. Tatsumi 201445
  46. 46. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Concluding Remarks The Analects - Chinese Text Project  • The Master said, "If a man keeps cherishing his old knowledge, so as continually to be acquiring new, he may be a teacher of others."  • The Master said, "Is it not pleasant to learn with a constant perseverance and application? Is it not delightful to have friends coming from distant quarters? Is he not a man of complete virtue, who feels no discomposure though men may take no note of him?"  • The Master said, "Learning without thought is labor lost; thought without learning is perilous."  • The Master said, "You, shall I teach you what knowledge is? When you know a thing, to hold that you know it; and when you do not know a thing, to allow that you do not know it - this is knowledge." (C) K. Tatsumi 201446
  47. 47. HISTORYOFSOFTWARETESTING HISTORYOFSOFTWARETESTING Now is Your Turn to Make History! Thank you ! (C) K. Tatsumi 201447