Michael Erichsen, profile picture
Uploaded byMichael Erichsen
2,076 views

How to test a Mainframe Application

This document provides an overview of testing methods for mainframe applications. It discusses how testing needs have changed from the past where testing involved examining batch job outputs or online screen outputs, to modern approaches like test automation, virtualization, and DevOps. It covers different levels of testing maturity and factors to consider when planning tests. Black box and white box testing techniques are described, such as model-based testing, use case testing, and debugging. The presentation emphasizes the importance of testing to ensure stability, reliability, and compliance with changing requirements.

Embed presentation

Downloaded 55 times
FOR MORE INFORMATION PLEASE CONTACT Xact Consulting A/S Arnold Nielsens Boulevard 68A DK-2650 Hvidovre +45 7023 0100 info@Xact.dk www.Xact.dk Enterprise Modernization S505. How to test a Mainframe Application GSE Nordic Conference May 2017 Michael Erichsen, Xact Consulting A/S, michael.erichsen@Xact.dk Welcome to a presentation about how to test a mainframe application. 1
Disclaimers • This is NOT a presentation about how to replatform from a mainframe to a Microsoft platform • If you want to hear about that subject, then you could spend the next hour much better anywhere else! There has been some confusion in the conference invitation about the presentation I was invited to give. If this had led you to expect a presentation on how to change platforms from z/OS to Microsoft, then I am going to disappoint you. 2
More Disclaimers IANAT – I Am Not A Tester… But I am co-author of one of the testing tools discussed This is a presentation about testing. I am not a tester myself, but I do spend most of my time these days setting up test environments and building test tools. 3
Remember back in the 2. Millennium? Let us start with a flash back to the recent past. Do you remember back in the second millennium, where we used to sail in Viking ships? We discovered America, but were careful enough not to tell anybody about it. 4
Remember back in the 2. Millennium? That was when applications were monolithic. 5
Remember back in the 2. Millennium? Code was spaghetti. 6
Remember back in the 2. Millennium? 7 And the user interface was Leporello lists in batch. 7
Remember back in the 2. Millennium? 8 And green screens on line. 8
Testing – Old Style Batch programs were tested by resetting the master file or database. Then you created an input file. You submitted the batch job. And finally, you examined the output and compared it to the expected result or the output of a previous run. 9
Testing – Old Style Online programs were tested by starting a transaction, entering data, pressing “Enter”, and then examining the output on the screen. Or perhaps taking a screen shot for documentation. 10
This was good enough for your Grandfathers, but probably not for you? This way of testing was perhaps sufficient in the millennium, where we sailed the long ships with dragon’s heads. But today we need something else. 11
Testing in the 3. Millennium • Stability and reliability have always been the hallmarks of the mainframe – Changing test requirements – Changed environment – Shorter time to market – Developers, testers and operations staff are not sitting close to each other any more Testing is changing completely now, and it has suddenly become very interesting. Stability and reliability have always been the hallmarks of the mainframe. But that is not enough anymore. 12
Testing in the 3. Millennium • Stability and reliability have always been the hallmarks of the mainframe – Changing test requirements – Changed environment – Shorter time to market – Developers, testers and operations staff are not sitting close to each other any more We have moved from green screens to client/server, monolithic and spaghetti to modular and structured, from small tight groups to globally dispersed, virtual teams. And compliance requirements are added on top of it all. We live and work in a time of liberalization, economic challenges, concentration of corporations and a focus on shareholder value rather than using the revenue for company building. 13
Testing in the 3. Millennium • Stability and reliability have always been the hallmarks of the mainframe – Changing test requirements – Changed environment – Shorter time to market – Developers, testers and operations staff are not sitting close to each other any more • Instead we got DevOps • Reduce time used to provide new solutions • Test faster and more completely – Test automation – Test virtualization – More flexible test environments – Promotion depending on test and quality assurance DevOps is the current buzzword. It means that we need to be faster and better. All the time. Testing needs to be automated to speed deliveries up. This requires more and more parallel test environments, and it requires that we use test virtualization to make them more independent of each other. 14
Testing in the 3. Millennium • Stability and reliability have always been the hallmarks of the mainframe – Changing test requirements – Changed environment – Shorter time to market – Developers, testers and operations staff are not sitting close to each other any more • Instead we got DevOps • Reduce time used to provide new solutions • Test faster and more completely – Test automation – Test virtualization – More flexible test environments – Promotion depending on test and quality assurance We now have new flexible and modern test environments available, like Micro Focus Enterprise, Compuware Topaz, and the IBM products formerly known as Rational. The Rational products includes the IBM developer, the IBM debugger, Rational Team Concert, all the tools in the Test Workbench, and zD&T, which is a complete mainframe on Intel hardware. 15
Testing in the 3. Millennium • Stability and reliability have always been the hallmarks of the mainframe – Changing test requirements – Changed environment – Shorter time to market – Developers, testers and operations staff are not sitting close to each other any more • Instead we got DevOps • Reduce time used to provide new solutions • Test faster and more completely – Test automation – Test virtualization – More flexible test environments – Promotion depending on test and quality assurance Finally, promotion is increasingly combined with test and quality assurance. Nobody will be allowed to put anything in production, unless they can prove that it is tested, that all the changes are covered by the test, and that coding standards have been followed. 16
Testing Maturity Introduction to Software Testing (Ammann & Offutt) Level 0 •There’s no difference between testing and debugging Level 1 •The purpose of testing is to show that the software works Level 2 •The purpose of testing is to show that the software doesn’t work Level 3 •The purpose of testing is not to prove anything specific, but to reduce the risk of using the software Level 4 •Testing is a mental discipline that helps all IT professionals develop higher quality software So, why do we test in the first place? This is an attempt to discuss testing maturity. If you sit there thinking that testing and debugging is the same thing, you are given a round zero. If you test to prove that the code that I have written does not work, you are some levels more mature. But not too respectful, I might say. If you focus on risk reduction and quality improvement, you score the highest marks. 17
Typical Software Quality Factors William C. Hetzel, The Complete Guide to Software Testing • Functionality (exterior quality) – Correctness – Reliability – Usability – Integrity • Engineering (interior quality) – Efficiency – Testability – Documentation – Structure • Adaptability (future quality) – Flexibility – Reusability – Maintainability Here are some good factors to consider when you plan your tests. The list itself demonstrates how far modern testing has moved. 18
Typical Software Quality Factors Hetzel, William C., The Complete Guide to Software Testing • Functionality (exterior quality) – Correctness – Reliability – Usability – Integrity • Engineering (interior quality) – Efficiency – Testability – Documentation – Structure • Adaptability (future quality) – Flexibility – Reusability – Maintainability •How many factors do we actually test for? And it puts the big question to all of us: How many factors do we actually test for? 19
Release Culture Company A (Old story, possibly an urban legend): – It is better economy to ship early and often – We have a monopoly, and it is cheaper to pay damages (and lawyers) than to have better quality Company B (Real life): – Every version must be both forward and backward compatible – Thorough testing – Wide beta programs The culture aspect is important. It is really a part of your corporate culture. If all that matters are shareholder values for the current quarter, then you risk losing focus on your customers. And in the longer run: Losing focus on your real business. Company A cocktail party. Company B is how IBM is expected to work. 20
Where do we validate our Input? • The 3270 model leaves no choice • The distributed model leaves nothing but choices – In the client? – At entry to the mainframe? – At entry to every module? The first question we meet is: Where do you validate your input? This is traditionally different from mainframes to distributed platforms. But it might be worth reconsidering? 21
Where do we validate our Input? • The 3270 model leaves no choice • The distributed model leaves nothing but choices – In the client? – At entry to the mainframe? – At entry to every module? • An all too true story – Mainframe developers have validated certain data for decades – Front end developers do not ask • They just code the same test in the front end In many shops, there is no consensus in this matter. This results in the same validation happening in different layers of the applications. And even a service catalog will not help you, if nobody is using it, anyway. 22
Quote of the Day • “If debugging is the process of removing software bugs, then programming must be the process of putting them in” – Edsger Dijkstra • Stolen from http://www.marcofolio.net/ Debugging is not the same as testing. But this quote probably goes for both efforts. 23
The Real Architecture Model Cycle Experience shows limitations Before we dig into some of the many methods, techniques and tools for testing, I would like to remind how architecture models evolve. First somebody gets annoyed by the limitations in their practical work. 24
The Real Architecture Model Cycle Experience shows limitations Improvement ideas Then someone gets some good ideas for improvement. 25
The Real Architecture Model Cycle Experience shows limitations Improvement ideas Bestseller with new terms for everything If one of them is ambitious enough, then he gives it a name and writes a book about it. It is always a “he”, and sometimes the book becomes a best seller. 26
The Real Architecture Model Cycle Experience shows limitations Improvement ideas Bestseller with new terms for everything Part of curriculum This is new and exciting, so some universities put in into their curriculum to show that they are on the leading edge. It is beaten into the heads of the poor students. 27
The Real Architecture Model Cycle Experience shows limitations Improvement ideas Bestseller with new terms for everything Part of curriculum Vendors create new tools No tool vendor wants to be left behind, so they all develop or buy a tool to support it. 28
The Real Architecture Model Cycle Experience shows limitations Improvement ideas Bestseller with new terms for everything Part of curriculum Vendors create new tools Students become architects In the meantime the students graduate. They get jobs, and since they are on the beat with the newest trend, they become architects in the companies, where you work. 29
The Real Architecture Model Cycle Experience shows limitations Improvement ideas Bestseller with new terms for everything Part of curriculum Vendors create new tools Students become architects Everybody must use new terms and tools Now everybody else must use the new tools and terms, or they are fast becoming obsolete. 30
The Real Architecture Model Cycle Experience shows limitations Very soon somebody gets annoyed by the limitations in their practical work, and the cycle starts all over. So, I will not have a slide in this presentation about Test Driven Development, because that might have become obsolete since I started speaking. 31
Processes Waterfall? – Test planning • Finding an error before it has been made is brilliant • Finding errors late in the project is expensive Agile? – ”Shift Left”? • Finding an error as soon as it has been made Can they be combined? – Or will that be too expensive? The big movement in the development and testing process today is going from waterfall to agile. We know the limitations of the waterfall model, and we are annoyed by them. It is too rigid and too slow for the requirements of today. Errors found in the test phase become expensive, and they can delay a project. But we should try not to throw out the baby with the waterfall. I have seen test planners finding errors that the coders had not yet made, because of the helicopter view of the testers. 32
Quote of the Day • “Walking on water and developing software from a specification are easy if both are frozen” – Edward V Berard • Stolen from http://www.marcofolio.net/ If an environment is too agile, you might fall through. 33
Testing Methods • Static testing – Review – Walk through – Inspection – Verification • Dynamic testing – Black box • Treats software under test as a black-box without knowing its internals. • Tests are using software interfaces and trying to ensure that they work as expected – White box • Looks inside the software that is being tested • Uses that knowledge as part of the testing process To start to discuss and categorize testing methods, we have a distinction between static and dynamic testing. Static is what we do at code review meetings, at the desk top, or at home with a print list and a glass of good single malt. Dynamic is computer aided. It comes in two flavours: Black box, and white box. 34
Black Box Testing Viktor Farcic: Technology Conversations • Efficient for large segments of code • Code access is not required • Separation between user’s and developer’s perspectives This is a listing of some of the advantages of black box testing. 35
Black Box Testing Viktor Farcic: Technology Conversations • Efficient for large segments of code • Code access is not required • Separation between user’s and developer’s perspectives • Limited coverage since only a fraction of test scenarios is performed • Inefficient testing due to tester’s lack of knowledge about software internals • Blind coverage since tester has limited knowledge about the application And some of the disadvantages. 36
Black Box Testing Model-based testing Use case testing Specification-based testing All-pairs testing Fuzz testing Exploratory testing These are the main methods for black box testing. They are much more advanced than just entering data into a green screen. They all look at both how to develop your test cases and how to automate them. Model-based testing is used by a method called “Model Driven Design” by defining test cases to test that the model is implemented. 37
Black Box Testing Model-based testing Use case testing Specification-based testing All-pairs testing Fuzz testing Exploratory testing Use case testing uses test cases based on analysis of success and some failure scenarios. Specification-based testing uses Boundary Value Analysis, Decision Tables, and State Transitioning tests. All-pairs testing tests all possible discrete pairs of input parameters. Fuzz testing is an automatic test of invalid, unexpected or random data. And Exploratory testing will develop test cases as you go and get new ideas from experience. This could be the basis of a long list of requirements to your test tool vendors. 38
White Box Testing Viktor Farcic: Technology Conversations • Efficient in finding errors and problems • Required knowledge of internals of the software under test is beneficial for thorough testing • Allows finding hidden errors • Programmers introspection • Helps optimizing the code • Due to required internal knowledge of the software, maximum coverage is obtained This is a listing of some of the advantages of white box testing. 39
White Box Testing Viktor Farcic: Technology Conversations • Efficient in finding errors and problems • Required knowledge of internals of the software under test is beneficial for thorough testing • Allows finding hidden errors • Programmers introspection • Helps optimizing the code • Due to required internal knowledge of the software, maximum coverage is obtained • Might not find unimplemented or missing features • Requires high level knowledge of internals of the software under test • Requires code access And some of the disadvantages. 40
White Box Testing • Debugging (single stepping) • Post-mortem debugging (Dump analysis) • Unit testing • Integration testing • Component Interface testing • System testing • Acceptance testing • Regression testing • Usability testing • Accessibility testing • Security testing • Internationalization testing • Code coverage • Fault injection White box testing calls for a discussion about testing techniques. We will only touch it very lightly, but go into more details about some of the tools. 41
Code Coverage Criteria • Function coverage • Statement coverage • Branch coverage • Condition coverage (or predicate coverage) Code coverage was one of the first methods for systematic software testing. Modern tools often report on code coverage after testing on a function and statement basis, but we might consider more advanced criteria as well. Branch coverage checks that each branch of each control structure has been executed. For example, given an if statement, have both the true and false branches been executed? Condition coverage (or predicate coverage) checks in the same way if each Boolean sub- expression has been evaluated both to true and false. 42
Fault Injection Parasoft JTest as an example (AD 1999) – Analyse Java class using introspection – Generate input data intended to break the code – Shoot at it – Create a report on what was broken and how to fix it This depends on the wish to validate on the unit level Fault Injection is wide spread in Java development. Java has a language feature called introspection, which reports which methods can be called with which arguments. This is used to generate automated test cases, that analyses a program, injects all interesting input, and reports on missing validations and error handling. I have once used the tool mentioned here to make a critical Java application bullet proof. 43
Test Automation Front end tools – Hiperstation – IDz Generic Service Client – TestRunner – SoapUI Host tools – Headless Code Coverage Lifecycle tools – Embed in lifecycle management Automated tests are important for speeding up the development and test life cycle. We can use front end tools that only need to know about the interfaces. Code coverage can run as part of automated test as what is called headless Code Coverage, where it is running as a batch job without a graphical user interface. And they can and should be embedded in your lifecycle management as parts of the promotion procedure. 44
Quote of the Day • “Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live” – Rick Osborne • Stolen from http://www.marcofolio.net/ Some programmers claim that if a program has been hard to code, it should also be hard to read. Maintainers do not agree. Neither do white box testers. 45
Security Testing • Authentication and authorization • Resilience (DoD attacks, injections etc.) – Chaos Monkey (Simian Army) A different area that needs testing is security. We now know that mainframes can be attacked and penetrated. We need to develop formalized tests of authentication, authorization and resilience. This is not only a task for network and firewall departments. 46
Security Testing • Authentication and authorization • Resilience (DoD attacks, injections etc.) – Chaos Monkey (Simian Army) An interesting tool has been developed by Netflix engineers to test the resiliency and recoverability of their Amazon Web Services. It was called “Chaos Monkey”, and now they have a complete suite, so they called it “Simian Army”. Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures. 47
Security Testing • Authentication and authorization • Resilience (DoD attacks, injections etc.) – Chaos Monkey (Simian Army) • Privacy – The EU ePrivacy Directive – The EU General Data Protection Regulation – Privacy By Design – Privacy By Default – The Right to be Forgotten But not only must we protect our companies and our systems. We also need to protect the security and privacy of our current customers and of others that use our systems and might be our future customers. 48
Security Testing • Authentication and authorization • Resilience (DoD attacks, injections etc.) – Chaos Monkey (Simian Army) • Privacy – The EU ePrivacy Directive – The EU General Data Protection Regulation – Privacy By Design – Privacy By Default – The Right to be Forgotten This has long been an ethical requirement, but thanks to visionary and resilient politicians in the European Union we now have “The EU ePrivacy Directive” and “The General Data Protection Regulation”. It must become law in each member state by May 6th 2018. Some of the important principles are “Privacy By Design”, “Privacy By Default”, and “The Right to be Forgotten”. These principles must be obeyed by our application design and our test data management. 49
Test Data Management • The Achilles’ heel of many test teams • Test data creation – By hand – By tool • CA Test Data Manager (Datamaker) • Compuware’s Test Data Management • InfoSphere Optim Test Data Management – From production data • Masking/Anonymization/Pseudonym ization – EU directive • Test data backup and restore Test Data Management is a very big discussion in itself. There are other presentations on that subject at the conference, such as S503 - The Impact of Inefficient, Ineffective and unsecure Test Data By Kim Mortensen and Gerald Pfeiffer, and S510 - Where does your test data come from? By Bill Alexander. 50
Test Data Management • The Achilles’ Heel of many test teams • Test data creation – By hand – By tool • CA Test Data Manager (Datamaker) • Compuware’s Test Data Management • InfoSphere Optim Test Data Management – From production data • Masking/Anonymization/Pseudonym ization – EU directive • Test data backup and restore • Synchronizing code changes, data structure changes, and test data – Synchronization Big Iron  zD&T I will just mention that when you are expanding your test environments to a farm of zD&T mainframe systems running on Intel, one of the biggest challenges is precisely test data management: How do you create a consistent and sufficiently large set of test data? And how do you update your test data, when applications change? 51
z series Development and Test Environment • zD&T, RD&T, RDzUT, zPDT, ADCD… • Fast and economic deployment of test environments Z Series Development and Test Environment is the mainframe on Intel hardware. It is a less expensive and much faster way of setting up all the parallel test environments that we need today. Setting up a new z/OS system with out-of-the-box settings can be done in a few days from scratch, and even faster when you have a golden image to clone. 52
z series Development and Test Environment • zD&T, RD&T, RDzUT, zPDT, ADCD… • Fast and economic deployment of test environments – The heavy lift is load and adaptation of data, applications, configurations, and customizations What takes time and effort is to identify and extract applications, data, configurations, and customizations on the big iron, and then to adapt them to the new test environment. 53
Tools • Service test • Performance test • Unit test Let us now look at some test tools. 54
Service test Product Company HiperStation Compuware IBM Developer for z Service Client feature IBM LoadRunner HPE Rational Test Workbench IBM SoapUI SmartBear Service test is done by tools running outside the mainframe. They are doing a black box test to check the interfaces and API’s you have exposed to front end clients. These are some of the most popular tools. 55
Performance Test Product Company Application Performance Analyzer IBM FreezeFrame Macro4 InTune BMC Mainframe Application Tuner CA SmartTune ASG Startool Serena Strobe Compuware Performance test is done by your usual on line monitors like MainView, Omegamon, Tmon, and Sysview. And with profiler products such as these on this list. 56
Unit test Product Company Topaz for Total Test Compuware XaTester Xact zUnit IBM Unit test tools have been top of the mainframe testers’ wish list for years. Now they are finally becoming available. IBM has zUnit, which is part of IDZ, IBM Developer for z systems. I will point a presentation about that one. Compuware recently announced Topaz for Total Test. I will point to a presentation on that one too. And Xact also recently announced XaTester, which I will go a bit more in detail on in a moment. 57
Quote of the Day “A bug is not an error, it is a test that was not written” – Extreme programming So, with a bit of luck, it might be time to say goodbye to some of the bugs? 58
The New Kids on the Block • Rational Integration Tester • zUnit • Topaz • XaTester Let us go into details om one integration tester and three unit testers. They are two different categories, but in practical use, they seem to overlap. Rational Integration Tester can be used for test virtualization at interfaces like a call to MQ or a distributed program link in CICS. 59
The New Kids on the Block • Rational Integration Tester • zUnit • Topaz • XaTester ZUnit tests single modules in batch. Topaz test single modules, but can also provide test virtualization for some calls. XaTester was built to test single modules, but user requirements have made it evolve into some integration testing and quite a bit of test virtualization. 60
Rational Integration Tester Rational Integration Tester is a scripting-free environment for developing tests for service-oriented architecture (SOA) messaging and business process integration projects. It can virtualize several messaging solutions (for example, JMS, TIBCO, MQ, CICS distributed program link, and CICS Transaction Gateway) by using a plug-in for the different environments. 61
A Sample Application Imagine an application that is called by a front end, and which communicates with another application using MQ. It could look like this. 62
Test Virtualization What you would like to do was to test your application independently of the other application. So, you would like to stub or mock up the MQ call and get the expected result back, even though the other application was unavailable. 63
RIT Installation You must set up a server for Rational Integration Tester with a repository. You must install an exit in your MQ system. And, you must install the Rational Test Workbench on your workstation. You can then define that certain MQ calls should be intercepted by the exit and that the requests and responses are recorded by the server when running the application. 64
RIT Simulation Now you can define that all requests are redirected to the server, and the proper responses are returned. You can also use an editor to define which data is returned in which fields by the server. 65
Rational Integration Tester The user interface is the Rational Test Workbench, which is shown here. I have been using it in a project, where the response time for the remote application was two to three days. The test virtualization reduced testing time to seconds. 66
More Sample Applications The limitations of the integration tester are seen, when you want to test single modules in batch or online. 67
Unit Test The unit tester is similar to the integration tester, but it operates at the module level. So, we need some more packaging around the module to test it. This is a generic view of a unit tester. 68
Unit Test Unit testers typically have graphical user interfaces and tools to record and edit test data. They can have parses that analyses the data definitions of sources of the modules under test. And they can have repositories to save test cases, test suites, and test data. 69
zUnit zUnit is not a separate product, but a feature of Developer for z Systems. It provides a code-driven unit testing framework for Enterprise COBOL and PL/I. zUnit is an adaptation of the xUnit framework, which is derived from the Java environment. It generates COBOL or PL/I programs that can call the modules under test. Bill Alexander will tell you much more about it in his presentation. 70
Topaz for Total Test Topaz for Total Test from Compuware automates both the creation and the execution of unit tests It integrates tightly with Xpediter, but can run stand alone or integrated with IBM Developer. It generates data stubs, program stubs and test result assertions. It can call subprograms with parameters collected by Xpediter It can integrate to tools like SonarCube, and Code Coverage can be used during the test- run. Kim Mortensen will tell you more in his presentation. 71
XaTester XaTester is a unit test and integration test tool for COBOL, PL/I, EGL, Fortran, RPG and Assembler programs on Mainframe and iSeries. This is the Eclipse based user interface, which can run stand-alone or embedded in IBM Developer for z. 72
XaTester Web Interface This is the web browser interface of XaTester. 73
XaTester Infrastructure And this is the infrastructure needed to support all the features. 74
XaTester Installation To install XaTester you need a separate server for the test engine and the repository. The test engine includes a scheduling feature. You also need to install a host agent, when running on z/OS. 75
XaTester Installation If you want to run XaTester in CICS, you also need to install some CICS components. IMS is handled as batch. If you want to run XaTester from Eclipse or from IBM Developer, you also need to install a plug-in on your workstation. 76
XaTester Extraction First, you must get the data definitions of the modules under test. COBOL, EGL and RPG can be extracted by XaTester and stored in the repository for later use. Structures for other languages like PL/I, Fortran and assembler can be defined in an editor in XaTester until somebody requires us to build a parser for this language. Or they plug in their own parser. 77
XaTester Testing You add the module or modules you want to test in an editor window, where you also define input data. This can be typed directly or you can define an SQL query to retrieve the data. If you have more than one module, you can define the output from one module as input to the next. Then you run the test and examine the result. 78
XaTester Regression Testing You can save the result from at test in the repository. Next time you run the test, the results will be compared to the results of the previous run. Fields like time and date can be excluded from the comparison. 79
XaTester Code Coverage If you have RDz or IDz on your system, and you have configured the headless code coverage feature, then XaTester can use it to display, which statements have not been tested. 80
XaTester Mock up When you test a module that calls submodules, you can ask XaTester to generate a mock up module, which can be called to return the recorded values, if that module is not available. This is supported both in batch and in CICS. 81
XaTester Demo But let us just spend a few minutes to see what such a unit tester looks like in a web browser interface. It is a test of a COBOL program that calculate a risk factor based on the age of the customer. 82
XaTester Demo 83
XaTester Demo Let us spend a few minutes more to see what it looks like in Eclipse. 84
XaTester Demo 85
Feature Comparison 3 Unit Testers, 1 Integration Tester Category XaTester TopazforTT zUnit RIT Runtime Main batch, called batch, CICS, IMS, iSeries Main batch Batch Any User Interfaces Eclipse, Browser Eclipse Eclipse Workbench and browser Programming Languages COBOL, PL/I, assembler, EGL, RPG COBOL COBOL & PL/I Any Execution Environments PC, host and a server PC and host PC and host Server and host Execution integration Stand alone, Idz, local scheduling, submitting to night batch Stand alone, Idz and Topaz Workbench/Xpediter Idz N/A Testing Test suites, Code coverage, Advanced test script support, SonarCube, External reporting, repository with test artefacts Test suites, code coverage, SonarCube, External reporting Test suites Test suites, Repository with test artefacts This is a short comparison chart between the products. Since all vendors are presenting at the conference, you can dig deeper yourselves, if you are interested. None of the tools support DB2 Stored Procedures or Websphere application server. 86
Feature Comparison 3 Unit Testers, 1 Integration Tester Category XaTester TopazforTT zUnit RIT Runtime Main batch, called batch, CICS, IMS, iSeries Main batch Batch Any User Interfaces Eclipse, Browser Eclipse Eclipse Workbench and browser Programming Languages COBOL, PL/I, assembler, EGL, RPG COBOL COBOL & PL/I Any Execution Environments PC, host and a server PC and host PC and host Server and host Execution integration Stand alone, Idz, local scheduling, submitting to night batch Stand alone, Idz and Topaz Workbench/Xpediter Idz N/A Testing Test suites, Code coverage, Advanced test script support, SonarCube, External reporting, repository with test artefacts Test suites, code coverage, SonarCube, External reporting Test suites Test suites, Repository with test artefacts Nor do they support an ISPF user interface, which I would not expect anybody to want anyway. None of them support C or C++. Both Topaz and XaTester can be invoked from tools like Jenkins. For XaTester external requests can be made using a REST API. 87
Feature Comparison 3 Unit Testers, 1 Integration Tester Category XaTester TopazforTT zUnit RIT Virtualization Submodules and DB2 (including rollback) Submodules and VSAM/QSAM (Automated using Xpediter) Subroutines, VSAM Really many Integration Apache Ant, CentraSite ActiveSOA, CICS DPL, CTG, DB2, Java, JDBC, Jenkins, MQ, SAG WebMethods, SAP, Tibco, WAS Service Integration Bus Test data creation File import, recording File import, recording File import File import, recording When reading this chart, you should keep in mind that the integration tester does not really compare directly to the unit testers. 88
The Final Test We are almost done now, but it is never over without a final test. 89
The Final Test Well known fact A study at Manchester Metropolitan University involving dropping 100 slices under laboratory conditions established that toast typically lands on the floor butter-side-down because of way it is typically dropped from a table, and the aerodynamic drag caused by the air pockets within the bread. 90
The Final Test Well known fact The toast is invariably butter-side-up when dropped. As it falls, it rotates; given the typical speed of rotation and the typical height of a table, a slice of toast that began butter-side-up on the table will land butter-side-down on the floor in 81% of cases. 91
The Final Test Well known Fact: Cat righting reflex Cats possess the ability to turn themselves right side up in mid-air if they should fall upside-down, known as the cat righting reflex. This enables them to land on their feet if dropped from sufficient height, about 30 cm. 92
The Final Test So, what happens if you tie a slice of toast to the back of a cat and push it down from a table? And how do you design a practical test for it? 93
To Summarize • Automated testing and quality assurance is a matter of survival • We are expanding the scope of our testing rapidly – But many areas are still more or less unhandled • The main issue is not tools, but processes and focus • Pinpoint the needs of your business – Then define your processes and methods – And finally select the tools that suit your requirements To summarize the presentation. Automated testing and quality assurance is not just something nice to have in this day and time. It is a matter of survival. The scope of our testing is expanding rapidly, but many areas are still more or less unhandled. The main issue is not the tools, but to have the right processes and to keep the focus. You should pinpoint the needs of your business. Then you should define your processes, methods and best practices. And finally, you could select the tools that suit your requirements. 94
Hand-on Session and more Information • If you want to know more about zUnit, go to session S510, or catch Bill Alexander • If you want to know more about Topaz for Total Test, go to session S608, or catch Kim Mortensen • If you want to play with XaTester, you can join the hands-on session at 15.15, or catch Steen Brahe at the demonstration booth in the coffee break area 95
FOR MORE INFORMATION PLEASE CONTACT Xact Consulting A/S Arnold Nielsens Boulevard 68A DK-2650 Hvidovre +45 7023 0100 info@Xact.dk www.Xact.dk Enterprise Modernization Moderniza- tion Conversion Business Software Consultants Thank you for listening to this presentation about testing. 96

More Related Content

PPTX
Oracle Sub-ledger Accounting : a Technical Overview
bydrdavidtaylor
 
PPT
Manual testing visonia
byVisoniaTechlab
 
PDF
Service Virtualization: What Testers Need to Know
byTechWell
 
PPT
Software Testing Presentation in Cegonsoft Pvt Ltd...
byChithraCegon
 
PPT
Software Engineering (Testing Overview)
byShudipPal
 
PDF
Continuous Testing through Service Virtualization
byTechWell
 
PPTX
10 Most Important Software Testing Lessons from 2025
byvishalmore1356
 
PPT
Software Testing- Principles of testing- Mazenet Solution
byMazenetsolution
 
Oracle Sub-ledger Accounting : a Technical Overview
bydrdavidtaylor
 
Manual testing visonia
byVisoniaTechlab
 
Service Virtualization: What Testers Need to Know
byTechWell
 
Software Testing Presentation in Cegonsoft Pvt Ltd...
byChithraCegon
 
Software Engineering (Testing Overview)
byShudipPal
 
Continuous Testing through Service Virtualization
byTechWell
 
10 Most Important Software Testing Lessons from 2025
byvishalmore1356
 
Software Testing- Principles of testing- Mazenet Solution
byMazenetsolution
 

Similar to How to test a Mainframe Application

PDF
DevOps Workshop - Addressing Quality Challenges of Highly Complex and Integra...
byAndrew Williams
 
PDF
The Developer’s Guide to Test Automation
byTechWell
 
PDF
Enhancing Quality and Test in Medical Device Design - Part 2.pdf
byICS
 
PPT
ISTQB / ISEB Foundation Exam Practice -1
byYogindernath Gupta
 
PPT
ISTQB, ISEB Lecture Notes
byonsoftwaretest
 
PPTX
QM in Software Projects
byAbhishek Prasoon
 
PPTX
An introduction to Software Testing and Test Management
byAnuraj S.L
 
PPT
Testing 1 - the Basics
byArleneAndrews2
 
PPTX
Software Testing
byVishal Singh
 
PPTX
Software Testing ppt
byPratibha Singh
 
PDF
Continuous Testing through Service Virtualization
byTechWell
 
PDF
Agile Acceptance testing with Fitnesse
byClareMcLennan
 
PPT
Testing- Fundamentals of Testing-Mazenet solution
byMazenetsolution
 
PDF
Galileo computing software testing
byCristiano Caetano
 
PPTX
Software testing
byNico Heidtke
 
PDF
Testingfor continuousdeliverywithvisualstudio2012
bySteve Xu
 
PPT
'Architecture Testing: Wrongly Ignored!' by Peter Zimmerer
byTEST Huddle
 
PPTX
A Software Testing Intro
byEvozon Test Lab
 
PDF
Test Automation: Investment Today Pays Back Tomorrow
byTechWell
 
PPTX
Introduction to Software Testing Techniques
byasadkhaann776
 
DevOps Workshop - Addressing Quality Challenges of Highly Complex and Integra...
byAndrew Williams
 
The Developer’s Guide to Test Automation
byTechWell
 
Enhancing Quality and Test in Medical Device Design - Part 2.pdf
byICS
 
ISTQB / ISEB Foundation Exam Practice -1
byYogindernath Gupta
 
ISTQB, ISEB Lecture Notes
byonsoftwaretest
 
QM in Software Projects
byAbhishek Prasoon
 
An introduction to Software Testing and Test Management
byAnuraj S.L
 
Testing 1 - the Basics
byArleneAndrews2
 
Software Testing
byVishal Singh
 
Software Testing ppt
byPratibha Singh
 
Continuous Testing through Service Virtualization
byTechWell
 
Agile Acceptance testing with Fitnesse
byClareMcLennan
 
Testing- Fundamentals of Testing-Mazenet solution
byMazenetsolution
 
Galileo computing software testing
byCristiano Caetano
 
Software testing
byNico Heidtke
 
Testingfor continuousdeliverywithvisualstudio2012
bySteve Xu
 
'Architecture Testing: Wrongly Ignored!' by Peter Zimmerer
byTEST Huddle
 
A Software Testing Intro
byEvozon Test Lab
 
Test Automation: Investment Today Pays Back Tomorrow
byTechWell
 
Introduction to Software Testing Techniques
byasadkhaann776
 

More from Michael Erichsen

PPTX
ZCX Hybrid Application
byMichael Erichsen
 
PDF
SMP/What?
byMichael Erichsen
 
PDF
ZD&T Survival Kit
byMichael Erichsen
 
PDF
A002 200 yearsofit
byMichael Erichsen
 
PPTX
SOA Mainframe Service Architecture and Enablement Practices Best and Worst Pr...
byMichael Erichsen
 
PPTX
Java is a new language on the mainframe
byMichael Erichsen
 
PPT
Gradueret adgang til EPJ
byMichael Erichsen
 
PPT
Websphere on z/OS and RACF security
byMichael Erichsen
 
PPT
A Natural Web Front End using CICS Transaction Gateway
byMichael Erichsen
 
PPT
Automating the Donut Donation
byMichael Erichsen
 
PPT
Trends but No Directions
byMichael Erichsen
 
PDF
Java on the Mainframe
byMichael Erichsen
 
PDF
Through CICS OS/2 with 50 Million Peas an Hour
byMichael Erichsen
 
PPTX
z13: New Opportunities – if you dare!
byMichael Erichsen
 
PDF
Cobol 5 presentation
byMichael Erichsen
 
ZCX Hybrid Application
byMichael Erichsen
 
SMP/What?
byMichael Erichsen
 
ZD&T Survival Kit
byMichael Erichsen
 
A002 200 yearsofit
byMichael Erichsen
 
SOA Mainframe Service Architecture and Enablement Practices Best and Worst Pr...
byMichael Erichsen
 
Java is a new language on the mainframe
byMichael Erichsen
 
Gradueret adgang til EPJ
byMichael Erichsen
 
Websphere on z/OS and RACF security
byMichael Erichsen
 
A Natural Web Front End using CICS Transaction Gateway
byMichael Erichsen
 
Automating the Donut Donation
byMichael Erichsen
 
Trends but No Directions
byMichael Erichsen
 
Java on the Mainframe
byMichael Erichsen
 
Through CICS OS/2 with 50 Million Peas an Hour
byMichael Erichsen
 
z13: New Opportunities – if you dare!
byMichael Erichsen
 
Cobol 5 presentation
byMichael Erichsen
 

Recently uploaded

PDF
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
PDF
2025-CB-NCAE-slide-deck.pptx-1-83.pdf Virtual Orientation on the Administrati...
byNelizabethEsplaguera1
 
PDF
Building Voice Agents with Google Agent Development Kit
bySuresh Peiris
 
PDF
Just Eat Data Scraping For Restaurant Reviews And Pricing.pdf
byfusiondata2026
 
PPTX
Accelerate Innovation with Engineering R&D Services
byChetu
 
PDF
How Automation Powers Data Quality and Governance at Scale
bySafe Software
 
PPTX
congo red stain nd PAS STAIN histopathology
bybushrariaz1240
 
PDF
Intro to CrafterQ - AI Agent Platform for the Enterprise.pdf
byCrafterQ
 
PPTX
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
PDF
Salesforce Sales Cloud and Generative AI Present a Fresh Approach to Personal...
byMinuscule Technologies
 
PDF
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
PDF
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
PDF
UiPath Coded Agents - A Developer Driven Agentic Automation Era
byUiPathCommunity
 
PDF
Hart, Inc. M&A Data Transition Checklist
byHart, Inc.
 
PDF
Effective Ai powered mock interview .pdf
byamazingnishusingh766
 
PDF
Zniber Partners. Audience Foresight for Confident Decisions
bySamuel Zniber
 
PPTX
Robot Vacuums are Cleaning Up. The global robot vacuum segment has high marke...
byRobert March
 
PDF
Philippine Agricultural Engineering Standard_Hand Tractor .pdf
by6xfrnmdp4b
 
PDF
AI Data Analyst: Smarter Insights for Modern Real Estate Decisions
byLeni Analyst
 
PDF
Top 10 Dedicated Server Providers in Stockholm (Updated Edition).pdf
byAtal Networks
 
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
2025-CB-NCAE-slide-deck.pptx-1-83.pdf Virtual Orientation on the Administrati...
byNelizabethEsplaguera1
 
Building Voice Agents with Google Agent Development Kit
bySuresh Peiris
 
Just Eat Data Scraping For Restaurant Reviews And Pricing.pdf
byfusiondata2026
 
Accelerate Innovation with Engineering R&D Services
byChetu
 
How Automation Powers Data Quality and Governance at Scale
bySafe Software
 
congo red stain nd PAS STAIN histopathology
bybushrariaz1240
 
Intro to CrafterQ - AI Agent Platform for the Enterprise.pdf
byCrafterQ
 
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
Salesforce Sales Cloud and Generative AI Present a Fresh Approach to Personal...
byMinuscule Technologies
 
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
UiPath Coded Agents - A Developer Driven Agentic Automation Era
byUiPathCommunity
 
Hart, Inc. M&A Data Transition Checklist
byHart, Inc.
 
Effective Ai powered mock interview .pdf
byamazingnishusingh766
 
Zniber Partners. Audience Foresight for Confident Decisions
bySamuel Zniber
 
Robot Vacuums are Cleaning Up. The global robot vacuum segment has high marke...
byRobert March
 
Philippine Agricultural Engineering Standard_Hand Tractor .pdf
by6xfrnmdp4b
 
AI Data Analyst: Smarter Insights for Modern Real Estate Decisions
byLeni Analyst
 
Top 10 Dedicated Server Providers in Stockholm (Updated Edition).pdf
byAtal Networks
 

How to test a Mainframe Application

  • 1.
    FOR MORE INFORMATIONPLEASE CONTACT Xact Consulting A/S Arnold Nielsens Boulevard 68A DK-2650 Hvidovre +45 7023 0100 info@Xact.dk www.Xact.dk Enterprise Modernization S505. How to test a Mainframe Application GSE Nordic Conference May 2017 Michael Erichsen, Xact Consulting A/S, michael.erichsen@Xact.dk Welcome to a presentation about how to test a mainframe application. 1
  • 2.
    Disclaimers • This isNOT a presentation about how to replatform from a mainframe to a Microsoft platform • If you want to hear about that subject, then you could spend the next hour much better anywhere else! There has been some confusion in the conference invitation about the presentation I was invited to give. If this had led you to expect a presentation on how to change platforms from z/OS to Microsoft, then I am going to disappoint you. 2
  • 3.
    More Disclaimers IANAT – IAm Not A Tester… But I am co-author of one of the testing tools discussed This is a presentation about testing. I am not a tester myself, but I do spend most of my time these days setting up test environments and building test tools. 3
  • 4.
    Remember back inthe 2. Millennium? Let us start with a flash back to the recent past. Do you remember back in the second millennium, where we used to sail in Viking ships? We discovered America, but were careful enough not to tell anybody about it. 4
  • 5.
    Remember back inthe 2. Millennium? That was when applications were monolithic. 5
  • 6.
    Remember back inthe 2. Millennium? Code was spaghetti. 6
  • 7.
    Remember back inthe 2. Millennium? 7 And the user interface was Leporello lists in batch. 7
  • 8.
    Remember back inthe 2. Millennium? 8 And green screens on line. 8
  • 9.
    Testing – OldStyle Batch programs were tested by resetting the master file or database. Then you created an input file. You submitted the batch job. And finally, you examined the output and compared it to the expected result or the output of a previous run. 9
  • 10.
    Testing – OldStyle Online programs were tested by starting a transaction, entering data, pressing “Enter”, and then examining the output on the screen. Or perhaps taking a screen shot for documentation. 10
  • 11.
    This was goodenough for your Grandfathers, but probably not for you? This way of testing was perhaps sufficient in the millennium, where we sailed the long ships with dragon’s heads. But today we need something else. 11
  • 12.
    Testing in the3. Millennium • Stability and reliability have always been the hallmarks of the mainframe – Changing test requirements – Changed environment – Shorter time to market – Developers, testers and operations staff are not sitting close to each other any more Testing is changing completely now, and it has suddenly become very interesting. Stability and reliability have always been the hallmarks of the mainframe. But that is not enough anymore. 12
  • 13.
    Testing in the3. Millennium • Stability and reliability have always been the hallmarks of the mainframe – Changing test requirements – Changed environment – Shorter time to market – Developers, testers and operations staff are not sitting close to each other any more We have moved from green screens to client/server, monolithic and spaghetti to modular and structured, from small tight groups to globally dispersed, virtual teams. And compliance requirements are added on top of it all. We live and work in a time of liberalization, economic challenges, concentration of corporations and a focus on shareholder value rather than using the revenue for company building. 13
  • 14.
    Testing in the3. Millennium • Stability and reliability have always been the hallmarks of the mainframe – Changing test requirements – Changed environment – Shorter time to market – Developers, testers and operations staff are not sitting close to each other any more • Instead we got DevOps • Reduce time used to provide new solutions • Test faster and more completely – Test automation – Test virtualization – More flexible test environments – Promotion depending on test and quality assurance DevOps is the current buzzword. It means that we need to be faster and better. All the time. Testing needs to be automated to speed deliveries up. This requires more and more parallel test environments, and it requires that we use test virtualization to make them more independent of each other. 14
  • 15.
    Testing in the3. Millennium • Stability and reliability have always been the hallmarks of the mainframe – Changing test requirements – Changed environment – Shorter time to market – Developers, testers and operations staff are not sitting close to each other any more • Instead we got DevOps • Reduce time used to provide new solutions • Test faster and more completely – Test automation – Test virtualization – More flexible test environments – Promotion depending on test and quality assurance We now have new flexible and modern test environments available, like Micro Focus Enterprise, Compuware Topaz, and the IBM products formerly known as Rational. The Rational products includes the IBM developer, the IBM debugger, Rational Team Concert, all the tools in the Test Workbench, and zD&T, which is a complete mainframe on Intel hardware. 15
  • 16.
    Testing in the3. Millennium • Stability and reliability have always been the hallmarks of the mainframe – Changing test requirements – Changed environment – Shorter time to market – Developers, testers and operations staff are not sitting close to each other any more • Instead we got DevOps • Reduce time used to provide new solutions • Test faster and more completely – Test automation – Test virtualization – More flexible test environments – Promotion depending on test and quality assurance Finally, promotion is increasingly combined with test and quality assurance. Nobody will be allowed to put anything in production, unless they can prove that it is tested, that all the changes are covered by the test, and that coding standards have been followed. 16
  • 17.
    Testing Maturity Introduction toSoftware Testing (Ammann & Offutt) Level 0 •There’s no difference between testing and debugging Level 1 •The purpose of testing is to show that the software works Level 2 •The purpose of testing is to show that the software doesn’t work Level 3 •The purpose of testing is not to prove anything specific, but to reduce the risk of using the software Level 4 •Testing is a mental discipline that helps all IT professionals develop higher quality software So, why do we test in the first place? This is an attempt to discuss testing maturity. If you sit there thinking that testing and debugging is the same thing, you are given a round zero. If you test to prove that the code that I have written does not work, you are some levels more mature. But not too respectful, I might say. If you focus on risk reduction and quality improvement, you score the highest marks. 17
  • 18.
    Typical Software QualityFactors William C. Hetzel, The Complete Guide to Software Testing • Functionality (exterior quality) – Correctness – Reliability – Usability – Integrity • Engineering (interior quality) – Efficiency – Testability – Documentation – Structure • Adaptability (future quality) – Flexibility – Reusability – Maintainability Here are some good factors to consider when you plan your tests. The list itself demonstrates how far modern testing has moved. 18
  • 19.
    Typical Software QualityFactors Hetzel, William C., The Complete Guide to Software Testing • Functionality (exterior quality) – Correctness – Reliability – Usability – Integrity • Engineering (interior quality) – Efficiency – Testability – Documentation – Structure • Adaptability (future quality) – Flexibility – Reusability – Maintainability •How many factors do we actually test for? And it puts the big question to all of us: How many factors do we actually test for? 19
  • 20.
    Release Culture Company A(Old story, possibly an urban legend): – It is better economy to ship early and often – We have a monopoly, and it is cheaper to pay damages (and lawyers) than to have better quality Company B (Real life): – Every version must be both forward and backward compatible – Thorough testing – Wide beta programs The culture aspect is important. It is really a part of your corporate culture. If all that matters are shareholder values for the current quarter, then you risk losing focus on your customers. And in the longer run: Losing focus on your real business. Company A cocktail party. Company B is how IBM is expected to work. 20
  • 21.
    Where do wevalidate our Input? • The 3270 model leaves no choice • The distributed model leaves nothing but choices – In the client? – At entry to the mainframe? – At entry to every module? The first question we meet is: Where do you validate your input? This is traditionally different from mainframes to distributed platforms. But it might be worth reconsidering? 21
  • 22.
    Where do wevalidate our Input? • The 3270 model leaves no choice • The distributed model leaves nothing but choices – In the client? – At entry to the mainframe? – At entry to every module? • An all too true story – Mainframe developers have validated certain data for decades – Front end developers do not ask • They just code the same test in the front end In many shops, there is no consensus in this matter. This results in the same validation happening in different layers of the applications. And even a service catalog will not help you, if nobody is using it, anyway. 22
  • 23.
    Quote of theDay • “If debugging is the process of removing software bugs, then programming must be the process of putting them in” – Edsger Dijkstra • Stolen from http://www.marcofolio.net/ Debugging is not the same as testing. But this quote probably goes for both efforts. 23
  • 24.
    The Real ArchitectureModel Cycle Experience shows limitations Before we dig into some of the many methods, techniques and tools for testing, I would like to remind how architecture models evolve. First somebody gets annoyed by the limitations in their practical work. 24
  • 25.
    The Real ArchitectureModel Cycle Experience shows limitations Improvement ideas Then someone gets some good ideas for improvement. 25
  • 26.
    The Real ArchitectureModel Cycle Experience shows limitations Improvement ideas Bestseller with new terms for everything If one of them is ambitious enough, then he gives it a name and writes a book about it. It is always a “he”, and sometimes the book becomes a best seller. 26
  • 27.
    The Real ArchitectureModel Cycle Experience shows limitations Improvement ideas Bestseller with new terms for everything Part of curriculum This is new and exciting, so some universities put in into their curriculum to show that they are on the leading edge. It is beaten into the heads of the poor students. 27
  • 28.
    The Real ArchitectureModel Cycle Experience shows limitations Improvement ideas Bestseller with new terms for everything Part of curriculum Vendors create new tools No tool vendor wants to be left behind, so they all develop or buy a tool to support it. 28
  • 29.
    The Real ArchitectureModel Cycle Experience shows limitations Improvement ideas Bestseller with new terms for everything Part of curriculum Vendors create new tools Students become architects In the meantime the students graduate. They get jobs, and since they are on the beat with the newest trend, they become architects in the companies, where you work. 29
  • 30.
    The Real ArchitectureModel Cycle Experience shows limitations Improvement ideas Bestseller with new terms for everything Part of curriculum Vendors create new tools Students become architects Everybody must use new terms and tools Now everybody else must use the new tools and terms, or they are fast becoming obsolete. 30
  • 31.
    The Real ArchitectureModel Cycle Experience shows limitations Very soon somebody gets annoyed by the limitations in their practical work, and the cycle starts all over. So, I will not have a slide in this presentation about Test Driven Development, because that might have become obsolete since I started speaking. 31
  • 32.
    Processes Waterfall? – Test planning •Finding an error before it has been made is brilliant • Finding errors late in the project is expensive Agile? – ”Shift Left”? • Finding an error as soon as it has been made Can they be combined? – Or will that be too expensive? The big movement in the development and testing process today is going from waterfall to agile. We know the limitations of the waterfall model, and we are annoyed by them. It is too rigid and too slow for the requirements of today. Errors found in the test phase become expensive, and they can delay a project. But we should try not to throw out the baby with the waterfall. I have seen test planners finding errors that the coders had not yet made, because of the helicopter view of the testers. 32
  • 33.
    Quote of theDay • “Walking on water and developing software from a specification are easy if both are frozen” – Edward V Berard • Stolen from http://www.marcofolio.net/ If an environment is too agile, you might fall through. 33
  • 34.
    Testing Methods • Statictesting – Review – Walk through – Inspection – Verification • Dynamic testing – Black box • Treats software under test as a black-box without knowing its internals. • Tests are using software interfaces and trying to ensure that they work as expected – White box • Looks inside the software that is being tested • Uses that knowledge as part of the testing process To start to discuss and categorize testing methods, we have a distinction between static and dynamic testing. Static is what we do at code review meetings, at the desk top, or at home with a print list and a glass of good single malt. Dynamic is computer aided. It comes in two flavours: Black box, and white box. 34
  • 35.
    Black Box Testing ViktorFarcic: Technology Conversations • Efficient for large segments of code • Code access is not required • Separation between user’s and developer’s perspectives This is a listing of some of the advantages of black box testing. 35
  • 36.
    Black Box Testing ViktorFarcic: Technology Conversations • Efficient for large segments of code • Code access is not required • Separation between user’s and developer’s perspectives • Limited coverage since only a fraction of test scenarios is performed • Inefficient testing due to tester’s lack of knowledge about software internals • Blind coverage since tester has limited knowledge about the application And some of the disadvantages. 36
  • 37.
    Black Box Testing Model-basedtesting Use case testing Specification-based testing All-pairs testing Fuzz testing Exploratory testing These are the main methods for black box testing. They are much more advanced than just entering data into a green screen. They all look at both how to develop your test cases and how to automate them. Model-based testing is used by a method called “Model Driven Design” by defining test cases to test that the model is implemented. 37
  • 38.
    Black Box Testing Model-basedtesting Use case testing Specification-based testing All-pairs testing Fuzz testing Exploratory testing Use case testing uses test cases based on analysis of success and some failure scenarios. Specification-based testing uses Boundary Value Analysis, Decision Tables, and State Transitioning tests. All-pairs testing tests all possible discrete pairs of input parameters. Fuzz testing is an automatic test of invalid, unexpected or random data. And Exploratory testing will develop test cases as you go and get new ideas from experience. This could be the basis of a long list of requirements to your test tool vendors. 38
  • 39.
    White Box Testing ViktorFarcic: Technology Conversations • Efficient in finding errors and problems • Required knowledge of internals of the software under test is beneficial for thorough testing • Allows finding hidden errors • Programmers introspection • Helps optimizing the code • Due to required internal knowledge of the software, maximum coverage is obtained This is a listing of some of the advantages of white box testing. 39
  • 40.
    White Box Testing ViktorFarcic: Technology Conversations • Efficient in finding errors and problems • Required knowledge of internals of the software under test is beneficial for thorough testing • Allows finding hidden errors • Programmers introspection • Helps optimizing the code • Due to required internal knowledge of the software, maximum coverage is obtained • Might not find unimplemented or missing features • Requires high level knowledge of internals of the software under test • Requires code access And some of the disadvantages. 40
  • 41.
    White Box Testing •Debugging (single stepping) • Post-mortem debugging (Dump analysis) • Unit testing • Integration testing • Component Interface testing • System testing • Acceptance testing • Regression testing • Usability testing • Accessibility testing • Security testing • Internationalization testing • Code coverage • Fault injection White box testing calls for a discussion about testing techniques. We will only touch it very lightly, but go into more details about some of the tools. 41
  • 42.
    Code Coverage Criteria •Function coverage • Statement coverage • Branch coverage • Condition coverage (or predicate coverage) Code coverage was one of the first methods for systematic software testing. Modern tools often report on code coverage after testing on a function and statement basis, but we might consider more advanced criteria as well. Branch coverage checks that each branch of each control structure has been executed. For example, given an if statement, have both the true and false branches been executed? Condition coverage (or predicate coverage) checks in the same way if each Boolean sub- expression has been evaluated both to true and false. 42
  • 43.
    Fault Injection Parasoft JTestas an example (AD 1999) – Analyse Java class using introspection – Generate input data intended to break the code – Shoot at it – Create a report on what was broken and how to fix it This depends on the wish to validate on the unit level Fault Injection is wide spread in Java development. Java has a language feature called introspection, which reports which methods can be called with which arguments. This is used to generate automated test cases, that analyses a program, injects all interesting input, and reports on missing validations and error handling. I have once used the tool mentioned here to make a critical Java application bullet proof. 43
  • 44.
    Test Automation Front endtools – Hiperstation – IDz Generic Service Client – TestRunner – SoapUI Host tools – Headless Code Coverage Lifecycle tools – Embed in lifecycle management Automated tests are important for speeding up the development and test life cycle. We can use front end tools that only need to know about the interfaces. Code coverage can run as part of automated test as what is called headless Code Coverage, where it is running as a batch job without a graphical user interface. And they can and should be embedded in your lifecycle management as parts of the promotion procedure. 44
  • 45.
    Quote of theDay • “Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live” – Rick Osborne • Stolen from http://www.marcofolio.net/ Some programmers claim that if a program has been hard to code, it should also be hard to read. Maintainers do not agree. Neither do white box testers. 45
  • 46.
    Security Testing • Authenticationand authorization • Resilience (DoD attacks, injections etc.) – Chaos Monkey (Simian Army) A different area that needs testing is security. We now know that mainframes can be attacked and penetrated. We need to develop formalized tests of authentication, authorization and resilience. This is not only a task for network and firewall departments. 46
  • 47.
    Security Testing • Authenticationand authorization • Resilience (DoD attacks, injections etc.) – Chaos Monkey (Simian Army) An interesting tool has been developed by Netflix engineers to test the resiliency and recoverability of their Amazon Web Services. It was called “Chaos Monkey”, and now they have a complete suite, so they called it “Simian Army”. Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures. 47
  • 48.
    Security Testing • Authenticationand authorization • Resilience (DoD attacks, injections etc.) – Chaos Monkey (Simian Army) • Privacy – The EU ePrivacy Directive – The EU General Data Protection Regulation – Privacy By Design – Privacy By Default – The Right to be Forgotten But not only must we protect our companies and our systems. We also need to protect the security and privacy of our current customers and of others that use our systems and might be our future customers. 48
  • 49.
    Security Testing • Authenticationand authorization • Resilience (DoD attacks, injections etc.) – Chaos Monkey (Simian Army) • Privacy – The EU ePrivacy Directive – The EU General Data Protection Regulation – Privacy By Design – Privacy By Default – The Right to be Forgotten This has long been an ethical requirement, but thanks to visionary and resilient politicians in the European Union we now have “The EU ePrivacy Directive” and “The General Data Protection Regulation”. It must become law in each member state by May 6th 2018. Some of the important principles are “Privacy By Design”, “Privacy By Default”, and “The Right to be Forgotten”. These principles must be obeyed by our application design and our test data management. 49
  • 50.
    Test Data Management •The Achilles’ heel of many test teams • Test data creation – By hand – By tool • CA Test Data Manager (Datamaker) • Compuware’s Test Data Management • InfoSphere Optim Test Data Management – From production data • Masking/Anonymization/Pseudonym ization – EU directive • Test data backup and restore Test Data Management is a very big discussion in itself. There are other presentations on that subject at the conference, such as S503 - The Impact of Inefficient, Ineffective and unsecure Test Data By Kim Mortensen and Gerald Pfeiffer, and S510 - Where does your test data come from? By Bill Alexander. 50
  • 51.
    Test Data Management •The Achilles’ Heel of many test teams • Test data creation – By hand – By tool • CA Test Data Manager (Datamaker) • Compuware’s Test Data Management • InfoSphere Optim Test Data Management – From production data • Masking/Anonymization/Pseudonym ization – EU directive • Test data backup and restore • Synchronizing code changes, data structure changes, and test data – Synchronization Big Iron  zD&T I will just mention that when you are expanding your test environments to a farm of zD&T mainframe systems running on Intel, one of the biggest challenges is precisely test data management: How do you create a consistent and sufficiently large set of test data? And how do you update your test data, when applications change? 51
  • 52.
    z series Developmentand Test Environment • zD&T, RD&T, RDzUT, zPDT, ADCD… • Fast and economic deployment of test environments Z Series Development and Test Environment is the mainframe on Intel hardware. It is a less expensive and much faster way of setting up all the parallel test environments that we need today. Setting up a new z/OS system with out-of-the-box settings can be done in a few days from scratch, and even faster when you have a golden image to clone. 52
  • 53.
    z series Developmentand Test Environment • zD&T, RD&T, RDzUT, zPDT, ADCD… • Fast and economic deployment of test environments – The heavy lift is load and adaptation of data, applications, configurations, and customizations What takes time and effort is to identify and extract applications, data, configurations, and customizations on the big iron, and then to adapt them to the new test environment. 53
  • 54.
    Tools • Service test •Performance test • Unit test Let us now look at some test tools. 54
  • 55.
    Service test Product Company HiperStationCompuware IBM Developer for z Service Client feature IBM LoadRunner HPE Rational Test Workbench IBM SoapUI SmartBear Service test is done by tools running outside the mainframe. They are doing a black box test to check the interfaces and API’s you have exposed to front end clients. These are some of the most popular tools. 55
  • 56.
    Performance Test Product Company ApplicationPerformance Analyzer IBM FreezeFrame Macro4 InTune BMC Mainframe Application Tuner CA SmartTune ASG Startool Serena Strobe Compuware Performance test is done by your usual on line monitors like MainView, Omegamon, Tmon, and Sysview. And with profiler products such as these on this list. 56
  • 57.
    Unit test Product Company Topazfor Total Test Compuware XaTester Xact zUnit IBM Unit test tools have been top of the mainframe testers’ wish list for years. Now they are finally becoming available. IBM has zUnit, which is part of IDZ, IBM Developer for z systems. I will point a presentation about that one. Compuware recently announced Topaz for Total Test. I will point to a presentation on that one too. And Xact also recently announced XaTester, which I will go a bit more in detail on in a moment. 57
  • 58.
    Quote of theDay “A bug is not an error, it is a test that was not written” – Extreme programming So, with a bit of luck, it might be time to say goodbye to some of the bugs? 58
  • 59.
    The New Kidson the Block • Rational Integration Tester • zUnit • Topaz • XaTester Let us go into details om one integration tester and three unit testers. They are two different categories, but in practical use, they seem to overlap. Rational Integration Tester can be used for test virtualization at interfaces like a call to MQ or a distributed program link in CICS. 59
  • 60.
    The New Kidson the Block • Rational Integration Tester • zUnit • Topaz • XaTester ZUnit tests single modules in batch. Topaz test single modules, but can also provide test virtualization for some calls. XaTester was built to test single modules, but user requirements have made it evolve into some integration testing and quite a bit of test virtualization. 60
  • 61.
    Rational Integration Tester RationalIntegration Tester is a scripting-free environment for developing tests for service-oriented architecture (SOA) messaging and business process integration projects. It can virtualize several messaging solutions (for example, JMS, TIBCO, MQ, CICS distributed program link, and CICS Transaction Gateway) by using a plug-in for the different environments. 61
  • 62.
    A Sample Application Imaginean application that is called by a front end, and which communicates with another application using MQ. It could look like this. 62
  • 63.
    Test Virtualization What youwould like to do was to test your application independently of the other application. So, you would like to stub or mock up the MQ call and get the expected result back, even though the other application was unavailable. 63
  • 64.
    RIT Installation You mustset up a server for Rational Integration Tester with a repository. You must install an exit in your MQ system. And, you must install the Rational Test Workbench on your workstation. You can then define that certain MQ calls should be intercepted by the exit and that the requests and responses are recorded by the server when running the application. 64
  • 65.
    RIT Simulation Now youcan define that all requests are redirected to the server, and the proper responses are returned. You can also use an editor to define which data is returned in which fields by the server. 65
  • 66.
    Rational Integration Tester Theuser interface is the Rational Test Workbench, which is shown here. I have been using it in a project, where the response time for the remote application was two to three days. The test virtualization reduced testing time to seconds. 66
  • 67.
    More Sample Applications Thelimitations of the integration tester are seen, when you want to test single modules in batch or online. 67
  • 68.
    Unit Test The unittester is similar to the integration tester, but it operates at the module level. So, we need some more packaging around the module to test it. This is a generic view of a unit tester. 68
  • 69.
    Unit Test Unit testerstypically have graphical user interfaces and tools to record and edit test data. They can have parses that analyses the data definitions of sources of the modules under test. And they can have repositories to save test cases, test suites, and test data. 69
  • 70.
    zUnit zUnit is nota separate product, but a feature of Developer for z Systems. It provides a code-driven unit testing framework for Enterprise COBOL and PL/I. zUnit is an adaptation of the xUnit framework, which is derived from the Java environment. It generates COBOL or PL/I programs that can call the modules under test. Bill Alexander will tell you much more about it in his presentation. 70
  • 71.
    Topaz for TotalTest Topaz for Total Test from Compuware automates both the creation and the execution of unit tests It integrates tightly with Xpediter, but can run stand alone or integrated with IBM Developer. It generates data stubs, program stubs and test result assertions. It can call subprograms with parameters collected by Xpediter It can integrate to tools like SonarCube, and Code Coverage can be used during the test- run. Kim Mortensen will tell you more in his presentation. 71
  • 72.
    XaTester XaTester is aunit test and integration test tool for COBOL, PL/I, EGL, Fortran, RPG and Assembler programs on Mainframe and iSeries. This is the Eclipse based user interface, which can run stand-alone or embedded in IBM Developer for z. 72
  • 73.
    XaTester Web Interface Thisis the web browser interface of XaTester. 73
  • 74.
    XaTester Infrastructure And thisis the infrastructure needed to support all the features. 74
  • 75.
    XaTester Installation To installXaTester you need a separate server for the test engine and the repository. The test engine includes a scheduling feature. You also need to install a host agent, when running on z/OS. 75
  • 76.
    XaTester Installation If youwant to run XaTester in CICS, you also need to install some CICS components. IMS is handled as batch. If you want to run XaTester from Eclipse or from IBM Developer, you also need to install a plug-in on your workstation. 76
  • 77.
    XaTester Extraction First, youmust get the data definitions of the modules under test. COBOL, EGL and RPG can be extracted by XaTester and stored in the repository for later use. Structures for other languages like PL/I, Fortran and assembler can be defined in an editor in XaTester until somebody requires us to build a parser for this language. Or they plug in their own parser. 77
  • 78.
    XaTester Testing You addthe module or modules you want to test in an editor window, where you also define input data. This can be typed directly or you can define an SQL query to retrieve the data. If you have more than one module, you can define the output from one module as input to the next. Then you run the test and examine the result. 78
  • 79.
    XaTester Regression Testing Youcan save the result from at test in the repository. Next time you run the test, the results will be compared to the results of the previous run. Fields like time and date can be excluded from the comparison. 79
  • 80.
    XaTester Code Coverage Ifyou have RDz or IDz on your system, and you have configured the headless code coverage feature, then XaTester can use it to display, which statements have not been tested. 80
  • 81.
    XaTester Mock up Whenyou test a module that calls submodules, you can ask XaTester to generate a mock up module, which can be called to return the recorded values, if that module is not available. This is supported both in batch and in CICS. 81
  • 82.
    XaTester Demo But letus just spend a few minutes to see what such a unit tester looks like in a web browser interface. It is a test of a COBOL program that calculate a risk factor based on the age of the customer. 82
  • 83.
  • 84.
    XaTester Demo Let usspend a few minutes more to see what it looks like in Eclipse. 84
  • 85.
  • 86.
    Feature Comparison 3 UnitTesters, 1 Integration Tester Category XaTester TopazforTT zUnit RIT Runtime Main batch, called batch, CICS, IMS, iSeries Main batch Batch Any User Interfaces Eclipse, Browser Eclipse Eclipse Workbench and browser Programming Languages COBOL, PL/I, assembler, EGL, RPG COBOL COBOL & PL/I Any Execution Environments PC, host and a server PC and host PC and host Server and host Execution integration Stand alone, Idz, local scheduling, submitting to night batch Stand alone, Idz and Topaz Workbench/Xpediter Idz N/A Testing Test suites, Code coverage, Advanced test script support, SonarCube, External reporting, repository with test artefacts Test suites, code coverage, SonarCube, External reporting Test suites Test suites, Repository with test artefacts This is a short comparison chart between the products. Since all vendors are presenting at the conference, you can dig deeper yourselves, if you are interested. None of the tools support DB2 Stored Procedures or Websphere application server. 86
  • 87.
    Feature Comparison 3 UnitTesters, 1 Integration Tester Category XaTester TopazforTT zUnit RIT Runtime Main batch, called batch, CICS, IMS, iSeries Main batch Batch Any User Interfaces Eclipse, Browser Eclipse Eclipse Workbench and browser Programming Languages COBOL, PL/I, assembler, EGL, RPG COBOL COBOL & PL/I Any Execution Environments PC, host and a server PC and host PC and host Server and host Execution integration Stand alone, Idz, local scheduling, submitting to night batch Stand alone, Idz and Topaz Workbench/Xpediter Idz N/A Testing Test suites, Code coverage, Advanced test script support, SonarCube, External reporting, repository with test artefacts Test suites, code coverage, SonarCube, External reporting Test suites Test suites, Repository with test artefacts Nor do they support an ISPF user interface, which I would not expect anybody to want anyway. None of them support C or C++. Both Topaz and XaTester can be invoked from tools like Jenkins. For XaTester external requests can be made using a REST API. 87
  • 88.
    Feature Comparison 3 UnitTesters, 1 Integration Tester Category XaTester TopazforTT zUnit RIT Virtualization Submodules and DB2 (including rollback) Submodules and VSAM/QSAM (Automated using Xpediter) Subroutines, VSAM Really many Integration Apache Ant, CentraSite ActiveSOA, CICS DPL, CTG, DB2, Java, JDBC, Jenkins, MQ, SAG WebMethods, SAP, Tibco, WAS Service Integration Bus Test data creation File import, recording File import, recording File import File import, recording When reading this chart, you should keep in mind that the integration tester does not really compare directly to the unit testers. 88
  • 89.
    The Final Test Weare almost done now, but it is never over without a final test. 89
  • 90.
    The Final Test Wellknown fact A study at Manchester Metropolitan University involving dropping 100 slices under laboratory conditions established that toast typically lands on the floor butter-side-down because of way it is typically dropped from a table, and the aerodynamic drag caused by the air pockets within the bread. 90
  • 91.
    The Final Test Wellknown fact The toast is invariably butter-side-up when dropped. As it falls, it rotates; given the typical speed of rotation and the typical height of a table, a slice of toast that began butter-side-up on the table will land butter-side-down on the floor in 81% of cases. 91
  • 92.
    The Final Test Well known Fact:Cat righting reflex Cats possess the ability to turn themselves right side up in mid-air if they should fall upside-down, known as the cat righting reflex. This enables them to land on their feet if dropped from sufficient height, about 30 cm. 92
  • 93.
    The Final Test So,what happens if you tie a slice of toast to the back of a cat and push it down from a table? And how do you design a practical test for it? 93
  • 94.
    To Summarize • Automatedtesting and quality assurance is a matter of survival • We are expanding the scope of our testing rapidly – But many areas are still more or less unhandled • The main issue is not tools, but processes and focus • Pinpoint the needs of your business – Then define your processes and methods – And finally select the tools that suit your requirements To summarize the presentation. Automated testing and quality assurance is not just something nice to have in this day and time. It is a matter of survival. The scope of our testing is expanding rapidly, but many areas are still more or less unhandled. The main issue is not the tools, but to have the right processes and to keep the focus. You should pinpoint the needs of your business. Then you should define your processes, methods and best practices. And finally, you could select the tools that suit your requirements. 94
  • 95.
    Hand-on Session andmore Information • If you want to know more about zUnit, go to session S510, or catch Bill Alexander • If you want to know more about Topaz for Total Test, go to session S608, or catch Kim Mortensen • If you want to play with XaTester, you can join the hands-on session at 15.15, or catch Steen Brahe at the demonstration booth in the coffee break area 95
  • 96.
    FOR MORE INFORMATIONPLEASE CONTACT Xact Consulting A/S Arnold Nielsens Boulevard 68A DK-2650 Hvidovre +45 7023 0100 info@Xact.dk www.Xact.dk Enterprise Modernization Moderniza- tion Conversion Business Software Consultants Thank you for listening to this presentation about testing. 96