Some of you are from IT background and some of you are from Non-IT, however there is no difference in the work that you will be doing once you are assigned to a project.
What is a software product? A software product is made up of software packages that are separately installable. Ex. Word, online sales application, What are the activities involved when you think about developing a project? Marketing Survey is done by marketing people in the market for various product and benchmark the product. MRS( Marketing requirement survey) is created at the end of the survey. Software Requirement Analysis This is also known as feasibility study. In this phase, the development team visits the customer and studies their system. For a few projects, technical feasibility is a significant concern: Is it technically feasible to build a Star Wars missile defense system? Is it technically feasible to build a natural language English-to-French translator? For most projects, however, feasibility depends on non-technical issues: Are the project’s cost and schedule assumptions realistic? Does the project have an effective executive sponsor? Does the company have a business case for the software when the real cost—rather than the initial blue-sky, wishful-thinking cost—is considered? Ex- Contractor A and B made flyover at Domlur but they did not exchange/discuss their blueprints hence there is a gap of around 6 mtrs. By the end of the feasibility study, the team furnishes a document that holds the different specific recommendations for the candidate system. It also includes the personnel assignments, costs, project schedule, and target dates. The requirements gathering process is intensified and focussed specially on software. To understand the nature of the program(s) to be built, the system engineer (&quot;analyst&quot;) must understand the information domain for the software, as well as required function, behavior, performance and interfacing. The essential purpose of this phase is to find the need and to define the problem that needs to be solved . System Analysis and Design In this phase, the software development process, the software's overall structure is defined. In terms of the client/server technology, the number of tiers needed for the package architecture, the database design, the data structure design etc are all defined in this phase. A software development model is created. Analysis and Design are very crucial in the whole development cycle. Any gap in the design phase could be very expensive to solve in the later stage of the software development. Much care is taken during this phase. The logical system of the product is developed in this phase.
Code Generation The design must be translated into a machine-readable form. The code generation step performs this task. If the design is performed in a detailed manner, code generation can be accomplished without much complication. Programming tools like Compilers, Interpreters, Debuggers are used to generate the code. Different high level programming languages like C, C++, Pascal, Java are used for coding. With respect to the type of application, the right programming language is chosen. Formal reviews are conducted after every 500 lines of code e.g. Code Inspection and code walkthrough What happen if we send this software directly to production? What all issues we may face? If we send the software directly to production without thorough testing, system may fail at the initial stage itself, and even if it doesn’t there may be some functionality which was not even thought of and tested. e.g. in online shopping system add products to cart and one page can hold description of ten carts only and then next page appears with additional product added to cart, but it was never tested, every time cart with less than 10 products was tested, hence testing of more than 10 products in a cart was skipped and leads to bad impression on customer. Will the customer again want to visit the site?? Off course NO What else a customer expect from any application other than perfect functionality? Look and Feel:- If the application is very dull, not very well presentable then also customer would not like to visit that site again. user-friendly:- If the application is not user friendly, instead of common language understood by people some other language is used or it is not easy to browse it, searching is not convenient etc then even though functionality of the application is perfect still this application would be a failure as the end- user is not at all satisfied. Testing Once the code is generated, the software program testing begins. Different testing methodologies are available to unravel the bugs that were committed during the previous phases. Different testing tools and methodologies are already available. Some companies build their own testing tools that are tailor made for their own development operations. Maintenance Software will definitely undergo change once it is delivered to the customer. There are many reasons for the change. Change could happen because of some unexpected input values into the system. In addition, the changes in the system could directly affect the software operations. The software should be developed to accommodate changes that could happen during the post implementation period.
Requirement analysis:- As we were talking about feasibility study, which also includes working in team, work cohesively with other people engaged in the similar or other activities. Eg. The flyover that is being constructed at Domlur airport. Contract was given to 2 contractors, they started constructing flyover without exchanging their blue prints, result was that once they finished their part of flyover there was a gap of almost 6 mtrs in height. Had they have exchanged their blueprints they would have constructed it much earlier and without any disruption and disturbance and loss of money and effort. Now they have to do some workaround as it has to be completed somehow… Why these situations occur??? Majorly due to lack of proper testing. Overconfidence Even the software developed by best of the developers may fail. It is not only the system that is being developed is tested but the system with which our product is interfaced that is also tested for interfacing not the product as a whole.
As per IEEE Standards, SDLC broadly consist of 3 phases:- Definition Development Maintainence
As we have talked about that we cannot launch any product successfully without testing. Even the product designed by best of the best developers needs to be tested and verified under various circumstances. The U.S. space shuttle Columbia with a seven-member crew, disintegrated in flames over central Texas shortly before it was scheduled to land at Cape Canaversal in Florida. On February 1, 2003, at an altitude of 63 kilometers and a velocity of 20,000 km/hr, Columbia disintegrated killing 7 astronauts. Columbia was designed and developed by most of the intelligent people employed in NASA. And the product was tested very well before its launch, still it failed leading to disaster and taking lives of 7 best astronauts that includes Kalpana Chawla from India. The product failed not because it was not tested well but it could have been tested more to avoid this disaster. Hence testing is very vital in all application/products however more aggressive where lives are involved for e.g medical equipments, space shuttles etc.
What Is Software Testing? Software testing is a process used to help identify the correctness, completeness and quality of developed computer software. With that in mind, testing can never completely establish the correctness of computer software. Only the process of formal verification can prove that there are no defects. What is the expected behavior of the system? Testing is to check if there is no difference between actual output and expected output. It is to put software under various test to check that it satisfies some requirements of end user/Client and hence detecting errors if it fails to satisfy a set of requirements.
Testing is to establish confidence that a system does what it is supposed to do and even if it “misbehaves” it should not crash, it should do it decently. It is to confirm that system performs its intended functions correctly, moreover testing does not guarantee bug free product and good testing cannot be a substitute for good programming ex. If I do a very poor programming assuming that if there are any bugs my testers would find it and I would fix them later. This idea would take more time plus more effort in debugging those many number of bugs and in all the cost would be much higher than it would have been if the code has been developed effectively. Testing is not a debugging or preventing process, it is only to ensure that within an application I have found/detect these many number of bugs and rest debugging/fixing of bugs is done by developers. It improves quality and identify risks as well.
As we were discussing about the need of software testing… hence we test our software to :- Detect programming errors:- Even though we have best developers developing the product still there may be flaws somewhere due to misconception, due to lack of understanding in integrating modules of 2 great developers. Most of the time developers think that their code is perfect, it cannot have bugs and that’s where the problem is. They think that finding bugs in their code means that some one is proving them inefficient. Hence we test code not to prove a developer inefficient but to find bug in code not in developer. Testing at earlier stage is much cheaper than testing at later stage. If Client find bugs which were uncovered earlier then we loses credibility and may lose business as well. There are several functionality related that occur in real life scenarios related only so while testing at customer site during beta testing we can uncover these defects there and then. To release a bug free product seems to be a dream yet to be fulfilled, hence it is a real challenge to release a product that is bug-free.
Testing is defined in a number of ways by various veterans. Of course, none of these definitions claims that testing shows that software is free from defects. Testing can show the presence, but not the absence of problems.
What is the criteria for a successful product? A product is successful only when it is adding any value to customer business, only if it increases its revenue, credibility in the market and so on. To develop a product we set a target time for completion and released to customer. If we can stick to that time without increasing budget and scope of the product then the product is successful. Using the developed product true business goals should be met else the product developed can be a failure leading to finance loss, customer satisfaction loss, and loss of business further from that client as well as associates. Look and feel should be accepted by user who will be finally using that product.
Product Success Criteria (PSC) so as to address the following. These criteria, once arrived at the initial stage of the project, shall be the basis for strategy and planning. Functionality: The business perceives that the system satisfactorily addresses the true business goals. The system is functionally compliant. The system was delivered on time and within budget and scope. Usability: The user perceives the system as value-added to his or her job and is easy to learn and use. Likeability: End user feels that look, feel, and navigation are easy. Configurability: All changes to project scope, schedule, and budget were handled under a well-defined and visible change control process. Maintainability: The operations staff is prepared to maintain and support the delivered system. Interoperability: The organizational interoperability groups perceive that the solution is consistent with interoperability and reusability standards and goals.
Correctness To claim that software is correct, we have to assure that the data entered, processed, and outputted is accurate and complete. We can achieve this through controlling data element across the entire transaction. This requires that: Well defined Functional specifications. Design conformance with user requirements. Program conformance to design specifications. Testing to ensure requirements are properly implemented. Availability of right programs and data in the released system. Proper management change requests Reliability This will ensure that the system will perform its intended function with the required precision over an extended period of time. This shall ensure: Establishing the system in operational environment to meet the expected level of accuracy and completeness. Designing and implementing process tolerance through the data integrity controls. Performing manual, regression, and functional tests to ensure proper functioning of the data integrity controls. Verification of the installation for system accuracy and completeness. Maintaining accuracy of system requirements and also, system updation. Ease of Use This will address effort required to learn, operate, prepare input for, and interpret output from the system. This test factor deals with the how quickly and easily the people interfacing with the application system will learn to use it. The points addressed here are: Defining the usability specifications for the application system. Design to optimize the usability related requirements Conformance of programs to the design in order to optimize the ease of use. Testing to ensure that the application is easy to use. Proper presentation and communication of usability instructions to appropriate individuals. Preserving usability as and when the system is maintained. File integrity This ensures appropriate storage and maintenance of data. This requires that: Defined file integrity requirements. Controls in design that ensure the integrity of the file. Proper implementation of specified file integrity controls. Proper testing to ensure proper functioning of the file integrity. Verification of file integrity to ensure delivery of right riles prior to system release. Preserving integrity of the files during the maintenance phase. Maintainability This factor addresses effort required to locate and fix an error in an operational system so that system operates smoothly. This addresses: Specification of the desired level of maintainability of the system. Development of design and program to achieve the desired level of maintainability. Inspection of system to ensure that it is maintainable. Verifying system documentation for completeness. Maintainability is preserved as the system is updated.
02/09/12 The development process must be iterative-- let’s see how that works The traditional development process involves a long building phase after you’ve specifiedd the application. After the app is built, you move to a testing phase where you find bugs and fix them. Many people view testing as a “washing machine” that you clean your code with at the end of the development phase.
As we have already discussed Software Development V- model, Let us compare it with v-Model of software testing life cycle.
The responsibility for testing Unit Test is the responsibility of the Development Team System Testing is the responsibility of SQA User Acceptance Testing is the Responsibility of the User Representatives Team Technology Compliance Testing is the responsibility of the Systems Installation & Support Group.
During acceptance lets see what do we mean by hot-Fixes. In acceptance phase product is installed and executed in users environment. There may be some failure to software might be due to some configuration files mismatch or some other interface. When Bill Gates was giving Live demo for Microsoft windows new version suddenly a Unwanted popup appears and whole world was watching this demo, Bill did some changes to config files and it worked fine. This is termed as HOT_FIX( to make small changes to software to continue testing activity)
Role of test engineer is to :-
02/09/12 module interfaces are tested to ensure that the information properly flows into and out of the program unit under test. The local data structure is examined to ensure that data stored temporarily maintains its integrity during all steps in an algorithm’s execution. Boundary conditions are executed to ensure that the module operates properly at boundaries established to limit or restrict processing. All independent paths (basis paths) through the control structure are exercised to ensure that all statements in a module have been executed at least once All error handling paths are tested.
02/09/12 Step1. Unit test criteria is a narrative describing what will be done to verify that the program code operates properly and according to specifications. The unit test plan is doc that defines the process for testing the code, including descriptions and procedures for all tests and evaluations. These above items and test specifications are prepared before the program or module is coded. Step2. Code walk through is done to review to the requirements, design, and coding before the testing to ensure compliance to the IS standards. Step3. Unit test data are specific values or transactions that are created for the purpose of testing the code.Test data should be meticulously prepared so that a full range of data values and combinations are considered, and all instruction paths within the module are executed. The unit test report records the results and findings of the test and provides the basis for determining if the unit is ready for system testing.
02/09/12 Select test data to exercise each aspect of functionality identified in the program’s requirements specifications Functional Testing of word processor calls for using each required feature of of the program at least once. Such features include editing, formatting, search etc
02/09/12 Tests module against functional and non-functional specifications Specification used to derive test cases Do not look at module code (except to debug) Attempt to force behavior that doesn't match specification Problem – how to select inputs that have a high probability of causing error
02/09/12 regression testing aims to assure that the software continues to perform according to specification after it has been modified.
02/09/12 Definitions ‘ Regression testing is the process of testing changes to computer programs to make sure that the older programming still works with the new changes.’ ‘ A technique for ensuring that all aspects of a software system operate as required after enhancement or repair.’
02/09/12 Manual Testing: : Here, the developer/quality assurance team manually tests each and every path that the software code can take and then compares the results by ‘eyeballing’. This approach takes a lot of time and a lot of manpower. It is successful most of the times, but not efficient enough. Hence the need for automating the whole process...
02/09/12 Why? -- If no errors are found during testing, the project team did not test the system sufficiently. Exhaustive testing is impractical, so the Project team must design and plan a testing strategy that utilizes a balance of testing techniques to cover a representative sample of the system. The test planning process is a critical step in the testing process. Without a documented test plan, the test itself cannot be verified, coverage cannot be analyzed, and the test is not repeatable Repeatable : Once the tests are documented, any member of the test team should be able to execute the tests. If the test must be executed multiple times, the plan ensures that all of the critical elements are tested correctly. Parts or the entire plan can be executed for any necessary regression testing. Controllable : Knowing what test data is required and what the expected results are. Coverage : Based on the risks and priorities associated with the parts of the system, the test plan is designed to insure that adequate test coverage is build into the test. The plan can be reviewed by the project team to insure that all are in agreement that the correct amount and types of tests are planned.
02/09/12 When? – Should begin at the same time requirements definition starts. The plan can then be detailed in parallel with application requirements.
Test plan identifier:- Shall contain full identification of the system and the software to which this document applies, including, as applicable, identification numbers(s), title(s), abbreviation(s), version number(s) and release number(s). Introduction:- 1. System Overview Shall briefly state the purpose and nature of the system and the software to which this document applies. Summarize the history of system development, operation, and maintenance. Identify the project sponsor, acquirer, user, developer, and support agencies, current and planned operating sites and list other relevant documents. 2. Document overview: Shall summarize the purpose and contents of this document and shall describe any security or privacy considerations associated with its use. 3. Relationship to other plans: Shall describe the relationship, if any, of the Software Test Plan to related project management plans. Test items/Integrated components:- Shall identify a Unit, subsystem, system or other entity by name and project-unique identifier and shall be divided into the following subparagraphs to describe the testing planned for the item(s). 1.Project-unique identifier of a test: Shall identify a test by project-unique identifier and shall provide the information specified below for the test. 1. Test objective 2. Test level 3. Test type or class 4. Qualification method(s) as specified in the requirements specification 5. Identifier for the System requirement and, if applicable software system requirements addressed by this test. 6. Special requirements (for example, 48 hours of continuous facility time, weapon simulation, extent of test, use of a special input or database) 7. Type of data to be recorded 8. Type of data recording/reduction/analysis to be employed 9. Assumptions and constraints, such as anticipated limitations on the test due to system or test conditions- timings, interfaces, equipment personnel, database etc. 10. Safety, security and privacy considerations associated with the test Features to be tested:- It list all the feature of application that needs to be tested. Features not to be tested:- It list all the features of application that are not to be tested.
02/09/12 Focus is on individual tests and small groups of related tests.
Bug Triage Meetings (sometimes called Bug Councils) are project meetings in which open bugs are divided into categories. The most important distinction is between bugs that will not be fixed in this release and those that will be. Triaging a bug involves: Making sure the bug has enough information for the developers and makes sense Making sure the bug is filed in the correct place Making sure the bug has sensible &quot;Severity&quot; and &quot;Priority&quot; fields Let us see what Priority and Severity means Priority is Business; Severity is Technical In Triages, team will give the Priority of the fix based on the business perspective. They will check “How important is it to the business that we fix the bug?” In most of the times high Severity bug is becomes high Priority bug, but it is not always. There are some cases where high Severity bugs will be low Priority and low Severity bugs will be high Priority. In most of the projects I worked, if schedule drawn closer to the release, even if the bug severity is more based on technical perspective, the Priority is given as low because the functionality mentioned in the bug is not critical to business. Priority and Severity gives the excellent metrics to identify overall health of the Project. Severity is customer-focused while priority is business-focused. Assigning Severity for a bug is straightforward. Using some general guidelines about the project, testers will assign Severity but while assigning a priority is much more juggling act. Severity of the bug is one of the factors for assigning priority for a bug. Other considerations are might be how much time left for schedule, possibly ‘who is available for fix’, how important is it to the business to fix the bug, what is the impact of the bug, what are the probability of occurrence and degree of side effects are to be considered.
Software Requirement Specifications IEEE 830 : Software Requirement Specification is a means of translating the ideas in the minds of the clients(the inputs) into a set of formal document (the output) of the requirement phase The Role Bridge the communication gap between the client the user and the developer
Conventional Testing Process Design Build Test & Fix * Here testing was happening only towards the end of the life cycle Spec
Distribution of Defects in the life cycle Source: IBM/TRW/Mitre 27% 7% 10% 56%
Software development life cycle Operational Testing Ongoing Support Requirement Analysis High level design Detailed Specifications Coding Unit Testing Integration Testing Review/Test
STLC-V Model Requirement Req. Review Design Design Review Code Code Review Develop Unit Test Review Unit Test Execute Unit Test Execute Integration tests Execute System Tests Develop integration Tests Review Integration Tests Develop Acceptance Tests Review Acceptance tests
Test Process :- The project under development or incorporation of accepted changes in the project or project under maintenance which implemented changes, use the testing process. Based on the nature of the project, adequate testing shall be arrived at the project level.
The Test Approach
sets the scope of system testing
the overall strategy to be adopted
the activities to be completed
the general resources required
the methods and processes to be used to test the release.
details the activities, dependencies and effort required to conduct the System Test.
valid, invalid and limit data input; screen & field
look and appearance, and overall consistency
with the rest of the application.
Vertical First Testing : - When the complete set of functionality is taken for one module and tested it is called Vertical First testing. Horizontal First Testing : - If a similar function is taken across all the modules and it is tested, it is called horizontal-first testing. Vertical Horizontal
Top-Down is an incremental approach to testing of the program structure. Modules are integrated by moving downward through the control hierarchy, beginning with the main control module, this could be done as depth- first or breadth-first manner.
Both normal cases and exceptions should be tested on both sides of the interface (if both sides exchange data). The interface should be tested for handling the normal amount and flow of data as well as peak processing volumes and traffic.
If appropriate, the batch processing or file transmission “window” should be tested to ensure that both systems complete their processing within the allocated amount of time.
If fixes or changes need to be made to either side of the interface, the decisions, deadlines and re-test procedures should be documented and distributed to all the appropriate organizations.
“ Selective retesting to detect faults introduced during modification of a system or system component, to verify that modifications have not caused unintended adverse effects, or to verify that a modified system or system component still meets its specified requirements.”
“ ...a testing process which is applied after a program is modified.”
During Y2K code changing, regression testing was the essence of the transition phase. What was typically done, was that code was changed at multiple places (it did not turn the original logic upside down, but made subtle changes). Now Regression testing was very important for the fact that even one small piece of code lying untested could lead to huge ramifications in the large amounts of data that is typically handled by these mainframe computers / programs.
Regression testing might even be required when one of the business associates changes his systems (might be new hardware). Since our system is hooked on to this transition system, our test engineers are also required to do regression testing on our system which has NOT been changed.
This example brings to light another fact with Regression testing, i.e., sometimes, even an unchanged system needs to be tested!
Includes a greater focus on regression testing, on keeping the users informed of specific fixes or changes that were requested.
Test process should be described in terms of the periodic release cycles that are part of the change control process.
Also describe a set of minimum tests to be performed when emergency fixes are needed (for instance, due to failed hardware or recovering from a database crash).
Test Strategy: Inputs & Deliverables Test Strategy Priority & Criticality Types of Applications Project Success Criteria Time Required for Testing No. & Levels of Resources Rounds of Testing Exit Criteria Test Suspension Criteria Resumption Criteria Deliverables Time Required for Testing No. & Levels of Resources Rounds of Testing Exit Criteria Test Suspension Criteria
Typical Test Issues Test Participation Test Environments Approach to Testing External Interfaces Approach to Testing COTS products Scope of Acceptance Testing Verification of Un-testable Requirements Criteria for Acceptance of the System Pilot of Field Testing Performances and Capacity Requirement/Testing Test Issues
Common Test Related Risks and Considerations Test Related & Risks & Considerations Poor Requirements Stakeholder Participation Test Staffing Testing of COTS External Interfaces Performance and Stress Testing Schedule Compression Requirement Testability Acceptance
Test planning process is a critical step in the testing process. Without a documented test plan, the test itself cannot be verified, coverage cannot be analyzed and the test is not repeatable Importance
Test plan To support testing, a plan should be there, which specifies
Scripts and test cases will be needed for most tests
Structure 1. Test plan identifier 2. Introduction 3. Test items / integration components 4. Features to be tested 5. Features not to be tested 6. Test Approach 7. Item pass/fail criteria 8. Suspension criteria and resumption requirements Continue.. * As defined by the IEEE 829 Test documentation Std
Structure 9. Test deliverables (PPlan) 10. Environmental needs (H/w & S/w) 11. Responsibilities (PPlan) 12. Staffing and Training needs (PPlan) 13. Schedule (PPlan) 14. Risks and Contingencies (PPlan) 15. Approvals Ref : Test Plan Template IEEE 829
Test Case Sheet To Capture Details: 1.Testcase ID (should be unique, e.g.: c_01.1, c_01.1a, c_01.2,…) 2.Feature functionality to be tested (each Requirement/feature could be from Usecase/COMP) 3.Test Description/ test input details (test input, test data, action to be performed to test the feature, complex test cases be split to more than one) 4.Expected behavior ( in messages, screens, data, to be with correct details) 5.Actual and Status
To evaluate a software program, check conformance to standards, guidelines and specifications
Educating / Training participants
Improve software program
Consider alternative implementation if required (not done in inspections)
Difference between Inspections and Walkthroughs Walkthrough process includes Overview, little or no preparation, examination (actual walkthrough meeting), rework and follow up. Inspection process includes Overview, preparation, inspection, rework and follow up. No checklist used in walkthroughs Checklist is used to find faults Usually team members of the same project take participation in the walkthrough. Author himself acts the walkthrough leader. A group of relevant persons from different departments participate in the inspection.
Difference between Inspections and Walkthroughs Contd. Shorter time is spent on walkthroughs as there is not formal checklist used to evaluate the program. Inspection takes longer time as the list of items in the checklist is tracked to completion. No formalized procedure in the steps. Formalized procedure in each step.