Define Necessary Reliability of RMS Reliability is a measure of continuous delivery of the correct service or equivalently, of the time to failure. In other words, it is the probability that the software will work without failure for a specified period of time (Far, 2005).
Administrator – Has the full right to setup different areas of the core system
Realtor - Searches for the properties by their own realtor company or may use the built-in query manager to inquire about the details of a particular property. The registered realtors get notification as soon as any property comes into the market
Customer - Inquires about their desired house in the preferred community based on several different home design criteria
How will you define the failure of the product (I)
Incorrect user input/selection/search criteria may result incorrect or invalid property information or end up with an error message or warning - Mild
Correct user selections but incorrect input registered by the system may result in incorrect/invalid property information/no property information - Mediocre to Severe
Correct user access Information but incorrect system access or access different user type preferences/information. Which may result severe damage of existing system information (Severe)
How will you define the failure of the product (II)
Failure in the initial connection to the online RMS (Severe)
Failure to print correct property information (Mediocre to Severe)
Failure administrative access for the System Management (Severe)
Failure to add new property information (Severe)
Failure to update existing property information [Such as Offer Made or Cancelled] may result in invalid information access (Severe)
Choose the time unit you will use for the product?
Serviced in 7X24 basis
7 Days x 24 Hours = 168 Hours per week
8760 hours yearly
Most transaction would happen between 09:00 AM – 09:00 PM [84 Hours/week]
Overall it is expected to service at least 100 hours/week
Set the Failure Intensity Objective (FIO) – (I)
Accommodates 600 Realtors/Customers
Each Property Search – approx. 30 min.
In total Search feature can have (100x600) / 0.5 [30 min] = 1,20,000 Transactions
Administrator uploads 50 Property/day of which each property – 5 Query Processes = 50x5x7 Days = 1,750 Transactions
Maximum transactions expected = 1,20,000+1,750 = 1,22,000 transactions/week
Set the Failure Intensity Objective (FIO) – (II)
Search Properties – Expected FIO is 1 Failure/1000 operational hours
Print Properties 1Failure/70,000 print jobs
So, 99.5% transactions without failure
Development Platform Type Platform Language C# Platform .NET Framework 1.2 Database SQL Server 2000 Server System Windows 2000 Server Database SQL Server 2000 Server System Windows 2000 Server Web Server Internet Information Server (IIS) Mail Server SMTP Mail Server
Hardware Specification Web server SQL server CPU: 2 x 2.8 GHz Pentium 4 Memory: 2 GB Disk: 80 GB Network: 100BaseT CPU: 4 x 2.8 GHz Pentium 4 Memory: 4 GB Disk: 4 x 40 GB RAID 0 Network: 100BaseT
Availability of Hardware Platforms Source: ( http://www.microsoft.com/technet/itsolutions/citsrv/ib/msib2tca.mspx ) Cluster One week One month Internet-facing firewall NLB cluster 0.998867 0.999735 Web NLB cluster 0.999092 0.999788 Search NLB cluster 0.999498 0.999883 Internal firewall NLB cluster 0.999197 0.999813 SQL Server cluster 0.999504 0.999884 Total availability 0.996164 0.999103
Internal System Access: 10 failures / 1,000,000 transactions
Determine the product-development software failure intensity objective (II)
Thus the target FIO of the system ranges from 10 failures / 1,000,000 transactions to 20 failures/1,000,000 transactions. Therefore the product-developed software failure intensity objective ( DEV ) is the target FIO minus the Platform FIO ( SYS ) which ranges from 0 to 15 failures / 1,000,000 transactions (20 – 5 = 15)
Lets say, one time cost is $50,000. So, According to COCOMO’s Design – 1 Failure/50,000 Transactions or 20 Failure/1,000,000 Transactions [It is based on Approximated Cost and Transaction Mapping Criteria]
How will you balance the fault prevention, fault removal and fault tolerance strategies
Fault Prevention: 60%
Initial Investigation and Requirement review: 30%.
Design review: 30%
Fault Removal: 30%
Code inspection: 20%
Unit test: 5%
System test: 5%
Fault Tolerance: 10%
ISO 9000 - 3: 4.1.1 Code compliance – Quality Assurance
ISO 9000- 3: 184.108.40.206 – Record of Issues and statistics
ISO 9000-3: 4.4.1 - Code convention and sharing
Better Exception Handling and ability to going back to it’s original state
The Realtor Management System (RMS) is the true solution for the Realty Industry in the recent years. It provides the valuable information of the desired property within a very short while. It is user-friendly and fast. It meets the requirement of different levels of user according to their requirements and access privileges. The Reliability of the system has been measured to produce target system performance. The Failure Intensity Objective (FIO) of RMS has been set at 15 failures/1,000,000 transactions.
Developing Operational Profile of RMS Developing operational profiles will give us the information about how users will employ the product that we are building so that we can focus our development and test resources. Operational profile is the set of operations (operation names and frequencies) and their probabilities of occurrences
The Operational mode of the system varies based on the following system design criteria: Type of User, Tasks Usage, Volume Time, External Systems etc. To make it brief and simple, the operational mode of RMS depends on three types of users of the system. The brief design of the system has been illustrated by using the class diagrams and sequence diagrams of each type of users (Administrator, Customer, and Realtor).
Preparing for Test In preparing for test, we apply the operational profile information that we have developed to planning for an efficient test. Our failure intensity should meet the FIO (Failure Intensity Objective) of RMS which is 15 failures / 1000,000 Transaction.
The process for preparing test cases involves:
Estimate the number of test cases we need to prepare
Allocate the number of test cases among the associated and sub-systems to be tested
Allocate the number of test cases for the developed product to its operations and
determine how will we divide hours of test among the associated systems you have defined
Specify the test profile for one of the operational modes
Document the test cases
Determine the number of hours we will devote to feature, regression (if needed), and load test for the product
Allocate the hours of load test among the operational modes
148 test cases are sub divided into the following sections
At this stage, we will decide what will be out testing phase timing. From thorough research and knowledge we can guess that the whole testing phase would take approximately 8 weeks with a team of 2 support people. Based on our experience, we can assume that it would take 4 hours to prepare for a test. In total, it the number of test cases would be:
N = 8 weeks X 2 People X 40 Hours/Week/4 Hours/Test = 160 test cases (Which is closed to 148 cases mentioned above)
Therefore, the theoretical number of test cases is 160 and the more realistic number is 148 test cases.
Number of test cases among the associated and sub-systems and hours of test (I)
Out of 148 test cases, we have found the following:
Since the total number of hours for a complete test is 592 hours (4 hours/test x 148 tests). So, the number of hours devoted to the testing of the system (Developed System) would be 528 Hours (592 Hours X (132 Tests/148 Tests)).
Hardware – 6 test cases x 4 hours/test = 24 hours
OS & IIS – 3 test cases x 4 hours/test = 12 hours
Network and Other systems – 7 test cases x 4 hours/test = 28 hours
Number of test cases among the associated and sub-systems and hours of test (II)
So, for theoretical calculations, 160 test cases equal a total testing time of 640 hours (160 tests x 4 hours/test). It includes: 24 hours for hardware, 12 hours for the Operating System and IIS, and 28 hours for Network and Other systems
Allocating number of test cases for the developed product to its critical operations
Upload new property listing
The selection criteria are being displayed according to the parent selections
The right property listing is being displayed based on specific search criteria
Taking the printout of the selected property and make sure that the necessary information are being printed correctly
Test profile for one of the operational modes (Realtor) Test Case Description Startup Realtor operates startup page Update Profile Realtor updates profile Search Property by Listing Number Realtor searches property by listing number View Today’s Listing Realtor can see today’s listed properties Search Property Realtor searches property Show Property Realtor receives selected property information
Test Objective: To test functionality for the Realtor to Log into the system
Item No: 1
Test Condition: Login is successful
Enters the login ID into the User ID text box
Enters password into the Password text box
Press Login button
Valid User ID
Message “Login Successful”
Shows Realtor Main Page
Pass or Fail: Pass
Hours devoted to feature, regression, and load test for the product (I)
Theoretically, 640 hours (160 tests x 4 hours/test) needed to perform 160 test cases. It includes: 24 hours for hardware, 12 hours for the Operating System and IIS, and 28 hours for Network and Other systems. Means in total 64 hours are for the associated systems. So, 576 hours should have to be allocated for Feature, Load and Regression testing of the product. Since load testing requires actual customer, it will take the maximum amount of time. So, we have allocated 91.7% (132 Test Cases for developed softwarex4 hours/each test=528 Hours). And, we have allocated 4.15% (24 Hours of testing for Feature and Regression Testing). The regression test may take more than the estimated time based on the number of bug fixes and the complexity of the fix.
Hours devoted to feature, regression, and load test for the product (II) Testing Type Hours Percentage Feature Testing 24 4.15% Load Testing 528 91.7% Regression Testing 24 4.15%
Allocating the hours of load test among the operational modes
We allocated 528 hours for load testing. Out of 528 Hours, we allocate 424 Hours for different operational Modes.
And, the rest 104 hours have been allocated (528-424 = 104) for the attached software/database system of the product. Operational Modes Test Cases Hours Administrator 68 272 Customer 14 56 Realtor 24 96 Total 106 424
A fully tested system can guarantee a stable system which is being perceived by the end users. So, Testing is a very important part of software development. The more error we can encounter and fix before going live, the less failure we will be having after implementation. This will reduce the cost and increase user satisfaction on the product. We are pretty confident that the allocated test cases and their corresponding hours can deliver a real stable system that will achieve 15 failures in 1,000,000 transactions as per our Failure Intensity Objective.
Execute Test Realtor Management System (RMS) is at the final phase. Development is completed and ready for Reliability Testing.
As discussed, we have three user types having different operational modes. Customers utilize the system mostly. But, we have taken the Realtor use cases as it covers almost all the functionalities of a customer with some additional features and modules, which has more complexity and functionality.
There is a supplier risk of 1% of wrongly accepting the software when its failure intensity objective is equal to or bigger than twice the failure intensity objective. Since the customer or realtor mostly selects the options of the query system rather than entering some valid values and internal search engine is responsible for correct query results, the system should have minimal errors to mention. The Administrator mostly enters the system values and he is regarded as a power user of the system, who is experienced of handling the system effectively and efficiently, the errors would be less. So, there is a very small percentage of chance of wrongly accepting the software. But, as the system is integrated with several different external systems there is still some chance of failure.
There is a risk of 1% of wrongly rejecting the software when its failure intensity objective is equal to or less than twice the failure intensity objective. Based on the design, development and testing phase of the system, we are confident of keeping very low failure intensity of the product. But, as it is integrated with external systems, still there is a risk of rejecting the software wrongly.
The testing phase has been completed and the failure data has been collected to guide decisions. We have tested the system only for Realtor mode. But, we need to test other two operational modes (Administrator and Customer) and accumulate their failure intensities to have the overall system reliability. As described in Part 2, we need to develop the system with failure intensity of 15 failures/1,000,000 transactions. Based on the Realtor test cases, the certification graph [Figure 1] demonstrates that the system would reach the accepted result within a very short while. Therefore, we may recommend that the system can be released if the Administrator and Customer test cases results similar to the Realtor test case. And, we are pretty confident to reach our goal as Realtor covers almost all components of a customer with some additional ones. And, Administrator uses the system twice a day while uploading the properties which is the minimal use of the system. And, other components of Administrator Mode are almost one time upload functionality.