Your SlideShare is downloading. ×
Software Reliability and Testing
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Software Reliability and Testing

758
views

Published on


0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
758
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Realtor Management System (RMS) (Project Presentation) Software Reliability and Testing Created By: Md. Fazlul Alam Chowdhury
  • 2. Sections of this Presentation
    • Abstract
    • RMS Workflow
    • Sample screen snapshots of RMS
    • Define Necessary Reliability
    • Develop Operational Profile
    • Prepare for Test
    • Execute Test
    • Apply Failure Data to Guide Decisions
    • Glossary
    • References
  • 3. Abstract
    • RMS is a Real state cataloging Solution designed specifically to help realtors and prospective homebuyers to find the right property at any point of time
    • Development Platform: Windows environment and using Object Oriented Technology
    • RMS helps realtors to increase the productivity by organizing the property information and inquire about the right property more accurately.
  • 4. RMS Workflow
  • 5. Sample Screen Snapshots (Home Page)
  • 6. Sample Screen Snapshots (Login Page)
  • 7. Sample Screen Snapshots (Admin Main Page)
  • 8. Sample Screen Snapshots (Realtor Main Page)
  • 9. Sample Screen Snapshots (Custom Property Search)
  • 10. Sample Screen Snapshots (Property Listing Page)
  • 11. Define Necessary Reliability of RMS Reliability is a measure of continuous delivery of the correct service or equivalently, of the time to failure. In other words, it is the probability that the software will work without failure for a specified period of time (Far, 2005).
  • 12. Components of RMS
    • Functionality of RMS
    • Reliability of RMS
  • 13. Reliability Components of RMS
    • How will you define failure for the product?
    • Choose the natural or time unit you will use for the product
    • Set the product failure intensity objective (FIO)
    • Find the expected product acquired failure intensity, based on the failure intensities of the hardware and acquired software components
    • Determine the product-developed software failure intensity objective
    • How will you balance fault prevention, fault-removal, and fault tolerance strategies?
  • 14. Functional Components (User Types & Interfaces)
    • Administrator – Has the full right to setup different areas of the core system
    • Realtor - Searches for the properties by their own realtor company or may use the built-in query manager to inquire about the details of a particular property. The registered realtors get notification as soon as any property comes into the market
    • Customer - Inquires about their desired house in the preferred community based on several different home design criteria
  • 15. Administrator Use Cases
    • Logging into the system
    • Update Province
    • Update District
    • Update Community
    • Update Property Types
    • Update Property Category
    • Update Property Sub Category
    • Update Realtor Company
    • Update Realtors
    • Update Application Users
    • Update Property Information
    • Update System Settings
  • 16. Realtor Use Cases
    • Logging into the system
    • Update Profile
    • View Today’s Listing
    • Search Property by Unique Listing Number
    • Custom Property Search
  • 17. Customer Use Cases
    • Search Property by Unique Listing Number
    • Custom Property Search
    • Realtor Search
  • 18. How will you define the failure of the product (I)
    • Incorrect user input/selection/search criteria may result incorrect or invalid property information or end up with an error message or warning - Mild
    • Correct user selections but incorrect input registered by the system may result in incorrect/invalid property information/no property information - Mediocre to Severe
    • Correct user access Information but incorrect system access or access different user type preferences/information. Which may result severe damage of existing system information (Severe)
  • 19. How will you define the failure of the product (II)
    • Failure in the initial connection to the online RMS (Severe)
    • Failure to print correct property information (Mediocre to Severe)
    • Failure administrative access for the System Management (Severe)
    • Failure to add new property information (Severe)
    • Failure to update existing property information [Such as Offer Made or Cancelled] may result in invalid information access (Severe)
  • 20. Choose the time unit you will use for the product?
    • Serviced in 7X24 basis
    • 7 Days x 24 Hours = 168 Hours per week
    • 8760 hours yearly
    • Most transaction would happen between 09:00 AM – 09:00 PM [84 Hours/week]
    • Overall it is expected to service at least 100 hours/week
  • 21. Set the Failure Intensity Objective (FIO) – (I)
    • 100 Hours/Week
    • Accommodates 600 Realtors/Customers
    • Each Property Search – approx. 30 min.
    • In total Search feature can have (100x600) / 0.5 [30 min] = 1,20,000 Transactions
    • Administrator uploads 50 Property/day of which each property – 5 Query Processes = 50x5x7 Days = 1,750 Transactions
    • Maximum transactions expected = 1,20,000+1,750 = 1,22,000 transactions/week
  • 22. Set the Failure Intensity Objective (FIO) – (II)
    • Search Properties – Expected FIO is 1 Failure/1000 operational hours
    • Print Properties 1Failure/70,000 print jobs
    • So, 99.5% transactions without failure
  • 23. Development Platform Type Platform Language C# Platform .NET Framework 1.2 Database SQL Server 2000 Server System Windows 2000 Server Database SQL Server 2000 Server System Windows 2000 Server Web Server Internet Information Server (IIS) Mail Server SMTP Mail Server
  • 24. Hardware Specification Web server SQL server CPU: 2 x 2.8 GHz Pentium 4 Memory: 2 GB Disk: 80 GB Network: 100BaseT CPU: 4 x 2.8 GHz Pentium 4 Memory: 4 GB Disk: 4 x 40 GB RAID 0 Network: 100BaseT
  • 25. Availability of Hardware Platforms Source: ( http://www.microsoft.com/technet/itsolutions/citsrv/ib/msib2tca.mspx ) Cluster One week One month Internet-facing firewall NLB cluster 0.998867 0.999735 Web NLB cluster 0.999092 0.999788 Search NLB cluster 0.999498 0.999883 Internal firewall NLB cluster 0.999197 0.999813 SQL Server cluster 0.999504 0.999884 Total availability 0.996164 0.999103
  • 26. Standard Availability vs Downtime Ratio Source: ( http://www.microsoft.com/technet/itsolutions/citsrv/ib/msib2tca.mspx ) Uptime Percentage Downtime /Day Downtime /Month Downtime /Year 95 72.00 minutes 36 hours 18.26 days 99 14.40 minutes 7 hours 3.65 days 99.9 86.40 seconds 43 minutes 8.77 hours 99.99 8.64 seconds 4 minutes 52.6 minutes 99.999 0.86 seconds 26 seconds 5.26 minutes
  • 27. Availability (A) & Downtime/Week (tm) of Hardware Systems
    • So, Summarizing Table 1 and Table 2, we get the following results:
    • Availability (A) = 0.996164/Week means 99.6% Availability/Week
    • Downtime/Week (tm)  3 Minutes x 7 Days = 21 Minutes = 0.35 Hours [As 99.9 represents 86.40 minutes/day so we assume it will be close to 130 minutes/day]
  • 28. Failure Intensity of Hardware Systems 1 – A 1 – 0.996164 0.000897  = = = = 0.00256 / Week A t m 0.996164 X 0.35 0.35 1 Week = 1,22,000 Transactions = 0.02 Failure/1,000,000 Transactions So,  HW = 0.02 Failure/1,000,000 Transactions
  • 29. Calculate Total Failure Intensity Source: http://www.enel.ucalgary.ca/People/far/Lectures/SENG635/ System/Platform Failure Intensity (  ) / 10 Million Transaction Hardware -  HW 0.2 Email Server-  EML 20 Other Systems -  OTH 30 Total Failure Intensity – (  SYS )  SYS = (  HW +  EML+  OTH ) i.e.  SYS  50.20 failure / 10 million transactions or 5 failure / 1,000,000 transactions
  • 30. Determine the product-development software failure intensity objective (I)
    • RMS consists of five system components:
    • External Inputs:10 - 20 failures / 1,000,000 transactions
    • External Outputs: 10 - 20 failures / 1,000,000 transactions
    • External Interface: 10 - 20 failures / 1,000,000 transactions
    • External Inquiries: 10 - 20 failures / 1,000,000 transactions
    • Internal System Access: 10 failures / 1,000,000 transactions
  • 31. Determine the product-development software failure intensity objective (II)
    • Thus the target FIO of the system ranges from 10 failures / 1,000,000 transactions to 20 failures/1,000,000 transactions. Therefore the product-developed software failure intensity objective (  DEV ) is the target FIO minus the Platform FIO (  SYS ) which ranges from 0 to 15 failures / 1,000,000 transactions (20 – 5 = 15)
    • Lets say, one time cost is $50,000. So, According to COCOMO’s Design – 1 Failure/50,000 Transactions or 20 Failure/1,000,000 Transactions [It is based on Approximated Cost and Transaction Mapping Criteria]
  • 32. How will you balance the fault prevention, fault removal and fault tolerance strategies
    • Fault Prevention: 60%
    • Initial Investigation and Requirement review: 30%.
    • Design review: 30%
    • Fault Removal: 30%
    • Code inspection: 20%
    • Unit test: 5%
    • System test: 5%
    • Fault Tolerance: 10%
    • ISO 9000 - 3: 4.1.1 Code compliance – Quality Assurance
    • ISO 9000- 3: 4.1.2.3 – Record of Issues and statistics
    • ISO 9000-3: 4.4.1 - Code convention and sharing
    • Better Exception Handling and ability to going back to it’s original state
  • 33. Conclusion
    • The Realtor Management System (RMS) is the true solution for the Realty Industry in the recent years. It provides the valuable information of the desired property within a very short while. It is user-friendly and fast. It meets the requirement of different levels of user according to their requirements and access privileges. The Reliability of the system has been measured to produce target system performance. The Failure Intensity Objective (FIO) of RMS has been set at 15 failures/1,000,000 transactions.
  • 34. Developing Operational Profile of RMS Developing operational profiles will give us the information about how users will employ the product that we are building so that we can focus our development and test resources. Operational profile is the set of operations (operation names and frequencies) and their probabilities of occurrences
  • 35. Operational Profile Development
    • Operation of RMS
    • The Operational Mode of RMS
    • The Operational Profile of RMS
    • Operation initiator of each operation mode
    • Determine the occurrence rate of each operation mode
  • 36. Operational Mode of RMS
    • The operational mode of RMS is how the system interacts based on several different types of user of the system. RMS has generally three different user types:
    • Administrator
    • Customer
    • Realtor
  • 37. Administrator
    • Creates/Updates Logon Users
    • Updates Provinces, Districts, Communities, Property Types, Property Categories/Sub Categories, Price ranges
    • Updates System Settings
    • Uploads property information
  • 38. Customer
    • Searches for the desired property based on several different search criteria’s
    • View / Print Property information
    • Searches Realtor and their listings
  • 39. Realtor
    • Searches for the desired property based on several different search criteria’s
    • View / Print Property information
    • View/Print Property History and Additional [Offer Status, Offer Made etc.] Information
    • Searches Realtor and their listings
    • Gets automatic email notification of new listings
  • 40.
    • Operational Profile of Realtor Management System (RMS)
    • Operational Modes, Initiators, Operation Lists, and Occurrence Rates
  • 41. Operational Profile of Realtor Management System (RMS) Operations Occurrence Rate (Transactions) Occurrence Probability (%) Search Properties by Customers 40,000 40 Search Properties by Realtors 30,000 30 Take printout of Properties 5,000 5 Email Property 2,000 2 Upload Property Information 20,000 20 Update Property Specific Information (Require Administrative Login) 1,500 1.5 Update Realtor Profile (Require Realtor/Administrative Login) 500 0.5 Auto Email Notification to Realtor 1,000 1 Total 1,00,000 100
  • 42. System Design Phase
    • Class Diagram for Administrator
    • Class Diagram for Customers
    • Class Diagram for Realtors
    • Sequence Diagram for Administrator
    • Sequence Diagram for Customers
    • Sequence Diagram for Realtors
  • 43. Conclusion
    • The Operational mode of the system varies based on the following system design criteria: Type of User, Tasks Usage, Volume Time, External Systems etc. To make it brief and simple, the operational mode of RMS depends on three types of users of the system. The brief design of the system has been illustrated by using the class diagrams and sequence diagrams of each type of users (Administrator, Customer, and Realtor).
  • 44. Preparing for Test In preparing for test, we apply the operational profile information that we have developed to planning for an efficient test. Our failure intensity should meet the FIO (Failure Intensity Objective) of RMS which is 15 failures / 1000,000 Transaction.
  • 45. The process for preparing test cases involves:
    • Estimate the number of test cases we need to prepare
    • Allocate the number of test cases among the associated and sub-systems to be tested
    • Allocate the number of test cases for the developed product to its operations and
    • determine how will we divide hours of test among the associated systems you have defined
    • Specify the test profile for one of the operational modes
    • Document the test cases
    • Determine the number of hours we will devote to feature, regression (if needed), and load test for the product
    • Allocate the hours of load test among the operational modes
  • 46. 148 test cases are sub divided into the following sections
    • Internal System of RMS – 130 Test Cases
    • - Administrator Tasks
    • - Customer Tasks
    • - Realtor Tasks
    • External System of RMS – 18 Test Cases
    • - Physical Database
    • - Transactions
    • - Hardware
    • - OS & IIS
    • - Printer
    • - Email System
    • Or, Developed System – 132 Tests
    • Hardware/Other Systems – 16 Tests
  • 47. No. of Test cases
    • At this stage, we will decide what will be out testing phase timing. From thorough research and knowledge we can guess that the whole testing phase would take approximately 8 weeks with a team of 2 support people. Based on our experience, we can assume that it would take 4 hours to prepare for a test. In total, it the number of test cases would be:
    • N = 8 weeks X 2 People X 40 Hours/Week/4 Hours/Test = 160 test cases (Which is closed to 148 cases mentioned above)
    • Therefore, the theoretical number of test cases is 160 and the more realistic number is 148 test cases.
  • 48. Number of test cases among the associated and sub-systems and hours of test (I)
    • Out of 148 test cases, we have found the following:
      • Since the total number of hours for a complete test is 592 hours (4 hours/test x 148 tests). So, the number of hours devoted to the testing of the system (Developed System) would be 528 Hours (592 Hours X (132 Tests/148 Tests)).
      • Hardware – 6 test cases x 4 hours/test = 24 hours
      • OS & IIS – 3 test cases x 4 hours/test = 12 hours
      • Network and Other systems – 7 test cases x 4 hours/test = 28 hours
  • 49. Number of test cases among the associated and sub-systems and hours of test (II)
    • So, for theoretical calculations, 160 test cases equal a total testing time of 640 hours (160 tests x 4 hours/test). It includes: 24 hours for hardware, 12 hours for the Operating System and IIS, and 28 hours for Network and Other systems
  • 50. Allocating number of test cases for the developed product to its critical operations
    • Upload new property listing
    • The selection criteria are being displayed according to the parent selections
    • The right property listing is being displayed based on specific search criteria
    • Taking the printout of the selected property and make sure that the necessary information are being printed correctly
  • 51. Test profile for one of the operational modes (Realtor) Test Case Description Startup Realtor operates startup page Update Profile Realtor updates profile Search Property by Listing Number Realtor searches property by listing number View Today’s Listing Realtor can see today’s listed properties Search Property Realtor searches property Show Property Realtor receives selected property information
  • 52. Documenting Test cases (Example)
    • Test Case ID: RMS - TEST- 01
    • Test Objective: To test functionality for the Realtor to Log into the system
    • Item No: 1
    • Test Condition: Login is successful
    • Operator Action:
    • Enters the login ID into the User ID text box
    • Enters password into the Password text box
    • Press Login button
    • Input Specification:
    • Valid User ID
    • Valid Password
    • Expected Result:
    • Message “Login Successful”
    • Shows Realtor Main Page
    • Pass or Fail: Pass
    • Comments: none
  • 53. Hours devoted to feature, regression, and load test for the product (I)
    • Theoretically, 640 hours (160 tests x 4 hours/test) needed to perform 160 test cases. It includes: 24 hours for hardware, 12 hours for the Operating System and IIS, and 28 hours for Network and Other systems. Means in total 64 hours are for the associated systems. So, 576 hours should have to be allocated for Feature, Load and Regression testing of the product. Since load testing requires actual customer, it will take the maximum amount of time. So, we have allocated 91.7% (132 Test Cases for developed softwarex4 hours/each test=528 Hours). And, we have allocated 4.15% (24 Hours of testing for Feature and Regression Testing). The regression test may take more than the estimated time based on the number of bug fixes and the complexity of the fix.
  • 54. Hours devoted to feature, regression, and load test for the product (II) Testing Type Hours Percentage Feature Testing 24 4.15% Load Testing 528 91.7% Regression Testing 24 4.15%
  • 55. Allocating the hours of load test among the operational modes
    • We allocated 528 hours for load testing. Out of 528 Hours, we allocate 424 Hours for different operational Modes.
    And, the rest 104 hours have been allocated (528-424 = 104) for the attached software/database system of the product. Operational Modes Test Cases Hours Administrator 68 272 Customer 14 56 Realtor 24 96 Total 106 424
  • 56. Conclusion
    • A fully tested system can guarantee a stable system which is being perceived by the end users. So, Testing is a very important part of software development. The more error we can encounter and fix before going live, the less failure we will be having after implementation. This will reduce the cost and increase user satisfaction on the product. We are pretty confident that the allocated test cases and their corresponding hours can deliver a real stable system that will achieve 15 failures in 1,000,000 transactions as per our Failure Intensity Objective.
  • 57. Execute Test Realtor Management System (RMS) is at the final phase. Development is completed and ready for Reliability Testing.
  • 58. Test Procedure (I)
    • As discussed, we have three user types having different operational modes. Customers utilize the system mostly. But, we have taken the Realtor use cases as it covers almost all the functionalities of a customer with some additional features and modules, which has more complexity and functionality.
  • 59. Test Procedure (II)
    • The test procedure is being subdivided into the following sections:
    • Test 1: Testing Realtor Login
    • Test 2: Testing Profile Update
    • Test 3: Testing Custom Property Search
    • Test 4: Testing Search Property by Listing Number
    • Test 5: Testing Today’s Property Listing
    • Test 6: Testing View Property
    • Test 7: Testing taking printout property
    • Test 8: Testing emailing property
  • 60. A Sample Test
    • Test 3.7: Custom Property Search
    • Objective:
    • To test that the searching of the right property based on several different selections is successful
    • Initial System State:
    • The Realtor is in the Custom Property Search Page
    • Inputs:
      • Selects Province
      • Selects District
      • Selects Community
      • Selects Property Type
      • Selects Category
      • Selects Sub Category
      • Selects Price Ranges
      • Selects Options and Amenities
      • Clicks on Search Button
    • Expected Output:
    • The property listing based on selection criteria is being loaded
    • Actual Output:
    • The property listing page is being displayed based on selection criteria
    • Test Result:
    • Pass
  • 61. Initial System State
  • 62. Actual Output
  • 63. Apply Failure Data to Guide Decisions
    • The following decisions are made based on the certification test:
    • Accepting or rejecting an acquired component
    • Accepting or rejecting a super system
  • 64. Certification Graph
    • Now we will apply the failure data to guide decisions. The following decisions are made based on the certification test.
      • Accepting or rejecting an acquired component
      • Accepting or rejecting a super system
    • As we know that when the risk levels (  and  ) or discrimination ratio (  ) decreases, the system will require more test before reaching accept or reject regions (FAR, 2006).
  • 65. Supplier Risk (  ):
    • There is a supplier risk of 1% of wrongly accepting the software when its failure intensity objective is equal to or bigger than twice the failure intensity objective. Since the customer or realtor mostly selects the options of the query system rather than entering some valid values and internal search engine is responsible for correct query results, the system should have minimal errors to mention. The Administrator mostly enters the system values and he is regarded as a power user of the system, who is experienced of handling the system effectively and efficiently, the errors would be less. So, there is a very small percentage of chance of wrongly accepting the software. But, as the system is integrated with several different external systems there is still some chance of failure.
    • So,  = 0.01
  • 66. Consumer Risk (  ):
    • There is a risk of 1% of wrongly rejecting the software when its failure intensity objective is equal to or less than twice the failure intensity objective. Based on the design, development and testing phase of the system, we are confident of keeping very low failure intensity of the product. But, as it is integrated with external systems, still there is a risk of rejecting the software wrongly.
    • So,  = 0.01
  • 67. Discrimination Ratio (  ):
    • The discrimination ratio has been set to 2 as the error factor should be close to half of the failure intensity of the product.
    • So,  = 2
  • 68. Table1: Certification Graph Value Failure Number Measure (million transactions) Normalized Measure (MTTF) 1 0.1 1.5 2 0.2 3 3 0.3 4.5 4 0.4 6 5 0.5 7.5 6 0.6 9 7 0.7 10.5 8 0.8 12 9 0.9 13.5 10 10.0 15
  • 69. Certification Graph: Figure 1 Failure Number Normalized Measures (MTTF) Reject Accept Continue
  • 70. Recommendation
    • The testing phase has been completed and the failure data has been collected to guide decisions. We have tested the system only for Realtor mode. But, we need to test other two operational modes (Administrator and Customer) and accumulate their failure intensities to have the overall system reliability. As described in Part 2, we need to develop the system with failure intensity of 15 failures/1,000,000 transactions. Based on the Realtor test cases, the certification graph [Figure 1] demonstrates that the system would reach the accepted result within a very short while. Therefore, we may recommend that the system can be released if the Administrator and Customer test cases results similar to the Realtor test case. And, we are pretty confident to reach our goal as Realtor covers almost all components of a customer with some additional ones. And, Administrator uses the system twice a day while uploading the properties which is the minimal use of the system. And, other components of Administrator Mode are almost one time upload functionality.
  • 71. Glossary
    • Feature Testing - Testing individual functions in each module of the system (Far, 2006)
    • Load Testing – Testing with actual input data and associated systems (Far, 2006)
    • Regression Testing – When an error or bug has been fixed all associated feature tests involving that one error must be retested to ensure that the system still passes all previous tests (Far, 2006)
    • Operational Mode – “Distinct pattern of system use and/or set of environmental conditions that may need separate testing due to likelihood of stimulating different failures” (Far, 2006)
  • 72. References
    • Lectures & Sample Project
    • Far, Behrouz. University of Calgary. 2006
    • http://www.enel.ucalgary.ca/People/far/Lectures/SENG635/
    • Handbook of Software Reliability Engineering Edited by Michael R. Lyu
    • Software Reliability Engineering
    • John D. Musa
    • Software-Reliability-Engineered Testing
    • John D. Musa, AT&T Bell Laboratories and James Widmaier, National Security Agency
    • http://www.stsc.hill.af.mil/crosstalk/1996/06/reliabil.asp
    • MCSE Training Kit (Exam 70-226): Designing Highly Available Web Solutions with Microsoft® Windows® 2000 Server Technologies
    • http://www.microsoft.com/mspress/books/sampchap/5357a.asp#110
    • Microsoft Solution for Internet Business
    • Performance and Capacity Planning
    • http://www.microsoft.com/technet/itsolutions/citsrv/ib/msib2tca.mspx
    • Planning Fault Tolerance and Avoidance
    • By Charlie Russel and Sharon Crawford
    • Chapter 7 from Microsoft Windows 2000 Server Administrator's Companion, Microsoft Press
    • http://www.microsoft.com/technet/prodtechnol/windows2000serv/plan/planning.mspx
    • ISO IEC 90003 2004 SOFTWARE STANDARD
    • http://www.praxiom.com/iso-90003.htm
    • Calgary Real-State Board
    • http://www.creb.com/
    • MLS.ca
    • http://www.mls.ca
  • 73. Thank you all for watching this presentation