Successfully reported this slideshow.

Software Reliability Engineering


Published on

Powerful model for software reliability engineering

Published in: Technology, Business

Software Reliability Engineering

  1. 1. Software Reliability Engineering DeFacto Model Hans Sassenburg (SE-CURE AG) Lucian Voinea (SolidSource BV) SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 1
  2. 2. Contents 1. Introduction 2. Basic Model 3. Measures and Metrics 4. Extended Model – DeFacto Appendix: Tool Examples SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 2
  3. 3. 1. Introduction • Software reliability is the probability of failure-free operation with respect to execution time and environment. • Software reliability engineering (SRE) is the quantitative study of the operational behavior of software-based systems with respect to user requirements concerning reliability. • SRE is increasingly adopted by many companies as it becomes an important economic consideration (competitive advantage, dependencies, safety regulations), others still regard reliability a cost rather than a value. • Creditable software reliability techniques are still in urgent need. • Hardware failures are repeatable and more predictable (wear-out) while software failures are largely unpredictable. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 3
  4. 4. System-of-System Characteristics • Continuous evolution and deployment – There is an increasing need to integrate new capabilities into a system while it is operating. – New and different capabilities are deployed, and unused capabilities will be dropped; the system evolves not in phases, but continuously. • Heterogeneous, inconsistent, and changing elements – A system is not constructed from uniform parts: there are always some misfits, especially as the system is extended and repaired. • Erosion of the people/system boundary – People are not just users of a system; they are elements of the system, affecting its overall emergent behavior. • Normal failures – Software and hardware failures are the norm rather than the exception. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 4
  5. 5. Errors, Defects and Failures • Error – Mistake made by a human action or tool during programming. • Defect (Fault) – Manifestation of an error during execution. • Failure – Incorrect software behaviour in operational context due to a defect. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 5
  6. 6. Errors, Defects and Failures Errors (actual number unknown) Manifested defects Manifested failures Defect and failure rates drop after releasing and continue to grow Testing depending on contexts of use (operational profile), and number of users Unit/integration Size test coverage Complexity System test coverage Time Pre-release Post-release SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 6
  7. 7. Post-release situation Released errors Theoretical Errors (actual number unknown) maximum Manifested defects (actual number unknown) Manifested failures Released defects Feature coverage during operation Released manifested failures Post-release SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 7
  8. 8. SRE Components 1. Error 2. Defect 3. Failure prevention removal forecasting • to avoid, by • to detect, by • to estimate, by construction, verification statistical error and validation, modeling, the occurrences the existence presence of of errors and defects and remove them occurrence of failures feedback loops SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 8
  9. 9. 2. Basic Model • For software reliability prediction, you need a model of the relationship between the predicted variable and other measurable variables. • Assumptions – We can accurately measure some property of software or process. – A relationship exists between what we can measure and what we want to know. – This relationship is understood, has been validated, and can be expressed in terms of a formula or model. • Few metrics have been demonstrated to be predictable or related to software reliability. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 9
  10. 10. Problems with Existing Models • Too many models, often overly complex • Wrong assumptions –Operation environment same as testing environment; –Once a failure occurs, the fault causing it is removed; –Fault removal will not introduce new faults; –Pre-release defects can be used as predictor for post- release failures; –… SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 10
  11. 11. Simplified Approach • Distinguish in the reliability model three phases from the product lifecycle Error introduction • Pre-release (especially during design and implementation) Defect identification • Pre-release (especially during testing) Failure manifestation • Pre and post-release • Determine for each phase the main factors that influence reliability SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 11
  12. 12. Basic Model with Reliability Factors Drivers Defect Operational profile • Problem complexity identification • System Testing • Design complexity • Contexts of use • Module and • Code complexity integration testing • Agility of demand • Domain knowledge • Model verification • Number of users • Project constraints Error Failure introduction Detection methods manifestation SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 12
  13. 13. 3. Measures and Metrics • Error introduction Factor: Design and code complexity unnecessarily high. Consequence: overlook achievable execution states during programming = more errors. • Defect identification Factor: Code limitedly verified Consequence: limited number of execution states investigated. • Failure manifestation Factor: Contexts of use not always considered/known. Consequence: clients use the product in ways that reveal unconsidered execution states. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 13
  14. 14. Error Introduction Factor: Design and code complexity unnecessarily high Measures Metrics Design complexity Fan-in Fan-out Code complexity Size McCabe cyclomatic complexity Code duplication Change propagation Halstead Volume SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 14
  15. 15. Defect Identification Factor: Code limitedly verified Measures Metrics Module testing Code coverage during testing completion Integration testing Code and interface coverage during testing completion SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 15
  16. 16. Failure Manifestation Factor: Contexts of use not always considered/known Measures Metrics System test Number of verified requirements with completion Boundary and Pairwise testing. Usage context Number of verified requirements (with Boundary and Pairwise testing) from the requirements that are needed to implement the functionality used 80% of the time. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 16
  17. 17. Feedback Loops • It can be argued that software parts with many errors being found are likely to have more errors remaining, since the same factors that produced the error affect the remainder of the code as well. • This means that failure occurrences are not independent, and in fact, will exhibit correlations that depend on the operational profile as well as on the system itself. • It is therefore crucial discovering these correlations across multi-dimensional data to improve the error prevention and defect removal processes. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 17
  18. 18. 4. Extended Model - DeFacto • Basic Model – Distinguishes three phases with reliability factors. • Extended Model - DeFacto – Assign measures to those reliability factors that are relevant and derive appropriate metrics; – Customize model to specific contexts by adding/removing factors and calibrating factor weights via correlations discovered using historical data. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 18
  19. 19. DeFacto – Example Metrics Defect • Design complexity identification • System Testing Fan-in, Fan-out Requirement • Module + coverage with Integration testing Boundary and • Code complexity Code coverage Pairwise testing McCabe, Duplication Error Failure introduction manifestation SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 19
  20. 20. Appendix: SolidFX = Fact eXtractor for C/C++ • The Fact eXtractor (SolidFX) is a Error standalone, non-intrusive solution for static analysis of industry-size projects introduction written in the C and C++ programming measurement languages. • SolidFX uses proprietary technology to analyze even the most complex C/C++ code bases efficiently and robustly. • SourceFX offers predefined analysis scenarios and metrics to measure C/C++ code quality, maintainability, modularity, detect potential bugs and Automatically extract design from source code – all for coding faster, cleaner, better. compute McCabe complexity, Fan-in, Fan-out,… SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 20
  21. 21. Appendix: SolidSDD = Software Duplication Detector • The Source Code Duplication Detector (SolidSDD) is a standalone application Error for detecting and managing source introduction code duplication (i.e., code clones) in software. measurement • It can be used to analyze large projects and detect code that has been cloned (e.g., via cut-n-paste operations) during development. • The currently supported programming languages are C, C++, C# and Java. In addition to identifying the code clone fragments, SolidSDD offers an intuitive graphical interface for assessing the code duplication characteristics and the location of the duplicated fragments in the code stack. Automatically • SolidSDD is able to cope with variations in the duplicated code; not only exact compute clones are detected but also duplicated fragments that have been modified by code duplication renaming variables or by inserting/removing code. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 21
  22. 22. Appendix: SolidSX = Software eXplorer • The Software Explorer (SolidSX) is a standalone Windows application that Model gives insight in large software systems. SolidSX creates high-quality calibration visualizations that simultaneously show the structure, dependencies, metrics on all types of source code elements (files, classes, methods, fields, etc.). • By using hardware-accelerated graphics, SolidSX is able to display large amounts of information in a clear and concise manner and provides fast and easy exploration through large source codes. • SolidSX uses plug-ins to import dependencies and metrics from Discover correlations different sources. across multidimensional data SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 22
  23. 23. Appendix: SolidSTA = Software Trend Analyzer • The Software Trend Analyzer (SolidSTA) is a standalone, non-intrusive solution Error for monitoring and investigating introduction software trends. measurement • SolidSTA uses a number of proprietary and standard metric analyses to assess Model calibration the evolution of software quality indicators for industry-size code versioning repositories. • The set of metrics can be extended with custom analyses via a plug-in system with an open API. SolidSTA presents the analyses results in an intuitive way to enable users to discover trend correlations and make Compute change propagation fact-based informed decisions. • Overviews of team activity or system and metrics can be produced in minutes. • No repository management expertise is analyze software required. history/trends. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 23
  24. 24. Software Benchmarking Organization • The Software Benchmarking Organization (SBO) was founded in 2009 by SE-CURE AG and SolidSource BV. • The objective is to provide benchmarking services to the software industry. • The portfolio includes benchmarking studies, workshops, and supporting products. • The portfolio is brought to the market through an international network with accredited partners. SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 24
  25. 25. Contact Information • Dr. Lucian Voinea T +31 (40) 203 4290 M +31 (6) 1436 3842 E • Dr. Hans Sassenburg: T +41 (33) 733 4682 M +41 (79) 231 6600 E SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 25