Your SlideShare is downloading. ×
Model for evaluating value at risk for
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Model for evaluating value at risk for


Published on

Published in: Technology, Business

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Model for evaluating value at risk for Risk Based Testing Keshava Moorthy Chandrashekar Test Consultant Abstract: Delivering high quality reliable software on time and within budget is the main focus of software testing. The quality of the software delivered is decided based on the test coverage, approach taken to discover the defects and effective risk evaluation. The major amount of rework done in testing phase is due to poor risk evaluation and risk management. The roles of development team, testing team and support teams are crucial in evaluating the risk and the magnitude of the risk in testing. The risks which are identified while testing is in progress consume 30-40% additional overall project effort and cost. Though testing is one kind of risk mitigation for the overall project lifecycle, there are lots of areas in testing where the risks are involved. Important areas can either be functions or functional groups, or properties such as performance, capacity, security etc. The result of this risk analysis is a list of functions and properties or combination of both that need attention. Risk based testing is getting popular since it is aimed at optimization of time and resources available for testing without comprising on the quality of testing. However, most approaches for Risk Based Testing today, suffer from the following drawbacks: • Lack of precise definition of risk. Risks are often confused with issues • Subjectivity in assessing the impact of risk, without any real, relevant quantification of impact • The dynamic nature of risk, in terms of differing impact and probability of occurrence of the risk factor, over different stages of the project lifecycle, is not captured This paper focuses on the deriving a model for risk evaluation and risk based testing, based on a simple value at risk model, addressing limitations of the traditional models. Introduction Software testing in many organizations is key and critical to delivering success in the current business environment. Because of the complexity of software or systems, the testing done on them is comprehensive but not exhaustive. Most of the test management fails to handle the risks on time due to lack of knowledge in identifying the risks and a risk based /modeled approach for testing. Risk can be broadly defined as the degree of uncertainty about the project meeting its goals, and is a function of probability of occurrence of the defined undesirable event and the magnitude of severity. A risk can arise almost anywhere and have a significant impact on the test schedules and quality. So it is important that testing is done in operational frame work gives the ability to mitigate the risks early in the test life cycle.
  • 2. Testing can reduce the probability of occurrence of certain risks by ensuring a wide coverage of test risk management wide across all possible risks that may occur due to development, testing approach, tools or test environment. Risk mitigation and compliance reduces costs, ensures quality, ensures timeliness and protects reputation, all of which are key factors to remain competitive The higher and more critical risks are practically visualized in reality either very close to the integration testing, system testing and performance testing phases, or when these phases are in progress. “Forward”[1] risks are those associated with the system operation and which involves the consequences of the software failure. But there are “Backward risks “[1] which are more of development life cycle originated risks. Risk based testing is aimed at the optimization of the time and resources available for testing without comprising on the quality of work. In standard approaches of testing all aspects are given equal priority. In the later phases of the software life cycle such as the System Testing phase there is a need to ensure completion testing in a short span of time. In such situations standard approaches of testing are not effective as it is difficult to take decisions about the aspects that have to be tested. Risk based testing has a major advantage of being the most suitable kind of testing for the System testing phase. The risk based testing procedure prioritizes risks and hence the testing effort is focused which leads to completion of testing in a lesser time span. Risk based testing also indicates the risk distribution across the application which is helpful in taking decisions about the release probability of the software application. To carryout this important factor is to understand the cost of each identified risk. The value at risk model for software testing will provide access to efficiently manage the testing risks. Why Risk based testing? Though risk based testing is contrasting from the traditional approach, it has got numerous benefits to the project in terms of cumulative improvements in quality. Risk based testing ensures that the applications are delivered to meet business and time requirements. This also helps in optimizing the testing resources in a better manner. Risk based testing allows the software organizations to manage and minimize the risks by tracking and addressing the risks ahead. Typical points of failure in new application development and testing effort are: 1. Meet the timelines 2. Meet the budget 3. Application logic 4. Keep up with business changes 5. To realize and implement business cases as expected 6. Measuring the reliability of the application 7. Value addition to the application due to testing
  • 3. Risk based testing outcome plays a major role in making a final decision to Go live or not, objectively. This is because, risk based testing gives the access to understand the status of the identified risks and depth of testing done. The more testing is done, the less value it adds. Most organizations find it comfortable to invent workaround solutions rather than addressing the risk. But these work around solutions in reality cost a lot compared to addressing the risk itself. The traditional risk model for testing The traditional risk model has two elements 1) The probability of a fault being present 2) The impact of a defect /fault in the corresponding function if it occurs during normal operation RE(f) = P(f) * I(f) Where RE(f) is the risk exposure function of f, P(f) is the probability of a fault occurring in function f and I(f) is the impact when a fault is executed in a function f in operational mode. The cost of software failure is very high. More number of defects or problems and defects can contribute to the software cost exponentially. To address this it might need a rigorous risk management to comprehensively mitigate and make contingency plans. “Heuristic reasoning is a reasoning not regarded as final and strict , but as provisional and Plausible only, whose purpose is to discover the solution for the present problem “ written by George Polya , Greek Philosopher. Using this approach, one need to identify the situations and related impacts by repeatedly asking “what could go wrong here?”. The three factors that can be considered are Vulnerabilities, threats and Victims. Begin with a set of potential risks and match them to the details of the situation. Basically the risks can be identified under three categories viz., Quality criteria, Generic risks and risk catalogs. Using following steps one can analyze the risks 1. Select a component or a function you want to analyze 2. Determine the severity of the concern 3. Gather more information or inputs on the concern 4. Visit each risk area on each list and determine its importance in the situation 5. Record unknowns In the traditional model, the risk exposure is seldom a quantified value but more a subjective value like low, medium high, based on the corresponding probabilities and impact.
  • 4. Value at Risk Model Value at Risk model is an extension of the traditional model, to of qualify, quantify and mitigate the risks in software testing. It follows a 4-step process 1. Risk identification 2. Risk description 3. Risk estimation 4. Risk mitigation Risk identification Risk identification process provides the access to the projects/software developments exposure to uncertainties that can occur during testing. This needs sound knowledge of the entire application (software) and the other systems with which the application interacts/interfaces and the deployment environment. This also requires a sound knowledge of performance requirements and business user’s perspectives. Test risk identification can be done systematically and methodically to ensure that the all the risks are identified and all the activities linked to the risks are defined. Developing a risk identification checklist : Generic risks • Complex: anything disproportionately large, intricate, or convoluted • New: anything that has no history in the product • Changed: anything that has been tampered with or “improved” • Upstream Dependency: anything whose failure will cause cascading failure in the rest of the system • Downstream Dependency: anything that is especially sensitive to failures in the rest of the system • Critical: anything whose failure could cause substantial damage • Precise: anything that must meet its requirements exactly • Popular: anything that will be used a lot • Strategic: anything that has special importance to your business, such as a feature that sets you apart from the competition • Third-party: anything used in the product, but developed outside the project • Distributed: anything spread out in time or space, yet whose elements must work together • Buggy: anything known to have a lot of problems • Recent Failure: anything with a recent history of failure Qualitative risks • Capability: Can it perform the required functions? • Reliability: Will it work well and resist failure in all required situations? • Usability: How easy is it for a real user to use the product? • Performance: How speedy and responsive is it? • Installability: How easily can it be installed onto its target platform?
  • 5. • Compatibility: How well does it work with external components and configurations? • Supportability: How economical will it be to provide support to users of the product? • Testability: How effectively can the product be tested? • Maintainability: How economical will it be to build, fix, or enhance the product? • Portability: How economical will it be to port or reuse the technology elsewhere? • Localizability: How economical will it be to publish the product in another language? Risk description Risk description is a method of displaying risks in a structured format. A well designed structure is necessary to ensure comprehensive risk identification, description and assessment process. By considering the consequence and probability of each of the risks set out in the table, it should be possible to prioritize the risks that would require further analysis # Item Description 1 Risk name 2 Scope Description of the event,size,dependencies 3 Nature/ Area of impact Quality,compliance,technical etc 4 Probability Probability of occurrence 5 Severity Severity of the risk(critical/High/Low) 6 Risk tolerance Potential impacted areas and losses(gains) 7 Risk mitigation/control Primary means of managing the risks, mechanism identification of protocols and other requirements/dependencies to control the risk 8 Policy development Identify the group( testing /development) to develop the policies, suggestions Risk Estimation Identified risks can be quantified in terms of the magnitude of the impact and the probability of occurrence. In the traditional model, they are typically represented in the following tabular format: Probability Rating Description Indicators High Likely / can occur more frequently Has occurred recently few times Medium Likely / can occur less frequently Has occurred some time back once/twice Low Not likely to occur in the near future Unlikely. No past experience
  • 6. Impact Rating Description High Significant impact on delivery/user/stake holder Medium Less impact on the delivery/user/stake holder Low Negligible or no impact on the delivery/user/stake holder The next step is to calculate the magnitude of the risk by developing the Probability /Impact matrix. Probability Impact RE (Risk Exposure) High High High High Medium High High Low Medium Medium High High Medium Medium Medium Medium Low Low Low High Medium This helps to understand and make decisions about the risks. This is called as risk profiling or risk estimation. Risk Estimation in the Value at Risk Model Where Value at Risk model extends the traditional model is in terms of quantifying high/ medium/lows into just a single number – the loss associated to a given probability. Organizations can further balance the cost of testing from the benefits derived through statistical analysis of the derived metrics. Value at risk has three components and they are • Approximate cost of the risk • Time over which the risk has to be monitored and measured • Confidence level Following factors play important role in identifying value at risk Complexity of the software: The complexity of the software plays a major role in detriming overall cost of the risk. Examples include complex control logic, data flow, dependency on the subsystems etc. This means the one needs to consider different aspects of complexity and problem areas. The amount and types of testing required depends on the complexity of the software Changed /Changing areas: Frequent changes to the software have a significant impact on testing and overall quality of the software. Areas with many defects before (Historical experience) : Defect fixes may potentially cause some other defects. The modules that had most defects are likely to have more defects in the future.
  • 7. Impact of number of people involved: If the testing involves more people the overhead for communication, retesting and downtime Impact of time pressures Time pressure makes people to take shortcuts and workarounds. This also may lead to overtime work and will result in poor quality job done by the resources Geographical usage People working on the same project in different geographical locations potentially have communication gaps. The cost contributed to this may be considerable Impact on business Due to system downtimes , unstable system or more iterations of the software releases , the cost to the business may be very high By considering and factoring the above items along with the other identified risks to determine the value at risk will make a significant difference in managing the risk. Sl Factors Impact Value (in USD) number 1 Complexity of the software 5000 2 Changed /Changing areas 3000 3 Areas with many defects before 10000 4 Impact of number of people involved 2000 5 Geographical usage 1000 6 Impact of time pressures 1000 7 Impact on business 10000 Total cost of a risk related to testing is = 32000 USD Obtain Value at Risk table Sl no Risk Value at Risk Risk qualification(Qualified /not qualified) 1 Unidentified business 32000 USD Qualified requirement By calculating the Value at risk for each of the identified risks, the overall cost of the risk in the testing area will be CRe = ∑ Value at risk of R1+ Value of risk at R2 + ……..Value of risk at Rn
  • 8. But in reality this is not true as some part of the Value at risk for each factors may be embedded in other factor by extrapolating the risk factors. Historical defects ComplexityMarket Ge Peopl Hence we need to normalize the value at risk Sl Factors Cost(in USD) number 1 Complexity of the software 1000 2 Changed /Changing areas Absorbed in complexity 3 Areas with many defects before 7000 4 Impact of number of people 2000 involved 5 Geographical usage Absorbed in impact of business 6 Impact of time pressures Absorbed in impact of business 7 Impact on business 10000 Now the actual value at risk = 11000 USD The final (optional) step is to capture the dynamic nature of risk by plotting a VAR distribution. There are three methods of calculating VAR distribution: the historical method, the variance-covariance method and the Monte Carlo simulation. The historical method takes into account past data for a risk factor, re-organizes actual historical impact, putting them in order from worst to best. It then assumes that history will repeat itself, from a risk perspective. The Variance-Covariance method assumes that impact is normally distributed. In other words, it requires that we estimate only two factors - an expected (or average) return and a standard deviation - which allow us to plot a normal
  • 9. distribution curve. The Monte Carlo Simulation method involves developing a model for future impact and running multiple hypothetical trials through the model. Based on the confidence level, which would be different at different stages of the project, one can calculate the overall value at risk and answer the key question – “what is the worst loss that I can expect during a specified time period with a certain confidence level?” Ilustrative plot of probabilities and VAR Risk Mitigation: • Projection of value of risk may provide the organization the visibility to the near future • Generation of the re-evaluated risk watch list based on the execution results of the risk based testing cycle. This watch list will also indicate the regression risks. • Projection of the risk lowering factors and product release probability. Case study One of the world’s largest Insurance companies Scenario : This organization has four applications which are interfaced and these applications are being used in different geographies. These applications are financial transaction critical applications. These applications were in the market for more than 10 years . The organization had recently re-platformed the application in J2EE technology. Risks Identified in testing area : 1. Shared UAT environment between all four applications 2. Unavailability of performance test environment 3. Technology upgrade was not done in the test environment 4. Unavailability of strategy for end to end testing 5. Unavailability of test data 6. Poor test coverage
  • 10. The above factors are important because it will lead to obvious software failure and rework. The values obtained after normalization given as below Sl Factors Cost of risk(in USD) number 1 Shared UAT environment 2.5 million 2 Unavailability of performance 1.6 million test environment 3 Technology upgrade was not 0.5 million done in the test environment 4 Unavailability of strategy for end 1 million to end testing 5 Unavailability of test data 0.25 million 6 Poor Test coverage 4 million This helped the management to come up with a mitigation plan based on the cost factors. The benefits derived : 1. Early mitigation of all the risks effectively 2. Good Time and space for test preparation based on the risk factors 3. The cost to business ( impact on business) is reduced drastically 4. No fear of reputation loss in the market due to extensive test coverage Summary Testing in a situation where management cuts both budget and time is a bad game. We have to endure and survive this game and turn it into a success. The general methodology for handling this situation is not to test everything in bits, but to concentrate on high risk areas and the worst impacted areas. The main objectives should be to return the product as fast as possible to the developers with a list of critical defects or deficiencies as possible and to make sure that best testing with good out put is achieved with in the dead line. So to do effective test risk management involves • Anticipate the risks • Asses the impact and plan to handle • Calculate the value at risk in the testing area • Use control /mitigation mechanisms to catch the ocuurance of risks • Proactively handle the risk whenever it triggers By doing the points discussed above a quality product can be delivered to any customer with minimum pain
  • 11. Reference: 1. Redmill 2004: Redmill F: Exploring risk based test and its implications . 2. Technical report CMU/SEI-93-TR-6,ESC-TR-93-183, Taxonomy based risk identification ,1993 3. Specification based regression testing measurement with Risk analysis by yanping chen , October 2002 4. Heuristic risk- based testing by James Bach. Software quality engineering Magazine,11/99 5. Joachim Karlsson & Kevin Ryan “ A cost –Value approach for prioritizing the requirements” IEEE software,Sept 1997 6. James Bach “ Risk based Testing “,STQE 6/1999, 7. Evaluation of value at risk Models using historical data ,FRBNY economic policy review /Apr 1996