Hypothesis Based Testing (HBT) Cookbook


Published on

A cookbook on Hypothesis Based Testing, a personal scientific test methodology. Presented by STAG Software.

Published in: Education, Technology
1 Comment
  • HBT - a revolutionary approach to software testing. Delivered stunning results to my clients. Must read for QA professionals.
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Hypothesis Based Testing (HBT) Cookbook

  1. 1. © 2011-12, STAG Software Private Limited. All rights reserved.STEM is the trademark of STAG Software Private Limited.HBT is the intellectual property of STAG Software Private Limited.This e-book is presented by STAG Software Private Limited www.stagsoftware.com 2
  2. 2. Hypothesis Based Testing (HBT) is a scientific personal test methodology that is unique in its approach to ensuring cleanliness of software. It is agoal focused approach, commencing with setting up of cleanliness criteria, hypothesising potential defect types that can impede this, and thenperforming activities to ensure that testing is purposeful and therefore effective and efficient. The central theme HBT is constructing a hypothesesof potential defects that may be probable, and then scientifically proving that they do not indeed exist. The activities of testing like test strategy,test design, tooling & automating become purposeful as these are focused on uncovering the hypothesised defect types ensuring that theseactivities are done scientifically and in a disciplined manner.HBT is based on sound engineering principles geared to deliver the promise of guaranteeing cleanliness. Its core value proposition is abouthypothesising potential defects that may be present in the software and then allow you to engineer a staged detection model to uncover thedefects faster and cheaper that other typical test methodologies.HBT fits into any development methodology and weaves into your organisational test process. HBT is powered by STEMTM (STAG TestEngineering Method) a collection of EIGHT disciplines of thinking. STEM provides the foundation for scientific thinking to perform the variousactivities. It is personal scientific inquiry process that is assisted by techniques, principles and guidelines to decompose the problem, identifycleanliness criteria, hypothesise potential defect types, formulate test strategy, design test cases, identify metrics and build appropriate automation. 3
  3. 3. Inspirations from natureHBT has been inspired by certain ideas and these are discussed below. The inspirations have come from “Properties of matter”, “Fractionaldistillation”, “Sherlock Holmes”, “Picture of baby growth”.Properties of matter Physical & Chemical properties of matter allow us to: ... classify “affected by” ... understand behaviours, interactions ... enable checking purity How can we use a similar train of thought to identify “properties of cleanliness” and then “types of defects”? “Properties of the system” End user expectations Cleanliness criteria Issues in specifications, structure, environment Potential Defect Types (PDT) and behaviour 4
  4. 4. Inspirations from natureFractional distillation This is a technique to separate mixtures that have components of different boiling points In software systems, there exists a variety of defect types that may be present in the system. How can we apply this thought process to optimally uncover the defects, by “fractionally distilling” them? Can we separate these types of defects on the basis of certain properties and optimally uncover the defects?From : http://withfriendship.com 5
  5. 5. Inspirations from naturePicture of baby growth The picture shows the health of the foetus/baby . This picture shows size, shape, parts and types of issues not present Seeking inspiration, can we depict the health of software system in a similar manner? Can we measure the ‘intrinsic quality’ at a stage? As we progressively evaluate in a staged manner, certain types of defects detected & removed and therefore quality grows. Can we chart this as “cleanliness index”?Source :http://www.environment.ucla.edu/media/images/Fetal_dev5.jpg 6
  6. 6. Inspirations Sherlock Holmes Sherlock Holmes was person who applied deductive logic to solve mysteries. How can we see inspirations from Holmes to hypothesise the types of defects that may be present and prove presence of these? 7
  7. 7. HBT - A Personal Scientific Test MethodologyTest methodologies focus on activities that are driven by a process which are powered by tools, yet successful outcomes stilldepend a lot on experience.Typically methodologies are at organisational level.On the other hand HBT is a personal scientific methodology enabled by STEMTM , a defect detection technology to deliver“Clean Software” 8
  8. 8. Scientific approach to detecting defectsCleanliness criteria What is the end user expectation of “Good Quality”?Potential Defect Types What types of issues can result in poor quality?Evaluation Stage When should I uncover them?Test Types How do I uncover them?Test Techniques What techniques to generate test cases?Scenarios/Cases What are the test cases? Are they enough?Scripts How do I execute them?Metrics & Management How good is it? How am I doing? 9
  9. 9. How is HBT different from other test methodologies?The typical test methodologies in vogue have relied on strength of the process and the capability of the individual to ensurehigh quality in the given cost and time constraints. They lack the scientific rigour to enable full cost optimisation and moreoften rely on automation as the means to driving down cost and cycle time. For example, they do not provide a strong basisfor assessing the quality of test cases in terms of their defect finding potential and therefore improve effectiveness andefficiency.HBT on the other hand enables you to set a clear goal for cleanliness, derive potential types of defect and then devise a“good net” to ensure that these are caught as soon as they get injected. It is intensely goal-oriented and provides you with aclear set of milestones allowing you to manage the process quickly and effectively. Goal drives la T ic B yp H Activities T Activities defect detection ...................................... ...................................... Powered by Powered by technology experience (STEM) ...................................... ...................................... ...................................... ...................................... ...................................... ...................................... ...................................... ...................................... ...................................... ...................................... hopefully results in Goal 10
  10. 10. Hypothesis Based Testing - HBT 2.0A Quick Introduction Personal, scientific test methodology. SIX stage methodology powered by EIGHT disciplines of thinking (STEMTM). Setup Hypothesise Cleanliness Criteria Potential Defect Types SUT Nine Stage Cleanliness Assessment Defect Removal Filter 11
  11. 11. A quick introduction to HBT SIX stages of DOING powered EIGHT disciplines of THINKING by D8 D1 S6 S1 Analysis & Business value Assess & Understand management understanding ANALYSE EXPECTATIONS D7 D2 Execution & Defect D8 D1 reporting STEM Core hypothesis Tooling D7 D2 S5 STEM Understand 32 core SUPPORT S2 D6 D3 CONTEXT concepts D5 D4 Strategy & Visibility planning D3 D6 Devise Formulate HYPOTHESIS Tooling Test design PROOF S4 S3 D5 D4 HBT powered STEM Personal test methodology by Defect detection technology 12
  12. 12. D1 Business value understanding D2 Defect hypothesis D3 Test strategy & planning Landscaping Orthogonality principle Viewpoints EFF model Tooling needs assessment Reductionist principle Defect centricity principle Defect centred AB Interaction matrix Negative thinking Quality growth principle Operational profiling Orthogonality principle Techniques landscape Attribute analysis Defect typing Process landscape GQMD4 Test design D5 Tooling Reductionist principle Input granularity principle Automation complexity Box model assessment Behaviour-Stimuli approach 32 core Minimal babysitting principle Techniques landscape concepts Separation of concerns Complexity assessment Tooling needs analysis Operational profiling Test coverage evaluation Visibility Execution & Reporting Analysis & ManagementD6 D7 D8 GQM Contextual awareness Gating principle Quality quantification model Defect rating principle Cycle scoping 13
  13. 13. Connecting HBT Stages to theScientific approach to detecting defects S1 Cleanliness criteria Potential defect types S3 S2 Expectations Staged & purposeful detection S4 Complete test cases S6 Goal directed measures Sensible automation S5 14
  14. 14. Clear baseline Set a clear goal for quality Cleanliness criteria Potential defect types Example: Clean Water implies S1, S2 1.Colourless Staged & purposeful 2.No suspended particles 3.No bacteria detection 4.Odourless Expectations What information(properties) can be used to identify this? Complete test cases ... Marketplace,Customers, End users ... Requirement(flows), Usage, Deployment ... Features, Attributes Goal directed ... Stage of development, Interactions Sensible automation measures ... Environment, Architecture ... Behaviour, Structure 15
  15. 15. A goal focused approach to cleanliness Identify potential defect types that can impede cleanliness Cleanliness criteria Potential defect types S3 Example: Data validation Timeouts Staged & purposeful Resource leakage detection Calculation Expectations Storage Presentation Complete test cases Transactional ... Scientific approach to hypothesising defects is about looking at Goal directed FIVE Aspects - Data, Logic, Structure, Environment & Usage Sensible automation measures from THREE Views - Error injection, Fault proneness & Failure Use STEM core concepts > Negative thinking (Aspect) > EFF Model (View) “A Holmes-ian way of looking at properties of elements” 16
  16. 16. Levels, Types & Techniques - STRATEGY NINE levels to Cleanliness Cleanliness criteria Potential defect types L9 End user value S4 Staged & purposeful L8 Clean Deployment detection Expectations L7 Attributes met Complete test cases L6 Environment cleanliness L5 Flow correctness Goal directed Sensible automation measures L4 Behaviour correctness L3 Structural integrity L3Quality Levels Test Techniques (T1-T4) PDT7 L2 Input interface cleanliness TT5 PDT6 TT4 TT3 T3 L2 PDT5 L1 Input cleanliness PDT4 TT3 TT2 L1 PDT3 T2 PDT2 TT2 TT1 T1 PDT1 TT1PDT: Potential Defect Types 17
  17. 17. Countable test cases & Fault coverageCountable test cases & Fault coverage Use STEM Core concepts Cleanliness criteria Potential defect types > Box model > Behaviour Stimuli approach > Techniques landscape > Coverage evaluation Staged & purposeful detection to Expectations - Model behaviour S4 - Create behaviour scenarios Complete test cases - Create stimuli (test cases) Irrespective of who designs, #scenarios/cases shall be same - COUNTABLE Goal directed Sensible automation Test Scenarios/Cases measures R1 PDT1 TS TC1,2,3 R2 PDT2 TT R3 TS TC4,5,6,7 PDT3 Requirements & Fault traceability That test cases for a given requirement shall have the ability to detect specific types of defects FAULT COVERAGE 18
  18. 18. Focused scenarios + Good Automation Architecture Level based test scenarios yield shorter scripts that are more flexible for change and easily maintainable. Cleanliness criteria Potential defect types L9 End user value Staged & purposeful detection L8 Clean Deployment Expectations L7 Attributes met Complete test cases L6 Environment cleanliness S5 L5 Flow correctness Goal directed Sensible automation measures L4 Behaviour correctness L3 Structural integrity L2 Input interface cleanliness L1 Input cleanliness 19
  19. 19. “Cleanliness Index” - Improved visibility L4 Cleanliness criteria Potential defect types PDT10 TT8 PDT9 TT7 Staged & purposeful L3 PDT9 TT6 Cleanliness detection PDT8 TT5 Expectations PDT7 TT4 Complete test cases L2 PDT6 PDT5 TT3 L1 PDT4S6 Goal directed PDT3 Sensible automation measures PDT2 TT2 PDT1 TT1 Quality report Stage CC1 CC2 CC3 CC4 R1 Met R2 Not met R3 “Growth of a baby” Partially met R4 R5 20
  20. 20. HBT StagesSix stages to produce clean software 21
  21. 21. Six staged methodology to produce clean softwareThe act of validation in HBT consists of “SIX Stages of DOING”. It commences with first two stages focused on a scientific approach tounderstanding of the customer expectations and the the context of the software. One of the key outcomes of the first two stages is“Cleanliness Criteria” that gives a clear understanding of the expectation of quality. In the third stage, the Cleanliness Criteria and theinformation acquired in the first two stages are used to hypothesise potential types of defects that are probable in the software. The fourthstage consists of devising a proof to scientifically ensure that the hypothesised defects can be indeed be detected cost-efficiently. The fifthstage focuses on building the tooling support needed to execute the proof. The last stage is about executing the proof and assessing if thesoftware does indeed meet the Cleanliness Criteria. Who are the customers, end users, what do they need, and S1 S6 S1 what do they expect? Assess & Understand ANALYSE What are the features of the system, what technologies are EXPECTATIONS S2 used, architecture? D8 D1 What types of defects may be present? D2 S3 Tooling D7 Understand What types of fishes to catch? S5 STEM SUPPORT S2 D3 CONTEXT D6 D5 D4 What is strategy, plan, test scenarios/cases? S4 Sherlock Holmes Devise Formulate PROOF HYPOTHESIS What tools do I need to detect the defects? S5 Boat in the fishing analogy S4 S3 How am I doing? How is quality? S6 Fisherman 22
  22. 22. Stage #1 : Understand EXPECTATIONS The perception that end-users have of how well the product delivers the needs denotes the quality of the Understand the software/system. "Needs" represent the various features that the software/system needs to have, to allow the marketplace for end-user to fulfill his tasks effectively and efficiently. "Expectations" on the other hand represent how well the the system needs are fulfilled. The final software/system may be deployed in different marketplaces addressing the needs of various types of Understand the customers. Hence it is imperative that we understand the various target markets (i.e marketplace) where the technology(ies) used software or system will be deployed. There could be different types of customers in the marketplace. Hence it is necessary to identify the various types of customers and then finally identify various types of end-users present in the customer. What we have done now is to start from outward direction i.e marketplace and adopt a customer/end-user centric view to understand the needs and expectations.Understand deployment environment Once we have identified the various types of customers and the corresponding end-users, we can move on to understand the various technologies that make up the software or the system and also a deployment environment. The intent is to get a good appreciation of the "construction components" and the target environment of deployment. It is imperative that we should have a good understanding of the internal aspectsIdentify end user types and not merely the external aspects of the system.& #users for each type Now were ready to go into a detailed analysis of the various types of end-users and the typical number of users for each of these end-users. Subsequent to this, we need to identify the various business requirements Identify business i.e. "needs" for each end-user.requirements for each user type At the end of the stage, the objective is to have a good understanding of the various end-users and their needs paving the way to understanding expectations clearly. 23
  23. 23. Needs & Expectations NEEDS Customers in End users Should write Should have a eraser Kids Education EXPECTATIONS Should be attractive Seniors Should be non-toxic Lead should not break easily Product Artists Drawing e.g Pencil NEEDS Draftsmen Should write Should not need sharpening Management EXPECTATIONS Corporate Thickness should be consistent Variety of thickness should be Engineering available Variety of hardness should be Admin availableNeeds typically features that allow to get the job done.Expectations are how well the need is satisfied.Remember Functional & Non-functional requirements ? 24
  24. 24. What does “understanding” involve?Good understanding of what is expected is key to effective testing. To accomplish this, it is imperative that we commence from understandingwho the various types of end users, their requirements and subsequently the expectations that they have from these. Having a deep domainknowledge helps immensely. But what if I this is a domain that I am not very conversant with? Is there a scientific way to undertand?Understanding is a non-linear activity, it is about identifying the various elements and establishing connections between these. In the process ofconnecting the dots, missing information is identified, leading to intelligent questions. Seeking answers to these questions aids in deepening theunderstanding. These are some of the elements that need to be understood. Some of the information elements are “external to the system” i.e. marketplace, customer types, end users, business requirements while some are “internal to the system” i.e. features, architecture, technology etc. Stage #1 (Understand EXPECTATIONS) focuses on “external information while Stage #2 (Understand CONTEXT) focuses on “internal information”. “Good testing is about asking intelligent questions leading to deeper understanding.” 25
  25. 25. Information extracted & artefacts generated Information At each stage, certain information is extracted, understood and transformed into artefacts useful to perform effective & efficient testing. Marketplace Customers Artefacts The key outcomes as demonstrated by the artefacts are: User types System overview ‣The big picture of the system ‣The various end users ascertained for Requirements different classes of customers in different HBT marketplaces Stage #1 User type list Deployment ‣A list of business requirements for each type of end user. environment. Requirement map Technology Lifecycle stage In Stage #1, the focus is on external information that relate to marketplace, #Users/type customers, end users andHBT Stage 1 business “Good understanding is key to effective requirements. This stage is useful to get the testing. Identifying who will use what is the bigger picture of the system and its potential beginning to become customer-focused” usage and the how it is deployed.
  26. 26. Deliverables from Stage #1 Should contain a a good overview of the marketplace, the various types of customers, end-users types, System overview deployment environment and technologies that will be used to build the system. Should contain a list of the various types of users for different types of customers in various market User type list segments. Should contain a list of the business requirements and high level technical features mapped to the various Requirement map individual types.STEM Discipline applied in Stage #1 The STEM discipline “Business value understanding” of STEM is applied in this stage of HBT. The two STEM core concepts of “Landscaping” and “Viewpoints” are useful in this stage to scientifically understand the expectations. 27
  27. 27. Stage #2 : Understand CONTEXT In this stage the objective is to understand the technical context in terms of the various features, their relative Identify technical business value, the profile of usage and ultimately arrive at the cleanliness criteria. Note that at this stage, we features and baseline are moving inward to get a better understanding of technical features of the system. them Having identified the various business requirements mapped by each type of end-user, the next logical step is to drill-down to the various technical features for each business requirement. It is important to understand Understand the various technical features that constitute the entire system do not really work in isolation. Therefore it is dependencies necessary to understand the interplay of the features i.e. understand the dependencies of a feature with other features. Understanding this dependency is very useful at later stages of the life cycle, particularly to regress Understand profile of optimally. usage We now have a list of requirements and the corresponding technical features mapped by each end-user. We are ready to proceed logically to understand the profile of usage of each of the features by the various end-Identify critical success users. To do this it is important to understand the typical in the maximum number of users for each user type factors and then the volume of usage by each user for every technical feature. Since we already have a mapping between the end-user type and the technical feature feature, all we have to do is to understand as to approximately how many times this feature will be used by typical end-user of that end user type. The intentPrioritize value of end of this is to gain a deeper understanding of the usage profile to enable an effective strategy formulation at theusers(s) and features later stage of HBT. It is not only sufficient that the features work correctly, it is equally important that the various attributes of Ensure attributes are the nonfunctional aspects of the various features are indeed met. Typically nonfunctional aspects of the testable system are identified in the highest system level, and typically turn out to be fuzzy. Good testing demands that each requirement is indeed testable. In HBT, attributes are identified for each key feature and then aggregated to form the complete set of nonfunctional requirements. We will do this in two stages: firstly identifying the Setup cleanliness critical success factors for the technical features and thereof the business requirement and then detailing the criteria critical success factors to arrive at the nonfunctional requirements or attributes. Hence after figuring out the usage profile, identify the success factors for each business requirement. 28
  28. 28. Stage #2 : Understand CONTEXTGood testing is not about testing all features equally, it is about learning to focus more on those requirements/features that affect the customerexperience significantly. This does not imply that some requirements/features are less important than the others, it simply means that somerequirements/features are more important . Before we start detailing the various attributes, it is worthwhile to the prioritize the variousrequirements /features and also various end-user types. To prioritize, start by prioritizing the various types of end users in terms of theirimportance to the successful deployment of the final system. Subsequently rank the importance of each of the requirement/feature for each ofthe end-user type. At the end of this exercise, we should have a very clear understanding of the business value of each requirement/feature. Notethat the understanding of usage profile comes in very handy here.Now we are ready to derive the various attributes from the previously identified success factors and ensure that they are testable. A testablerequirement simply means that it is an unambiguously possible to state whether it failed all passed after executing it. In the context of attributes,testability implies that each attribute does indeed have a clear measure/metric. Therefore it is necessary to identify the measures and theexpected value of the measures for each of the attribute.Having identified the various technical features and the corresponding attributes, the usage profile in the ranking of the requirements/features,we are now set to identify the various criteria that constitute the cleanliness of the intended software. Cleanliness criteria in HBT representstestable expectations. Cleanliness criteria provides a very strong basis for ensuring a goal-focused testing. This allows one to identify potentialtypes of defects and then formulate an effective strategy in the complete set of test cases It is important that the cleanliness criteria is not vagueor fuzzy. 29
  29. 29. Information extracted & artefacts generated At each stage, certain information is extracted, Artefacts understood and transformed into artefacts useful to perform effective & efficient testing. Information Feature list The key outcomes as demonstrated by the Features artefacts are: Value prioritization ‣A clear list of technical features matrix ‣Ranking of features to focus on high risk Usage areas ‣Profile of usage HBT Usage profile ‣List of attributes Focus areas Stage #2 ‣Feature interactions Attributes list ‣Clarity of expectations outlined as Attributes “Cleanliness criteria” Interaction matrix Interactions Cleanliness criteria In Stage #2, the focus is on internal information that relate to technical features, their interactions, focus areas, attributes, architecture, technology. 30
  30. 30. Deliverables from Stage #2 Feature list Should contain the list of technical features, that forms the technical features baseline. Value prioritization Should contain a set of users, requirements and features ranked by importance. matrix Usage profile Should contain a the profile of various operations by various end users over time. Should contain the key attributes stated objectively i.e. state expected value for all the measures Attributes list for each attribute. Should contain the which feature affects what. Note that this should list the interactions and not the details Interaction matrix of interactions. The objective is to get a rapid understanding of the linkages. Cleanliness criteria Should contain criteria that need to be met to ensure that the deployed system is indeed clean.STEM Discipline applied in Stage #2 The STEM discipline “Business value understanding” of STEM is applied in this stage of HBT. The STEM core concepts of “Interaction matrix”, “Operational profiling”, “Attribute analysis” and “GQM” are useful in this stage to scientifically understand the context. 31
  31. 31. Cleanliness criteriaCleanliness criteria is a mirror of expectations, The intention is to come up with criteria that if met will ensure that system meets theexpectations of the the various end users. This is not be confused with “Acceptance criteria”, as “Acceptance criteria” is typically at a higherlevel. Acceptance criteria is typically “extrinsic” in nature i.e. it describes aspects like long duration running, migration of existing data, cleaninstallation and running in the final deployment environment, delivering stated performance under real-life load conditions.Cleanliness criteria represents the “intrinsic quality” i.e. what properties should the final system have to ensure that it is deemed clean?Use the properties of the FIVE aspects of Data, Business logic, Structure, Environment, Usage as applied to your application to arrive at thesecriteria specific to your application.Note that the cleanliness criteria should both the the functional and non-functional requirements. The recommended style of writing Cleanliness criteria is: “That the system shall meet ....” Examples: That the system is able to handle large data (need to qualify large) That the system releases resources after use. That the system displays meaningful progress for long duration activities. That the system is able to detect inappropriate environment/configuration. 32
  32. 32. Stage #3 : Formulate HYPOTHESIS Having understood the expectations and the context resulting in the formulation of cleanliness criteria, we are ready to hypothesize the potential defects that could affect the cleanliness criteria. This is one of the important stages of HBT resulting in a clear articulation of the various types of defects and forms the basis for the remaining stages of HBT. The key idea is to use the external information like the feature’s behaviour, environment, attributes, usage and internal information like construction material i.e technology, architecture to hypothesize the potential defects that may be present in the software under construction. Also note that the history of the previous versions of the software or similar systems can also be used to construct and strengthen the hypothesis. Having hypothesized the potential defects, it is possible to scientifically construct a validation strategy and design adequate test cases, thereby ensuring that the final system to be deployed is indeed clean. The FIVE key aspects useful for constructing hypotheses of defects are: data, business logic, structure, environment and usage. This HBT stage allows us to follow a structured &scientific approach to the hypothesize the potential effects ensuring that we do not miss any. 33
  33. 33. Stage #3 : Formulate HYPOTHESIS (continued) Firstly use the external information like data specification and business logic Identify potential faults for the five specification to identify the potential defects. The information related to data that could aspects - Data, Business logic, Structure, help are: data type, boundaries, volumes, rate, format, data interrelationships. The intent Environment, Usage should be to get into a "negative mentality" and think of what can go wrong with respect to all the information related to the data and then produce a list of potential Identify potential failures of the five defects. aspects - Data, Business logic, Structure, Environment, Usage Now use the information related to the business logic to identify the potential effects. Business logic or the intended behaviour primarily transforms the various inputs i.e. input data to outputs that the user values. The intention is to identify potential Identify potential errors that could be transformation losses. The information specific to business logic that is useful for injected in the five aspects - Data, arriving at potential defects are : the various conditions and their linkages, values of Business logic, Structure, Environment, conditions, exception handling conditions, access control and dependencies on the Usage other parts of the software. Once again, the intent is to get into a "negative mentality", and identify erroneous business flows of logic. Now identify potential defects (PD) & Up to now the focus has been on using external information like the specification of combine PDs, remove duplicate PDs data and business logic to identify the potential defects. Now focus on the internal information like structure of the system and construction materials(i.e. language, technology) used to build the system to hypothesize potential defects. Structure at the Group similar PD to form Potential highest level represents the deployment architecture while structure at the lowest level Defect Types (PDT) represents the structure of the code. Some of the structural information that could be useful to hypothesize are: flow of control, resource usage, distributed architecture, interfacing techniques, exception handling, timing information, threading, layering. As Map PDTs to the elements-under-test i.e.features/requirements explained above, continue with the similar train of thought of examining these information with intent to identify potential problems in the structure. 34
  34. 34. Stage #3 : Formulate HYPOTHESIS (continued)Having identified potential defects using the behavioural and structural information, examine information related to environment and how theycan affect the deployed system. By environment, we mean the associated hardware and software on which the system is deployed and thehardware, software and application resources used by the system. The objective is to examine carefully how these can affect the finally deployedsystem. Some of the key information related to the environment that could be useful are: hardware/software versions, system access control,application configuration information, speed of hardware (CPU, memory, hard disk, communication links), environment configuration information(e.g. #handles, cache size etc), system resources (hardware, OS and other applications).Up till now we have taken a fault-centric approach of looking for potential faults (aka defects) by examining external or internal information. Inaddition to a fault-centric approach, we can also view the system from potential failure points and then identify the potential defects.Additionally, it is also possible to examine the system from an error injection point of view. That is, understand the kinds of potential errors thatcould be injected into the system to irritate the potential defects if any. The objective is to ensure that we have examined the system from allthree views (error centric, fault centric & failure centric) and thereby ensure that we have not missed any potential defects.A failure centric approach demands that we wear an end-user hat and identify the potential failures that could cause business loss. The cleanlinesscriteria formulated earlier could come in very handy as this would force us to think like a customer/end-user. What we trying to do is to ensurethat we have considered all the potential failures and therefore hypothesized the potential defects.Now move to a user centric view to examine the various ways that an end-user could abuse the system by identifying the various ways errorscould be injected into the system. Not that an end user does not always connote a physical person, it could be another system that interacts withthe system via some interface. so examine the various points of interaction and look at the possibilities of their injection and then hypothesize thepotential defects that could get irritated by these errors. The kinds of information that could be useful here are: workflows, data access, interestingways of using the system, accessibility, environmental constraints faced by the physical end-user and potential deviant ways of using the system.Then consolidate the potential defects and group similar ones into potential defect types (PDT). Finally map the PDTs to the various elements-under-test i.e. feature/requirements. Now we have a clear notion as to what types of defects that we should look forward to uncovering in whatparts of the system. 35
  35. 35. Information extracted & artefacts generated At each stage, certain information is extracted, understood and transformed into artefacts useful to perform effective & efficient testing. Information Data Structure Artefacts The key outcomes as demonstrated by the Environment artefacts are: HBT PD catalog Stage #3 ‣List of potential defect types Business logic ‣Mapping between PDTs & the elements- Fault traceability under-test i.e. Feature/Requirement matrix Usage Attributes Past defects In Stage #2, the focus is on hypothesizing PDTs using the FIVE aspects of Data, Business logic, Structure, Environment & Usage from THREE views - Error-centric, Fault-centric & Failure-centric. 36
  36. 36. Deliverables from Stage #3 PD catalog Should contain the list of potential defects and the potential defect types Fault traceability Should contain the mapping between the potential defect types/potential defects and features/requirements. matrixSTEM Discipline applied in Stage #3 The STEM discipline “Defect hypothesis” of STEM is applied in this stage of HBT. The STEM core concepts of “Negative thinking”, “EFF model”, “Defect centricity principle” and “Orthogonality principle” are useful in this stage to scientifically hypothesize defects. 37
  37. 37. Stage #4 : Devise PROOF (Part #1: Test Strategy & Planning)HBT being a goal focused test methodology, the intent is about figuring out an optimal approach to detect the potential of defects in thesystem. Therefore strategy in HBT is about staging the order of defect detection, identifying tests that are needed to uncover the specificdefect types and finally choosing test techniques best suited for each type of test.Typically we have always looked at the levels of testing like unit, integration and system from the aspect of the “size” of entity-under test. Unittest is typically understood as being done on the smallest component that can be independently tested. Integration test is typically understoodas being done once the various units have been integrated. System test is typically seen as the last stage of validation and is done on the wholesystem.What is not very necessarily very clear is the specific types of defects that are expected to be uncovered by each of these test levels. In HBT,the focus shifts to specific types of defects to be detected, and therefore the act of detection is staged to ensure an efficient detectionapproach.In HBT, the notion is of quality levels, where each quality level represents a milestone towards meeting the final cleanliness criteria. In otherwords each quality level represents a step in the staircase of quality. The notion is to ensure that the defects that can be caught earlier isindeed caught. So the first step to formulation of strategy is to stage the potential defects and thereby formulating the various quality levels.However in HBT, there are NINE pre-defined quality levels where the lowest quality level focuses on input correctness progressively goingonto the highest quality level to validate of the intended business value is indeed delivered. 38
  38. 38. Stage #4 : Devise PROOF (Part #1: Test Strategy & Planning) Understand scope Having identified the various types of potential defect types to be detected at various levels, it is now necessary to understand the specific types of tests needed to uncover these potential defects. In HBT each test shall be intensely goal focused. This means that a type of test shall only uncover specific type of defects. Choose quality levels The act of test type identification results in specific types of tests to be done at each of the quality levels. Now that we know what types of defects need to be detected when and where what type of tests, we need Identify test types to know how to design sufficient yet adequate test cases for each type of test. In HBT, a test technique is one that allows us to design test cases. Based on the types of defects i.e. types of tests, we have to identify the test technique(s) that is best suited for uncovering these types of the defects. Identify test techniques Now we have a clearer idea of various types of defects, the levels of detection, types of tests and test techniques., we are now ready to identify the optimal detection process best suited for design/execution of Identify detection process test cases. The the act of identifying detection process also allows us to understand whether we need technology support to be able to execute test cases and therefore pave the way for automation strategy. Identify tooling needs At this point in time we have a strategy and are ready to develop the detailed test plan. Some of the key elements of the test plan is the estimation of effort and time and formulating the various test cycles. In HBT cycles are formulated first and then effort and time estimated. Formulate cycles Finally potential risks that could come in the way of executing the test plan are identified and the risk management plan put in place. Estimate effort In summary, a strategy in HBT is a clear articulation of the quality levels, test types test techniques and detection process model. Identify risks 39
  39. 39. Information extracted & artefacts generated At each stage, certain information is extracted, understood and transformed into artefacts useful Information to perform effective & efficient testing. Cleanliness criteria PDT Artefacts Attributes The key outcomes as demonstrated by the HBT Test strategy artefacts are: Techniques Stage #4 ‣Test strategy ‣Test plan Deployment env. Test plan Scope of work #Scenarios Risks In Stage #4 (Part 1) the focus is on 1 HBT Stage identifying the quality levels, test types, test techniques and the detection process. 40
  40. 40. Deliverables from Stage #4 (Part #1) Test strategy Should contain the quality levels, test types, test techniques & detection process Test plan Should contain the test effort estimate, cycle details and the potential risk & mitigation plan.STEM Discipline applied in Stage #4 (Part #1) The STEM discipline “Strategy & Planning” of STEM is applied in this stage of HBT. The STEM core concepts of “Orthogonality principle”, “Quality growth principle”, “Defect centered activity breakdown” , “Cycle scoping” are useful in this stage to scientifically developing the strategy & plan. 41
  41. 41. Stage #4 : Devise PROOF (Part #2: Test Design) The act of designing test cases is a crucial activity in the test life cycle. Effective testing demands that the test cases possess the power to uncover the hypothesized potential defects. It is necessary that the test cases are adequate and also optimal. In HBT the design is done level-wise and within each level test-type wise. Based on the level & type, the test entity may be different. The test design activity for an entity for a type of test at a quality level consists of two major steps, firstly to design test scenarios and then generate these test cases for each scenario. Test scenarios are designed entity-wise and therefore there is a built-in notion of requirements traceability. In addition to requirements traceability, it is expected that the test scenarios and corresponding test cases are traced to the potential types of defects that they are expected to uncover. This is termed “Fault traceability”. 42
  42. 42. Stage #4 : Devise PROOF (Part #2: Test Design) Identify test level to design The act of test design commences with the identification of the quality level and then the specific type consider & identify entities of test for which the test cases are to be designed. This allows us to identify the various test entities for which test cases have to be designed. Identify conditions & data Having identified the test entities it is then required to identify the conditions that govern the business logic and the data elements that drive these conditions. Subsequent to this, build the behavioral model. Use the behavioral model to generate test scenarios. Then for every scenario, identify the data thatModel the intended behaviour varies and then generate values for each data element. Finally combine the data values to generate thesemi-formally test cases. Since we have designed scenarios/cases entity-wise, requirements traceability is built-in i.e. the designedGenerate the test scenarios scenarios/cases automatically trace to the entity (or requirement). Now map the scenarios/cases to the hypothesized PDTs to build the fault traceability matrix.For each scenario, generate Finally assess the test adequacy of the designed scenarios/cases by checking test breadth, depth &test cases porosity.Trace scenarios to PDT &entity-under -testAssess the test adequacy byfault coverage analysis 43
  43. 43. Information extracted & artefacts generated At each stage, certain information is extracted, understood and transformed into artefacts useful Information to perform effective & efficient testing. Conditions Artefacts Data Test scenarios & Logic cases The key outcomes as demonstrated by the HBT artefacts are: Stage #4 Requirements ‣Test scenarios & cases Structure ‣Requirements traceability matrix traceability matrix ‣Fault traceability matrix PDT Fault traceability matrix Defect escapes Attributes In Stage #4 (Part 2), the focus is on designing test scenarios/cases that can be proved to be adequate and have the power to uncover the hypothesized PDTs. 44
  44. 44. Deliverables from Stage #4 (Part #2) Test scenarios & Should contain the test scenarios/cases for each entity for all types of tests at various quality levels cases Requirements Should contain the mapping between the scenarios/cases and the entity-under-test traceability matrix Fault traceability Should contain the mapping between the scenarios/cases and the PDTs matrixSTEM Discipline applied in Stage #4 (Part #2) The STEM discipline “Test design” of STEM is applied in this stage of HBT. The STEM core concepts of Reductionist principle, Input granularity principle, Box model , Behavior-Stimuli approach, Techniques landscape, Complexity assessment,Operational profiling, Test coverage evaluation are useful in design test scenarios/cases scientifically. 45
  45. 45. Stage #4 : Devise PROOF (Part #3: Metrics Design) In this stage, the objective is to design measurements to manage the process of validation in an Identify progress aspects effective and efficient manner. Since HBT is a good focused test methodology, it is necessary to device measurements that enable us to clearly show the progress towards this goal. Identify adequacy(coverage) The measurements in HBT are categorized into progress related measures, test effectiveness aspects measures and system risk measures. Therefore it is necessary to identity the various aspects related to progress, effectiveness and the system health.Identify progress aspects Once the aspects are identified, key goals related to these are identified and then the metrics formulated. Finally it is necessary to understand when to measure and how to measure.For each of the aspects identifythe intended goal to meetFor each of these goals, identifyquestions to askTo answer these questions,identify metricsIdentify when you want tomeasure and how to measure 46
  46. 46. Information extracted & artefacts generated At each stage, certain information is extracted, understood and transformed into artefacts useful to perform effective & efficient testing. Information Quality aspects Progress aspects Artefacts HBT The key outcomes as demonstrated by the Process aspects Stage #4 artefacts are: Metrics chart ‣Chart of metrics that are goal-focused Organization goals When & how to measure In Stage #4 (Part 3), the focus is on designing metrics that will ensure that we stay on course towards the goal. 47
  47. 47. Deliverables from Stage #4 (Part #3) Metrics chart Should contain the list of metrics, collection frequency and a how this meets the goal.STEM Discipline applied in Stage #4 (Part #3) The STEM discipline “Visibility” of STEM is applied in this stage of HBT. The STEM core concepts of GQM, Quality quantification model are useful in design metrics that are goal-focused. 48
  48. 48. Stage #5 : TOOLING support Perform tooling benefit In this stage, the objective is to analyze the support that we need from tooling/technology to analysis perform the tests. Automation does always imply scripting that is typically automating the designed scenarios. It could also involve development of test bench, custom tooling to enable the system to be tested. Identify automation scope This stage of HBT allows you to identify the tooling needs, understand issues/complexity involved, perform cost-benefit analysis, evaluate existing tools for suitability/fitment and finallyAssess automation complexity devising a good architecture that provides for flexibility/maintainability before embarking onto automation.Identify the order in whichscenarios need to be automatedEvaluate toolsDesign automation architectureDevelop scriptsDebug and baseline scripts 49
  49. 49. Information extracted & artefacts generated At each stage, certain information is extracted, Information understood and transformed into artefacts useful Artifacts to perform effective & efficient testing. Automation Needs & benefits objectives document Scope Complexity assessment report The key outcomes as demonstrated by the Scenarios to artefacts are: automate Tooling requirements ‣The reason for tooling & automation HBT ‣Challenges involved Stage #5 ‣Requirements of tooling Scenario fitness ‣Scope of tooling & automation Automation scope ‣Architecture of automation Technologies used ‣Automated scripts Automation architecture Tool info. Tooling & Scripts Complexity info. In Stage #5, the focus is on identifying tooling requirements and building automated scripts that is delivers value & ROI. 50
  50. 50. Deliverables from Stage #5 Needs & benefits Should contain the technical & business need for automation. document Complexity Should contain the technical challenges of automation assessment reportTooling requirements Should contain the requirements expected out of automation Automation scope Should contain scope of automation Automation Should contain the architecture adopted to building tooling/scripts architecture Tooling & Scripts The actual tools/scripts for performing automated testingSTEM Discipline applied in Stage #5 The STEM discipline “Tooling” of STEM is applied in this stage of HBT. The STEM core concepts of Automation complexity assessment, Minimal babysitting principle, Separation of concerns, Tooling needs analysis are useful in adopting a disciplined approach to tooling & automation and deliver the ROI.. 51
  51. 51. Stage #6 : Assess & ANALYZE Identify test cases/scripts This stage is where you execute the test cases, record defects, report to the team and take to be executed appropriate action to ensure that the system/application is delivered on time with the requisite quality. Execute test cases, record outcomes Record defects Record learnings from the activity and the context Record status of execution Analyze execution progressQuantify quality and identify risk to delivery Update strategy, plan, scenarios, cases/scripts 52
  52. 52. Information extracted & artefacts generated At each stage, certain information is extracted, Artifacts understood and transformed into artefacts useful to perform effective & efficient testing. Execution status report Information Defect report The key outcomes as demonstrated by the artefacts are: Execution ‣Report of test execution & progress Progress report ‣Defect report information HBT ‣Report on cleanliness aka quality Defect Stage #6 ‣Learnings from execution resulting in information Cleanliness report improved strategy, scenarios & cases ‣Any other key learnings Context Updated strategy, plan, scenarios & cases Key learnings In Stage #6, the focus is on ensuring a disciplined execution, intelligent analysis and continuous learning to ensure that the goal is reached. 53
  53. 53. Deliverables from Stage #6 Execution status Should contain the status of test execution report Defect report Should contain defect information Progress report Should contain progress of execution and thereof the cycle Cleanliness report Should contain the cleanliness index and how well the cleanliness criteria has been met Updated strategy, Updated strategy, plan, scenarios, cases based on learnings from execution plan, scenarios & cases Key learnings Key observations/learnings that could be useful in the futureSTEM Discipline applied in Stage #6 The STEM disciplines of “Execution & reporting” and “Analysis and management” of STEM are applied in this stage of HBT. The STEM core concepts of Contextual awareness, Defect rating principle, Gating principle, Cycle scoping enable a disciplined execution, fosters continual learning and stay focused on the goal. 54
  54. 54. STEM Disciplines 55
  55. 55. Discipline #1 : Business value understandingHow to This discipline enables one to understand the system, create a baseline of features, attributes andUnderstand a system finally expectations. This discipline consists of SEVEN tools, each of which uses certain STEM core concepts to ensure these are done in a scientific and disciplined manner.Landscaping | Viewpoints Good quality implies meeting expectations. This requires that we understand expectations inHow to additions to the needs as delivered by the requirements. Understanding the intended businessCreate a functional baseline value to delivered is key to this.Viewpoints | Reductionist principleHow toCreate an attribute baselineViewpoints | Reductionist principleHow to How toIdentify focus areas Understand interdependenciesValue prioritisation | Viewpoints Interaction matrixHow to How toUnderstand usage Baseline expectationsOperational profiling | Viewpoints Goal-Question-Metric | Viewpoints 56
  56. 56. Baseline provides the basis for future workWhat is to be tested needs to be clear.Remember Functional & Non-functional requirements? Functional Baseline Attribute Baseline Consists of list of features to be tested. The non-functional aspects. Essentially a agreed upon list of features. Agreed upon attributes & their values. 57
  57. 57. Tools in D1 -Business value understanding STEM CoreTools Description Concepts System is viewed as a collection of information elements that areHow to Landscaping interconnected. This tool enables you to come up with intelligent questions toUnderstand a system Viewpoints understand the various information elements and their interconnections. Commencing from an external view of end users, various use casesHow to Viewpoints (requirements) are identified and then technical features that constitute theCreate a functional baseline Reductionist principle use cases. This tool enables you to clearly setup a functional baseline that is used as a basis for strategy, plan, design, tooling, reporting & management. In addition to functional correctness, it is imperative that the attributes areHow to Attribute analysis met,. This tool enables you to identify the attributes and ensure that these areCreate an attribute baseline Viewpoints testable. All requirements/features are not equally valued by the end users. This toolHow to Viewpoints allows you to rank the end users, requirements, features thereby enablingIdentify focus areas Value prioritisation prioritisation of testing based on the risk and perceived value. Understanding the real life usage profile is about knowing what operations,How to Viewpoints #concurrent operations, rate of arrival are in progress at a point in time. ThisUnderstand usage Operational profiling tool allows to arrive at the closer to reality potential usage profile of the system to ensure effective non-functional tests. Understanding how a feature/requirement affects/is-dependent on otherHow to Interaction matrix feature/requirements is useful to understand impact & re-testing effort. ThisUnderstand interdependencies tool allows you to rapidly understand the interdependencies.How to Viewpoints This tool allows to derive cleanliness criteria that reflect the expectations.Baseline expectations Goal-Question-Metric 58
  58. 58. Customers & End Users Customers in End users Kids Education Seniors Product Artists Drawing e.g Pencil Draftsmen Management Corporate Engineering AdminA product or an application may be sold in different market places made up of different kinds of customers.Each class of customer may have different types of end users who use the product.It is important to understand that each end user may have different needs & expectations.Testing is about ensuring that the product will indeed satisfy the variety of needs & expectations 59
  59. 59. Needs & Expectations NEEDS Customers in End users Should write Should have a eraser Kids Education EXPECTATIONS Should be attractive Seniors Should be non-toxic Lead should not break easily Product Artists Drawing e.g Pencil NEEDS Draftsmen Should write Should not need sharpening Management EXPECTATIONS Corporate Thickness should be consistent Variety of thickness should be Engineering available Variety of hardness should be Admin availableNeeds typically features that allow to get the job done.Expectations are how well the need is satisfied.Remember Functional & Non-functional requirements ? 60
  60. 60. Customer Profile Customer #1 Customer #2 Customer #3 Customer #4 Different customers have different types of end users, and differing number of users for type of end user. 61
  61. 61. Customer Profile & Usage How many What does each one use? What types users What is order of importance? of users What is the usage frequency? F1 F2 F3 F4 System F5 F6 F7 F8 Different end users may use the system differently in terms of what they use, frequency of usage and how they value each each feature. 62
  62. 62. Business ValueUltimately end users need the system to do their jobBETTER, FASTER, CHEAPER and deliver value to their customers.Understand that it is about “business value” of system - how does the systemhelp my business to do BETTER, FASTER, CHEAPER. 63
  63. 63. Discipline #2 : Defect hypothesis 64