Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Continuous Testing
Maximising Velocity, Quality
and Customer Happiness
June 19, 2018
About Me
|Gerie Owen
• VP, Knowledge & Innovation--US at QualiTest
Group
• Speaker and Writer on Testing topics
• Experien...
Agenda
| Why Continuous Testing?
| Continuous Testing Explained
| Implementing Continuous Testing
| Streamlining the Test ...
Why Continuous Testing?
|Velocity
|AND
|Quality
4
Increasing Velocity AND Quality
| Agile and DevOps
• Testing Cycles must be shortened yet quality must be increased
• Incr...
Why Continuous Testing?
Customer Happiness
6
Customer Happiness
| Customers demand Quality more than ever before
| When customers encounter bugs, performance issues an...
Continuous Testing
| A core capability in Agile and DevOps where all testing activities run continuously in an
integrated ...
Continuous Testing
| Continuous testing is an approach to managing risk by focusing on not only on
improving testing effic...
Continuous Testing vs Test Automation
| Continuous testing requires automation, however, it encompasses much more.
10
Attributes of Continuous Testing
| Assesses business risk coverage as its primary goal
| Establishes a safety net that hel...
Implementing Continuous Testing
| Engage in continuous risk analysis and process improvement
| Increase the velocity of cu...
Key Practices of Continuous Testing
|Defect Management
|Discuss their “observations” prior to logging defects.
|Fix defect...
Key Practices of Continuous Testing
|Environment Access and Availability
|Requires easy access to production-like environm...
Key Practices of Continuous Testing
|Advanced Analytics
|Using code change impact analysis for test optimization to determ...
Streamlining the Test Process
| Focus on Mitigating Business Risk
| Optimize Test Scope
| Identify and Mitigate Bottlenecks
Focus on Mitigating Business Risk
| Focus on the customer; aspects of the applications that are most important to the
cust...
Focus on Mitigating Business Risk
| Map risks to application components, requirements and tests.
| Optimize the test suite...
Test Optimization
| An ongoing process:
19
Test Optimization
| Test Optimization encompasses risk assessment of the entire test scope:
| Analyze Test cases for quali...
Optimize Test Scope
| Select end-to-end tests that focus on the critical and high-risk features of the
application.
| Unde...
Optimizing Test Cases
| Tests should be focused on the functionality they are validating.
| Test should be as simple as po...
Optimizing Test Suites
| Include test cases that validate high risk areas of code.
| Include test cases that validate cust...
Identify and Mitigate Bottlenecks
| Constraints to the test process can be found throughout from requirements through
post...
Continuous Testing Strategy
| Components
| Increase Velocity AND Improve Quality.
| Integrate Testing throughout the Build...
Increase Velocity AND Improve Quality
| Increase velocity
| “Shift Left”; Push automation to the base of the testing pyram...
Improve Quality
| Developers need to embrace testing.
| Entire team is responsibility for quality.
| Implement post-produc...
Focus on the Customer
| Exploratory Testing
| Use exploratory test charters to find bugs in new features
| Usability Testi...
Integrate Testing Throughout the Delivery Pipeline
| Integrate automated tests in each stage the Continuous Integration Pi...
Implement Automated Testing Checkpoints
| Automated Testing Checkpoints
| Unit tests verify code.
| Component tests verify...
Continuous Non-Functional Testing
| “Shift Left”
| Begin Non-Functional testing as early as possible in order to correct d...
Continuous Monitoring in Test and Production
| Continuous Monitoring
| Comes from the DevOps principle of amplifying feedb...
Summary
|Focus on Risk
|Optimize the Test Process.
|Automated Checkpoints.
|Automated feedback loops.
|Integrate testing t...
References
| Continuous Testing for IT Leaders, Wayne Ariola and Cynthia Dunlop
| A Practical Guide to Testing in DevOps, ...
www.QualiTestGroup.com
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Continuous testing maximising velocity, quality and customer happiness

Download to read offline

Organizations turn to Agile and DevOps to increase value and improve the customer experience by maximizing the speed of delivery without sacrificing quality.  As the champions of quality, testers achieve this goal through continuous testing.  But just what is continuous testing?

Spring OnlineTestConf 2018 session by Gerie Owen - VP Knowledge & Innovation--US at QualiTest Group

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Continuous testing maximising velocity, quality and customer happiness

  1. 1. Continuous Testing Maximising Velocity, Quality and Customer Happiness June 19, 2018
  2. 2. About Me |Gerie Owen • VP, Knowledge & Innovation--US at QualiTest Group • Speaker and Writer on Testing topics • Experienced Tester, Test Lead, & Test Architect • Gerie can be reached: • gowen@qualitestgroup.com • www.gerieowen.com, • her blog, Testing in the Trenches, is at https://testinggirl.wordpress.com/ • @GerieOwen on Twitter and on • http://www.linkedin.com/in/gerieowen 2
  3. 3. Agenda | Why Continuous Testing? | Continuous Testing Explained | Implementing Continuous Testing | Streamlining the Test Process | Developing a Test Strategy for Continuous Testing | Monitoring in Production
  4. 4. Why Continuous Testing? |Velocity |AND |Quality 4
  5. 5. Increasing Velocity AND Quality | Agile and DevOps • Testing Cycles must be shortened yet quality must be increased • Increase in microservices architectures require high-quality functional testing including automated API testing • Builds, Integrations and deployments are automated; testing must be integrated into the pipeline 5
  6. 6. Why Continuous Testing? Customer Happiness 6
  7. 7. Customer Happiness | Customers demand Quality more than ever before | When customers encounter bugs, performance issues and usability issues, they • Choose your competitor • Expound about their issue with your website in the social media | Customer Happiness is critical to the success of your organization | Testers are the champions of quality | Testers are the guardians of the organization’s reputation 7
  8. 8. Continuous Testing | A core capability in Agile and DevOps where all testing activities run continuously in an integrated fashion with development and delivery. Immediate bug fixing is enabled, test environments are provisioned instantly, and unit, functional, and nonfunctional tests are run in an automated way, orchestrated by continuous integration and continuous delivery tools. The Forrester Wave™: Modern Application Functional Test Automation Tools, Q4 2016 8
  9. 9. Continuous Testing | Continuous testing is an approach to managing risk by focusing on not only on improving testing efficiency but more importantly, increasing the effectiveness of our test processes. 9
  10. 10. Continuous Testing vs Test Automation | Continuous testing requires automation, however, it encompasses much more. 10
  11. 11. Attributes of Continuous Testing | Assesses business risk coverage as its primary goal | Establishes a safety net that helps the team protect the customer experience | Requires a stable test environment to be available on demand | Integrates seamlessly into the software delivery pipeline and DevOps toolchain | Delivers actionable feedback appropriate for each stage of the delivery pipeline 11
  12. 12. Implementing Continuous Testing | Engage in continuous risk analysis and process improvement | Increase the velocity of current quality assurance process | Implement automated testing through the entire software development process | Develop a culture in which the entire team is responsible for quality. 12
  13. 13. Key Practices of Continuous Testing |Defect Management |Discuss their “observations” prior to logging defects. |Fix defects immediately rather than logging and tracking. |Test Case Management and Optimization |Analyze individual test cases for for their quality and effectiveness. |Evaluate test suites to determine if they are providing the maximum amount of coverage with the fewest number of test cases.
  14. 14. Key Practices of Continuous Testing |Environment Access and Availability |Requires easy access to production-like environments. |Automated processes for creating and verifying these environments. |Service Virtualization and Data Management |Processes include deployment automation, test data creation. and service virtualization. Embraces shift left because integration testing can begin even though all the integrations may not be completed or available. 14
  15. 15. Key Practices of Continuous Testing |Advanced Analytics |Using code change impact analysis for test optimization to determine: •what new tests are needed, •what old tests need updating or replacement, •what other areas of the application have been affected and require regression testing. |Automated Testing |Includes automating not only all layers of testing; but also, •automated testing of the deployment process • automated health checks •feedback at each step. 15
  16. 16. Streamlining the Test Process | Focus on Mitigating Business Risk | Optimize Test Scope | Identify and Mitigate Bottlenecks
  17. 17. Focus on Mitigating Business Risk | Focus on the customer; aspects of the applications that are most important to the customer are areas of highest risk. | Understand the competitive climate of the organization; how important is it to be first to market? 17
  18. 18. Focus on Mitigating Business Risk | Map risks to application components, requirements and tests. | Optimize the test suite to provide the greatest risk coverage with the fewest number of test cases. | Always report status in a way that highlights the business risk including, technical, performance, and compliance perspectives. 18
  19. 19. Test Optimization | An ongoing process: 19
  20. 20. Test Optimization | Test Optimization encompasses risk assessment of the entire test scope: | Analyze Test cases for quality, effectiveness and automation potential | Evaluating the test suites to determine if they are providing the maximum amount of coverage with the fewest number of test cases.
  21. 21. Optimize Test Scope | Select end-to-end tests that focus on the critical and high-risk features of the application. | Understand all integrations involved in critical user workflows including the technologies employed in those applications (web, mobile, message/API-layer etc.). | Implement service virtualization for integrations and components that aren’t readily available in order to run end-to-end test continuously. | Use the most realistic data possible for each test, include synthetic data generation where needed. | Include exploratory testing to find user-experience and other bugs not easily found by automated tests. 21
  22. 22. Optimizing Test Cases | Tests should be focused on the functionality they are validating. | Test should be as simple as possible; the more complex the test, the more validation is required for the test code itself. | Even though service virtualization can be used to fill in for missing components in integration testing, test should as independent as possible.
  23. 23. Optimizing Test Suites | Include test cases that validate high risk areas of code. | Include test cases that validate customer-centric features. | Include test cases that validate critical business workflows. | Use Functional Automation tools that create multi-purpose scripts: | Cross-Platform Testing • Scripts should run on Desktop, Web, Mobile & Server | Cross-Browser Testing • Scripts should run on multiple browsers including Chrome, Edge, Safari and any others required | Performance Testing • Scripts should collect performance statistics including page loads etc.
  24. 24. Identify and Mitigate Bottlenecks | Constraints to the test process can be found throughout from requirements through post production checkout. Some examples: | Testers aren’t included in backlog refinement so they don’t have the opportunity help build the acceptance criteria. | Testers aren’t included in sprint planning, story points may not include testing which affects velocity. | Defects may not be addressed on a timely basis. | Regression suite runs too long or is ineffective. | Post production checkout is still done manually. | Constraints impact velocity. 24
  25. 25. Continuous Testing Strategy | Components | Increase Velocity AND Improve Quality. | Integrate Testing throughout the Build Pipeline. | Implement Automated Testing Checkpoints. | Implement “Shift-Left” and “Shift Right” Non-Functional Testing. | Implement Continuous Monitoring in Test and Production. 25
  26. 26. Increase Velocity AND Improve Quality | Increase velocity | “Shift Left”; Push automation to the base of the testing pyramid • TDD, BDD. | Include risked-based manual and exploratory testing. | Implement continuous non-functional testing including: • Performance • Security • Availability • Accessibility | Amplify automated feedback loops to enable risk-based decision-making. 26
  27. 27. Improve Quality | Developers need to embrace testing. | Entire team is responsibility for quality. | Implement post-production monitoring. | Implement testing in production, based on application risk. 27
  28. 28. Focus on the Customer | Exploratory Testing | Use exploratory test charters to find bugs in new features | Usability Testing | Effectiveness | Efficiency | Learnability | Error Prevention | User Experience | Satisfaction | Enjoyment | Value 28
  29. 29. Integrate Testing Throughout the Delivery Pipeline | Integrate automated tests in each stage the Continuous Integration Pipeline 29
  30. 30. Implement Automated Testing Checkpoints | Automated Testing Checkpoints | Unit tests verify code. | Component tests verify key features. | Integration test verify workflows involving multiple applications. | Risk-based, Optimized Regression tests run prior to each deployment. | Post production health checks verify applications are up and running after deployment. | Production monitoring finds defects and performance issues before they impact the customer. 30
  31. 31. Continuous Non-Functional Testing | “Shift Left” | Begin Non-Functional testing as early as possible in order to correct design flaws more easily. This is critical in • Performance • Security | “Shift Right” | Post-Production testing to provide critical metrics and data about the performance of the application in the real world. 31
  32. 32. Continuous Monitoring in Test and Production | Continuous Monitoring | Comes from the DevOps principle of amplifying feedback loops in both test and production. | Enables collection and analysis of data and metrics throughout the lifecycle that are used to optimized user experience. | The most important data is about how users are interacting with the application and feedback from customers. This data allows stakeholders and DevOps teams to make targeted adjustments such as: • Functional enhancements to remove customer pain points • Upgrades to environments to improve application performance. | Testing in production includes • Monitoring data and transactions • Simulating transactions • Experimentation with real users • A/B Testing.
  33. 33. Summary |Focus on Risk |Optimize the Test Process. |Automated Checkpoints. |Automated feedback loops. |Integrate testing through the delivery pipeline including deployments at each stage. |Focus on Customer Happiness |Include exploratory and usability throughout the process
  34. 34. References | Continuous Testing for IT Leaders, Wayne Ariola and Cynthia Dunlop | A Practical Guide to Testing in DevOps, Katrina Clokie | What Testers Need To Know About Continuous Testing, Wayne Ariola | The Forrester Wave™: Modern Application Functional Test Automation Tools, Q4 2016, Diego Lo Guidice 34
  35. 35. www.QualiTestGroup.com

Organizations turn to Agile and DevOps to increase value and improve the customer experience by maximizing the speed of delivery without sacrificing quality.  As the champions of quality, testers achieve this goal through continuous testing.  But just what is continuous testing? Spring OnlineTestConf 2018 session by Gerie Owen - VP Knowledge & Innovation--US at QualiTest Group

Views

Total views

602

On Slideshare

0

From embeds

0

Number of embeds

210

Actions

Downloads

13

Shares

0

Comments

0

Likes

0

×