Alex syvorotka - QA: Customer Oriented Testing Approaches in theory and practice


Published on

This presentation is intended to share the experience around customer oriented testing gained in one of Lohika projects. It will describe how important such approach is, what are the outcomes and which tools, processes and technics can be used.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Alex syvorotka - QA: Customer Oriented Testing Approaches in theory and practice

  1. 1. Customer Oriented Testing Approaches in theory and practiceby Alex Syvorotkaformer SQA @ Lohika2011<br />Lohika Confidential<br />
  2. 2. Contents<br />Presentation Description and Short words intro<br />HP UCMDB Quick Overview<br />COST aims<br />Methodology<br />Program stages<br />On-Site visit<br />Program requirements<br />COST testing<br />Met goals<br />08/20/11<br />Lohika Confidential<br />2<br />
  3. 3. Presentation Description<br />There are many approaches in Quality Assurance are being used in enterprise software development which include many different testing types. Starting from classic functional testing and going beyond with different techniques of non functional testing.<br />This presentation is intended to share the experience around customer oriented testing, aka COST, gained in one of Lohika projects – HP UCMDB.<br />It will describe how important such approach is, what are the outcomes and which tools, processes and techniques can be used.<br />08/20/11<br />Lohika Confidential<br />3<br />
  4. 4. Short words intro<br />Every workday QA tries to provide testing coverage of software components by test cases as full as it possible. In order to make coverage really high HP has agreement with premier customers to make EA cycles during on-site visits. And along with other benefits of such visits - to retrieve dumps of their software in production. Obviously owing to security reasons customer hides vulnerable details such as credentials, business critical pieces of information, etc.<br />QA analyses DB dumps to clarify in what way (HOW?) particularly a certain customer uses HP UCMDB (BSM). QA creates tablesmaps of: DB capacity, used configurations, amounts. This allows to use replicated customer’s environments for testing and also to understand which of the configurations is the most effective for a specific area of testing coverage.<br />Cross-team testing – a method applicable to mass testing of software product components. The main idea is to make a joint effort of all teams to cover their areas of responsibility. An experienced QA guy is being challenged as a cycle owner. He prepares cycle time periods, focus points, servers, presentation. And, if necessary, sheds a light onto possible technical issues and “bottlenecks”. The owner assigns details of the cycle and tasks to participants and sets timelines. As a result of cross-cycle an aggregated report is generated and sent to management of the particular area.<br />08/20/11<br />Lohika Confidential<br />4<br />
  5. 5. HP UCMDB Quick overview<br />The complexity of today’s IT infrastructure places enterprises at great risk of prolonged and frequent IT outages as a result of change. For example, market research has found that upwards of 80 percent of IT outages are caused by human error that stems from both planned and unplanned IT changes, as well as shortfalls in testing and process inadequacies. <br />Yet simply avoiding change is not possible.  Things break.  Software needs patches.  Servers will always need memory upgrades.<br />Agility in IT means responding to business needs when a company builds, buys, divests or merges lines of business.  It is defined by the ability to quickly deploy and manage new services for example, a Web application that the business launches in order to pursue an emerging market opportunity and stay ahead of the competition.  IT’s role as a strategic business partner stems from its ability to make changes quickly – and less risk. <br />If change is mandatory, then IT organizations need a means to control change and manage the impact of change when it does occur.  This is the value proposition of a CMDB. <br />02/20/09<br />Lohika Confidential<br />5<br />Painful process called “Change”<br />
  6. 6. HP UCMDB Quick overview<br />A Configuration Management DB visually models information about IT infrastructure components in order to understand the interdependencies among these components in the context of IT service. <br />If IT understands the relationships among its IT components – if it has the ability to map components to applications to the business – then IT can control change and manage impact. IT needs to understand the impact of change prior to implementing changes in their production environment. <br />For example, IT needs to understand impact the installation of a new O/S software patch to a single box in a server farm will have on a business service – such as online trading – and can take proactive steps to mitigate the risk of an outage.  Similarly, should outages arise from unplanned changes or for any other reason, such as fault failures, the relationships the CMDB maps allows IT operations to quickly find the root cause of IT outages or degradations – often before performance, availability or end-user satisfaction are adversely affected.<br />02/20/09<br />Lohika Confidential<br />6<br />CMDB’s Purpose<br />
  7. 7. HP UCMDB Quick overview<br />02/20/09<br />Lohika Confidential<br />7<br />CMDB and discovery relationships<br />
  8. 8. HP UCMDB Quick overview<br />02/20/09<br />Lohika Confidential<br />8<br />
  9. 9. HP UCMDB Quick overview<br />02/20/09<br />Lohika Confidential<br />9<br />
  10. 10. HP UCMDB Quick overview<br />02/20/09<br />Lohika Confidential<br />10<br />
  11. 11. So…<br />08/20/11<br />Lohika Confidential<br />11<br />
  12. 12. COST aims<br />08/20/11<br />Lohika Confidential<br />12<br />Test products from the customer’s perspective, using real business scenarios executed on authentic data<br />Provide version status as reflected on customers<br />Bring customer’s use cases expertise in house to improve prior release testing<br />
  13. 13. COST Methodology<br />Understand customers usage, needs and pain<br />Learn customers methodology and implementation<br />Simulate customers deployments using real data<br />Execute customer oriented scenarios<br />Focus R&D on the real usage of our products<br />08/20/11<br />Lohika Confidential<br />13<br />
  14. 14. COST Program Stages<br />08/20/11<br />Lohika Confidential<br />14<br /><ul><li>Visit customer site
  15. 15. Study Product implementation
  16. 16. Understand the organization testing process
  17. 17. Receive feedback/enhancement requests directly from product users
  18. 18. Retrieve Product Dump</li></li></ul><li>COST Program Stages Cont.1<br />08/20/11<br />Lohika Confidential<br />15<br /><ul><li>Replicate customer environment using real data
  19. 19. Document business processes and usage scenarios
  20. 20. Validate information</li></li></ul><li>COST Program Stages Cont.2<br />08/20/11<br />Lohika Confidential<br />16<br /><ul><li> Upgrade environment and data
  21. 21. Run user scenarios
  22. 22. Validate new features</li></li></ul><li>COST Program Stages Cont.3<br />08/20/11<br />Lohika Confidential<br />17<br /><ul><li>Install new version in QA Laboratory
  23. 23. Perform upgrade over customer’s dumps
  24. 24. E2E cross teams testing
  25. 25. Massive bug hunt on customer’s dumps</li></li></ul><li>COST Program Stages Cont.4<br />08/20/11<br />Lohika Confidential<br />18<br /><ul><li>Install new version on site
  26. 26. Perform upgrade
  27. 27. Post upgrade tests with users</li></li></ul><li>On Site Testing Activities<br />Selecting areas for ‘on site testing’<br />Critical areas that a large percent of customers are exposed to.<br />Areas containing high number of fixes or where refactoring has been done (risky areas).<br />Areas that suffer from ‘bad reputation’ (are more prone to defects) <br />By demand – from QA or Dev<br />02/20/09<br />Lohika Confidential<br />19<br />
  28. 28. On Site Testing Activities Cont.1<br />Planning the testing activity<br />Analyze the feature with relevant QA FL and Dev FL; understand current problems and risk factor related to them.<br />Get buy in of Dev (CORD, feature team) and set criteria for build that will be used for this activity (including fixing QA holders)<br />Determine environment coverage needed for this feature (DB, OS and etc.)<br />Identify actions that can be done on customer site and verify that these actions cover the risk areas.<br />Create specific use cases to be tested at each customer<br />Learn the feature thoroughly including troubleshooting knowledge<br />Select between remote activity and on site activity<br />Set available support level for customers participating in this activity.<br />02/20/09<br />Lohika Confidential<br />20<br />
  29. 29. On Site Testing Activities Cont.2<br />Selecting customers – requirements and profile<br />Customer characteristics: environment parameters (if relevant to tested feature), projects structure, special configuration, challenges and problems and etc.<br />Customer is willing to cooperate and accepts relevant requirements<br />Match between feature and usage patterns at customer site.<br />Location (when relevant)<br />02/20/09<br />Lohika Confidential<br />21<br />
  30. 30. On Site Testing Activities Cont.3<br />Execution phase<br />Before conducting the activity make sure all preparations are done at customer site.<br />Log all defects and problems detected on site, make sure to take all relevant data (logs, configuration).<br />If possible, obtain a copy of production project.<br />02/20/09<br />Lohika Confidential<br />22<br />
  31. 31. On Site Testing Activities Cont.4<br />Wrap up<br />Maintain support level as agreed with the customer before the activity<br />Wrap up communication with customer – provide status of issues detected on his environment.<br />In case of open Support Requests – involve CSO and CORD.<br />02/20/09<br />Lohika Confidential<br />23<br />
  32. 32. COST Testing Process<br />Requirements definition (obtained during customer’s visit)<br />Installation and configuration <br />Upgrade<br />Critical business scenarios per project<br />Testing scenarios and features validation<br />Business flows by user role<br />E2E testing<br />08/20/11<br />Lohika Confidential<br />24<br />
  33. 33. Met goals<br />Get overall understanding of HP UCMDB implementation at the customer.<br />Understand challenges faced by the customer.<br />Define specific testing requirements for upgrade and critical areas in HP UCMDB.<br />Obtain project specific business processes to be translated into testing scenarios.<br />Create somewhat like Customers Knowledge Base<br />08/20/11<br />Lohika Confidential<br />25<br />
  34. 34. Q&A?<br />WELCOME TO LOHIKA!<br />Lohika Confidential<br />26<br />08/20/11<br />