Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
 
Software Testing Presentation About Us Our Offerings Our Methodology Metrics Case Study Way Forward
<ul><li>About Us </li></ul><ul><li>NetServ Applications – outsourced software services was designed to help businesses ove...
Offerings
Customized Test scripts  Framework for Test  automation Client Server  Applications Functional  Testing Web based applicat...
<ul><li>Functional Testing </li></ul><ul><ul><li>Product Functionality </li></ul></ul><ul><ul><ul><li>Understand product f...
<ul><li>Functional Testing…. </li></ul><ul><ul><li>Testing of web services  </li></ul></ul><ul><ul><ul><li>Identify the va...
Web Service Testing <ul><ul><li>SOAPUI  automates the functional testing of Web services and measures the performance of W...
<ul><li>Functional Testing…. </li></ul><ul><ul><li>Testing of Reports  </li></ul></ul><ul><ul><ul><li>Identify the various...
Non Functional Testing – Performance Testing <ul><ul><ul><li>Understand the problem faced by the customer and the need for...
Non Functional Testing – Load Testing <ul><ul><ul><li>Define the various operations that needs to be tested for load condi...
<ul><li>Non Functional Testing… </li></ul><ul><ul><li>Compatibility & Interoperability Testing  </li></ul></ul><ul><ul><ul...
<ul><li>Non Functional Testing </li></ul><ul><li>Usability Testing </li></ul>Analysis Vision, Goals, Objectives, Challenge...
<ul><li>Test Automation </li></ul><ul><li>Test framework Components </li></ul><ul><ul><li>Initialization functions </li></...
<ul><li>Testing Tools Used </li></ul><ul><ul><li>Test Automation </li></ul></ul><ul><ul><ul><li>Test Complete  and Seleniu...
Methodology
Testing Process - Methodology
Identify functional and non functional requirements Develop a test strategy Develop test scenarios document  Submit  test ...
Key Metrics
<ul><li>Total Number of Defects detected during System Testing Phase </li></ul><ul><li>Defects detected by Priority of the...
Case Study
ERP Re-Engineering – A case study   <ul><li>Problem Statement </li></ul><ul><li>Legacy application, to be re-engineered to...
<ul><li>Solution provided </li></ul><ul><li>Service oriented architecture  </li></ul><ul><li>System Framework has been dev...
<ul><li>Project Scope   </li></ul><ul><li>Modules  –  </li></ul><ul><li>AP, AR, GL, Deals, Inventory, Pricing </li></ul><u...
<ul><li>System Testing for the ERP </li></ul><ul><li>Scope of Testing </li></ul><ul><ul><li>Functionality Testing of </li>...
<ul><li>Metrics Tracked </li></ul><ul><li>During Test Case design and Execution  </li></ul><ul><ul><li>Effort and Schedule...
Defect Removal Efficiency
Defects Detected by Priority
Defect Trend for a milestone Release
Case Study 2 –  System testing of Web Application <ul><li>Product </li></ul><ul><ul><li>Online Art Shopping Portal </li></...
Case Study - 2 <ul><li>Scope of the Project </li></ul><ul><ul><li>Designed Test Strategy </li></ul></ul><ul><ul><li>Functi...
Case Study 2 - Metrics
<ul><li>Product  </li></ul><ul><ul><li>Portal for Order Entry and Processing </li></ul></ul><ul><li>Problem Definition </l...
<ul><li>We simulated simultaneous users to perform the processes under study using Open STA  </li></ul><ul><li>Setup the v...
<ul><li>There were many instances where unnecessary usage of “Rowlocks” were identified. </li></ul><ul><li>Process blocks ...
<ul><li>Login Process </li></ul><ul><li>When we started the performance testing of the login process </li></ul><ul><ul><li...
<ul><li>Login Process </li></ul><ul><li>After identifying the bottle necks and tuning the code / queries,  we found that t...
<ul><li>Login Process  -  Response Time </li></ul><ul><li>The Response time for 100 concurrent users was well within the r...
<ul><li>Customer Search </li></ul><ul><li>Initially (3 rd  Feb) the Customer search operation would freeze when the concur...
Thank You Netserv Applications Inc. Norcross (Atlanta), GA 30071 ww.netserv-appl.com SM Netserv Technologies Bangalore, In...
Upcoming SlideShare
Loading in …5
×

Netserv Software Testing

1,023 views

Published on

The acute software testing process, tools we use and tools we\'ve developed. We test with both open source and licensed-based products, such as Selenium and Mercury.

  • Be the first to comment

Netserv Software Testing

  1. 2. Software Testing Presentation About Us Our Offerings Our Methodology Metrics Case Study Way Forward
  2. 3. <ul><li>About Us </li></ul><ul><li>NetServ Applications – outsourced software services was designed to help businesses overcome limits – of time, of technology, of resources – to meet their business goals. </li></ul><ul><li>NETSERV Offshore Services was established in 2001 and has grown rapidly in .NET Application Development & Testing. </li></ul><ul><li>We provide compelling value to our clients by employing proven quality processes, excellence in project management, flexible project models and a blended global delivery model that leverages our strengths in US and Indian operations. </li></ul>
  3. 4. Offerings
  4. 5. Customized Test scripts Framework for Test automation Client Server Applications Functional Testing Web based application Mobile and Pocket PC solutions Performance Tuning Load Testing Usability Testing Non Functional Testing Test Automation Interoperability Testing Offerings
  5. 6. <ul><li>Functional Testing </li></ul><ul><ul><li>Product Functionality </li></ul></ul><ul><ul><ul><li>Understand product functionality </li></ul></ul></ul><ul><ul><ul><li>Write Test Scenarios </li></ul></ul></ul><ul><ul><ul><li>Design Test Cases </li></ul></ul></ul><ul><ul><ul><li>Check for Traceability of the test case against Requirements </li></ul></ul></ul><ul><ul><ul><li>Review Test Cases </li></ul></ul></ul><ul><ul><ul><li>Execute the test cases and log the results </li></ul></ul></ul>
  6. 7. <ul><li>Functional Testing…. </li></ul><ul><ul><li>Testing of web services </li></ul></ul><ul><ul><ul><li>Identify the various inputs passed to the web service </li></ul></ul></ul><ul><ul><ul><li>Create scenarios to execute all the paths in the web service </li></ul></ul></ul><ul><ul><ul><li>Create Queries to check the service output based on the inputs passed to the service </li></ul></ul></ul><ul><ul><ul><li>Design the input data and output data to simulate the scenarios </li></ul></ul></ul><ul><ul><ul><li>Design test cases for the above scenarios </li></ul></ul></ul><ul><ul><ul><li>In case of calculations, design test cases to verify them </li></ul></ul></ul><ul><ul><ul><li>Execute the test cases from the various places where the web service is invoked </li></ul></ul></ul><ul><ul><li>Tool Used </li></ul></ul><ul><ul><ul><li>SOAPUI </li></ul></ul></ul>
  7. 8. Web Service Testing <ul><ul><li>SOAPUI automates the functional testing of Web services and measures the performance of Web services at runtime </li></ul></ul><ul><ul><li>We can perform Load ,Functional and Compliance Testing of Web Service. </li></ul></ul><ul><ul><li>SOAP UI </li></ul></ul><ul><ul><ul><li>Soap UI is a free and open source desktop application for inspecting, invoking, developing, and functional/load/compliance testing of web services over HTTP. </li></ul></ul></ul><ul><ul><li>To test any web service, we need to feed WSDL URL of the web service to SOAPUI tool. </li></ul></ul><ul><ul><li>It automatically detects all the web methods of the web service and adds the default request for each method. </li></ul></ul><ul><ul><li>We need to pass the parameters required to call the web service in SoapUI tool manually or from any file through groovy script. </li></ul></ul><ul><ul><li>Soap UI will invoke the web service and return with SOAP Response. </li></ul></ul>
  8. 9. <ul><li>Functional Testing…. </li></ul><ul><ul><li>Testing of Reports </li></ul></ul><ul><ul><ul><li>Identify the various inputs passed to the report </li></ul></ul></ul><ul><ul><ul><li>Create Queries to match the output of the report </li></ul></ul></ul><ul><ul><ul><li>Write individual queries to test the various filters, group bys etc. that can be applied to the report </li></ul></ul></ul><ul><ul><ul><li>Design the input data and output data to simulate the report </li></ul></ul></ul><ul><ul><ul><li>Design a checklist to test the report and use the queries appropriately </li></ul></ul></ul><ul><ul><ul><li>In case of calculations, design test cases to verify them </li></ul></ul></ul><ul><ul><ul><li>Create the required data using the application and then execute the test cases </li></ul></ul></ul><ul><ul><ul><li>Check for correct alignment of the report, exporting feature, performance etc. </li></ul></ul></ul>
  9. 10. Non Functional Testing – Performance Testing <ul><ul><ul><li>Understand the problem faced by the customer and the need for performance testing and arrive at a problem statement </li></ul></ul></ul><ul><ul><ul><li>Define the performance goal clearly (Expected Response time for individual pages , reports, process etc.) </li></ul></ul></ul><ul><ul><ul><li>Create test scripts to simulate the scenario being analyzed like order entry process, activation process etc. </li></ul></ul></ul><ul><ul><ul><li>Identify the various performance counter that will be monitored (on the client machine, IIS server and db server) </li></ul></ul></ul><ul><ul><ul><li>Setup the test environment </li></ul></ul></ul><ul><ul><ul><li>Execute the tests for the various load conditions and monitor the performance counters in the test environment </li></ul></ul></ul><ul><ul><ul><li>Analyze the test result and fine tune the test script if required </li></ul></ul></ul><ul><ul><ul><li>Execute the tests in the production environment </li></ul></ul></ul><ul><ul><ul><li>Collate the data and present it to the development team </li></ul></ul></ul>
  10. 11. Non Functional Testing – Load Testing <ul><ul><ul><li>Define the various operations that needs to be tested for load conditions </li></ul></ul></ul><ul><ul><ul><li>Identify the various load that will be applied </li></ul></ul></ul><ul><ul><ul><li>Create test scripts to simulate the scenario being tested for load </li></ul></ul></ul><ul><ul><ul><li>Identify the various performance counter that will be monitored (on the client machine, IIS server and db server) </li></ul></ul></ul><ul><ul><ul><li>Setup the test environment </li></ul></ul></ul><ul><ul><ul><li>Execute the tests for the various load conditions and monitor the counters in the test environment </li></ul></ul></ul><ul><ul><ul><li>Analyze the test result and fine tune the test script if required </li></ul></ul></ul><ul><ul><ul><li>Execute the tests in the production environment </li></ul></ul></ul><ul><ul><ul><li>Collate the data and analyze the behavior of the system under study for the various load conditions </li></ul></ul></ul><ul><ul><ul><li>We can also detect the peak load condition if required </li></ul></ul></ul><ul><ul><ul><li>May need to fine tune some parameters like # of threadpool etc. for achieving best performance when loaded </li></ul></ul></ul>
  11. 12. <ul><li>Non Functional Testing… </li></ul><ul><ul><li>Compatibility & Interoperability Testing </li></ul></ul><ul><ul><ul><li>Multi Browser Testing </li></ul></ul></ul><ul><ul><ul><ul><li>We test the application on various browsers like IE5, IE6, </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Mozilla Firefox, Netscape etc. </li></ul></ul></ul></ul><ul><ul><ul><li>Database Independence </li></ul></ul></ul><ul><ul><ul><ul><li>We have designed a db layer (Netdirect) which ensures database independence </li></ul></ul></ul></ul><ul><ul><ul><ul><li>For testing it, we setup the required test environment and test the software using different databases (SQL, Oracle etc.) </li></ul></ul></ul></ul><ul><ul><ul><li>On different Operating Systems </li></ul></ul></ul><ul><ul><ul><ul><li>We test the software for interoperability on different OS like </li></ul></ul></ul></ul><ul><ul><ul><ul><li>2000 server, XP etc. </li></ul></ul></ul></ul>
  12. 13. <ul><li>Non Functional Testing </li></ul><ul><li>Usability Testing </li></ul>Analysis Vision, Goals, Objectives, Challenges, User Categories List, Profiles, Personas / Characterization and more Evaluation Field Study, Contextual Inquiry, Design Walkthroughs, Heuristic Evaluation, Usability testing - paper, low fidelity - high fidelity; informal – formal Implementation Paper Prototype, Click Through Prototype, User Feedback and more Deployment Debrief developers on the design aspect, Handover reports and more
  13. 14. <ul><li>Test Automation </li></ul><ul><li>Test framework Components </li></ul><ul><ul><li>Initialization functions </li></ul></ul><ul><ul><li>Script reads input values from an Excel file </li></ul></ul><ul><ul><li>Script performs the test </li></ul></ul><ul><ul><li>Reads the “expected output” from the excel file </li></ul></ul><ul><ul><li>Compare the results </li></ul></ul><ul><ul><li>Logs the result </li></ul></ul><ul><li>We develop application specific customized scripts using </li></ul><ul><li>the test framework components </li></ul><ul><li>Tool Used </li></ul><ul><ul><li>Test Complete and Selenium </li></ul></ul>
  14. 15. <ul><li>Testing Tools Used </li></ul><ul><ul><li>Test Automation </li></ul></ul><ul><ul><ul><li>Test Complete and Selenium </li></ul></ul></ul><ul><ul><li>Performance and Load </li></ul></ul><ul><ul><ul><li>Open STA for web based Application </li></ul></ul></ul><ul><ul><ul><li>Ethereal to analyze the packets </li></ul></ul></ul><ul><ul><ul><li>ANTS Profiler on the web server </li></ul></ul></ul><ul><ul><ul><li>SQL Profiler </li></ul></ul></ul><ul><ul><li>Web Services Testing </li></ul></ul><ul><ul><ul><li>SOAPUI </li></ul></ul></ul>
  15. 16. Methodology
  16. 17. Testing Process - Methodology
  17. 18. Identify functional and non functional requirements Develop a test strategy Develop test scenarios document Submit test scenarios document for client review/ feedback Baseline the Test Scenario Document Design the test cases for all the test scenarios Develop “Tractability matrix” Test cases are reviewed by the development team and client Sanity Test the product before accepting the code for ST Execute the test cases and Log / Track Defects Release the product on meeting the “Release Criteria” Detailed Study of Requirement Spec Document Participate in the Requirement walk through -client Analyse all the Post Release Defects Do a Root cause Analysis and Implement Process Improvements System Testing Process Develop Test Strategy Develop Test Scenarios Implementation Phase Test Execution Phase Retrospective Understand the Requirements
  18. 19. Key Metrics
  19. 20. <ul><li>Total Number of Defects detected during System Testing Phase </li></ul><ul><li>Defects detected by Priority of the defect </li></ul><ul><li>Defects per module </li></ul><ul><li>Defect Removal Efficiency </li></ul><ul><li>Defect Arrival Curve </li></ul><ul><li>Effort Variance and Schedule Variance </li></ul><ul><li>Analyze defects based on Defect Type (Requirements defect, design defect etc.) </li></ul>Key Metrics
  20. 21. Case Study
  21. 22. ERP Re-Engineering – A case study <ul><li>Problem Statement </li></ul><ul><li>Legacy application, to be re-engineered to meet the latest features and functionalities, with complete integration with existing users. </li></ul><ul><li>Development should be in phases to ensure that we take up new functionalities in Phase 1 of development that are not available with existing customers. They wanted to sell these as add-on to their existing products thus, increasing their revenue flow, as well as keep their existing customers happy with additional features and facilities. </li></ul><ul><li>Application to work on multiple databases - SQL / Oracle / DB2 / Universe. </li></ul><ul><li>Architecture should enable uniformity across multiple geographic and multi location development activities </li></ul><ul><li>Ease of application maintenance </li></ul>
  22. 23. <ul><li>Solution provided </li></ul><ul><li>Service oriented architecture </li></ul><ul><li>System Framework has been developed for the entire architecture </li></ul><ul><li>Database independent application supports SQL 2005, Oracle, DB2 and Universe </li></ul><ul><li>Multilingual Support and Globalization and Localization implemented. </li></ul><ul><li>Multi user multi session license support </li></ul><ul><li>Web services are secured using  MS WSE3.0 (Microsoft Web service Enhancement Version 3.0) </li></ul><ul><li>Concurrency handling </li></ul><ul><li>Report framework for all reporting purpose using crystal report. Log4Net - used for all logging activities </li></ul><ul><li>Net Direct Component – Provides database independence </li></ul>ERP Re-Engineering – A case study
  23. 24. <ul><li>Project Scope </li></ul><ul><li>Modules – </li></ul><ul><li>AP, AR, GL, Deals, Inventory, Pricing </li></ul><ul><li>Order Entry ,Transportation, purchase </li></ul><ul><li>Project Effort - 100,000+ Person hrs </li></ul><ul><li>Total Forms Estimated – 1500+ </li></ul><ul><li>Total tables - 1009 </li></ul><ul><li>Project Scope delivered to date </li></ul><ul><li>Modules – </li></ul><ul><li>AP, AR, GL, System Framework </li></ul><ul><li>Project Effort to date </li></ul><ul><li>25,000+ Person hrs </li></ul><ul><li>(System Test Effort – 7500+) </li></ul><ul><li>Forms delivered – 410 </li></ul><ul><li>Framework Controls - 87 </li></ul><ul><li>Complex Web services – 37 </li></ul><ul><li>Current number of Tables - 239 </li></ul>ERP Re-Engineering – A case study
  24. 25. <ul><li>System Testing for the ERP </li></ul><ul><li>Scope of Testing </li></ul><ul><ul><li>Functionality Testing of </li></ul></ul><ul><ul><ul><li>General Ledger Module </li></ul></ul></ul><ul><ul><ul><li>Accounts Payable Module </li></ul></ul></ul><ul><ul><ul><li>Accounts Receivable Module </li></ul></ul></ul><ul><ul><ul><li>System Admin </li></ul></ul></ul><ul><ul><li>Current Phase </li></ul></ul><ul><ul><ul><li>Order Entry </li></ul></ul></ul><ul><ul><ul><li>Pricing </li></ul></ul></ul><ul><ul><li>Testing of all reports </li></ul></ul><ul><ul><li>Total Test Effort – 7500+ phrs </li></ul></ul><ul><ul><li>Total Test Cases Designed – 1000+ per module </li></ul></ul>
  25. 26. <ul><li>Metrics Tracked </li></ul><ul><li>During Test Case design and Execution </li></ul><ul><ul><li>Effort and Schedule Variance </li></ul></ul><ul><ul><li>% of test cases designed (planned versus actual) </li></ul></ul><ul><ul><li>% of test cases executed (planned versus actual) </li></ul></ul><ul><ul><li>Feature Coverage using traceability matrix </li></ul></ul><ul><ul><li>Status of defects (# Open, # Close, # Ready for Testing) </li></ul></ul><ul><ul><li>Defect Arrival Curve </li></ul></ul><ul><li>Product Metrics </li></ul><ul><ul><li>Total Number of defects per module </li></ul></ul><ul><ul><li>Defect Priority Distribution (# of High, Medium and Low defects detected) </li></ul></ul><ul><ul><li>Total Number of system test defects </li></ul></ul><ul><ul><li>Defect Removal Efficiency </li></ul></ul>
  26. 27. Defect Removal Efficiency
  27. 28. Defects Detected by Priority
  28. 29. Defect Trend for a milestone Release
  29. 30. Case Study 2 – System testing of Web Application <ul><li>Product </li></ul><ul><ul><li>Online Art Shopping Portal </li></ul></ul><ul><li>Product Details </li></ul><ul><ul><li>“ Product Admin Module” is used to configure product details, Vendor details,User details </li></ul></ul><ul><ul><li>User can order </li></ul></ul><ul><ul><ul><li>“ Art”, “Frames” </li></ul></ul></ul><ul><ul><ul><li>“ Customized portraits” </li></ul></ul></ul><ul><ul><ul><li>View the art so as to get a visual effect using a tool (Art tool) </li></ul></ul></ul><ul><ul><li>Product uses the “Paypal Direct / Express” checkout method </li></ul></ul><ul><ul><li>Can view order status </li></ul></ul>
  30. 31. Case Study - 2 <ul><li>Scope of the Project </li></ul><ul><ul><li>Designed Test Strategy </li></ul></ul><ul><ul><li>Functional Testing of the Product </li></ul></ul><ul><ul><li>Volume Testing on the Database </li></ul></ul><ul><ul><li>Testing Payment using Paypal </li></ul></ul><ul><ul><li>Testing the “Order Return process” </li></ul></ul><ul><ul><li>Order History </li></ul></ul>
  31. 32. Case Study 2 - Metrics
  32. 33. <ul><li>Product </li></ul><ul><ul><li>Portal for Order Entry and Processing </li></ul></ul><ul><li>Problem Definition </li></ul><ul><ul><li>Test for performance of </li></ul></ul><ul><ul><ul><li>Login process </li></ul></ul></ul><ul><ul><ul><li>Customer Search and Order Entry Process </li></ul></ul></ul><ul><ul><li>Identify the bottleneck </li></ul></ul><ul><ul><li>Tune the code </li></ul></ul><ul><ul><li>Demonstrate improvement in performance </li></ul></ul>Case Study 3 – Performance Tuning
  33. 34. <ul><li>We simulated simultaneous users to perform the processes under study using Open STA </li></ul><ul><li>Setup the various parameters being studied on the IIS (web server and db server), as well as on the client machine (CPU processor time, MemoryPage faults per second, lock requests / sec, response time and code etc.) </li></ul><ul><li>Instrumented the code so as to log the time taken by critical queries that were being used on these pages </li></ul><ul><li>Monitored the Response code and response time (at the server and client end) </li></ul><ul><li>Plotted the graphs for the various load conditions and analyzed for bottlenecks </li></ul><ul><li>Analyzed the profiler results for dead locks, process locks etc. Also studied the time taken for each queries </li></ul><ul><li>Identified the problem areas. </li></ul><ul><li>Developer’s </li></ul><ul><ul><li>reviewed the relevant code, </li></ul></ul><ul><ul><li>suggested the fixes, </li></ul></ul><ul><ul><li>Implemented them </li></ul></ul><ul><ul><li>Retested the application for improvement in numbers </li></ul></ul>Case Study 3 – Process followed
  34. 35. <ul><li>There were many instances where unnecessary usage of “Rowlocks” were identified. </li></ul><ul><li>Process blocks were identified when concurrent users were accessing the site. </li></ul><ul><ul><li>We recommended </li></ul></ul><ul><ul><li>There were appropriate indexing missing - we suggested them to be mandatory. </li></ul></ul><ul><ul><li>Advised them to use &quot;nolock&quot; hint for the queries - which were on less critical business paths. </li></ul></ul><ul><li>There were unnecessary indexes created on the columns that has larger datalength – which takes more database resource and will contribute to poor server’s response time. </li></ul><ul><li>There were database hits happening inside certain functions – as these functions are invoked inside the queries - this is a wrong approach. Advised them not to use functions for these scenarios, as it will reduce the query run performance. </li></ul><ul><li>Hardware was not sufficient both at the web server and database server. As a quick fix, advised them to upgrade the memory. </li></ul>Case Study 3 – DBA Findings
  35. 36. <ul><li>Login Process </li></ul><ul><li>When we started the performance testing of the login process </li></ul><ul><ul><li>It could not scale beyond 90 concurrent users </li></ul></ul><ul><ul><li>The loading of the post login page (default) was taking more than 28 seconds </li></ul></ul>
  36. 37. <ul><li>Login Process </li></ul><ul><li>After identifying the bottle necks and tuning the code / queries, we found that the application could scale upto 500 concurrent users and was able to sustain the load </li></ul><ul><ul><li>The CPU utilization was within 5% </li></ul></ul>
  37. 38. <ul><li>Login Process - Response Time </li></ul><ul><li>The Response time for 100 concurrent users was well within the required 6 seconds and that of post login page was reduced to 15 seconds. </li></ul>
  38. 39. <ul><li>Customer Search </li></ul><ul><li>Initially (3 rd Feb) the Customer search operation would freeze when the concurrent users were increased beyond 50 users </li></ul><ul><li>During our second test (17 th Feb), Customer Search operations would not scale beyond 100 concurrent users but the search would work well for 100 concurrent users </li></ul><ul><li>During the final tests, we were able to scale the search operation for 200 concurrent users with a reasonable response time. </li></ul>
  39. 40. Thank You Netserv Applications Inc. Norcross (Atlanta), GA 30071 ww.netserv-appl.com SM Netserv Technologies Bangalore, India www.smnetserv.com

×