Your SlideShare is downloading. ×
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
ERP Testing Failures - Questionnaire
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

ERP Testing Failures - Questionnaire

393

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
393
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. University of Colombo School of Computing ICT 3001 ENTERPRISE RESOURCE PLANNING THIRD YEAR GROUP ASSIGNMENT ERP TESTING FAILURES - QUESTIONNAIRE GROUP 9 Lecturer: - Mr. Samantha Mathara Arachchi Date of Submission 10 th of June 2013 Name Registration Index Number Number SANDARUWAN J.W.C. 2010/ICT/018 10020187 WICKRAMASINGHE W.M.M. 2010/ICT/038 10020381 MENDIS D.G.T. 2010/ICT/057 10020578 UPENDRA K.A.N. 2010/ICT/070 10020705 KARUNARATHNE H.B.K.A. 2010/ICT/082 10020829 WIJESUNDARA W.M.D.M. 2010/ICT/092 10020926 KARUNARATHNA S.A. 2010/ICT/102 10021027
  • 2. ERP TESTING FAILURES | UCSC | GROUP 9 Page 1 of 30 We are a research group from University of Colombo School of Computing (UCSC). We would like to invite you to participate in a research study which aims at collecting data for a project as a part of Enterprise Resource Planning (ICT 3001). Before you begin, take a few minutes to read why we are inviting you to participate and what will be done with the information you provide. You will be asked to fill out a questionnaire that aims to investigate the ERP testing failures which could be beneficial you all as you are a part of the ERP community. Please feel free to stop at any time if you have questions about the study. This project is conducted only on sites which are engaging in ERP development. Your individual privacy and confidentiality of the information you provide will be maintained in all published and written data analysis resulting from the study. Your participation should take approximately 10-15 minutes for each phase mentioned in the questionnaire. We ask you to fill it out with as much accuracy as possible. Please understand your participation is entirely on a voluntary basis and you have the right to withdraw your consent or discontinue participation at any time without penalty. At no instance you will be asked to reveal any personal information. Participation in this study involves no major risks whatsoever, be it physical or emotional. This study is directly beneficial to the participants, engaging in ERP development. You will receive no incentive or payment for your participation. If at any time and for any reason, you would prefer not to answer any questions, please feel free to skip those questions. If at any time you would like to stop participating, please tell us. We can take a break, stop and continue at a later date, or stop altogether. You will not be penalized for deciding to stop participation at any time. If you have any questions, you are free to ask them now. If you have questions later, you may contact: If you have any questions about your rights as a participant in this study, or you want to talk to someone outside the research, please contact the: University of Colombo School of Computing No. 35, Reid Avenue Colombo 07. Tel: +94 112- 581245-7 I have read and understood the above, and I agree to participate in the study. The consent form is to be detached and kept with the participant. Name Phone Number Email address Saman Jayantha 0771-362587 samanjayantha@gmail.com Dilruk Karunarathne 0715-897423 dilrukkaru@gmail.com WELCOME TO THE ERP TESTING FAILURES SURVEY, 2013
  • 3. ERP TESTING FAILURES | UCSC | GROUP 9 Page 2 of 30 Table of Contents WELCOME TO THE ERP TESTING FAILURES SURVEY, 2013 .......................................1 1) Unit Testing...........................................................................................................................4 2) Integration Testing.................................................................................................................6 3) System Testing ......................................................................................................................9 3.1) Usability Testing.............................................................................................................9 3.2) Load testing and Stress testing........................................................................................9 3.3) Parallel Testing .............................................................................................................10 3.4) Configuration testing and installation testing ...............................................................10 3.5) Security Testing ............................................................................................................11 4) Documentation Testing........................................................................................................11 4.1) Master Plan ...................................................................................................................11 4.1.1) Incomplete Documentation ....................................................................................11 4.1.2) Unsuitable scope ....................................................................................................11 4.1.3) Mess of testing .......................................................................................................11 4.1.4) Incorrect testing methods .......................................................................................12 4.2) Level test plan...............................................................................................................13 4.3) Test report.....................................................................................................................14 4.3.1) Ambiguity...............................................................................................................14 5) Other Testing Failures .........................................................................................................15 Justification..............................................................................................................................17 1) Unit Test ..............................................................................................................................17 2) Integration Test Failures......................................................................................................18 2.1) Interface Testing Failures .............................................................................................18 2.2) Acceptance Testing Failures.........................................................................................18 2.3) Interoperability Testing Failures Question ..................................................................18 2.4) Infrastructure Testing Issues.........................................................................................19 3) System Testing Failures.......................................................................................................20 3.1) Failures with Usability Testing.....................................................................................20 3.1.1) Poor Task Design ...................................................................................................20 3.1.2) Not well structured test cases.................................................................................20 3.2) Load testing and Stress testing......................................................................................21 3.2.1) Bugs in the Application under test .........................................................................21
  • 4. ERP TESTING FAILURES | UCSC | GROUP 9 Page 3 of 30 3.2.2) Software Interaction failures ..................................................................................21 3.2.3) Problems with the Load Generation.......................................................................22 3.3) Parallel Testing Failures ...............................................................................................22 3.3.1. Higher complexity of the system............................................................................22 3.3.2. Less interaction with sub system component testing..............................................22 3.4) Configuration testing and installation testing failures ..................................................22 3.4.1. Tests are getting too much time for their completion.............................................22 3.4.2. Compatibility issues in the system. ........................................................................23 3.4.3. Failures of identifying configuration changes in the system..................................23 3.5) Security testing failures ................................................................................................23 3.5.1 Core security concepts are not followed..................................................................23 4) Documentation testing failures............................................................................................24 4.1. Incomplete Documentation........................................................................................24 4.2. Unsuitable scope........................................................................................................24 4.3. Mess of testing...........................................................................................................24 4.4. Incorrect testing methods...........................................................................................24 4.5. Under and over estimations of resources...................................................................25 4.6) Budget allocation.......................................................................................................25 4.7) Validation and verification problem .........................................................................25 4.8) Estimation failures.....................................................................................................25 4.9) Incomplete report ......................................................................................................26 5) Other Testing Failures .........................................................................................................27 5.1) E-commerce and security compliance failures .............................................................27 5.2) Communication failures................................................................................................27 5.3) Failures of third party tools...........................................................................................27 Mind Map.................................................................................................................................28 Reference List ..........................................................................................................................29
  • 5. ERP TESTING FAILURES | UCSC | GROUP 9 Page 4 of 30 1) Unit Testing 1. Does unit test catch each and every bug in the code? 2. This question is regarding the test cases. Tick your opinion/s. 3. Do you agree or disagree with this statement. (1 being highly agree and 5 being highly disagree) “The test code also may have bugs same as the code it testing. Therefore if any error of bug found it is difficult to identify whether it is an error with the code or test code”. 1 2 3 4 5 4. Do you believe before start unit testing is it necessary to remember as the test is behaved as a part of the complete system to exercise the code in a realistic scenario? 5. What are the best practices you follow in unit testing? 6. What are the biggest challenges you will be facing in ERP unit testing? Yes No  The test cases for unit testing are normally written by developers with the same programming language, before implementation of each unit.  Test cases should be developed for every exception path and for every possible input.  Then programmers have to write 3 to 5 lines of test code for every written line of the code.  This required much time and effort.  This doesn’t require much time and effort. Yes No  Following a rigorous way of working through development process.  Keep records of the entire test that have been performed.  No need of writing test cases for every exception path and for every possible input.  Make records of all the changes that have been made in the source code during the unit testing. If others, mention them briefly: - ----------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------  Writing test cases for every exception path and for every possible input.  Software is being developed and tested in one platform and should be deployed in to another platform.  Make records of all the changes that have been made in the source code during the unit testing.  Writing tests for legacy codes are not an easy task and it is necessary to refer documentation of those legacy systems and may have to change the code to fit with the tests. If others, mention them briefly: - ----------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------
  • 6. ERP TESTING FAILURES | UCSC | GROUP 9 Page 5 of 30 7. Tick the choices that you feel more comfortable with system test failures? 8. Select most suitable choices regarding unit test automation?  Developing tests for each and every unit may not be possible with the time and budget constraints of the project which in such situations developers may use same test cases to test many units.  If there is an error with a test case there will be many testing failures with many units. And if two or more units use same resources simultaneously test failure may increase.  Tests those are hard to understand cause failures. If others, mention them briefly: - ----------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------  Unit test automation can solve most of unit testing failures.  Test automation cannot separate each component and validate its correctness well.  Using framework is a popular method of unit test automation. It simplifies the unit test process and has been developed for wide variety of languages.  Unit test along cannot solve most of the failures and are only developed for few languages. If others, mention them briefly: - ----------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------
  • 7. ERP TESTING FAILURES | UCSC | GROUP 9 Page 6 of 30 2) Integration Testing 9. Have you been provided the scope of the integration testing? 10. How clear does your defined scope? Very clear Clear Neither clear nor not clear Not very clear Not clear at all 11. How many number of “items” tested and not tested? I. Number of items that are tested ……… II. Number of items that are not tested ……… 12. How many number of “features” tested and not tested? I. Number of features that are tested ……… II. Number of features that are not tested ……… 13. Is the system tested on possible access platforms? 14. Did you identify data mismatches in the system? a) If “Yes”, how many data mismatches did you correct? ……… 15. Was the system tested by the client representatives and system users? 16. Do all the business functions work correctly? If not, list the functions that fail to perform. ………………………………………………………………………………………………… ………………………………………………………………………………………………… 17. Did you achieve the enterprise wide compatibility? If not, list the functions that fail to perform. ………………………………………………………………………………………………… ………………………………………………………………………………………………… Yes No Yes No Yes No Yes No Yes No Yes No
  • 8. ERP TESTING FAILURES | UCSC | GROUP 9 Page 7 of 30 18. Is each functional module tested with its corresponding interoperating external module? If not, list the modules that were left untested. ………………………………………………………………………………………………… ………………………………………………………………………………………………… 19. What are the QA related reference documents referred? (tick “” appropriate documents)  IEEE Std. 1465-1998 // ISO/IEC 12119 (IEEE Standard for Quality requirements and testing) IEEE Std. 829-2008 (IEEE Standard for Software Test Documentation) Other (state name) ………………………………………………………………………………………… ………………………………………………………………………………………… 20. What are other applicable documents referred within the system? (tick “” appropriate documents) Architectural design document Detailed design document System requirement specification Problem report Modification request Other (state name) ………………………………………………………………………………………… ………………………………………………………………………………………… 21. What are the planned deliverables before starting integration testing (tick “” if delivered) Architectural design document Detailed design document Unit test pan and report Integration test plan Other (state name) ………………………………………………………………………………………… ………………………………………………………………………………………… 22. Did you check the pre-installation compatibility? 23. Did you carry out a contingency planning? 24. Does testing team have adequate qualified hardware testers? Yes No Yes No Yes No Yes No
  • 9. ERP TESTING FAILURES | UCSC | GROUP 9 Page 8 of 30 25. What are the deliverables have to be submitted after finishing the integration test? (tick “” if delivered) Integration test report Integration test output data Problem report Other (state name) ………………………………………………………………………………………… ………………………………………………………………………………………… 26. Environmental needs definition was,  Excellent  Good  Average  Poor  Not defined at all 27. Test case pass fail criteria definitions were, Excellent Good Average Poor Not defined at all 28. Test procedure definitions were, Excellent Good Average Poor Not defined at all
  • 10. ERP TESTING FAILURES | UCSC | GROUP 9 Page 9 of 30 3) System Testing 3.1) Usability Testing 29. Did you use any usability testing tool/tools? 30. Did you consider the system that you are going to test, when selecting testing tools? 31. Did you find any changes required to improve user performance from those testing? I. If YES, Did you notify those changes to the user? 32. Did you use an essential task designing component when designing tasks? 33. Following question is based on test cases. I. Do you have following characteristics with your test cases? a) It is accurate and tests what is intended to test. b) No unnecessary steps included to it. c) It is reusable. d) It is traceable to requirements. e) It is simple and clear, any tester is able to understand it by reading once. 3.2) Load testing and Stress testing 34. What is the loading test category that your system belongs to?  Quantification of risk  Determination of minimum configuration Yes No If Yes, mention them briefly: - ----------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------ - Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No If others, mention them briefly: - -------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------
  • 11. ERP TESTING FAILURES | UCSC | GROUP 9 Page 10 of 30 35.Did you test the system from both load and unload execution under load testing? 36.Can you certify that the load generator is working correctly? 37.Does the system interact with other applications, when execution? I. If YES, did you check impact of other applications, with the load test? 3.3) Parallel Testing 38. Do you use complex methodologies in parallel testing? 39. Are you monitoring the tests in sub systems? 40. Is there a mechanism to measure that you test all the parts of the system? 3.4) Configuration testing and installation testing 41. Are you using business process based scenarios for testing? 42. Are you using different platforms, supported drivers and different computer configurations? 43. Have you clearly identified where configuration changes might take place? 44. Have you defined test cases for each configuration tests? 45. Have you used an automated mechanism for configuration tests? Yes No Yes No Yes No Yes No Yes No If Yes, mention them briefly: - ----------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------ Yes No Yes No Yes No Yes No Yes No Yes No Yes No
  • 12. ERP TESTING FAILURES | UCSC | GROUP 9 Page 11 of 30 46. What are the system components that are tested for compatibility issues? 3.5) Security Testing 47. What are the types of security testing are you performing in the system? 4) Documentation Testing 4.1) Master Plan 4.1.1) Incomplete Documentation 48. What is the template do you use for test plan documentation?  IEEE 829 Standard  Organizational custom template  If any other standard (mention it here)…………………………………………………… 4.1.2) Unsuitable scope 49. Have you go through the requirement specification report before determine the scope? 50. Have you identified what are the testable functions and non-testable functions? 4.1.3) Mess of testing 51. Have you prioritized the modules and functions to be tested? 52. Have you prepared a work breakdown schedule for testing?  Hardware platforms  Software platforms If any, mention in briefly: ---------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------  Confidentiality  Integrity  Authentication  Availability  Authorization  Non-repudiation If any, mention in briefly: ---------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------ Yes No Yes No Yes No Yes No
  • 13. ERP TESTING FAILURES | UCSC | GROUP 9 Page 12 of 30 4.1.4) Incorrect testing methods 53. Have you determined what methods you are going to use for each test function? 4.1.5) Under and over estimations of resources 54. Have you identified all resources that you need to conduct the test? 55. What are the resources you have identified mentioned in last question? Hardware resources (number of machines, performance, storages, etc) Required Platforms (OS / versions ) Human resource (number of team members, skills, expertise, etc) Budget Mention any other resources you determine in documentation (if any) ………………………………………………………………………………………………… 56. How much do you allocate for testing from budget? (As percentage of whole budget) ………………………………………………………………………………………………… 57. Have you measured the time required for whole testing process? . 58. Have you set buffer time to accommodate unexpected delays? 4.1.6) Ignore Risks 59. Have you determine critical functions to be tested? 60. Have you identified risks with the product? 4.1.7) Validation and verification problem 61. What is the test pass percentage of testing respect to all test cases? …………………………………………………………………………………………… 62. Have you validated the test plan with respect to the requirement specification? 63. Have you reviewed the test plan? Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No
  • 14. ERP TESTING FAILURES | UCSC | GROUP 9 Page 13 of 30 4.1.8) Ignore future requirement 64. What are the deliverables of test?  Test cases  Test scripts  Test report  Test log  Work load analysis If any other deliverables mention them here …………………………………………………………………………………………… 4.2) Level test plan These questions are regarding the testing documentation of ERP components testing. 4.2.1) Inadequate testing 65. What are the methods you are going to use for testing?  Black box testing  White box testing  Gray box testing  Manual testing  Automated testing Mention any other methods you have determine in test plan documentation ……………………………………………………………………………………………… 66. Have you determined the methods going to use for testing each phase? 67. Have you set test pass percentage of each testing units? 4.2.2) Incorrect estimations 68) Have you identified unique resources required in each test unit? 69) Have you estimated testing effort of each test unit? Yes No Yes No Yes No Yes No
  • 15. ERP TESTING FAILURES | UCSC | GROUP 9 Page 14 of 30 70) How do you estimate the effort?  Based on previous experience  Based on standard effort estimate models  Based on expertise knowledge  Based on typical values like importance of functions, criticality of function or number of test cases 4.3) Test report This section is regarding the test reporting. 4.3.1) Ambiguity 71) Have you clearly defined what, when, who, where, how and to whom reporting? If you said some are not, what are the things you defined? …………………………………………………………………………………………… 4.3.2) Inadequate Report 72) What are the charts and diagrams you are going to generate in tests?  Pareto chart  Cause and effect Diagrams (fishbone diagram, Ishikawa diagram, characteristics diagram)  Check sheets  Histogram  Run chart  Control chart  Scatter plot diagrams If any other diagram mention it here. …………………………………………………………………………………………… 73) What are the analysis tools used to report test results?  Cluster analysis  Regression analysis  Multivariate analysis If any other analysis mention it here ………………………………………………………………………………………….. Yes No Some are not
  • 16. ERP TESTING FAILURES | UCSC | GROUP 9 Page 15 of 30 74) What are the reports you generate during the testing?  Function test matrix  Defect status report  Functional testing status report  Testing action report If any other report mention it here ………………………………………………………………………………………….. 75) What are the reports you generate as final test reports?  Unit test report  Integration test report  System test report If any other report mention it here ………………………………………………………………………………………….. 4.3.3) Incomplete report 76) Are you going to review test report before finalizing it? 5) Other Testing Failures 77) What parts of testing are you going to automate?  Management part of testing  Test specification part  Static testing part  Test execution and logging part  Performance and monitoring part If any other parts mention them here ……………………………………………………………………………………….... 78) Is testing team experienced with test automating tools? Yes No Yes No
  • 17. ERP TESTING FAILURES | UCSC | GROUP 9 Page 16 of 30 79) Did you go through all rules and regulations to determine e-commerce compliance? 80) Are you going to use third party tools for testing? a) If “Yes”, do you check the quality, reliability and the testability of that tools? If your answer is “yes”, answer for these questions. 1. Can the selected tool handle the required functions? 81) How long the vendor stable in industry? _______________________________________ 82) Are the selected tools upgraded continuously? 83) Are selected tools compatible with your environment? 84) Is the testing team familiar with the selected automating tools? 85) Have you obtained any CPI compliant certificate? If “yes”, mention it here………………………………………… 86) Do developers and testers have peer communication method? If yes, what is the present communicate system? ................................................................... 87) Are you satisfied about experience and knowledge of testing team members?  Extremely satisfied  Moderately satisfied  Neither satisfied nor dissatisfied  Dissatisfied Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No
  • 18. ERP TESTING FAILURES | UCSC | GROUP 9 Page 17 of 30 Justification 1) Unit Test 1.1) Unit test failures Question Do u agree with this statement is correct or not. (1 being highly agree and 5 being highly disagree) “The test code also may have bugs same as the code it testing. Therefore if any error of bug found it is difficult to identify whether it is an error with the code or test code”. 1 2 3 4 5 This question is asked to catch whether the tester have good knowledge on test code and program code separately. From the response we can identify the tester knowledge on program code and test code. Question Before start unit testing is it necessary to remember as the test is behaved as a part of the complete system to exercise the code in a realistic scenario? Tester should have a good knowledge in real world business process before writing the test code. The aim of this question is to catch that answer. If not the system won’t be able to match with the defined framework. Question What are the biggest challenges you will be facing in ERP unit testing? From this multiple choice question we can measure the readiness of the system implementers. Based on the answers to this question we can get an idea of their knowledge regarding ERP challenges where it would be useful to build a better system. Question What are the best practices you follow in unit testing? Best practices are mandatory in ERP implementation. Failures can be occur due to mal practices followed by people. The aim of this question is to measure the level of following best practices. Yes No
  • 19. ERP TESTING FAILURES | UCSC | GROUP 9 Page 18 of 30 Question Select most suitable answers regarding unit test automation? With this multiple choice question we are trying to find the unit test automation need for the organization. Based on the answers we can come to a conclusion whether the development clearly caught the test automation or not. 2) Integration Test Failures 2.1) Interface Testing Failures Question Is the system tested on possible access platforms? The system is being accessed by users using various platforms. Thus it should be tested in order to eliminate the platform issues. This question is used to identify those issues. 2.2) Acceptance Testing Failures Question Was the system tested by the client representatives and system users? Acceptance of the system by its users should be verified. These question find out how the accuracy of the system and performance are being verified. 2.3) Interoperability Testing Failures Question Is each functional module tested with its corresponding interoperating external module? If not, list the modules that were left untested. ………………………………………………………………………………………………… ………………………………………………………………………………………………… This question finds out all the scenarios where mismatches between modules and business functions have been occurred. By this, mismatches between modules that interoperate in different departments can be identified. Yes No Yes No Yes No
  • 20. ERP TESTING FAILURES | UCSC | GROUP 9 Page 19 of 30 Question Did you achieve the enterprise wide compatibility? If not, list the functions that fail to perform. ………………………………………………………………………………………………… ………………………………………………………………………………………………… This question finds out all the scenarios where mismatches between modules and business functions have which can lead to the system being isolated from the whole network (international) due to module mismatches and resulting in transaction failures. 2.4) Infrastructure Testing Issues Question Did you check the pre-installation compatibility? This question finds out whether the compatibility of the hardware components and the software components being used in the system are verified. This is known as device compatibility testing. Question Did you carry out a contingency planning? This question derives the necessary information regarding the performance of the system in disaster situations and the stated guidelines to follow in a such situation and the drawbacks of the guidelines. Yes No Yes No Yes No
  • 21. ERP TESTING FAILURES | UCSC | GROUP 9 Page 20 of 30 3) System Testing Failures 3.1) Failures with Usability Testing Question Did you consider the system that you are going to test, when selecting testing tools? The testing tool should be based on the type of the testing method. So the appropriate testing type should be chosen. Question Did you find any changes required to improve user performance from those tests? In lots of situations, overall testing process will identify changes required to improve. It is like a successful character in usability testing. Therefore, from this question analyst can identify the results of modifications in system. Question Did you find any changes required to improve user performance from those testing? I. If YES, Did you notify those changes to the user? From this question, analyst can make sure that all component of the system meet the user requirements. Sometime tester may not notice recently changed user requirements. Hence, some components that do not meet the user requirements could be included in the final product. This question is asked to track that kind of failures. 3.1.1) Poor Task Design Question Did you use essential task designing component when designing tasks? In order to create effective design task, essential task designing component should be included. As instances task scope, task support, task resource and task execution can be mentioned. Ensuring that the tasks have been designed well or not is the core aim of this question. 3.1.2) Not well structured test cases Question Do you have following characteristics with your test cases? Yes No Yes No Yes No Yes No Yes No
  • 22. ERP TESTING FAILURES | UCSC | GROUP 9 Page 21 of 30 a) It is accurate and tests what is intended to test. b) No unnecessary steps included to it. c) It is reusable. d) It is traceable to requirements. e) It is simple and clear, any tester is able to understand it by reading once. Every tester writes test cases however many times the test cases are rejected by reviewers because of bad quality, in order to write good test cases one should know what are the characteristics of a good test case. Using this question, analyst can make sure that the test cases are well structured and cover all what are intended to test. 3.2) Load testing and Stress testing 3.2.1) Bugs in the Application under test Question Did you test the system from both load and unload execution under load testing? Most systems use load testing for identify sensitivity errors. And these sensitivity errors appear only under the load or extended execution. Hence tester also tests under the load or extended execution always. Sometime there can be errors without loading execution. Those errors can also be identified from the above question. 3.2.2) Software Interaction failures Question Does the system interact with other applications, when execution? This question is related to load testing. Load testing is showing all problems that are related in handling load. It does not consider how many programs interact with the system while showing result as a whole. Thus tester cannot certify that the problems are related to which software. Hence from this question analysts can know whether system is interacted with other application at the execution. If YES, did you check impact of other application, with the load test? If there is another application running same time, then first attention has to be paid for that application. If that application works correctly then there is no impact for our load testing result. Hence from this question, analyst can make sure that the load balancing is not showing false results. Yes No Yes No Yes No Yes No Yes No Yes No Yes No
  • 23. ERP TESTING FAILURES | UCSC | GROUP 9 Page 22 of 30 3.2.3) Problems with the Load Generation Question Can you certify that the load generator that is used is working correctly? Load generators are used for generating hundreds of concurrent request to access the application. And then load the execution and check results. If load generator is not properly working as intended, then end results can be false. Therefore the purpose of this question is to identify whether load generating system works properly or not. 3.3) Parallel Testing Failures 3.3.1. Higher complexity of the system Question Do you use complex methodologies in parallel testing? Based on the answer to this question, we can measure the complexity of the parallel testing. There is a high possibility of parallel testing failure due to the complexity. Methodology complexities can be simplified in order to minimize the potential failures. 3.3.2. Less interaction with sub system component testing Question Are you monitoring the tests in sub systems? This question is asked to get an idea about sub system testing. If we get the answer as “No”, we can assume there can be possible testing failures in sub systems. In parallel testing, sub system test monitoring is a crucial factor for identifying the potential failures. 3.4) Configuration testing and installation testing failures 3.4.1. Tests are getting too much time for their completion. Question Have you used an automated mechanism for configuration tests? This question is asked to catch the testing mechanism of the system. It means the manual testing and the automated testing. Configuration and installation tests are taking much time in manual testing as we have to look various kinds of test components. In order to get a maximum throughput we can automate those tests. Yes No Yes No Yes No
  • 24. ERP TESTING FAILURES | UCSC | GROUP 9 Page 23 of 30 3.4.2. Compatibility issues in the system. Question What are the system components that are tested for compatibility issues? Based on the answer for this question we can identify whether the system is properly tested for compatibility issues. 3.4.3. Failures of identifying configuration changes in the system. Question Have you clearly identified where configuration changes might take place? Due to the complexity of the system sometimes people couldn’t identify the places where configuration changes should take place. From this question we can come to an idea, whether configuration changes are correctly identified or not. 3.4.4. Test cases are not defined for every configuration test due to complexity Question Have you defined test cases for each configuration tests? Whatever we have to test in front of us we have to do it. There are instances where people are not defining test cases for small configuration changes that will cause serious problems in system functionality. 3.5) Security testing failures 3.5.1 Core security concepts are not followed. Question What are the types of security testing are you performing in the system? Based on the user answer we can get an idea of the system’s security. If the system is not following the desired standards there will be drawbacks in security.  Hardware platforms  Software platforms If any, mention in briefly: -------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------ Yes No Yes No  Confidentiality  Integrity  Authentication  Availability  Authorization  Non-repudiation If any, mention in briefly: -------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------
  • 25. ERP TESTING FAILURES | UCSC | GROUP 9 Page 24 of 30 4) Documentation testing failures 4.1. Incomplete Documentation Question What is the template you use for test plan documentation?  IEEE 829 Standard  Organizational custom template  Any other standard (mention it here)…………………………………………………… There should be a standard format to any business documentation. It is very important to ensure they don’t miss any important thing to document with a standard. This question asks to check whether they use proper standard template for documentation. 4.2. Unsuitable scope Question Have you go through the requirement specification report before determine the scope? This question checks whether test plan cover all factors which are should test in the testing phase. If testing team misses any important items to test, it will cause to fail the ERP system. 4.3. Mess of testing Question Have you prioritized the modules and functions to be tested? It is very important to correctly identify what are the most important and what are the primary testing functions to test. If not it will raise two major problems. I. Some interdependent function may lose to test. II. Some very important and critical function may escape to test within the estimated time. Hence this question asks to ensure that they don’t miss any critical function to test. 4.4. Incorrect testing methods Question Have you determined what methods you are going to use for each test function? It is very important to determine what method will be used to what functions. Some functions should be automated; some require human inspection while some are both. Yes No Yes No Yes No
  • 26. ERP TESTING FAILURES | UCSC | GROUP 9 Page 25 of 30 Therefore this question ask check whether testing team clearly define the testing methods which they going to use. 4.5. Under and over estimations of resources Question Have you identified all resources you want to do testing? Testing phase requires lot of resources and it will deviate from ERP system to system. Some ERP system tests require simulation software or very advance software having artificial intelligent. Some system tests require install new platforms for testing environment. Hence it is very important to verify all resource requirements and availability of current resources. 4.6) Budget allocation Question How much do you allocate for testing phase from budget? (As percentage of whole budget) …………………………………………………………………………………………………………… It is important to allocate at least 40% of budget from whole to testing phase. Therefore this question is check whether they allocate enough budgets for testing phase. 4.7) Validation and verification problem Question What is the test pass percentage of testing respect to all test cases? ………………………………………………………………………………………………….. Some ERP systems require more reliability while some are not much. As well as with time constraint user may like to accept system with less testing as first version of system in order to continuously improve. Therefore it is very important to set proper percentage to pass the test. Most of time the ERP system cannot achieve 100% bug free system. Therefore this question check whether set percentage is accurate or not. 4.8) Estimation failures Question How do you estimate the effort?  Based on previous experience  Based on standard effort estimate models  Based on expertise knowledge  Based on typical values like importance of functions, criticality of function or number of test cases Yes No
  • 27. ERP TESTING FAILURES | UCSC | GROUP 9 Page 26 of 30 These two questions are asked to ensure whether they do estimation using correct technique or not. Different estimations use different methods to estimate effort. 4.9) Incomplete report Question Are you going to review test report before finalizing it? This question is asked to check whether they do reviews on report before finalizing it. It is very important to properly do the reporting, because it will use to make decision on next steps of testing and maintenance processes. Yes No
  • 28. ERP TESTING FAILURES | UCSC | GROUP 9 Page 27 of 30 5) Other Testing Failures 5.1) E-commerce and security compliance failures Questions Did you go through all rules and regulations to determine e-commerce compliance? Have you obtained any CPI compliant certificate? If “yes”, mention it here………………………………………… E-commerce involving ERP systems must be very secure. If not it will raise fraud through the system. As well as confidence of user privacy is very important factor in ERP systems, especially when e-commerce involve in the processes. If system lose to ensure confidence of data whole ERP system is useless. Thus these questions are asked to check the security level of ERP system. 5.2) Communication failures Question Do developers and testers have peer communication method? If yes, what is the present communicate system? ................................................................... Communication problems arise during the quality assurance (QA) process. This is the result of using different tools and technologies, or working in disparate locations. But even when development and testing take place in the same building, communication problems still exist. To overcome these issues we can use communication systems with standardized frameworks. 5.3) Failures of third party tools Question Are you going to use third party tools for testing? Most of all the time we have to use third party tools for testing. When we are choosing those tools there should be a well-defined set of rules. That means we have to check them for their quality, testability and the reliability. The aim of this question is to track the third party tools usage of the testing process. Yes No Yes No Yes No Yes No
  • 29. ERP TESTING FAILURES | UCSC | GROUP 9 Page 28 of 30 Mind Map
  • 30. ERP TESTING FAILURES | UCSC | GROUP 9 Page 29 of 30 Reference List 1. Configuration Testing | James D. McCaffrey. 2013. Configuration Testing | James D. McCaffrey. [ONLINE], Available at: http://jamesmccaffrey.wordpress.com/2009/01/31/configuration-testing/. [Accessed 05th June 2013]. 2. ERP Testing | Testing ERP | Test ERP | ERP Test | ERP Validation. 2013. ERP Testing | Testing ERP | Test ERP | ERP Test | ERP Validation. [ONLINE], Available at: http://www.qualitestgroup.com/ERP-Testing. [Accessed 04th June 2013]. 3. James D. McCaffrey, 2009, Configuration Testing | James D. McCaffrey, [E-book], Available at: http://jamesmccaffrey.wordpress.com/2009/01/31/configuration-testing/, [Accessed 07th June 2013]. 4. Test driven guy: Common Unit Testing Problems. 2013. Test driven guy: Common Unit Testing Problems. [ONLINE], Available at:http://testdrivenguy.blogspot.com/2009/05/common-unit-testing-problems.html, [Accessed 03rd June 2013]. 5. Top 10 reasons why teams fail with Acceptance Testing . 2013. Top 10 reasons why teams fail with Acceptance Testing . [ONLINE], Available at: http://gojko.net/2009/09/24/top-10- reasons-why-teams-fail-with-acceptance-testing/, [Accessed 04th June 2013]. 6. Writing Great Unit Tests: Best and Worst Practices - Steve Sanderson’s blog - As seen on YouTube™. 2013. Writing Great Unit Tests: Best and Worst Practices - Steve Sanderson’s blog - As seen on YouTube™. [ONLINE], Available at: http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst- practises/, [Accessed 03rd June 2013]. 7. Why do performance tests fail? @ 1202Performance. 2013. Why do performance tests fail? @ 1202Performance. [ONLINE], Available at: http://www.1202performance.com/2012/02/why-do-performance-tests-fail/, [Accessed 04th June 2013]. 8. What is Regression Testing?. 2013. What is Regression Testing?. [ONLINE], Available at: http://www.guru99.com/regression-testing.html, [Accessed 04th June 2013]. 9. Usability testing mistakes | Information & Design. 2013. Usability testing mistakes | Information & Design. [ONLINE], Available at: http://infodesign.com.au/usabilityresources/usabilitytestingmistakes/, [Accessed 05th June 2013]. 10. Zhen Ming Jiang, Ahmed E. Hassan, Gilbert Hamann, Parminder Flora, 2006, Automatic Identification of Load Testing Problems, [E-book], Available at: http://sail.cs.queensu.ca/publications/pubs/ICSM2008Jiang.pdf, [Accessed 05th June 2013].

×