ERP Testing Failures - Questionnaire

University of Colombo School of Computing
ICT 3001
ENTERPRISE RESOURCE PLANNING
THIRD YEAR GROUP ASSIGNMENT
ERP TESTING FAILURES - QUESTIONNAIRE
GROUP 9
Lecturer: - Mr. Samantha Mathara Arachchi
Date of Submission
10
th
of June 2013
Name Registration Index
Number Number
SANDARUWAN J.W.C. 2010/ICT/018 10020187
WICKRAMASINGHE W.M.M. 2010/ICT/038 10020381
MENDIS D.G.T. 2010/ICT/057 10020578
UPENDRA K.A.N. 2010/ICT/070 10020705
KARUNARATHNE H.B.K.A. 2010/ICT/082 10020829
WIJESUNDARA W.M.D.M. 2010/ICT/092 10020926
KARUNARATHNA S.A. 2010/ICT/102 10021027

ERP TESTING FAILURES | UCSC | GROUP 9 Page 1 of 30
We are a research group from University of Colombo School of Computing (UCSC). We
would like to invite you to participate in a research study which aims at collecting data for a
project as a part of Enterprise Resource Planning (ICT 3001).
Before you begin, take a few minutes to read why we are inviting you to participate and what
will be done with the information you provide. You will be asked to fill out a questionnaire
that aims to investigate the ERP testing failures which could be beneficial you all as you are
a part of the ERP community. Please feel free to stop at any time if you have questions about
the study. This project is conducted only on sites which are engaging in ERP development.
Your individual privacy and confidentiality of the information you provide will be maintained
in all published and written data analysis resulting from the study.
Your participation should take approximately 10-15 minutes for each phase mentioned in the
questionnaire. We ask you to fill it out with as much accuracy as possible. Please understand
your participation is entirely on a voluntary basis and you have the right to withdraw your
consent or discontinue participation at any time without penalty. At no instance you will be
asked to reveal any personal information. Participation in this study involves no major risks
whatsoever, be it physical or emotional. This study is directly beneficial to the participants,
engaging in ERP development. You will receive no incentive or payment for your
participation.
If at any time and for any reason, you would prefer not to answer any questions, please feel
free to skip those questions. If at any time you would like to stop participating, please tell us.
We can take a break, stop and continue at a later date, or stop altogether. You will not be
penalized for deciding to stop participation at any time. If you have any questions, you are
free to ask them now. If you have questions later, you may contact:
If you have any questions about your rights as a participant in this study, or you want to talk
to someone outside the research, please contact the:
University of Colombo School of Computing No. 35, Reid Avenue
Colombo 07.
Tel: +94 112- 581245-7
I have read and understood the above, and I agree to participate in
the study.
The consent form is to be detached and kept with the participant.
Name Phone Number Email address
Saman Jayantha 0771-362587 samanjayantha@gmail.com
Dilruk Karunarathne 0715-897423 dilrukkaru@gmail.com
WELCOME TO THE ERP TESTING FAILURES SURVEY, 2013

Table of Contents
WELCOME TO THE ERP TESTING FAILURES SURVEY, 2013 .......................................1
1) Unit Testing...........................................................................................................................4
2) Integration Testing.................................................................................................................6
3) System Testing ......................................................................................................................9
3.1) Usability Testing.............................................................................................................9
3.2) Load testing and Stress testing........................................................................................9
3.3) Parallel Testing .............................................................................................................10
3.4) Configuration testing and installation testing ...............................................................10
3.5) Security Testing ............................................................................................................11
4) Documentation Testing........................................................................................................11
4.1) Master Plan ...................................................................................................................11
4.1.1) Incomplete Documentation ....................................................................................11
4.1.2) Unsuitable scope ....................................................................................................11
4.1.3) Mess of testing .......................................................................................................11
4.1.4) Incorrect testing methods .......................................................................................12
4.2) Level test plan...............................................................................................................13
4.3) Test report.....................................................................................................................14
4.3.1) Ambiguity...............................................................................................................14
5) Other Testing Failures .........................................................................................................15
Justification..............................................................................................................................17
1) Unit Test ..............................................................................................................................17
2) Integration Test Failures......................................................................................................18
2.1) Interface Testing Failures .............................................................................................18
2.2) Acceptance Testing Failures.........................................................................................18
2.3) Interoperability Testing Failures Question ..................................................................18
2.4) Infrastructure Testing Issues.........................................................................................19
3) System Testing Failures.......................................................................................................20
3.1) Failures with Usability Testing.....................................................................................20
3.1.1) Poor Task Design ...................................................................................................20
3.1.2) Not well structured test cases.................................................................................20
3.2) Load testing and Stress testing......................................................................................21
3.2.1) Bugs in the Application under test .........................................................................21

3.2.2) Software Interaction failures ..................................................................................21
3.2.3) Problems with the Load Generation.......................................................................22
3.3) Parallel Testing Failures ...............................................................................................22
3.3.1. Higher complexity of the system............................................................................22
3.3.2. Less interaction with sub system component testing..............................................22
3.4) Configuration testing and installation testing failures ..................................................22
3.4.1. Tests are getting too much time for their completion.............................................22
3.4.2. Compatibility issues in the system. ........................................................................23
3.4.3. Failures of identifying configuration changes in the system..................................23
3.5) Security testing failures ................................................................................................23
3.5.1 Core security concepts are not followed..................................................................23
4) Documentation testing failures............................................................................................24
4.1. Incomplete Documentation........................................................................................24
4.2. Unsuitable scope........................................................................................................24
4.3. Mess of testing...........................................................................................................24
4.4. Incorrect testing methods...........................................................................................24
4.5. Under and over estimations of resources...................................................................25
4.6) Budget allocation.......................................................................................................25
4.7) Validation and verification problem .........................................................................25
4.8) Estimation failures.....................................................................................................25
4.9) Incomplete report ......................................................................................................26
5) Other Testing Failures .........................................................................................................27
5.1) E-commerce and security compliance failures .............................................................27
5.2) Communication failures................................................................................................27
5.3) Failures of third party tools...........................................................................................27
Mind Map.................................................................................................................................28
Reference List ..........................................................................................................................29

1) Unit Testing
1. Does unit test catch each and every bug in the code?
2. This question is regarding the test cases. Tick your opinion/s.
3. Do you agree or disagree with this statement. (1 being highly agree and 5 being highly
disagree)
“The test code also may have bugs same as the code it testing. Therefore if any error of bug
found it is difficult to identify whether it is an error with the code or test code”.
1 2 3 4 5
4. Do you believe before start unit testing is it necessary to remember as the test is behaved as
a part of the complete system to exercise the code in a realistic scenario?
5. What are the best practices you follow in unit testing?
6. What are the biggest challenges you will be facing in ERP unit testing?
Yes No
 The test cases for unit testing are normally written by developers with the same
programming language, before implementation of each unit.
 Test cases should be developed for every exception path and for every possible input.
 Then programmers have to write 3 to 5 lines of test code for every written line of the code.
 This required much time and effort.
 This doesn’t require much time and effort.
Yes No
 Following a rigorous way of working through development process.
 Keep records of the entire test that have been performed.
 No need of writing test cases for every exception path and for every possible input.
 Make records of all the changes that have been made in the source code during the unit
testing.
If others, mention them briefly: - -----------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------
 Writing test cases for every exception path and for every possible input.
 Software is being developed and tested in one platform and should be deployed in to another
platform.
 Make records of all the changes that have been made in the source code during the unit
testing.
 Writing tests for legacy codes are not an easy task and it is necessary to refer documentation
of those legacy systems and may have to change the code to fit with the tests.
---------------------------------------------------------------------------------------------------------------------

7. Tick the choices that you feel more comfortable with system test failures?
8. Select most suitable choices regarding unit test automation?
 Developing tests for each and every unit may not be possible with the time and budget
constraints of the project which in such situations developers may use same test cases to test
many units.
 If there is an error with a test case there will be many testing failures with many units. And if
two or more units use same resources simultaneously test failure may increase.
 Tests those are hard to understand cause failures.
---------------------------------------------------------------------------------------------------------------------
 Unit test automation can solve most of unit testing failures.
 Test automation cannot separate each component and validate its correctness well.
 Using framework is a popular method of unit test automation. It simplifies the unit test process
and has been developed for wide variety of languages.
 Unit test along cannot solve most of the failures and are only developed for few languages.
---------------------------------------------------------------------------------------------------------------------

2) Integration Testing
9. Have you been provided the scope of the integration testing?
10. How clear does your defined scope?
Very clear
Clear
Neither clear nor not clear
Not very clear
Not clear at all
11. How many number of “items” tested and not tested?
I. Number of items that are tested ………
II. Number of items that are not tested ………
12. How many number of “features” tested and not tested?
I. Number of features that are tested ………
II. Number of features that are not tested ………
13. Is the system tested on possible access platforms?
14. Did you identify data mismatches in the system?
a) If “Yes”, how many data mismatches did you correct? ………
15. Was the system tested by the client representatives and system users?
16. Do all the business functions work correctly?
If not, list the functions that fail to perform.
…………………………………………………………………………………………………
…………………………………………………………………………………………………
17. Did you achieve the enterprise wide compatibility?
…………………………………………………………………………………………………
…………………………………………………………………………………………………
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No

18. Is each functional module tested with its corresponding interoperating external module?
If not, list the modules that were left untested.
…………………………………………………………………………………………………
…………………………………………………………………………………………………
19. What are the QA related reference documents referred? (tick “” appropriate documents)
 IEEE Std. 1465-1998 // ISO/IEC 12119
(IEEE Standard for Quality requirements and testing)
IEEE Std. 829-2008
(IEEE Standard for Software Test Documentation)
Other (state name)
…………………………………………………………………………………………
…………………………………………………………………………………………
20. What are other applicable documents referred within the system? (tick “” appropriate
documents)
Architectural design document
Detailed design document
System requirement specification
Problem report
Modification request
…………………………………………………………………………………………
…………………………………………………………………………………………
21. What are the planned deliverables before starting integration testing (tick “” if delivered)
Architectural design document
Detailed design document
Unit test pan and report
Integration test plan
…………………………………………………………………………………………
…………………………………………………………………………………………
22. Did you check the pre-installation compatibility?
23. Did you carry out a contingency planning?
24. Does testing team have adequate qualified hardware testers?
Yes No
Yes No
Yes No
Yes No

25. What are the deliverables have to be submitted after finishing the integration test? (tick
“” if delivered)
Integration test report
Integration test output data
Problem report
…………………………………………………………………………………………
…………………………………………………………………………………………
26. Environmental needs definition was,
 Excellent
 Good
 Average
 Poor
 Not defined at all
27. Test case pass fail criteria definitions were,
Excellent
Good
Average
Poor
Not defined at all
28. Test procedure definitions were,
Excellent
Good
Average
Poor
Not defined at all

3) System Testing
3.1) Usability Testing
29. Did you use any usability testing tool/tools?
30. Did you consider the system that you are going to test, when selecting testing tools?
31. Did you find any changes required to improve user performance from those testing?
I. If YES, Did you notify those changes to the user?
32. Did you use an essential task designing component when designing tasks?
33. Following question is based on test cases.
I. Do you have following characteristics with your test cases?
a) It is accurate and tests what is intended to test.
b) No unnecessary steps included to it.
c) It is reusable.
d) It is traceable to requirements.
e) It is simple and clear, any tester is able to understand it by reading once.
3.2) Load testing and Stress testing
34. What is the loading test category that your system belongs to?
 Quantification of risk
 Determination of minimum configuration
Yes No
If Yes, mention them briefly: - -----------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
-
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
If others, mention them briefly: - --------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------

35.Did you test the system from both load and unload execution under load testing?
36.Can you certify that the load generator is working correctly?
37.Does the system interact with other applications, when execution?
I. If YES, did you check impact of other applications, with the load test?
3.3) Parallel Testing
38. Do you use complex methodologies in parallel testing?
39. Are you monitoring the tests in sub systems?
40. Is there a mechanism to measure that you test all the parts of the system?
3.4) Configuration testing and installation testing
41. Are you using business process based scenarios for testing?
42. Are you using different platforms, supported drivers and different computer
configurations?
43. Have you clearly identified where configuration changes might take place?
44. Have you defined test cases for each configuration tests?
45. Have you used an automated mechanism for configuration tests?
Yes No
Yes No
Yes No
Yes No
Yes No
If Yes, mention them briefly: - -----------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No

46. What are the system components that are tested for compatibility issues?
3.5) Security Testing
47. What are the types of security testing are you performing in the system?
4) Documentation Testing
4.1) Master Plan
4.1.1) Incomplete Documentation
48. What is the template do you use for test plan documentation?
 IEEE 829 Standard
 Organizational custom template
 If any other standard (mention it here)……………………………………………………
4.1.2) Unsuitable scope
49. Have you go through the requirement specification report before determine the scope?
50. Have you identified what are the testable functions and non-testable functions?
4.1.3) Mess of testing
51. Have you prioritized the modules and functions to be tested?
52. Have you prepared a work breakdown schedule for testing?
 Hardware platforms  Software platforms
If any, mention in briefly: ----------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
 Confidentiality  Integrity  Authentication
 Availability  Authorization  Non-repudiation
If any, mention in briefly: ----------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
Yes No
Yes No
Yes No
Yes No

4.1.4) Incorrect testing methods
53. Have you determined what methods you are going to use for each test function?
4.1.5) Under and over estimations of resources
54. Have you identified all resources that you need to conduct the test?
55. What are the resources you have identified mentioned in last question?
Hardware resources (number of machines, performance, storages, etc)
Required Platforms (OS / versions )
Human resource (number of team members, skills, expertise, etc)
Budget
Mention any other resources you determine in documentation (if any)
…………………………………………………………………………………………………
56. How much do you allocate for testing from budget? (As percentage of whole budget)
…………………………………………………………………………………………………
57. Have you measured the time required for whole testing process?
.
58. Have you set buffer time to accommodate unexpected delays?
4.1.6) Ignore Risks
59. Have you determine critical functions to be tested?
60. Have you identified risks with the product?
4.1.7) Validation and verification problem
61. What is the test pass percentage of testing respect to all test cases?
……………………………………………………………………………………………
62. Have you validated the test plan with respect to the requirement specification?
63. Have you reviewed the test plan?
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No

4.1.8) Ignore future requirement
64. What are the deliverables of test?
 Test cases
 Test scripts
 Test report
 Test log
 Work load analysis
If any other deliverables mention them here
……………………………………………………………………………………………
4.2) Level test plan
These questions are regarding the testing documentation of ERP components testing.
4.2.1) Inadequate testing
65. What are the methods you are going to use for testing?
 Black box testing
 White box testing
 Gray box testing
 Manual testing
 Automated testing
Mention any other methods you have determine in test plan documentation
………………………………………………………………………………………………
66. Have you determined the methods going to use for testing each phase?
67. Have you set test pass percentage of each testing units?
4.2.2) Incorrect estimations
68) Have you identified unique resources required in each test unit?
69) Have you estimated testing effort of each test unit?
Yes No
Yes No
Yes No
Yes No

70) How do you estimate the effort?
 Based on previous experience
 Based on standard effort estimate models
 Based on expertise knowledge
 Based on typical values like importance of functions, criticality of function or
number of test cases
4.3) Test report
This section is regarding the test reporting.
4.3.1) Ambiguity
71) Have you clearly defined what, when, who, where, how and to whom reporting?
If you said some are not, what are the things you defined?
……………………………………………………………………………………………
4.3.2) Inadequate Report
72) What are the charts and diagrams you are going to generate in tests?
 Pareto chart
 Cause and effect Diagrams (fishbone diagram, Ishikawa diagram, characteristics
diagram)
 Check sheets
 Histogram
 Run chart
 Control chart
 Scatter plot diagrams
If any other diagram mention it here.
……………………………………………………………………………………………
73) What are the analysis tools used to report test results?
 Cluster analysis
 Regression analysis
 Multivariate analysis
If any other analysis mention it here
…………………………………………………………………………………………..
Yes No Some are not

74) What are the reports you generate during the testing?
 Function test matrix
 Defect status report
 Functional testing status report
 Testing action report
If any other report mention it here
…………………………………………………………………………………………..
75) What are the reports you generate as final test reports?
 Unit test report
 Integration test report
 System test report
If any other report mention it here
…………………………………………………………………………………………..
4.3.3) Incomplete report
76) Are you going to review test report before finalizing it?
5) Other Testing Failures
77) What parts of testing are you going to automate?
 Management part of testing
 Test specification part
 Static testing part
 Test execution and logging part
 Performance and monitoring part
If any other parts mention them here
………………………………………………………………………………………....
78) Is testing team experienced with test automating tools?
Yes No
Yes No

79) Did you go through all rules and regulations to determine e-commerce compliance?
80) Are you going to use third party tools for testing?
a) If “Yes”, do you check the quality, reliability and the testability of that tools?
If your answer is “yes”, answer for these questions.
1. Can the selected tool handle the required functions?
81) How long the vendor stable in industry? _______________________________________
82) Are the selected tools upgraded continuously?
83) Are selected tools compatible with your environment?
84) Is the testing team familiar with the selected automating tools?
85) Have you obtained any CPI compliant certificate?
If “yes”, mention it here…………………………………………
86) Do developers and testers have peer communication method?
If yes, what is the present communicate system? ...................................................................
87) Are you satisfied about experience and knowledge of testing team members?
 Extremely satisfied
 Moderately satisfied
 Neither satisfied nor dissatisfied
 Dissatisfied
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No

Justification
1) Unit Test
1.1) Unit test failures
Question
Do u agree with this statement is correct or not. (1 being highly agree and 5 being highly
disagree)
“The test code also may have bugs same as the code it testing. Therefore if any error of bug
found it is difficult to identify whether it is an error with the code or test code”.
1 2 3 4 5
This question is asked to catch whether the tester have good knowledge on test code and
program code separately. From the response we can identify the tester knowledge on program
code and test code.
Question
Before start unit testing is it necessary to remember as the test is behaved as a part of
the complete system to exercise the code in a realistic scenario?
Tester should have a good knowledge in real world business process before writing the test
code. The aim of this question is to catch that answer. If not the system won’t be able to
match with the defined framework.
Question
What are the biggest challenges you will be facing in ERP unit testing?
From this multiple choice question we can measure the readiness of the system implementers.
Based on the answers to this question we can get an idea of their knowledge regarding ERP
challenges where it would be useful to build a better system.
Question
What are the best practices you follow in unit testing?
Best practices are mandatory in ERP implementation. Failures can be occur due to mal
practices followed by people. The aim of this question is to measure the level of following
best practices.
Yes No

Question
Select most suitable answers regarding unit test automation?
With this multiple choice question we are trying to find the unit test automation need for the
organization. Based on the answers we can come to a conclusion whether the development
clearly caught the test automation or not.
2) Integration Test Failures
2.1) Interface Testing Failures
Question
Is the system tested on possible access platforms?
The system is being accessed by users using various platforms. Thus it should be tested in
order to eliminate the platform issues. This question is used to identify those issues.
2.2) Acceptance Testing Failures
Question
Was the system tested by the client representatives and system users?
Acceptance of the system by its users should be verified. These question find out how the
accuracy of the system and performance are being verified.
2.3) Interoperability Testing Failures
Question
Is each functional module tested with its corresponding interoperating external module?
If not, list the modules that were left untested.
…………………………………………………………………………………………………
…………………………………………………………………………………………………
This question finds out all the scenarios where mismatches between modules and business
functions have been occurred. By this, mismatches between modules that interoperate in
different departments can be identified.
Yes No
Yes No
Yes No

Question
Did you achieve the enterprise wide compatibility?
…………………………………………………………………………………………………
…………………………………………………………………………………………………
This question finds out all the scenarios where mismatches between modules and business
functions have which can lead to the system being isolated from the whole network
(international) due to module mismatches and resulting in transaction failures.
2.4) Infrastructure Testing Issues
Question
Did you check the pre-installation compatibility?
This question finds out whether the compatibility of the hardware components and the
software components being used in the system are verified. This is known as device
compatibility testing.
Question
Did you carry out a contingency planning?
This question derives the necessary information regarding the performance of the system in
disaster situations and the stated guidelines to follow in a such situation and the drawbacks of
the guidelines.
Yes No
Yes No
Yes No

3) System Testing Failures
3.1) Failures with Usability Testing
Question
Did you consider the system that you are going to test, when selecting testing tools?
The testing tool should be based on the type of the testing method. So the appropriate testing
type should be chosen.
Question
Did you find any changes required to improve user performance from those tests?
In lots of situations, overall testing process will identify changes required to improve. It is
like a successful character in usability testing. Therefore, from this question analyst can
identify the results of modifications in system.
Question
Did you find any changes required to improve user performance from those testing?
I. If YES, Did you notify those changes to the user?
From this question, analyst can make sure that all component of the system meet the user
requirements. Sometime tester may not notice recently changed user requirements. Hence,
some components that do not meet the user requirements could be included in the final
product. This question is asked to track that kind of failures.
3.1.1) Poor Task Design
Question
Did you use essential task designing component when designing tasks?
In order to create effective design task, essential task designing component should be
included. As instances task scope, task support, task resource and task execution can be
mentioned. Ensuring that the tasks have been designed well or not is the core aim of this
question.
3.1.2) Not well structured test cases
Question
Do you have following characteristics with your test cases?
Yes No
Yes No
Yes No
Yes No
Yes No

a) It is accurate and tests what is intended to test.
b) No unnecessary steps included to it.
c) It is reusable.
d) It is traceable to requirements.
e) It is simple and clear, any tester is able to understand it by reading once.
Every tester writes test cases however many times the test cases are rejected by reviewers
because of bad quality, in order to write good test cases one should know what are the
characteristics of a good test case. Using this question, analyst can make sure that the test
cases are well structured and cover all what are intended to test.
3.2) Load testing and Stress testing
3.2.1) Bugs in the Application under test
Question
Did you test the system from both load and unload execution under load testing?
Most systems use load testing for identify sensitivity errors. And these sensitivity errors
appear only under the load or extended execution. Hence tester also tests under the load or
extended execution always. Sometime there can be errors without loading execution. Those
errors can also be identified from the above question.
3.2.2) Software Interaction failures
Question
Does the system interact with other applications, when execution?
This question is related to load testing. Load testing is showing all problems that are related
in handling load. It does not consider how many programs interact with the system while
showing result as a whole. Thus tester cannot certify that the problems are related to which
software. Hence from this question analysts can know whether system is interacted with other
application at the execution.
If YES, did you check impact of other application, with the load test?
If there is another application running same time, then first attention has to be paid for that
application. If that application works correctly then there is no impact for our load testing
result. Hence from this question, analyst can make sure that the load balancing is not showing
false results.
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No

3.2.3) Problems with the Load Generation
Question
Can you certify that the load generator that is used is working correctly?
Load generators are used for generating hundreds of concurrent request to access the
application. And then load the execution and check results. If load generator is not properly
working as intended, then end results can be false. Therefore the purpose of this question is to
identify whether load generating system works properly or not.
3.3) Parallel Testing Failures
3.3.1. Higher complexity of the system
Question
Do you use complex methodologies in parallel testing?
Based on the answer to this question, we can measure the complexity of the parallel testing.
There is a high possibility of parallel testing failure due to the complexity. Methodology
complexities can be simplified in order to minimize the potential failures.
3.3.2. Less interaction with sub system component testing
Question
Are you monitoring the tests in sub systems?
This question is asked to get an idea about sub system testing. If we get the answer as “No”,
we can assume there can be possible testing failures in sub systems. In parallel testing, sub
system test monitoring is a crucial factor for identifying the potential failures.
3.4) Configuration testing and installation testing failures
3.4.1. Tests are getting too much time for their completion.
Question
Have you used an automated mechanism for configuration tests?
This question is asked to catch the testing mechanism of the system. It means the manual
testing and the automated testing. Configuration and installation tests are taking much time in
manual testing as we have to look various kinds of test components. In order to get a
maximum throughput we can automate those tests.
Yes No
Yes No
Yes No

3.4.2. Compatibility issues in the system.
Question
What are the system components that are tested for compatibility issues?
Based on the answer for this question we can identify whether the system is properly tested
for compatibility issues.
3.4.3. Failures of identifying configuration changes in the system.
Question
Have you clearly identified where configuration changes might take place?
Due to the complexity of the system sometimes people couldn’t identify the places where
configuration changes should take place. From this question we can come to an idea, whether
configuration changes are correctly identified or not.
3.4.4. Test cases are not defined for every configuration test due to complexity
Question
Have you defined test cases for each configuration tests?
Whatever we have to test in front of us we have to do it. There are instances where people are
not defining test cases for small configuration changes that will cause serious problems in
system functionality.
3.5) Security testing failures
3.5.1 Core security concepts are not followed.
Question
What are the types of security testing are you performing in the system?
Based on the user answer we can get an idea of the system’s security. If the system is not
following the desired standards there will be drawbacks in security.
 Hardware platforms  Software platforms
If any, mention in briefly: --------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
Yes No
Yes No
 Confidentiality  Integrity  Authentication
 Availability  Authorization  Non-repudiation
If any, mention in briefly: --------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------

4) Documentation testing failures
4.1. Incomplete Documentation
Question
What is the template you use for test plan documentation?
 IEEE 829 Standard
 Organizational custom template
 Any other standard (mention it here)……………………………………………………
There should be a standard format to any business documentation. It is very important to
ensure they don’t miss any important thing to document with a standard. This question asks
to check whether they use proper standard template for documentation.
4.2. Unsuitable scope
Question
Have you go through the requirement specification report before determine the scope?
This question checks whether test plan cover all factors which are should test in the testing
phase. If testing team misses any important items to test, it will cause to fail the ERP system.
4.3. Mess of testing
Question
Have you prioritized the modules and functions to be tested?
It is very important to correctly identify what are the most important and what are the primary
testing functions to test. If not it will raise two major problems.
I. Some interdependent function may lose to test.
II. Some very important and critical function may escape to test within the estimated
time.
Hence this question asks to ensure that they don’t miss any critical function to test.
4.4. Incorrect testing methods
Question
Have you determined what methods you are going to use for each test function?
It is very important to determine what method will be used to what functions. Some
functions should be automated; some require human inspection while some are both.
Yes No
Yes No
Yes No

Therefore this question ask check whether testing team clearly define the testing methods
which they going to use.
4.5. Under and over estimations of resources
Question
Have you identified all resources you want to do testing?
Testing phase requires lot of resources and it will deviate from ERP system to system. Some
ERP system tests require simulation software or very advance software having artificial
intelligent. Some system tests require install new platforms for testing environment. Hence it
is very important to verify all resource requirements and availability of current resources.
4.6) Budget allocation
Question
How much do you allocate for testing phase from budget? (As percentage of whole budget)
……………………………………………………………………………………………………………
It is important to allocate at least 40% of budget from whole to testing phase. Therefore this
question is check whether they allocate enough budgets for testing phase.
4.7) Validation and verification problem
Question
What is the test pass percentage of testing respect to all test cases?
…………………………………………………………………………………………………..
Some ERP systems require more reliability while some are not much. As well as with time
constraint user may like to accept system with less testing as first version of system in order
to continuously improve. Therefore it is very important to set proper percentage to pass the
test. Most of time the ERP system cannot achieve 100% bug free system. Therefore this
question check whether set percentage is accurate or not.
4.8) Estimation failures
Question
How do you estimate the effort?
 Based on previous experience
 Based on standard effort estimate models
 Based on expertise knowledge
 Based on typical values like importance of functions, criticality of function or
number of test cases
Yes No

These two questions are asked to ensure whether they do estimation using correct technique
or not. Different estimations use different methods to estimate effort.
4.9) Incomplete report
Question
Are you going to review test report before finalizing it?
This question is asked to check whether they do reviews on report before finalizing it. It is
very important to properly do the reporting, because it will use to make decision on next steps
of testing and maintenance processes.
Yes No

5) Other Testing Failures
5.1) E-commerce and security compliance failures
Questions
Did you go through all rules and regulations to determine e-commerce compliance?
Have you obtained any CPI compliant certificate?
If “yes”, mention it here…………………………………………
E-commerce involving ERP systems must be very secure. If not it will raise fraud through the
system. As well as confidence of user privacy is very important factor in ERP systems,
especially when e-commerce involve in the processes. If system lose to ensure confidence of
data whole ERP system is useless. Thus these questions are asked to check the security level
of ERP system.
5.2) Communication failures
Question
Do developers and testers have peer communication method?
If yes, what is the present communicate system? ...................................................................
Communication problems arise during the quality assurance (QA) process. This is the result
of using different tools and technologies, or working in disparate locations. But even when
development and testing take place in the same building, communication problems still exist.
To overcome these issues we can use communication systems with standardized frameworks.
5.3) Failures of third party tools
Question
Are you going to use third party tools for testing?
Most of all the time we have to use third party tools for testing. When we are choosing those
tools there should be a well-defined set of rules. That means we have to check them for their
quality, testability and the reliability. The aim of this question is to track the third party tools
usage of the testing process.
Yes No
Yes No
Yes No
Yes No

Mind Map

Reference List
1. Configuration Testing | James D. McCaffrey. 2013. Configuration Testing | James D.
McCaffrey. [ONLINE], Available at:
http://jamesmccaffrey.wordpress.com/2009/01/31/configuration-testing/. [Accessed 05th
June
2013].
2. ERP Testing | Testing ERP | Test ERP | ERP Test | ERP Validation. 2013. ERP Testing |
Testing ERP | Test ERP | ERP Test | ERP Validation. [ONLINE], Available at:
http://www.qualitestgroup.com/ERP-Testing. [Accessed 04th
June 2013].
3. James D. McCaffrey, 2009, Configuration Testing | James D. McCaffrey, [E-book],
Available at: http://jamesmccaffrey.wordpress.com/2009/01/31/configuration-testing/,
[Accessed 07th
June 2013].
4. Test driven guy: Common Unit Testing Problems. 2013. Test driven guy: Common Unit
Testing Problems. [ONLINE], Available
at:http://testdrivenguy.blogspot.com/2009/05/common-unit-testing-problems.html, [Accessed
03rd
June 2013].
5. Top 10 reasons why teams fail with Acceptance Testing . 2013. Top 10 reasons why teams
fail with Acceptance Testing . [ONLINE], Available at: http://gojko.net/2009/09/24/top-10-
reasons-why-teams-fail-with-acceptance-testing/, [Accessed 04th
June 2013].
6. Writing Great Unit Tests: Best and Worst Practices - Steve Sanderson’s blog - As seen on
YouTube™. 2013. Writing Great Unit Tests: Best and Worst Practices - Steve Sanderson’s
blog - As seen on YouTube™. [ONLINE], Available
at: http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst-
practises/, [Accessed 03rd
June 2013].
7. Why do performance tests fail? @ 1202Performance. 2013. Why do performance tests fail?
@ 1202Performance. [ONLINE], Available at:
http://www.1202performance.com/2012/02/why-do-performance-tests-fail/, [Accessed 04th
June 2013].
8. What is Regression Testing?. 2013. What is Regression Testing?. [ONLINE], Available at:
http://www.guru99.com/regression-testing.html, [Accessed 04th
June 2013].
9. Usability testing mistakes | Information & Design. 2013. Usability testing mistakes |
Information & Design. [ONLINE], Available at:
http://infodesign.com.au/usabilityresources/usabilitytestingmistakes/, [Accessed 05th
June
2013].
10. Zhen Ming Jiang, Ahmed E. Hassan, Gilbert Hamann, Parminder Flora, 2006, Automatic
Identification of Load Testing Problems, [E-book], Available at:
http://sail.cs.queensu.ca/publications/pubs/ICSM2008Jiang.pdf, [Accessed 05th
June 2013].

ERP Testing Failures - Questionnaire

More Related Content

Recently uploaded

Featured

ERP Testing Failures - Questionnaire