Test Methods and Tools for ERP Implementations - Proposed Research Project Paul Gerrard Gerrard Consulting 1 Old Forge Close Maidenhead Berkshire SL6 2RD UK e: firstname.lastname@example.org w: http://gerrardconsulting.com t: 01628 639173
Introduction to ERP and the ERP Market
Outline of Work Already Done
Proposed Research Project.
Agenda Assurance with Intelligence Slide
Enterprise Resource Planning (ERP) systems represent a huge market in the commercial arena
SAP, Oracle and more recently, Microsoft, dominate the market
Testing is a significant effort but is hardly supported by methods and tools
Experience shows that testing in these projects is critical, but often neglected
A Benefits, Risk and Coverage-based test approach could significantly reduce the risk of failures.
Summary Assurance with Intelligence Slide
Enterprise Resource Planning systems (ERPs) aim to integrate all data and processes of an organization into a unified system
Typical ERP system will use multiple components of computer software and hardware to achieve the integration
Key ingredient of most ERP systems is the use of a unified database to store data for the various system modules.
An Introduction to ERP Assurance with Intelligence Slide
ERP scope has expanded dramatically and usually includes:
Supply Chain Management
Customer Relationship Management
Large organisations want to implement standard processes and systems across divisions and subsidiaries
ERP offers attractive, integrated solutions
The market for these solutions is huge.
Scope of ERP Assurance with Intelligence Slide
ERP market dominated by SAP, Oracle, Microsoft worth EUR 40 billion
SAP is the biggest and most popular
Founded in 1972 by five ex-IBM employees, by 2005, there were 100,600 installations worldwide, with over 25 industry-specific modules in the SAP portfolio. SAP is the world's third-largest independent software vendor
Founded in 1977, Oracle were first to market a reliable relational database
The second largest software company after Microsoft
Since the 1990s, they have marketed Financial, HR and other solutions and through a series of acquisitions, including PeopleSoft and Siebel is now the second largest ERP vendor after SAP. 
Microsoft, also started in 1977
dominate the proprietary software market on the desktop and small to medium servers.
Microsoft were a big Siebel software user but since Oracle acquired Siebel, they have worked hard to build an ERP product suite of their own and are now probably in third place in ERP
The ERP Market Assurance with Intelligence Slide
ERP services market possibly worth more than EUR 100 billion
Accenture, IBM, CapGemini, LogicaCMG, CSC, HP and others have substantial ERP practices
ERP services include
Implementation and deployment
Projects tend to be large, long term, business critical and big-budget. EUR 200m implementation projects are not unusual.
ERP Services Market Assurance with Intelligence Slide
If the big three get their way, the integration of desktop (Windows), ERP and associated applications builders are poised to transform the use of IT in business
SAP and Microsoft Duet project integrates SAP back office data with the ubiquitous Office suite from Microsoft
Currently, these applications are small in number, but full integration of desktop office products and ERP systems appear to be the next phase of development in the market.
The (Near) Future of ERP Assurance with Intelligence Slide
ERP implementation projects typically have a mix of the following disciplines:
Package configuration and/or parameterisation
Package customisation (including programming)
Integration with legacy systems
Business change (process design and training)
Testing (typically unit, sub-system integration, system-system integration, acceptance)
Most projects are large scale and critical, the testing budget is substantial – perhaps 50%.
The Nature of ERP Implementation Projects Assurance with Intelligence Slide
ERP Testing Assurance with Intelligence Slide
Remarkably, very little research into the testing of packaged ERP suites
Few books on the subject; those that do exist focus on bundled tools
The software vendors have their own methodologies for testing, but these focus on process flows through configured components
In effect, such tests prove the software ‘does what it does’
The behaviour of ERP suites are typically controlled by configuration parameters and do not involve ‘software’ maintenance at all.
ERP Testing 1 Assurance with Intelligence Slide
However, there are specific areas of software change and risk
Where the ERP system must trigger an interface and import from or export to legacy systems
Amendments to the legacy systems themselves are required.
Much of the testing has to focus on
The behaviour of the system as experienced by its users
The outputs generated
Integration with legacy systems
Data migration is a challenging area of ERP projects
A lot of the functionality in integrated systems is master-data driven
Data migration efforts need evaluation, reconciliation, and testing using the integrated applications.
ERP Testing 2 Assurance with Intelligence Slide
‘ How to approach testing of ERP systems to demonstrate its behaviour, to meet the needs of its users and address the risks of failure?’
An integrated set of test methods and tools is required:
Results-based management (aka Benefits Realisation), modelling of results-chains to provide input to a benefits-based test approach
Experience-based risk assessments that can provide input to a risk-based test approach.
Modelling of integrated user, automated and manual processes and interactions in a format useful to testers.
Derivation of objective, measurable and manageable test objectives from these three models.
To integrate these three test objectives into a non-proprietary life-cycle test methodology that can be managed systematically.
Some foundation/visionary work has already been performed by the author, but the approach needs more research, rigor and proving in a commercial environment.
The ERP Testing ‘Problem’ Assurance with Intelligence Slide
Outline of Work Already Done Assurance with Intelligence Slide
Research in four areas has already been performed
Risk-Based Testing approach
Testing as intelligence gathering
Managing projects with intelligence
Lessons Learned in ERP Projects
Papers/presentations referenced at the end of this slide pack.
Work Already Done Assurance with Intelligence Slide
The first book focusing on Risk-Based Testing, ‘Risk-Based E-Business Testing’ , set out a systematic approach to testing, driven by requirements and product risk where the business environment was focused on the risky real-world business environment rather than esoteric safety-critical landscape.
1. Risk-Based Testing Defined Assurance with Intelligence Slide
Testing is an information-gathering exercise whose value is in the information gathered, not the activity we call testing was presented
The term, ‘Project Intelligence’ was coined to represent the information on business benefits, product risk and test coverage that test strategies should focus on.
2. Testing as Intelligence Gathering Assurance with Intelligence Slide
Project Intelligence (PI) is the key project information gathered by testers that helps managers to control their projects
Analogous to military intelligence
Commander needs to have intelligence before formulating his plan
As events unfold, intelligence reports affect the campaign
Final intelligence reports indicate the campaign is won
Software projects need risk analyses as input to testing and QA
As testing proceeds, it provides the intelligence to measure progress and support decision making
Final test reports indicate the delivered system is acceptable
Test process is a systematic set of activities to achieve test objectives.
3. Managing Projects with Intelligence Assurance with Intelligence Slide
Successful projects depend on
Clear and change-managed objectives
Attention to risk management
Monitoring of quality, schedule and cost.
Successful project managers manage achievements, not activities
The readiness and acceptability of project deliverables are what must be monitored, rather than the completion status of activities.
3 Managing Projects with Intelligence 2 Assurance with Intelligence Slide
Decision making in projects is critically dependent on accurate information
Key role of QA and testing is to gather Project Intelligence and present this to Project and Stakeholder management
Project Intelligence refers to the three information gathering activities
QA and test
benefits analysis and
ongoing product risk management.
3 Managing Projects with Intelligence 3 Assurance with Intelligence Slide
These three fundamental activities promote better communication within projects
They also enable stakeholders to make better decisions when setbacks occur early and when change becomes inevitable and testing is squeezed
The PI approach transforms your perception of testing, promotes closer collaboration between development and test and better inform management decision making.
3 Managing Projects with Intelligence 4 Assurance with Intelligence Slide
Based on research into ERP (in particular SAP) project experiences
To identify common patterns of failure in (typically) large scale, long-term projects
Sources included integration companies, project manager staff and user management
Over seventy actual project failure modes and risk mitigation actions experienced by interviewees were identified.
4 Lessons Learned in ERP Projects Assurance with Intelligence Slide
The purpose was to provide a coherent set of ‘challenges’ to projects
For each lesson/risk the question is
‘ has this risk been addressed?’
intention would be to ensure all the foreseeable risks of concern are addressed
Lessons Learned ‘areas’ include Organisation, Preparation, User Issues, Change Management as well as Testing and QA
With further development, will set out the requirements for an ERP Test Strategy.
4 Lessons Learned in ERP Projects 2 Assurance with Intelligence Slide
Proposed Research Project Assurance with Intelligence Slide
Foundation work has already been performed by the author, but the approach needs more research, rigor and proving in a commercial environment
Academic and commercial partners are sought to refine, develop and prove these methods and tools.
Challenge to Academia and Industry Assurance with Intelligence Slide
One business school with a focus on change management involving ERP systems
One software engineering school with focus on building tools to support business change, decision making and testing
5 large ERP user organisations
Interested in the results-based approach to business change
Experience of problems in the past, and/or are risk-averse
Have high commitment to testing and QA
Want greater control over ERP integration partners.
Research team Assurance with Intelligence Slide
Currently talking to potential industry partners
There is NO SHORTAGE of companies wanting to solve the ERP Testing Problem
Plan to approach business schools
Conference pitch to taicpart.org in September
If it’s not of interest to you, do you have colleagues for who might be interested?