3. Phase 1- Planning/Mobilization
Careful planning, particularly in the early stages of a project, is
necessary to coordinate activities and manage project risks effectively.
The depth and formality of project plans should be commensurate with
the characteristics and risks of a given project.
Outline Project Plan
Define Roles and Responsibilities
Define Project Communication and Reporting Requirements
Define Deliverables and Expectations – Involvement of all Key Players
Outline Risk Acceptance - Manage Internal and External Risks
Define Project oversight activities – Definition of Standards
Define Tollgates and Requirements
Define Budget and estimated Project Costs
Define Project Change Procedures
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
4. Phase 1 Planning/ Mobilization – Lessons Learned
Putting a proper project governance structure in place with
sufficient "checks and balances".
Proper Executive and Senior Management buy-in and
involvement in project and milestones reached
Projects are often comprised of international teams and must
consider both cultural issues and compliance with local laws
and regulations
Broader industry and business issues must be taken into
consideration
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
5. Phase 1 Planning/Mobilization – Lessons Learned cont.
Underlying Data Model Consideration (e.g. US GAAP versus
IFRS)
Downstream impact on support functions such as internal
audit and security administration
Additional Considerations to be aware of during the planning
stage:
41% of projects fail to meet management’s objectives
Only 28% of project fulfill management's expectations
Only 16% of IT projects hit all their targets
50% of projects end up late or over budget
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
6. Planning/Mobilization – Lessons Learned cont.
Reasons for project failure in the planning stage:
Bad estimates
Scope changes
Change in environment
Insufficient resources
Change in strategy
Imprecise goals/ Insufficient budget
Poor communication
Insufficient support
Wrong project management
Insufficient motivation
Stakeholders not adequately defined
Poor quality of deliverables
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
7. Project Phase 2 - Design/Blueprint
The design phase involves converting the informational, functional, and network
requirements identified during the initiation and planning phases into unified design
specifications that developers use to script programs during the development
phase
Application Control Standards
Designing appropriate security, audit, and automated controls
Standards should be in place to ensure end users, network
administrators, auditors, and security personnel are appropriately
involved during initial project phases.
Application control standards enhance the security, integrity, and
reliability of automated systems by ensuring input, processed, and
output information is authorized, accurate, complete, and secure.
Automated input controls help ensure employees accurately input
information, systems properly record input, and systems either reject,
or accept and record, input errors for later review and correction (e.g.
Check Digits, Completeness Checks, Duplication Checks, Validity
Checks, Reasonableness Checks, etc.)
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
8. Project Phase 2 - Design/Blueprint cont.
Processing Controls - Automated processing controls help ensure systems
accurately process and record information and either reject, or process and
record, errors for later review and correction.
•
•
Error Reporting
•
Transaction Logs
•
Run-to Run Totals
•
Batch Controls
Sequence Checks
Output Controls - Automated output controls help ensure systems securely
maintain and properly distribute processed information
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
9. Phase 2 Design/Blueprint – Lessons Learned
Avoid excessive customization - companies desire to
"re-invent the wheel"
Many key controls are application driven (e.g. controls which
depend on system generated reports, configuration settings
such as for the three-way match in the procurement cycle)
Effective process to prioritize all the business "wish-lists”
Decision Making from “Middle Management” – Timely
Decisions
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
10. Project Phase 3 - Realization/Build & Test
Development
Development standards should be in place to address the responsibilities of
application and system programmers. Application programmers are responsible for
developing and maintaining end-user application.
Library Controls - Libraries are collections of stored documentation, programs,
and data. Program libraries include reusable program routines or modules stored
in source or object code formats.
Automated Password Controls – Management should establish logical
access controls for all libraries or objects within libraries
Automated Library Applications – When feasible, management should
implement automated library programs, which are available from
equipment manufacturers and software vendors
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
11. Project Phase 3 - Realization/Build & Test – cont.
Version Controls
Software Documentation
System Descriptions – System descriptions provide narrative
explanations of operating environments and the interrelated input,
processing, and output functions of integrated application systems
System Documentation – System documentation includes system
flowcharts and models that identify the source and type of input
information, processing and control actions (automated and manual), and
the nature and location of output information.
System File Layouts – System file layouts describe collections of related
records generated by individual processing applications
Naming Convention - critical part of program documentation
End-User Instructions – Organizations should establish end-user instructions
that describe how to use an application.
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
12. Project Phase 3 - Realization/Build & Test
Build & Test
The testing phase requires organizations to complete various tests to ensure the
accuracy of programmed code, the inclusion of expected functionality, and the
interoperability of applications and other network components. Thorough testing is
critical to ensuring systems meet organizational and end-user requirements.
Acceptance Testing – to assess the overall functionality and interoperability of
an application
End-to-End Testing - to assess the interoperability of an application and other
system components such as databases, hardware, software, or communication
devices
Functional Testing - to assess the operability of a program against predefined
requirements
Integration Testing - to assess the interfaces of integrated software
components
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
13. Project Phase 3 - Realization/Build & Test – cont.
Parallel Testing - to compare the output of a new application against a
similar, often the original, application
Regression Testing - to assess functionality after programmers make
code changes to previously tested applications
Stress Testing - to assess the maximum limits of an application
String Testing - to assess the functionality of related code modules
System Testing - to assess the functionality of an entire system
Unit Testing - to assess the functionality of small modules of code
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
14. Phase 3 Realization/Build & Test – Lessons Learned
Project streams reporting 99% completion of tasks which, if
subject to deeper analysis, does not hold water
Incomplete testing which can have a devastating post go-live
impact when "too lightly" tested configurations fail and disrupt
the business
Data conversion is a task which many times are underestimated
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
15. Project Phase 4 - Pre Go-live/Deliver Phase
The implementation phase involves installing approved applications into
production environments.
Primary tasks include…
announcing the implementation schedule,
training end users, and
installing the product.
Additionally, organizations should…
input and verify data,
configure and test system and security parameters
Management should circulate implementation schedules to all affected
parties and should notify users of any implementation responsibilities.
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
16. Phase 4 Pre Go-live/Deliver Phase – Lessons Learned
Training is a key area where projects tend to cut corners:
Insufficient training can be disastrous for the morale of
users, acceptance of the new application and company
productivity which can seriously hamper the pre-go-live
promises of more efficient post go-live environment.
Strong personalities, ego's, compensation structures and a
mentality of "nothing will stop us from going live on x-date" can
mean that pre-determined exit factors for the deliver phase
such as successfully completed testing and completed cut-over
activities can be compromised
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
17. Project Phase 5 - Post Go-live/ Maintenance Phase
Management should…
conduct post-implementation reviews at the end of a project to validate the
completion of project objectives and assess project management activities.
interview all personnel actively involved in the operational use of a product and
document and address any identified problems.
analyze the effectiveness of project management activities by comparing,
among other things, planned and actual costs, benefits, and development times.
document the results and present them to senior management.
The maintenance phase involves…
making changes to hardware, software, and documentation to support its
operational effectiveness.
making changes to improve a system’s performance, correct problems, enhance
security, or address user requirements.
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
18. Phase 5 Post Go-live/Maintenance Phase – Lessons
Learned
PwC was able to categorize post go-live issues in the
following 35 buckets, sorted by number of incidents, highest
number first:
Locked user/UID validity date required resetting
Abend related issues
Report generation
Authentication
Batch processing/upload issues
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
19. Phase 5 Post Go-live/Maintenance Phase – Lessons
Learned cont.
Interface processing issues
Transaction Processing issues - mostly FI, FI-AP, SD
PO/EBP GR IR Processing issues
Access - General
SAP Mail/Inbox/Workflow Issues
Process Chain Issues
Authorization Issue
Shopping Cart PTP
Master Data issue
Systems Implementation Assurance – Lessons Learned
PricewaterhouseCoopers
24. Understanding Your Objectives
Draft
The company is making a significant investment to implement a single pricing, billing,
invoicing, accounts receivable and cash management and collection system, utilizing
SAP as the core technology. With Business Blueprint of Phase II of Project SAP
complete, Executive Management would like to gain the appropriate assurance that
the project achieves it’s stated objectives:
Realize the tangible and intangible business benefits outlined in the business case with the
priority to increase customer satisfaction with billing and an enhanced ability to efficiently and
effectively launch new products and services in the future.
Deliver the project on time, within budget, with agreed critical functionality for the business as
quickly as possible.
Leverage standard SAP business process design and core infrastructure to reduce risk and cost.
Provide a standard platform to allow for ease of integration and reporting.
Deliver a compliant system that addresses key stakeholder requirements, including financial and
regulatory reporting requirements.
SDLC Selection Framework • IT Process Maturity
25. Issues on Your Mind
Draft
Issue
Data Quality
Possible Area of Assurance
Customer master conversion/migration
•
Customer rate accuracy
•
•
Share independent perspective on data conversion activities and
provide recommendations throughout the process.
•
Assess key interfaces identified and controls supporting
completeness, accuracy, validity, and restricted access risks.
•
Review controls and system configurations associated with
invoice generation and shipment rating and provide
recommendations related to validity, completeness, accuracy,
efficiency, and evidence of duplication.
Share independent perspective on good practices associated with
revenue cycle and billing/invoicing.
•
Share other client experiences regarding security, internal control
and risk management associated with SAP upgrade to ECC 6.0.
•
Provide independent perspective on technical strategy for cash
application.
•
Assess process to define key financial and management reporting
requirements and assess the effectiveness of the reporting
designed to meet these requirements.
Billing data quality and accuracy
•
Review controls around data cleansing and conversion for billing
and customer master data.
•
•
•
Interfacing of information to legacy systems
Customer First Focus
•
Invoice Presentation Quality and Accuracy
•
Shipment Rating Timeframe
Financial Reporting
•
Inaccurate Bad Debt Provision Calculation
•
Excessive Unapplied Cash Balance
•
Current system Upgrade
SDLC Selection Framework • IT Process Maturity
26. Draft
Project Assurance – A Suggested Approach
•
Ongoing review of the project, control and business
outcomes focusing on the stated Project SAP business
objectives, risks, and priorities.
•
Project
Management
Provide Executive Management with ongoing project
assurance reporting.
•
We would work along side the project identifying
potential issues as early as possible and hence allowing
Executive Management adequate time to consider, and if
necessary address such issues. This is critical if the
independent project assurance role is to add value to the
project and help assist in its successful outcome. To this
end we believe the independent assurance function
should:
–
Attend and provide input to key project team meetings
–
Provide a rolling progress report on issues identified
through our work
–
Business
Case
Project
Governance
Functional
Readiness
Technical
Readiness
Project
Outcomes
Organizational
Readiness
Benefits
Realization
Plan
Business
Outcomes
Implementation
Methodology
Brief key program stakeholders on the status of our work
and issues arising on a regular basis
Data
Quality
Controls
Outcomes
Interfaces
Project
Structure
ITGCs
Business
Processes
SDLC Selection Framework • IT Process Maturity
27. Our Value Proposition to the company
Draft
• Flexible, tailored approach to focus on management’s priorities for assurance regarding the achievement of Project
SAP objectives.
– Efforts embedded in and integrated with overall Project SAP approach with a focus on value-add
– “One touch” integration of effort with external audit requirements to minimize disruption to project and avoid
surprises
– Evaluate and leverage work performed by others (e.g., Parent Company Internal Audit, SAP, etc.)
• “Hub and Spoke” deployment of world class functional and technical capabilities from PwC to the project:
– SAP Risk Management, Security, and Control
– Transportation & Logistics
– Business Process
– Data Assurance
– Program/Project Management
– Internal Control and Financial Reporting
• Distinguished history of providing independent project assurance services to the company and the parent company.
– Experience navigating the Demand and Supply IT Model
– Invested in relationships throughout the service center and the company.
– Teams deployed alongside of the company in Houston, Scottsdale, and Plantation.
SDLC Selection Framework • IT Process Maturity
28. Integrating our Audit into Project SAP
Draft
Control Design/Gap
Analysis
Blueprint
Agreement of expected key
controls within the draft
documentation during the
Blueprint and Realization
phases of the project allows
maximum opportunity to
correct any issues within the
design.
Realization
TIMETABLE
Business Process/ IT General Controls
Management Reporting
Testing Framework
Data Conversion/ Cleansing
Management Reporting
Many key business process
controls rely upon system
generated data. The
requirement to manipulate this
data as part of its use adds
additional risk. Effective design
and implementation of system
reports maximises process
efficiency and reduces the audit
risk.
SDLC Selection Framework • IT Process Maturity
Go-live & support
Testing Framework
Our experience of large
implementations has
found that the proving of
the system is complex and
difficult to manage
effectively. A key factor
are the controls around
the remediation of issues
reported during the testing
phase.
Security and Access Control
Data Conversion
and Cleansing
Security and Access Control
As greater use of system based controls
are built into the control environment, the
reliance upon the proper allocation of
access increases. Getting this right from
day one both for business and support
users reduces the risk that gaps are
found post live that affect our strategy.
Data integrity is a key risk
within any environment;
this risk is increased
during periods of
changes such as a
system replacement.
29. Example Workplan
Draft
Business Process
/IT General Controls
Review proposed
business process control
documentation
containing the following
types of controls:
configurable, reports,
manual procedures,
automated, and
interfaces.
Evaluate key controls
over financial reporting
(selected by the
company) for
completeness, accuracy,
validity, restricted
access, efficiency,
resilience, and evidence
of duplication.
Management
Reporting
Assess process to
define key financial and
management reporting
requirements and
assess the effectiveness
of the reporting designed
to meet these
requirements.
Baseline key custom
reports used to support
the operation of manual
controls for financial
reporting (completeness,
accuracy).
Review of SAP screens
to confirm settings of
configurable controls.
Walkthrough of business
process controls to
confirm
existence/operation of
the automated and
manual controls.
Assess SAP ITGCs
SDLC Selection Framework • IT Process Maturity
Testing Framework
Ensure requirements for unit
testing, integration testing,
system testing, UAT,
interface and performance
testing are adequately
considered with a focus on
testing of key controls.
Assess whether an adequate
testing monitoring system is
in place.
Assess coordination of
testing between business
and IT.
Review configuration
management and change
control strategy and plan.
Review sample of testing
scenarios and results
focusing on consistency in
approach and compliance
with policy in relation to key
controls.
Data
Conversion/Cleansing
Security/Access Controls
Review scope,
approach, and
requirements for data
cleansing and
conversion.
Review proposed SAP
access related controls
for sensitive access (SA)
and Segregation of
Duties (SOD) rule set;
role maintenance; and
user provisioning.
Assess quality controls
within the conversion,
setup and cleansing
processes to ensure
data integrity.
Assess SAP user roles
against SA and SOD rule
sets.
Review controls over
the data cleansing and
conversion process.
Review sample of data
cleansing and
conversion results.
Review strategy for
master data
maintenance.
Walkthrough user
provisioning and role
maintenance process.
Assess existence of
processes to manage
access during
implementation and
during early stages of live
operation.
30. Draft
Questions
Contact Information
– Peter Harries, Partner 213 – 356 – 6760
– Charles Lewis, Partner 602 – 364 – 8290
– Pablo Hernandez, Senior Manager 602 – 364 – 8064
– JJ Marais, Senior Manager 602 – 364 – 8232
SDLC Selection Framework • IT Process Maturity
Editor's Notes
Putting a proper project governance structure in place with sufficient "checks and balances". Herewith a recent example of a failed governance structure. The project manager (PM) for the project was a VP level person at the company (say ABC Inc) implementing SAP but a also a former partner at the consultant (say XYZ) implementing SAP at ABC Inc. ABC Inc did not have a sufficient counter balance on the overall steering committee to look out for their interests due to the influence of both the PM and XYZ on the overall project. Costs soon spiraled out of control with a 100% overrun in project costs.
Many projects span business units or organizations; we have seen that lack of executive buy-in/support can seriously hamper project progress or an eventual successful outcome Projects are often comprised of international teams and must consider both cultural issues and compliance with local laws and regulations. For example: where work is performed in low cost countries our experience has been that instructions for work to be performed by the local low cost team must be very detailed and step by step; do not assume that foreign resources can "fill in the blanks" or make accurate interpretations of less that detailed instructions.
Many SAP implementations span numerous countries and the initial country "template"; e.g. the first country to deploy SAP may have been the US, and this initial implementation acts as template for other countries. This template must be adjusted for some quirky requirements in other countries; e.g. (a) in Finland, assets can not be depreciated in the books below 10% of their cost. (b) Requirement that countries' transactional data are kept in that country, (c) Belgium requires the use of a mandatory and unique chart of accounts.
During planning, companies can ignore broader business issues such as a manufacturing company which highly dependent on the winter holiday season planning their go-live in November.
Another example is that many companies do not consider IFRS when implementing SAP. Although SAP can support multiple GAAP regimes, your underlying data model must still be able to support IFRS and GAAP. We know of a company who went live on SAP in 2008 and now they are spending millions more to perform their IRFS conversion, a task which would have been less expensive if it was included in the original implementation. As a side note, 66% of IFRS inquiries we as a firm respond currently are from companies who are contemplating a SAP implementation.
Failure to consider the downstream impact of SAP on support functions such as internal audit and security administration. Typically, the implementation of SAP requires a major skill-set retooling in the aforementioned departments.
Other sobering thoughts to consider during the planning stage:
41% of projects fail
Only 28% of project fulfill management's expectations
50% of projects end up late or over budget
Only 16% of IT projects hit all their targets
8. Reasons for failures above:
Bad estimates
Scope changes
Change in environment
Insufficient resources
Change in strategy
Imprecise goals
Insufficient budget
Poor communication
Insufficient support
Wrong project management
Insufficient motivation
Stakeholders not adequately defined
Poor quality of deliverables
Many designs we have seen contemplate excessive customization; companies desire to "re-invent the wheel" where SAP's industry solution would work for the vast majority of scenario's. Excessive customization will put project timelines and budgets at risk.
Internal controls are not part of the design and/or the compliance group is not part of the implementation effort. Many key controls in the SAP environment are application driven; e.g. controls which depend on system generated reports, configuration settings such as for the three-way match in the procurement cycle. If controls are not baked into the design the project runs the risk of implementing controls after go-live which will require additions to originally contemplated design, realization and pre-go-live phases. Access controls in SAP is best managed via automated processes enabled by products such as SAP GRC, Approva BizRights and SecurityWeaver and there is case to be made for including these tools as part of the broader implementation.
Companies do not have an effective process to prioritize all the business "wish-lists" which should make it to the design. A lack of such a process may mean that critical requirements are not baked in the design while non-critical items may actually be included in the design. One of our client's recently went live on SAP and a component of their revenue recognition process was just as deficient as it was on the legacy system but easily fixable in SAP.
Related to three above, many companies' operational mode is a kind of paralysis where middle management is not empowered to make decisions. A company with this culture will struggle to implement SAP at all, since a system implementation project is nothing but taking a series of timely decisions and acting upon decisions made
This is true of all project phases but could be very prevalent during realization: Project streams reporting 99% completion of tasks which, if subject to deeper analysis, does not hold water; e.g. if an 80 hour task is "99% complete" is it really true that this task only requires 48 more minutes (0.8 hr) to be completed? More often than not, "99% completion" means less than 85%.
Incomplete testing which can have a devastating post go-live impact when "too lightly" tested configurations fail and disrupt the business. We frequently encounter projects where test script completion for unit, integration, UAT is 50% deficient. Examples of deficiencies: test scripts have blank results fields, "N/A" entries in results fields, tests with a result of "fail" with no corresponding follow-up. How can PM's be confident of project success in circumstances such as these? One of our clients who went live with SAP in 2008 took two months to close their books for the first month, a clear indicator of weak, insufficient testing.
Data conversion is a task which many times are under-estimated; in fact the maxim in the project management community is that task is frequently 10x bigger than what was anticipated. Consider for example that an SAP implementation not only requires the conversation of legacy data but also the creation of brand new data; SAP's material master has 300+ fields and most legacy systems do not have such a deep material master which necessitates the creation of new data.
Training: together with testing, training is a key area where projects tend to cut corners. Insufficient training can be disastrous for the morale of users, acceptance of the new application and company productivity which can seriously hamper the pre-go-live promises of more efficient post go-live environment. At one of our clients which skimped on training, client personnel would wonder when the "old system" would "come back" to replace SAP.
Strong personalities, ego's, compensation structures and a mentality of "nothing will stop us from going live on x-date" can mean that pre-determined exit factors for the deliver phase such as successfully completed testing and completed cut-over activities can be compromised. The example of the company who could not close their books for two months after go-live is useful here too: The project was run by a very forceful SVP and part of his bonus was tied to going live on a specific date.
Companies may need to operate for a time with a more relaxed controls regime - e.g. Superusers with broader access to help out of "normal" users are "stuck" with a transaction. Consideration should be given for stricter monitoring controls; e.g. monitoring key SAP tables such as BKPF where accounting documents are logged, table CDHDR where change transactions are logged and transactions/reports such as SM20, ST03N etc.