This session shares the experiences of a very successful upgrade of Oracle E-Business Suite 11.5.10 to Oracle E-Business Suite 12.2.4. Learn about the best practices from a planning, project management, and technical perspective. The session shares the do’s and don’ts for an upgrade, and covers best practices regarding the following:
• Decision matrix for upgrade vs. reimplementation
• Test scenario planning
• Project planning
• Cut-over planning and testing
• Simulations
• Optimizing time for upgrade
• Post go-live support
2. GE HEALTHCARE TODAY
• $18 billion: global business unit of GE
• 46,000: number of employees worldwide
• $1 billion+ per year investment in R&D
• 35,800 patents filed by GE since 2000
• Core strengths in bio-sciences, medical imaging
& information technologies
• Headquartered in Chicago, USA
3. 3
9/15/2016
BROAD SOLUTIONS
FOR HEALTHCARE
Diagnostic and Interventional Imaging
Patient Monitoring
Maternal and Infant Care
Anesthesia and Respiratory Care
Diagnostic Cardiology
Healthcare IT
Life Sciences
Healthcare Consulting
Global Services
4. CUSTOMERS
OUR KEY CUSTOMERS INCLUDE:
• Healthcare systems, hospitals, and clinics
– Physicians
– Clinicians
– Nurses
– Executives and administrators
– IT managers
– Biomedical engineers
– Construction managers
• Government officials
• Pharmaceutical firms
• Genetics researchers
• Bio-science researchers
5. 5
9/15/2016
GE Healthcare ERP Context
4 Pillar instances – SAP, PS & EBS ( 2)
Largest EBS instance upgraded to 12.2.4
80% of revenue transacted
53 countries
Approx. 10000 users
DB Size approx. 14 TB
150+ companies
Upgraded from 11.5.10 to 12.2.4
7. Upgrade vs Re-implement
R12 Technical Upgrade R12 Upgrade Plus R12 Reimplementation
Shortest
Tech upgrade to R12.2
• Customization remediation
• Integration and Bolt-on checks
• Regression & Performance Testing
• New Features: SLA, inv org, multi-
org, role- based access controls
• Retire Customizations:
• Convert GL to Corporate COA
New R12.2 Instance
• Rationalize customizations
• Leverage new R12 features
• Potential for Phased Roll-out
Lowest
Lowest
Supportability
Real-time Patching
Timeline
Cost
Funct.
Resource
Needs
Business
Benefits
New
Features
Retire
Customizations
Corporate
COAs
+ +
Medium - Long Longest
Medium - High Highest
Medium - High Highest
Supportability+ Re-engineering
Technical Upgrade allowed for other business priorities.
Incremental Change in Business process instead of Big Bang
8. 8
9/15/2016
Initial Project Timeline
Code Remediation
2014 2015
Q3 Q4 Q1 Q2
CoE
Tech
Steady increase in program resourcing needs over time
Global
Teams
Core
Team
Planned Timeline
Resources
Leveraging a small core team to maximize value
R12 Upgrade Testing
9. 9
9/15/2016
Global Functions
SVCGSC FIN Commercial ENG
IT Technical /
Quality Team
Global
Management
Team
Senior Leadership
Team
Program
Management
Region / P&L
Region/ Modality CIOs
Functional
Program Manager
VP-
Service
VP-IT
VP- Supply
Chain
CFO
CTO Regional GMs
Supply chain
GSC
Reginal GM
Fin
FIN
Sales GM
C.E.
GM Engg
ENG
CIO
Fin Leader
IT Fin Leader
GSC Leader
IT GSC Leader
Service Leader
IT SVC Leader
ENG Leader
IT Eng Leader
Bus Integration
Business
Integration Leader
Region / P&L
Integration Leader
IT
Program Leader
Quality &
Compliance Leader
Commercial Ex
Leader
Comm. IT Leader
Program Organization
SVC
Service GMS
Technical
Program Manager
10. 10
9/15/2016
Team Functional IT
Senior
Leadership
Team (SLT)
• Top Down Support – CAP /
Communication
• Strategic Direction
• Help with staffing engagement
• Top Down Support – CAP /
Communication
• Strategic Direction
• Help with staffing engagement
Guidance
Management
Team (GMT)
• Drive func leadership selection
• Monitor progress, provide help
needed
• Authorize major project variances
• Manage conflicting priorities
• Assist with functional staffing
• Make the final Go / No-Go decision
• CIO engagement of functional
leadership
• Drive IT leadership selection
• Approve budget changes
• Resolve cross program issues
• Monitor progress, provide help needed
• Make the final Go / No-Go decision
Overall
Program
Executive
• Manages SLT engagement
• Organize and Lead GMT Sessions
• Develops overall program strategy & organization
• Manages top-level risk mitigation
Program
Managers
• Organizes the overall program and mobilizes teams
• Drives overall program operating mechanisms, issues, and risk mitigation
• Manages program Plan, risk mitigation, tollgates
• Manages program budget
• Manages communication plan
Business
Integration
Leader
• Creates Overall Program Test strategy
• Establishes and drives test reporting
• Drives cross functional integration
• Identifies simplification opportunities for future releases
Region
Integration
Leader
• Drives regional engagement through best practices and common work structure
• Pivots between program, global processes and region / P&L leaders
• Plans the work for the regional leads and monitors the overall region plan
• Identifies top regional issues and drives to closure
• Leads commercial staffing, supports functional staffing
Top
Management
Support
Staff Leaders
Make Go/No-
go
Leadership
Engagement
Mobilize &
Orchestrate
Cross-
functional
Consistency &
Integration
Lead
Region
Engagement
Roles & Responsibilities
11. 11
9/15/2016
Team Functional IT
Global Process
Leader
• Recommend Go / No-Go
• Assure functional coverage &
obtains resources
• Validate functional test plan
• Escalates critical risks
• Organizational readiness
• Identifies process eco-system
• Develop staffing coverage strategy and
resource plan
• Staffs GE & contractor positions
• Drives issue resolution
• Liaison's to function
Global Track
Leader
• Defines critical processes
• Defines test scenarios
• Manages test execution
• Escalates critical issues
• Identifies testing coverage for
customizations & interfaces
• Gathers data for testing
• Drives fixing defects
• Drives system documentation
maintenance
Regional
Leader
• Engages regional leadership and commercial teams for testing & readiness
• Identifies local system integrations, country localizations, and CFD’s
• Coordinates regional testing as needed – particularly where local tools are involved
• Identifies regional risks and issues and drives to closure
• Define the region go-live readiness (acceptance) criteria
IT Technical
Program
Leader
• N/A • Plans & manages outage window
execution
• Performs stress test & tunes system
• Governs customization designs
• Drives Overall eco-system connectivity
• Manages validation & compliance
• Leads central break/fix for Oracle defects
• Identify & modify BI fixes
Organizational
Readiness
Process &
Module
Readiness
Run the business…
Find the Bugs!
Run the Systems…
Fix the Bugs!
Assure
Regional
Readiness
Upgrade
Execution &
System
Performance
Roles & Responsibilities
12. 12
9/15/2016
2014 2015 2016
Actual Program Timeline
NovOct Nov Dec Jan Feb Mar Apr May DecJun Jul Sep OctAug
Go Live
Jan
CRP1 CRP2 UAT Validation Test
Development End to End Process Testing User Acceptance
Test
Compliance documentation
Training & Communication
Defect
Resolution
Defect
Resolution
Key Activities:
CRP1 – Code remediation and functionality exploration
CRP2 – End to end testing and Bug fixes
UAT – User acceptance testing
Upgrade simulations to optimize the time
Back out planning and Cutover planning
Simulate Back out plans.
Communication & Training
Go-Live
6 Upgrade Simulations
13. 13
9/15/2016
Upgrade completed on time – system came up 6pm Sunday as planned
Critical Business transactions exceeded daily averages (starting day 1 )
No post go-live outages
Successful January Close
Program Go-Live Highlights
Post Go-Live Executive Summary
“Just an awesome, awesome job!!!!” ~ CFO GEHC
“Best upgrade I have seen in 20 Years” ~CFO SC P&L
“As close to perfect as you could expect in an ERP upgrade” ~ VP &
CIO GEHC
15. 15
9/15/2016
Key Learnings
ERP is not standalone
350 Interfaces ;Inbound and outbound
120 connecting systems
3 BI tools
Transactions flow from Source to ERP to destination systems
Code remediation and testing done across systems
…… But…
How do we shutdown this web of systems?
Key is to understand dependencies and cross-impacts
How do we startup this web of systems?
A pilot can fly alone easily but when 15 pilots fly together, it’s a
different experience…. That’s how it feels like in the ecosystem !!
Think about your Ecosystem; not just the ERP
16. 16
9/15/2016
Key Learnings
Use of tools to remediate code
Automation of code remediation saved us months in the
project schedule
Code remediation activity could be run like a factory
Total number of code Objects remediated ~ 2300
…… But…
We found that like all software, our tools had issues..
We missed code in ecosystem, missed SSWA objects,
missed embedded queries in apps like QA, Pricing etc.
Online patching compliance
Trust your tools but verify the results !
17. 17
9/15/2016
Key Learnings
Test plan and Test coverage
We executed 13000 test scenarios in CRP’s
Fantastic functional team engagement
Defect monitoring and root cause analysis helped
Exited UAT with 12 defects across all tracks.
Load testing and Regression Testing
…… But…
Initially we did not test transactions which start in 11i but
end in R12. Practice period close. Get used to period close
analyzers.
Our root cause analysis did not look at underlying trends
and systemic defects…. We improved on that.
Start your Load testing early.
If using i-modules, please test them from outside the
firewall.
Test boundary conditions & exceptions !
18. 18
9/15/2016
Key Learnings
Appreciate your team
Any Oracle upgrade is very demanding on people
Work hours will be long and often at odd hours
Workload for different teams like developers, middleware team,
testers, users will be fluctuating.
Workload for DBA’s tends to be high all the times.
Keep morale up. Take pride in milestone accomplishments and
learn from every incident.
This is team work. Trust is key. Help in building it
…… But…
Think about back ups
Think about shifting work hours
Think if you run a risk of knowledge concentration in a few
Think about your communication plan across the globe if you are
global
Remember DBA’s are the centerpiece of this project
19. 19
9/15/2016
Key Learnings
Functional Users
Need a Strong buy-in.
Knowledgeable super users are the key to success
Users need to test the system as they use it everyday
Reports, labels are the key
Spend ample time in communications, awareness and training.
Posters and banners at major hubs and Mfg/ Dist sites.
Think about having a dedicated communications leader
Involve them in sanity testing during simulations
Use your super users as your messengers and damage controllers
in case of small hick ups.
Think about your communication plan across the globe if you are
global
Project success = Happy Users
20. 20
9/15/2016
Key Learnings
Upgrade Considerations from a DBA perspective
Archive or Purge unwanted data
Refresh all MVs to reduce upgrade time
Concentrate on long running jobs
Parallelize tasks if possible
Clean up database objects before enabling ADOP (duplicate tables
etc.)
Implement SPMs and SQL profiles for long runners
…… But…
Do end to end simulations
Plan for backups before critical steps
Check logs always for any unexpected errors
Follow the upgrade manual from Oracle. No Short
cuts please !!
21. 21
9/15/2016
Key Learnings
Cutover planning & execution
We had a 1900 line project plan just for the cutover tasks
Understand the dependencies of tasks.
Understand dependencies of people. They need to trust each
other. Handshakes on tasks are important
Co-location helps a lot
Simulate like it is Production. Avoid the temptation of shortcuts.
…… But…
Note every single incident for investigation later
Think user lock down is simple…it might not be.
Plan your simulations for every single failure. Don’t take things for
granted.
Practice, practice and practice….
22. 22
9/15/2016
R12 Upgrade Outage plan
Provide high quality instance as efficiently as possible
What actions were taken?
• Practiced and improvised the shutdown
process.
• Created save points which were key to
BCP
• Worked with Oracle to get best practices
• Oracle fix was practiced and verified
• Conducted multiple simulations to iron
out the process
• Added time for GS and introduced a
functional checkout process
Save Points
What Did We Learn?
Go/No Go
Monday Tuesday Wednesday Thursday Friday Saturday Sunday
24-Jan
Monday
25-Jan
R12.2
Code,
Patches
Check
Out
System
shutdown
R12.2.4/Online
Patching
3rd
Party,
M/W
• Graceful shutdown adds 12 hours
(flushing data through system)
• Issues need to be recreated. Painful.
• VCP first collection was slow
• Issues with R12.2.driver in the first 6
months vs the last 6 months
• Code migration, VCP collection, Sabrix
install and Brazil localizations took 42
hrs.
• System startup was not smooth
VCP
What were the outcomes?
• Finally reduced it for 7 hrs.
• Gained ability to run multiple
simulations at same time.
• VCP collections was not a bottleneck
• Reduced the time for the last phase
to 27 hrs. from 42 hrs.
• Resulted in a very high user
confidence
Practice to optimize the plan and understand
dependencies
Outage at ~5 Days…additional time needed to ensure Quality
23. 23
9/15/2016
Key Learnings
Where is your Plan B?
We had a 300 line back out plan
Plan for back out from every step of the upgrade
Playbook for executive engagement in the worst case
Have Plan B incase of delays. Understand steps where you could
lose time and where you could gain time
Use the time gained wisely. Try to stay ahead and be ready for
contingencies.
Have playbooks for concurrent activities going out of sequence.
Practice your back out plans and Plan B’s to make sure they can
work.
Your Back out plans should cover the ecosystem not just the ERP.
Have a Plan B for every possible thing you can think
about.
24. 24
9/15/2016
Key Learnings
System Performance
Comprehensive load testing
Verified performance between 11i & R12 up to the second
API and Interface performance is important too.
Watch out for first collection run for ASCP.
…… But…
Watch out for performance post-upgrade
Allow time for post upgrade steps to improve performance
Simulate post upgrade performance to take corrective actions
Productions system will be faster than your test environments
Gather Stats and Table re-orgs will help. Ask your DBA’s to plan for
those pre and post Upgrade
Plan for post-upgrade performance. Don’t
make it an after thought !
25. 25
9/15/2016
Key Learnings
Post Go-live support
Engage Production support teams during UAT
Pair the project SME’s with the support SME’s
Be ready for a high call volume
Have regional contact points by defect types for level 1 support
If you are changing java versions, be ready for login Issues.
Over communicate about go-live . Communicate not only during
the project but even after go-live.
Be ready with swat teams especially for finance.
All the best for your upgrade & hope that our learnings
will make you smarter.
27. 27
9/15/2016
Operating
Mechanism
Purpose / Objective Participants Frequency
Senior Leadership &
GMT Review
Inform corporate officers of project status,
issues, and help needed
Senior Leadership Team, aligned
GMT members, selected team
members, Global functional & IT
leads, Gilles, Jason
Monthly
Program Leadership
Forum
Strategic review, develop recovery plans,
make course corrections
Program management, Global
functions, Gilles, Region Leaders,
Jason
Monthly
Program Review Review scorecard status at end of week;
weekly status report preparation
Program management, Global
functions, Gilles, Jason
Weekly
Global functional
Review
Manage track progress, issue escalation &
mitigation, update on program information
Global functions, track leads,
region integration leader,
break/fix manager
Weekly
Region Leadership
Review
Manage region progress, issue escalation &
mitigation, update on program information
Region leader, Gilles, Brian,
Optional: Functional leads
Weekly
Training & Improvement
Forum
Add / update training & improvement lists Functional Leads, Gilles, Brian,
Architecture leader
Bi-Weekly
IT Technical Integration
Session
Manage IT Platform progress, issue
escalation & mitigation, dependencies
IT Technical Team Weekly
Defect Management
Review
Defect assignment, status, critical issues by
module
Defect manager, IT track leads, BA
COE architects, BI COE, GAMS
Weekly
Project Plan Review Progress vs plan update on activities Program management, Gilles,
Jason
Weekly
Operating Mechanisms