PowerCenter Upgrades with
Zero Downtime
Presented by Greg Wade & Ed
Wagner, Lockheed Martin
Topics
• About Us
• Our Environment
• Requirements
• Approach
• Development/ Test Transition
• Emergency Release Strategy
...
About Us
3
About Us
• Who we are
• Greg Wade, Information Systems Architect
• Ed Wagner, ETL Lead Developer
• What we do
• Build larg...
Our Environment
5
Our Environment
6
© 2013 Lockheed Martin Corporation. All Rights Reserved.
Input
Data Acquisition
EDW
Teradata
Staging
Ora...
Our Environment
7
© 2013 Lockheed Martin Corporation. All Rights Reserved.
Informatica Powercenter
1 Domain Per 6 Environm...
Requirements
8
Requirements
• Develop a reusable upgrade strategy
• Version 8.6.1 HF 11 to 9.1 HF 3 … and future versions
• Reusable for ...
Requirements
• Complete or partial roll-back must be supported
• Must be able to support any Production Emergency
Release ...
Approach
Plan, Development, Testing, Production
11
In Place Upgrades
• “Go Big or Go Home” Approach
• Pros
• Complex orchestration not required
• Can be accomplished quickly...
High Level Approach
13
© 2013 Lockheed Martin Corporation. All Rights Reserved.
Dual installations not possible under
Wind...
Approach – The Upgrade Begins
• Install 9.1 in development environment
• Clean install … DO NOT UPGRADE 8.6
• Create a sec...
Approach – The Upgrade Begins
• Created second Oracle schema
• v8.6.1 – IN86_RPS v9.1.0 – IN91_RPS
• Ensure sufficient fil...
Open new ports in host based firewalls and
any infrastructure firewalls.
Approach – The Upgrade Begins
• Established secon...
Approach – The Upgrade Begins
• Resolve any import problems
• None were encountered from 8.6 -> 9.1
• Validate a sampling ...
Approach – Regression Testing
• Clean install of Version 9.1
• Content imported from SCM
• Baseline created in development...
Approach – Regression Testing
• Environment
• Do both server versions successfully start up?
• Validates dual profiles
• H...
Approach – Regression Testing
20
© 2013 Lockheed Martin Corporation. All Rights Reserved.
Name Workflows Mapping
Command
T...
Approach – Performance Testing
• Identical test datasets were repeatedly run against both
PowerCenter versions using ident...
Approach – Repository SQL Review
22
© 2013 Lockheed Martin Corporation. All Rights Reserved.
• Executed SQL against reposi...
Approach – Testing
• Test results reviewed with engineering team
• GO / NO GO Decision point
23
© 2013 Lockheed Martin Cor...
Approach – Ready for Production?
24
© 2013 Lockheed Martin Corporation. All Rights Reserved.
• 9.1 baseline does not
match...
Approach – Ready for Production?
25
© 2013 Lockheed Martin Corporation. All Rights Reserved.
Resync 9.1
Baseline
Migrate
D...
Approach – Install Production Content
26
© 2013 Lockheed Martin Corporation. All Rights Reserved.
During content migration...
Approach – Interface Cutover
27
© 2013 Lockheed Martin Corporation. All Rights Reserved.
DO NOT run a workflow
simultaneou...
Approach – Interface Cutover
• Risk reduction
• Selecting the best time
• Having support on hand
• Internal
• Oracle Datab...
Approach – Interface Cutover
• Checklist – Repeated for each Workflow
 Unschedule Workflow in 8.6
 Wait for Workflow to ...
Development / Test Transition
30
Final Dev and Test Migration
• v8.6.1 was no longer needed to support Emergency
Releases – Production at v9.1
• v8.6.1 Dev...
Emergency Release Strategy
32
Approach - Emergency Releases
• Development PowerCenter components were maintained
in both the new and old Repository form...
Lessons Learned
34
Lessons Learned
• Regression Testing Results may be surprising
• Direct comparison of ETL runs may reveal problems not
cau...
Lessons Learned
• Production Cut Over
• Requires large manual effort
• Best to work in pairs to avoid mistakes
• Can be do...
Lessons Learned
• The upgrade was considered a success by all
stakeholders
• Reach out to the Pro’s at Informatica when th...
Questions and Answers
38
Don’t forget to rate my session in
the mobile app!
Upcoming SlideShare
Loading in …5
×

PowerCenter Upgrades with Zero Downtime

2,971 views

Published on

Keeping your PowerCenter version current is essential. However, performing an upgrade in a real time EDW environment requires data loads to be suspended. This costs money. Productivity is lost and penalties may be assessed for missed SLAs. It does not have to be this way. Using techniques proven with mission critical applications for the Department of Defense, PowerCenter can be upgraded with zero service interruption.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,971
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
196
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

PowerCenter Upgrades with Zero Downtime

  1. 1. PowerCenter Upgrades with Zero Downtime Presented by Greg Wade & Ed Wagner, Lockheed Martin
  2. 2. Topics • About Us • Our Environment • Requirements • Approach • Development/ Test Transition • Emergency Release Strategy • Lessons Learned • Q&A 2 © 2013 Lockheed Martin Corporation. All Rights Reserved.
  3. 3. About Us 3
  4. 4. About Us • Who we are • Greg Wade, Information Systems Architect • Ed Wagner, ETL Lead Developer • What we do • Build large scale active data warehouses with near real time data loads and high availability for Department of Defense (DoD) customers • Where we work • Lockheed Martin IS&GS; A leading federal services and information technology contractor 4 © 2013 Lockheed Martin Corporation. All Rights Reserved.
  5. 5. Our Environment 5
  6. 6. Our Environment 6 © 2013 Lockheed Martin Corporation. All Rights Reserved. Input Data Acquisition EDW Teradata Staging Oracle ETL • SPARC Enterprise M5000 • 144 GB RAM • Oracle Solaris 10 OS • Informatica toolsets • PowerCenter • B2BDT • Metadata Manager Excess ETL server and repository capacity needed to perform a Zero Downtime Upgrade.
  7. 7. Our Environment 7 © 2013 Lockheed Martin Corporation. All Rights Reserved. Informatica Powercenter 1 Domain Per 6 Environments 7 Integration Services 24 x 7 Operations Data Acquisitions ~125K Flat FilesXML Staging Oracle Operational Data Store Oracle EDW Teradata TPT Files ~65K Oracle Change Files 400+ WORKFLOWS Multiple Sessions (200+ Trans) Finalize Command Tasks Initialize Command Tasks
  8. 8. Requirements 8
  9. 9. Requirements • Develop a reusable upgrade strategy • Version 8.6.1 HF 11 to 9.1 HF 3 … and future versions • Reusable for major upgrades … not hot fix installation • Orchestration of upgrade in six environments • Minimize / eliminate production system outage • Zero data loss • Complete regression and performance testing required prior to production install 9 © 2013 Lockheed Martin Corporation. All Rights Reserved. Production must be upgraded while it is running with fully tested process and baseline.
  10. 10. Requirements • Complete or partial roll-back must be supported • Must be able to support any Production Emergency Release (ER) contingency during any point in the upgrade lifecycle • Test and development of ongoing products had to continue uninterrupted 10 © 2013 Lockheed Martin Corporation. All Rights Reserved. . Development, test, and deployment on both versions must be possible for length of upgrade.
  11. 11. Approach Plan, Development, Testing, Production 11
  12. 12. In Place Upgrades • “Go Big or Go Home” Approach • Pros • Complex orchestration not required • Can be accomplished quickly • Preserves history (statistics, logs, etc) • Cons • Roll-back complex • Requires production downtime • Baseline differences in each environment may produce unpredictable results • May loose SCM controlled baseline 12 © 2013 Lockheed Martin Corporation. All Rights Reserved. Too much risk to SLAs … we were scared!
  13. 13. High Level Approach 13 © 2013 Lockheed Martin Corporation. All Rights Reserved. Dual installations not possible under Windows. Dual instances of 8.6 and 9.1 stood up in each environment Production baseline migrated from external CM (8.6 to 9.1) Regression and performance testing performed before Go-Live Data sources migrated one at a time 9.1 objects migrated between environments via XML Start Date: Oct-2011 Complete Date: Mar-2012
  14. 14. Approach – The Upgrade Begins • Install 9.1 in development environment • Clean install … DO NOT UPGRADE 8.6 • Create a second PowerCenter repository • Load SCM controlled production baseline into 9.1 • XML Objects migration from 8.6 to 9.1 • Only possible when no repository changes • Same version migration and then upgrade required when repository changes between versions • Authoritative dev baseline ( ) in 8.6 14 © 2013 Lockheed Martin Corporation. All Rights Reserved. Before you proceed make sure you are licensed to install multiple repositories.
  15. 15. Approach – The Upgrade Begins • Created second Oracle schema • v8.6.1 – IN86_RPS v9.1.0 – IN91_RPS • Ensure sufficient file system space available for two complete PowerCenter installations • Use symbolic links • Change installation directory • Update UNIX Profile • Created a script to source the v8 instead of the new default v9 profile • INFA_HOME • IFCMPath (Content Manager / B2BDT) • PATH / CLASSPATH 15 © 2013 Lockheed Martin Corporation. All Rights Reserved. Better space allocation during next HW refresh will make future upgrades easier.
  16. 16. Open new ports in host based firewalls and any infrastructure firewalls. Approach – The Upgrade Begins • Established second set of ports • PowerCenter Web-based Administration Console - SSL • v8.6.1 – XXXX • v9.1.0 – ZXXX • Gateway Host • v8.6.1 – YYYY • v9.1.0 – ZYYY • Remember to target new ports during Server Installation • Install new GUI client tools, configured to use new Gateway ports • Old and new client tools available concurrently to monitor and correct issues with either existing or new installation 16 © 2013 Lockheed Martin Corporation. All Rights Reserved.
  17. 17. Approach – The Upgrade Begins • Resolve any import problems • None were encountered from 8.6 -> 9.1 • Validate a sampling of data exchanges for functionality • Export version 9.1 mappings and workflows and place under SCM 17 © 2013 Lockheed Martin Corporation. All Rights Reserved. Initial baseline of 9.1 configuration complete
  18. 18. Approach – Regression Testing • Clean install of Version 9.1 • Content imported from SCM • Baseline created in development • Authoritative dev baseline ( ) remains in 8.6 • Authoritative dev baseline ( ) remains in 8.6 18 © 2013 Lockheed Martin Corporation. All Rights Reserved. 9.1 installed into test and ready for regression and performance testing.
  19. 19. Approach – Regression Testing • Environment • Do both server versions successfully start up? • Validates dual profiles • How to select what to test • Test it all? • Select a representative subset? • What to look for when you test • Rows in/out • Do the rows match? 19 © 2013 Lockheed Martin Corporation. All Rights Reserved.
  20. 20. Approach – Regression Testing 20 © 2013 Lockheed Martin Corporation. All Rights Reserved. Name Workflows Mapping Command Task Logging Parameter Files Source/ Target Files Connections Joiners Expressions Update Strategies Aggregators wf_A.XML        N/A  wf_B.XML  X       N/A wf_C.XML X       N/A N/A SELECT DISTINCT REP_WORKFLOWS.SUBJECT_AREA, REP_WORKFLOWS.WORKFLOW_NAME, OPB_TASK_INST_RUN.INSTANCE_NAME, OPB_TASK_INST_RUN.TASK_ID, OPB_TASK_VAL_LIST.PM_VALUE, OPB_TASK_VAL_LIST.EXEC_ORDER FROM REP_WORKFLOWS REP_WORKFLOWS, OPB_TASK_INST_RUN OPB_TASK_INST_RUN, OPB_TASK_VAL_LIST OPB_TASK_VAL_LIST WHERE UPPER(SUBJECT_AREA) IN ('FOLDER_A', 'FOLDER_B', 'FOLDER_C') AND REP_WORKFLOWS.WORKFLOW_ID = OPB_TASK_INST_RUN.WORKFLOW_ID AND OPB_TASK_INST_RUN.TASK_ID = OPB_TASK_VAL_LIST.TASK_ID AND OPB_TASK_INST_RUN.INSTANCE_NAME NOT IN ('Start', 'tmr_Wait') AND OPB_TASK_VAL_LIST.PM_VALUE NOT IN ('$$cmd_a', '$$cmd_b $PMWorkflowRunId', '$$cmd_c') ORDER BY REP_WORKFLOWS.SUBJECT_AREA; Review subset of workflows
  21. 21. Approach – Performance Testing • Identical test datasets were repeatedly run against both PowerCenter versions using identical SCM-sourced components • Intentional backlog created an opportunity to measure all affected system component behaviors under stressed conditions • Created and ran a Unix script to capture real-time ETL Server memory / CPU utilization • Executed Repository queries • Determine workflow run times • Input and output rows per second • Compared all results from both versions to assess scope and depth of potential differences 21 © 2013 Lockheed Martin Corporation. All Rights Reserved.
  22. 22. Approach – Repository SQL Review 22 © 2013 Lockheed Martin Corporation. All Rights Reserved. • Executed SQL against repository tables in both 8.6 and 9.1 • Regression testing by comparing input_rows, output_rows, failed_rows, and trans_errs • Same rows, failures, and errors should be same between 8.6 and 9.1 • Depending on your design, some failures and errors may be “normal” • Performance testing by comparing number of runs and runtimes • Runtimes will not match, looking for outliers • Input and output rows per second provide a good picture of performance
  23. 23. Approach – Testing • Test results reviewed with engineering team • GO / NO GO Decision point 23 © 2013 Lockheed Martin Corporation. All Rights Reserved. Validation of 9.1 configuration complete Are we ready for production?
  24. 24. Approach – Ready for Production? 24 © 2013 Lockheed Martin Corporation. All Rights Reserved. • 9.1 baseline does not match current production baseline • Development and Testing of new projects continued in 8.6 while testing 9.1 Baseline merges need to be performed.
  25. 25. Approach – Ready for Production? 25 © 2013 Lockheed Martin Corporation. All Rights Reserved. Resync 9.1 Baseline Migrate Development to 9.1 Begin testing new releases on 9.1 Production deployment Production changes need to be minimized during final resync
  26. 26. Approach – Install Production Content 26 © 2013 Lockheed Martin Corporation. All Rights Reserved. During content migration ensure 8.6 infrastructure is not modified. 8.6 processes production workload. Final 9.1 baseline loaded. 8.6/9.1 configured to use same file system store. 8.6/9.1 configured to use same target DB
  27. 27. Approach – Interface Cutover 27 © 2013 Lockheed Martin Corporation. All Rights Reserved. DO NOT run a workflow simultaneously in 8.6 and 9.1. • Workflows cutover one at a time or in groups by IS • Cutover performed manually • Decided not to automate • Rollback by moving interface back to 8.6
  28. 28. Approach – Interface Cutover • Risk reduction • Selecting the best time • Having support on hand • Internal • Oracle Database Administrators /Unix System Administrators • Ops Team aware of, and involved in, the Production upgrades • Corporate and Customer Management – Available to make critical go / no-go decisions as issues were identified and actioned • External - Informed Informatica Support that the major upgrade was ongoing and that time-sensitive support may be needed needed • Balancing act to track down problems ourselves versus getting the Pros involved early weighted more heavily towards Informatica Support than usual • All SRs were dispositional quickly and effectively 28 © 2013 Lockheed Martin Corporation. All Rights Reserved.
  29. 29. Approach – Interface Cutover • Checklist – Repeated for each Workflow  Unschedule Workflow in 8.6  Wait for Workflow to complete, if running  Disable operational alerts for 8.6 workflow  Remove IS from Workflow in 8.6  Record sequencer values for 8.6 workflow  Assign workflow to IS in 9.1  Reset sequencer values in 9.1 workflow  Schedule workflow in 9.1  Enable operational alerts for 9.1 workflow  Validate successful completion of workflow 29 © 2013 Lockheed Martin Corporation. All Rights Reserved. Steps may vary slightly in your environment for external schedulers, etc
  30. 30. Development / Test Transition 30
  31. 31. Final Dev and Test Migration • v8.6.1 was no longer needed to support Emergency Releases – Production at v9.1 • v8.6.1 Development and Test work complete; snapshots checked into SCM • v8.6.1 Repository build complete; Import into v9.1 Repository • Goodbye v8.6.1 and Hello v9.1 in all environments! 31 © 2013 Lockheed Martin Corporation. All Rights Reserved.
  32. 32. Emergency Release Strategy 32
  33. 33. Approach - Emergency Releases • Development PowerCenter components were maintained in both the new and old Repository formats • Required fixes were made in v8.6.1 until successful testing was completed • Changes were made in the v8.6.1 version and propagated through the upstream environments for testing and Production deployment • Final version was also imported into v9.1 Repositories 33 © 2013 Lockheed Martin Corporation. All Rights Reserved.
  34. 34. Lessons Learned 34
  35. 35. Lessons Learned • Regression Testing Results may be surprising • Direct comparison of ETL runs may reveal problems not caused by the upgrade. • Bad mapping logic produced nondeterministic results • Performance Testing • Many factors not part of the upgrade may impact performance results • Other test environment activity (DB, Network) • HW configuration issues • New and old PowerCenter versions may require different OS tuning • System may need to be reconfigured between runs • More system resources may need to be added 35 © 2013 Lockheed Martin Corporation. All Rights Reserved.
  36. 36. Lessons Learned • Production Cut Over • Requires large manual effort • Best to work in pairs to avoid mistakes • Can be done in multiple live environments without user impact … they never noticed! • Have engineering, SA, DBA support on hand • Development and Test • Extra coordination required across projects • Are all changes made to the correct baseline? • Emergency changes need to be made in multiple baselines • Ensure all developers have both sets of development tools installed and functioning early • Create additional integration services names due to PM Locks 36 © 2013 Lockheed Martin Corporation. All Rights Reserved.
  37. 37. Lessons Learned • The upgrade was considered a success by all stakeholders • Reach out to the Pro’s at Informatica when things happen that you don’t understand • Would we do it again? 37 © 2013 Lockheed Martin Corporation. All Rights Reserved. PowerCenter can successfully be upgraded in a production environment with ZERO downtime.
  38. 38. Questions and Answers 38
  39. 39. Don’t forget to rate my session in the mobile app!

×