Partners Enterprise Caché Unix Migration Public


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • I’m going to cover these 5 topics There were many steps in the migration process. I’m going to focus on the production migration events
  • We had to press the red button too often Availability was the primary business driver
  • Our monthly downtime for a Windows ISM did not meet our business need Rebooting our Production environment for monthly Windows security patches took between 1 and a half to two hours
  • Here’s how we’ve decreased the planned and unplanned downtime for the Caché production environment
  • Other business drivers were scalability and performance Our Caché 5.0 Windows environment <click> could no longer deliver the performance demanded by the LMR application
  • Over the past 5 years we’ve had a 7-fold increase in the number of monthly LMR transactions
  • During the same time, the entire platform had an even higher increase in database references
  • Here’s how we improved performance and increased scalability In a 32-bit O/S, we were limited to about 1.5 GB of global buffers, configured up to 30 GB on some application servers
  • This is a diagram of our production environment pre-migration There are 61 Windows app servers and 5 Windows database servers All infrastructure is in Needham except for 12 IIS servers and the SAN replication target Symmetrix
  • This project wasn’t just an operating system migration
  • We faced a number of challenges bringing HP-UX into the Partners environment
  • The solution was CIFS Our file IO utility automatically mounts CIFS shares and creates them, if necessary In production there have been more than 41,000 dynamic CIFS share mounts in three months
  • Another challenge was printing Printing to Windows print servers through CIFS would require maintaining hundreds of print queues on each HP-UX host We had to find a better way to do it
  • Here’s our solution to printing We use Cache as a bridge between HP-UX and Windows The Windows printer Daemon Queue is redundant for high-availability
  • We had to get the data from Windows to UNIX The solution to this is covered in upcoming slides
  • Here is the timeline for the phase 1 migration Phase 1 is the database tier migration The Caché version remained 5.0 The first 4 months consisted of Proof of Concept and benchmarking tests, followed by DEV, QA and 4 production database migration events So one part of the solution to the data transfer issue was to break it up into 4 events
  • I’m going to describe the process we used for each of the four migration events On the left are the 5 Windows database servers In the middle are the Windows InfoMover servers. Infomover is a san-based, host-assisted cross-platform file transfer utility. This is the other essential part of the solution to our file transfer problem On the right is the HP-UX database server cluster where Caché instances are already installed Note that the Windows servers are attached to the DMX-2 and the HP-UX server is attached to the DMX-3 To get enough throughput, I configured 4 InfoMover servers with 24 InfoMover processes among them writing to 24 filesystems on the HP-UX host First, we cloned the drives for the server, kept them in synch until the scheduled downtine, activated and terminated the clones Then we started the Infomover scripts to transfer the files and then Perform an Endian conversion. Afterward, we started a shadow process to keep the Windows database server up to date in case a back-out was required.
  • Here’s the final phase 1 configuration with the 5 Windows database servers replaced by 1 HP-UX Serviceguard cluster
  • This is the timeline for Phase 2 Phase 2 is the application tier migration to HP-UX and the Caché upgrade to 2008.1. It actually started before Phase 1 was completed We built two new Caché 2008.1 environments for the application testing and certification efforts In the Fall and early Winter we built the application tier Then we migrated DEV, QA and production
  • Now I’d like to walk you through the Phase 2 production migration process Note that green indicates Caché 5.0.20 and blue indicates Caché 2008.1 We start with Windows application servers and an HP-UX Serviceguard cluster all running Caché 5.0 instances We build the Ensemble and application server cluster on HP-UX Integrity blades and go live on Ensemble 2008.1 We build the remaining Integrity blades and install 2008.1 on Windows Proliant servers Install 5 2008.1 database instances Create routine datasets for 2008.1 Export 5.0 routines and classes and import them into the 2008.1 datasets Now we’re in the downtime window
  • Here is a diagram of production environment post-migration
  • Phase 3 is the build out of the remaining DR infrastructure we completed post-migration
  • I have 3 slides of accomplishments, so I’m just going to highlight a few Nagios First LAN-free
  • Here are all of the software upgrades in the migration in more detail
  • Partners Enterprise Caché Unix Migration Public

    1. 1. <ul><li>Partners Enterprise Caché UNIX Migration </li></ul><ul><li>Walter Halvorsen </li></ul><ul><li>Systems Programming Lead II </li></ul><ul><li>[email_address] </li></ul><ul><li>May 7, 2009 </li></ul>
    2. 2. In the next half hour… <ul><li>Business and Technical Drivers </li></ul><ul><li>Scope of the Migration </li></ul><ul><li>Technical Challenges </li></ul><ul><li>Migration Process </li></ul><ul><ul><li>Vendor Selection </li></ul></ul><ul><ul><li>Staffing and Training </li></ul></ul><ul><ul><li>Proof of Concept </li></ul></ul><ul><ul><li>Phase 1 Migration (database tier) </li></ul></ul><ul><ul><li>Phase 2 Migration (application tier) </li></ul></ul><ul><ul><li>Phase 3 (DR configuration) </li></ul></ul><ul><li>Accomplishments and Results </li></ul>
    3. 3. Why Migrate off Windows? Availability
    4. 4. Business and Technical Drivers - Availability Average Monthly Downtime Windows ISM Uptime In Hours In Minutes In Seconds 99.000% 7.3 99.500% 3.6 99.750% 1.8 99.900% 44.0 99.990% 4.4 99.999% 26 Business need
    5. 5. Business and Technical Drivers - Availability <ul><li>High-Availability Configuration </li></ul><ul><ul><li>Redundant NICs/APA </li></ul></ul><ul><ul><li>Serviceguard Clustering for databases and IP-dependent workloads </li></ul></ul><ul><li>Less Scheduled Downtime </li></ul><ul><ul><li>HP-UX Patches twice a year </li></ul></ul><ul><ul><li>Caché configuration changes twice a year </li></ul></ul><ul><ul><li>Infrequent Caché upgrades </li></ul></ul><ul><li>Non-Disruptive Backup </li></ul><ul><ul><li>Caché database updates during quiesce </li></ul></ul>
    6. 6. Business and Technical Drivers – Scalability and Performance Caché 5.0/ Windows LMR
    7. 7. Unprecedented Growth - The LMR ~56 million ~8 million
    8. 8. Caché Application Tier Database References ~59.9 billion ~4.5 billion Billions of Global References
    9. 9. Business and Technical Drivers – Performance and Scalability <ul><li>Improved Performance </li></ul><ul><ul><li>64 bit OS, more memory for Caché </li></ul></ul><ul><ul><li>Faster FC HBA cards, Symmetrix frame and disk </li></ul></ul><ul><ul><li>Caché upgrade, improved ECP performance </li></ul></ul><ul><li>Increased Scalability </li></ul><ul><ul><li>New SAN storage solution – tiered storage and more expansion potential </li></ul></ul><ul><ul><li>More FC HBA cards per host </li></ul></ul><ul><ul><li>On-demand addition of processor cores </li></ul></ul>
    10. 10.
    11. 11. Migration Scope <ul><li>Windows to HP-UX O/S Migration </li></ul><ul><li>Application platform-independence </li></ul><ul><li>Caché Upgrade from 5.0 to 2008.1 </li></ul><ul><li>Ensemble Upgrade from 3.1 to 2008.1 </li></ul><ul><li>SAN Migration from DMX-2 to DMX-3 </li></ul><ul><li>Web tier/ Weblink </li></ul><ul><li>VB OCX </li></ul>
    12. 12. Operating System Challenges <ul><li>Windows-centric environment </li></ul><ul><ul><li>Windows is the primary O/S at Partners </li></ul></ul><ul><ul><li>Caché account originally had Domain Admin rights </li></ul></ul><ul><li>File Access </li></ul><ul><ul><li>Shared File Areas, home servers are Windows shares </li></ul></ul><ul><ul><li>No NAS or UNIX File Services for Windows </li></ul></ul><ul><li>Printing </li></ul><ul><ul><li>All print servers at Partners are Windows </li></ul></ul><ul><ul><li>Can’t print directly to Windows Print Servers from UNIX </li></ul></ul>
    13. 13. Operating System Challenges – Windows File Access How do we access Windows file shares from HP-UX?
    14. 14. Operating System Challenges – Windows File Access CIFS C ommon I nternet F ile S ystem Dynamic CIFS share creation and mounting 41,000+ dynamic CIFS mounts in 3 months
    15. 15. Operating System Challenges – Printing <ul><li>Print from UNIX to Windows? </li></ul>Just Say No!
    16. 16. HP-UX Windows Caché is the bridge between HP-UX and Windows Operating System Challenges – Printing
    17. 17. Phase 1 (database tier) Migration - Data Transfer How do we move 3.5 Terabytes of Data within an ISM Window?
    18. 18. Migration Process – Timeline Phase 1
    19. 19. Phase 1 (Database Tier) Migration Process Windows Database Servers Windows InfoMover Servers HP-UX Database Server Cluster EMC Sym 1547 DMX-2 Timefinder Clone EMC Sym 1149 DMX-3 24 InfoMover file transfer processes Caché Shadow Process Endian Conversion Caché instances
    20. 20.
    21. 21. Migration Process – Timeline Phase 2
    22. 22. Phase 2 (Application Tier) Production Migration Process Windows Proliant Application Servers HP-UX 8640 Database Server Serviceguard Cluster Globals 61 5.0.20 Routines 2008.1 Routines G2G HP-UX Integrity Blade Servers Application Server/ Ensemble/ Ancillary System Serviceguard Cluster Caché 5.0.20 Caché 2008.1 Config. File LEGEND Application Servers Reconfiguration of Job scheduling databases, VB and BICS service pools Export Routines and Classes Import Routines and Classes Caché instances 9
    23. 23.
    24. 24. Phase 3 (DR)
    25. 25. Accomplishments <ul><li>Hardware Upgrades </li></ul><ul><ul><li>Migrate all platform data to new enterprise tiered storage solution (DMX3) </li></ul></ul><ul><ul><li>Migrated all platforms from 80+ 32-bit Windows to 12 64-bit HP-UX servers </li></ul></ul><ul><ul><li>Implemented a Nagios server monitoring and alerting system </li></ul></ul><ul><ul><li>Implemented Serviceguard clustering with automatic failover </li></ul></ul><ul><ul><li>Implemented new Web server tier for all platforms </li></ul></ul><ul><ul><li>Implemented first LAN-free backup solution at Partners </li></ul></ul><ul><ul><li>Implemented new EMC Clone technology </li></ul></ul><ul><ul><li>Implemented new UNIX QA Citrix solution </li></ul></ul><ul><ul><li>Implemented a complete Disaster Recovery solution </li></ul></ul>
    26. 26. Accomplishments (cont.) <ul><li>Software Upgrades </li></ul><ul><ul><li>Migrated all Caché platforms from Windows Server 2003 to HP-UX 11i V2.3 </li></ul></ul><ul><ul><li>Migrated all Ensemble platforms from Windows Server 2003 to HP-UX 11i V2.3 </li></ul></ul><ul><ul><li>Migrated all Caché platforms from version 5.0.20 to 2008.1 </li></ul></ul><ul><ul><li>Migrated all Ensemble platforms from version 3.1 to 2008.1 </li></ul></ul><ul><ul><li>Migrated Production web servers from Windows 2000 to Windows Server 2003 </li></ul></ul><ul><ul><li>Migrated Production web servers from IIS V5 to IIS V6 </li></ul></ul><ul><ul><li>Upgraded InterSystems Weblink from version 401 to 458 </li></ul></ul><ul><ul><li>Migrated the Caché VISM-OCX from version 3.2 to 2008.1 </li></ul></ul>
    27. 27. <ul><li>Application Updates </li></ul><ul><ul><li>Replaced hard-coded URL references with relative path references using a new centralized platform-independent core utility </li></ul></ul><ul><ul><li>Replaced all Caché hard-coded file I/O and print references with a new centralized platform-independent core utility </li></ul></ul><ul><ul><li>Replace Caché hard-coded system functions ($ZU) with a new centralized platform-independent core utility </li></ul></ul><ul><li>Budget </li></ul><ul><ul><li>2+ year effort completed on time and under budget </li></ul></ul>Accomplishments (cont.)
    28. 28. Production Metrics – LMR Daily Outliers Pre-migration Post-migration
    29. 29. Partners Enterprise Caché UNIX Migration Walter Halvorsen Systems Programming Lead II [email_address]