One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day
DeltaV Version 10.3.1 Upgrade in 24hrs Version 8.4.2 – Version 10.3.1
Presenters Gus Dionne, Automation ARI Dan DiMatteo, IS ARI Josh Gangl, OIS Platform Lead David Maglaya, Emerson LBP
Introduction DeltaV Version Upgrade Planning the Upgrade – Objectives, Deliverables, Schedule, Resources Executing the Plan – Developing a Team, Direct and Manage the Execution Monitoring and Controlling – Verify scope, Integrate Changes Closing the Project – Lessons Learned These are all requirements of a successful upgrade.
ARI Mfg. Site 75 Acre Footprint At a site of this size, planning is key to success
Planning and Defining Our Scope Our goal was to upgrade two (2) DeltaV Systems with minimum impact to users Our constraints were Time (24hrs) , Resources, and Space We needed to upgrade not only the Software Application but we needed to upgrade most of the hardware that it would run on And we also needed to successfully complete these goals in the middle of a major project, which had extensive software impact
Additional Scope 8 of our 18 Servers would need to be updated from Dell 2850s to R710 (18 servers in the environment– 16 DV servers plus 2 dedicated domain controllers which were updated to PE2950 hardware) 70 out of 92 controllers would need to be upgraded from MD’s to MD Plus and MX models All of our 65 OITs would need to be upgraded from 5 and 6 year old models to Optiplex 780s Add an additional 19th server to manage Critical Data Backup Application (CDBA) as part of the upgrade
Tools and Resources We would need high performance tools to complete the upgrade in a 24hr window. The tool bag would have the following + – Imaging tool for OITs – A Broadcast Server for deploying images – Racks for Server buildup and testing – Spare OIT hardware for testing – SI Dongles to turn all OITs to Development stations – A team of Emerson Support Engineers – Hard copy checklist for sequential steps
Tools and Resources (additional) – A (Time smart) detailed DeltaV Excel Add-in spreadsheet for saving and restoring critical controller online parameters – A controller map depicting model, new Mac address of hardware and installation location – A team of hands on Tech’s and Engineers was created – A timed schedule was created with each task for each person on the team
The DeltaV Excel Add In Tool example Having a spreadsheet with macros simplifies work and reduces risk.
The Controller Tracking ID ToolController Tracker
The Simple Checklist goes a long wayOrganized tasks with priorities are a must for capturingessentials and ensuring nothing is missed.
The Early Execution Offline Migration Testing from 8.4.2 to 10.3.1 at LBP office 60+ new OITs would be deployed with the 8.4.2 image prior to the upgrade day 10.3.1 Images would be created and tested in plans for broadcasting to the new OITs on the upgrade day Each new controller to be deployed was labeled and Mac address recorded for its intended new location. This saved the light flashing process at each location for the new controllers during commissioning.
The Early Execution cont. A disk was pulled from the Domain Controller and used to copy the domain users onto the new DC box, (this was done for offline code development in the Dev environment, we used the live DC to migrate the Active Directory during the outage) Critical parameters that needed to be restored would be listed in Time Smart XL add in tool Server class boxes would be pre-built, racked, stacked and configured as far as possible prior to execution and left powered on until upgrade The Pro Plus Servers had Database imported at Version 8.4.2 and then the 10.3.1 upgrade wizard run
The Execution prep(the night before) Critical parameters that needed to be restored would be uploaded to the XL add in tool with an individual time record for each time-sensitive value Controllers with outputs that needed to be maintained would have the manual field overrides enabled 70 of the 92 controllers would be decommissioned and replaced with the new pre-labeled controllers
The Execution The execution started at 7:00am with the team concurrently working the plan from a central automation lab room, server room, and plant floor Servers previously racked and stacked are placed online starting with the Pro Plus boxes – Legacy DeltaV 8.4.2 PPS servers were “failed” • Metadata cleanup, FSMO roles seized by dedicated DC – New PPS R710 hardware racked & stacked • DCPROMO / DNS config, FSMO roles transferred to new PPS
The Execution 10.3.1 Images are concurrently broadcasted to multiple OIT stations seven (7) at a time A team of IS engineers are running OIT configuration and Sys Prep at pre-scheduled OIT locations
The Execution continued As OITs became available controllers were commissioned with no field assistance due to Mac address reservation, then flashed, then downloaded Each OIT coming on line was used to RDP to field OITs to multiply the Flashing capacity of the automation lab (SI) Critical online controller parameters were restored using the DeltaV Time Smart Add-in tool. Online values that were time sensitive were updated from within Excel, based on the time the value was stored and the current time of restoration.
The Closing As Systems came back online they were turned over to Validation then manufacturing A Lessons Learned session was held shortly after
Lessons learned Some Good Points Have a well defined plan and schedule Trained resources Pre building and Server Prep is key to saving time RAID 1 (mirroring) with hot spare disk setup to support DC transfer (easily export Active Directory from dedicated domain controller – Pro Plus and other DeltaV servers use standard RAID10 config) DeltaV Excel Add-in was key in restoring online values quickly without manual calculations Broadcasting of OIT image saved time over upgrade wizard
Lessons learned cont. Some Bad Points We were overstaffed the morning of outage. Staffing schedule could be improved, (Pro Plus delay due to Active Directory DCPROMO issue added to this) VX dongles were needed and not known right away Some user accounts got corrupted due to Schema mis- match Servers pulled from service should have been left in racks for a week or two since some data configuration recovery was required
Lessons learned cont. Some more Bad Points Implementing an upgrade with concurrent complex software changes is extremely difficult (Requirements were changed by people outside of core project team – caused problems with keeping plan locked down) – point here is to get everyone to commit to providing requirements up-front and early in the planning
Business Results Achieved 1. There were no Safety related incidents due to any upgrade associated actions 2. The DeltaV System went from running version 8.4.2 one evening to running version 10.3.1 the following evening 3. 18 Servers, 65 OITs, and 70 Controllers were upgraded (8 R710 replacements and 8 PE2950s were upgraded in place using Upgrade Wizard, 2 DCs were upgraded to PE2950 hardware) 4. There was no loss of production outside the planned outage due to the upgrade 5. There was no loss of utility access due to the upgrade
Summary We have done our best to share how we planned, managed, and executed a successful DeltaV software and hardware upgrade within a large scale Bio-Tech manufacturing Facility Our benefits are faster, better performing, and more capable software and hardware to the end user as well as a more robust system for the administrators
Questions I’d like to take this time to ask if anyone has any questions that pertain to the project upgrade?
Feedback If any of you would not mind sharing any feedback you might have we would all like to know if you enjoyed the presentation or not? Thank you all so much for attending.
Where To Get More Information Other training sessions List books, articles, electronic sources Consulting services, other sources