2. Presenters
Gus Dionne, Automation ARI
Dan DiMatteo, IS ARI
Josh Gangl, OIS Platform Lead
David Maglaya, Emerson LBP
3. Introduction
DeltaV Version Upgrade
Planning the Upgrade
– Objectives, Deliverables, Schedule, Resources
Executing the Plan
– Developing a Team, Direct and Manage the Execution
Monitoring and Controlling
– Verify scope, Integrate Changes
Closing the Project
– Lessons Learned
These are all requirements of a successful upgrade.
4. ARI Mfg. Site 75 Acre Footprint
At a site of this size, planning is key to success
5. Planning and Defining Our Scope
Our goal was to upgrade two (2) DeltaV Systems with
minimum impact to users
Our constraints were Time (24hrs) , Resources, and
Space
We needed to upgrade not only the Software Application
but we needed to upgrade most of the hardware that it
would run on
And we also needed to successfully complete these goals
in the middle of a major project, which had extensive
software impact
6. Additional Scope
8 of our 18 Servers would need to be updated from Dell
2850s to R710 (18 servers in the environment– 16 DV
servers plus 2 dedicated domain controllers which were
updated to PE2950 hardware)
70 out of 92 controllers would need to be upgraded from
MD’s to MD Plus and MX models
All of our 65 OITs would need to be upgraded from 5 and
6 year old models to Optiplex 780s
Add an additional 19th server to manage Critical Data
Backup Application (CDBA) as part of the upgrade
7. Tools and Resources
We would need high performance tools to complete the
upgrade in a 24hr window. The tool bag would have the
following +
– Imaging tool for OITs
– A Broadcast Server for deploying images
– Racks for Server buildup and testing
– Spare OIT hardware for testing
– SI Dongles to turn all OITs to Development stations
– A team of Emerson Support Engineers
– Hard copy checklist for sequential steps
8. Tools and Resources (additional)
– A (Time smart) detailed DeltaV Excel Add-in spreadsheet for
saving and restoring critical controller online parameters
– A controller map depicting model, new Mac address of hardware
and installation location
– A team of hands on Tech’s and Engineers was created
– A timed schedule was created with each task for each person on
the team
9. The DeltaV Excel Add In Tool example
Having a spreadsheet with macros simplifies work and
reduces risk.
11. The Simple Checklist goes a long way
Organized tasks with priorities are a must for capturing
essentials and ensuring nothing is missed.
12. The Early Execution
Offline Migration Testing from 8.4.2 to 10.3.1 at LBP office
60+ new OITs would be deployed with the 8.4.2 image
prior to the upgrade day
10.3.1 Images would be created and tested in plans for
broadcasting to the new OITs on the upgrade day
Each new controller to be deployed was labeled and Mac
address recorded for its intended new location. This saved
the light flashing process at each location for the new
controllers during commissioning.
13. The Early Execution cont.
A disk was pulled from the Domain Controller and used to
copy the domain users onto the new DC box, (this was
done for offline code development in the Dev
environment, we used the live DC to migrate the Active
Directory during the outage)
Critical parameters that needed to be restored would be
listed in Time Smart XL add in tool
Server class boxes would be pre-built, racked, stacked
and configured as far as possible prior to execution and
left powered on until upgrade
The Pro Plus Servers had Database imported at Version
8.4.2 and then the 10.3.1 upgrade wizard run
14. The Execution prep(the night before)
Critical parameters that needed to be restored would be
uploaded to the XL add in tool with an individual time
record for each time-sensitive value
Controllers with outputs that needed to be maintained
would have the manual field overrides enabled
70 of the 92 controllers would be decommissioned and
replaced with the new pre-labeled controllers
15. The Execution
The execution started at 7:00am with the team
concurrently working the plan from a central automation
lab room, server room, and plant floor
Servers previously racked and stacked are placed online
starting with the Pro Plus boxes
– Legacy DeltaV 8.4.2 PPS servers were “failed”
• Metadata cleanup, FSMO roles seized by dedicated DC
– New PPS R710 hardware racked & stacked
• DCPROMO / DNS config, FSMO roles transferred to new PPS
16. The Execution
10.3.1 Images are concurrently broadcasted to multiple
OIT stations seven (7) at a time
A team of IS engineers are running OIT configuration and
Sys Prep at pre-scheduled OIT locations
17. The Execution continued
As OITs became available controllers were commissioned
with no field assistance due to Mac address reservation,
then flashed, then downloaded
Each OIT coming on line was used to RDP to field OITs to
multiply the Flashing capacity of the automation lab (SI)
Critical online controller parameters were restored using
the DeltaV Time Smart Add-in tool. Online values that were
time sensitive were updated from within Excel, based on
the time the value was stored and the current time of
restoration.
18. The Closing
As Systems came back online they were turned over to
Validation then manufacturing
A Lessons Learned session was held shortly after
19. Lessons learned
Some Good Points
Have a well defined plan and schedule
Trained resources
Pre building and Server Prep is key to saving time
RAID 1 (mirroring) with hot spare disk setup to support DC
transfer (easily export Active Directory from dedicated
domain controller – Pro Plus and other DeltaV servers use
standard RAID10 config)
DeltaV Excel Add-in was key in restoring online values
quickly without manual calculations
Broadcasting of OIT image saved time over upgrade
wizard
20. Lessons learned cont.
Some Bad Points
We were overstaffed the morning of outage. Staffing
schedule could be improved, (Pro Plus delay due to Active
Directory DCPROMO issue added to this)
VX dongles were needed and not known right away
Some user accounts got corrupted due to Schema mis-
match
Servers pulled from service should have been left in racks
for a week or two since some data configuration recovery
was required
21. Lessons learned cont.
Some more Bad Points
Implementing an upgrade with concurrent complex
software changes is extremely difficult
(Requirements were changed by people outside of core
project team – caused problems with keeping plan locked
down) – point here is to get everyone to commit to
providing requirements up-front and early in the planning
22. Business Results Achieved
1. There were no Safety related incidents due to any
upgrade associated actions
2. The DeltaV System went from running version 8.4.2 one
evening to running version 10.3.1 the following evening
3. 18 Servers, 65 OITs, and 70 Controllers were upgraded
(8 R710 replacements and 8 PE2950s were upgraded in
place using Upgrade Wizard, 2 DCs were upgraded to
PE2950 hardware)
4. There was no loss of production outside the planned
outage due to the upgrade
5. There was no loss of utility access due to the upgrade
23. Summary
We have done our best to share how we planned,
managed, and executed a successful DeltaV software and
hardware upgrade within a large scale Bio-Tech
manufacturing Facility
Our benefits are faster, better performing, and more
capable software and hardware to the end user as well as
a more robust system for the administrators
24. Questions
I’d like to take this time to ask if anyone has any questions
that pertain to the project upgrade?
25. Feedback
If any of you would not mind sharing any feedback you might
have we would all like to know if you enjoyed the
presentation or not?
Thank you all so much for attending.
26. Where To Get More Information
Other training sessions
List books, articles, electronic sources
Consulting services, other sources