The city of Lima, Ohio wastewater treatment plant was running
off tribal knowledge and decades-old equipment. Facing
increased downtime, high overtime expenses for maintenance
staff, and new regulations on the horizon, the plant needed
a new control infrastructure. By upgrading to a modern,
information-enabled DCS, including PlantPAx,® FactoryTalk®
Historian and AssetCentre, the plant has cut more than 50 hours
of downtime in the first three months, saved over $2,000 on
maintenance costs with remote functionality, and is on track
to meeting future compliance goals.
4. • Background of the Project
• Goals for this Project
• Challenges
• Ways we Minimize Downtime
• How we hit our Goals
• How we overcame Challenges
• Best Practices to Share
• Benefits Achieved
• Future Compliance/Updates
• Questions & Answers
• Introduction of City of Lima & CCI
• Matt Fiedler – City of Lima
• Darren Smith – CCI, Project Manager
• Warren Woodcox – CCI, SCADA Specialist
6. Water Infrastructure
Fresh Water Treatment and Distribution
Wastewater Collections and Treatment
Automotive & Industrial Process
Paint Shop Systems
Utility and HVAC Systems
7.
8. • Matt Fiedler - Bio
• Computer Programmer/IT/Process Control Specialist
• Been with the city about 5 years
• Background in IT/IT Security and AV Systems
• 4th Year at Rockwell Automation TechED
• Prior to this job I had no PLC experience. I have learned
everything from TechED training sessions and on the job
training. Also with the help of our local Kirby Risk, I have
learned a lot from their staff with in house training.
9. • Matt Fiedler – Job Function
• My main job is maintaining and assisting staff at
WWTP and WTP. I help handle the IT/Servers
and Process Control Network at each plant.
Assist plant staff with acquiring data for
reporting. Lending a hand to our Electronics
Techs that help with the Physical Hardware and
troubleshooting. I also assist in normal IT duties
citywide with the rest of our Data System
Group(DSG).
10. • Established in 1831, named after Lima Peru, it is the county seat for Allen County.
• Mascot is a 7’ 1” Lima Bean named Beanie.
• Lima Services about 40,000 people within city limits, and a few other communities
for water service.
• We have had the same Mayor for 8 consecutive terms.
• Businesses we service
• General Dynamics – Makers of the M1A1 Tank
• Proctor and Gamble
• Husky Refinery
• Many other fine hospitals and service industries.
• All water treated comes from 4 above ground reservoirs equal to 14.5 billion
gallons of available water.
11. City of Lima Wastewater Treatment Plant
First constructed in 1930, the Lima Wastewater Treatment Plant consisted of screening, grit
removal, primary sedimentation, and anaerobic digestion. Throughout the years, secondary
treatment, tertiary treatment and additional sludge digestion were added. In 1973, the plant
was expanded to an average dry weather flow capacity of 18.5 million gallons per day (MGD),
and a peak flow capacity of 53 MGD. The design concept called for secondary and advanced
treatment processes of the plant to operate at a peak rate of 33 MGD, with any remaining flow
receiving primary settling and chlorination. Since 1973 the plant has provided primary,
secondary and tertiary treatment, as well as bio solids and digester gas recycling.
Newest addition was the upgrade to the headworks at the beginning of the plant. Upgraded
to now support 70 MGD and started the journey of this Process Upgrade.
13. 1. EPA Mandated – New regulations on discharging into the Ottawa River via Consent Decree
1. Decrease CSO discharge events into river to 4 a year. Adding 13 MGD Underground
basin – Project starting this year 2018.
2. Increase WWTP Capacity from 55 MGD to 70 MGD
3. Add wet weather maximization at plant – In Progress
4. Upgrade Remote Lift Stations
2. Create a City of Lima Process Control Standard
1. Bring DCS into industry standard
2. Limit most or all one off code in PLC’s
3. Implement remote capabilities at CSO’s, underground gate structure’s, and lift
stations.
4. Eliminate tribal knowledge to allow us to get outside help without days of learning
existing code to figure out how they programmed it.
14. • Increase Plant flow to 70 MGD from 53 MGD
• Replace old, outdated, aging, and failing process control network at WWTP
• ie. - PLC5, PLC500, DH+ on old fiber optic cable.
• Upgrade all lift station’s, CSO’s, and underground gate structures.
• Including radios and PLC Racks
• Separate WWTP and WTP radio communications.
• Existing equipment was PLC500 and serial MDS orbit radios.
• Upgrade plant network and servers
• Increase Productivity and Decrease Downtime
• Add remote troubleshooting and maintenance
• Replace aging equipment to reduce downtime and maintenance
• Decrease overtime/call outs to reset old equipment
• Replace coding to allow easier troubleshooting.
• Have good copies of programs for all PLC’s
15. • Minimize downtime with remote site upgrades to maintain sewer operation with
minimal discharges into the river. Also keep safe sewer levels without backing
up into basements.
• Maintain a safe flow thru plant while bar screens and primaries were being built.
• Split the flow and built bar screens 2 at a time
• Maintain 2 SCADA networks during the building process with overlap with
minimal downtime.
• Maintain 2 radio networks for remote sites and reduce downtime during
transitions.
• Allowing remote access to integrator to complete work remotely
• Integrate existing equipment and have it be a PlantPAx® Object
• Bring together existing lift station upgrades into new project.
16. • Plan, plan, and PLAN!!
• Looked years out and planned each project the best we could.
• Attended Rockwell Automation TechED to learn about software and
equipment.
• Used the CMAR general delivery method
• CMAR=Construction Manager At Risk
• This allowed us to get all bids up front
• Give the city more input to equipment and subs
• Allowed the city to bring in SCADA integrator (CCI) into the pre-
construction phase to help write scope of work for controls.
• Helped catch possible hold ups early and create work arounds.
• Reviewed all equipment to make sure it would all work together.
• Brought in local Kirby Risk and Rockwell Automation reps to review
drawings and equipment list to verify we were not missing anything.
• Reduced the amount of change orders we had to bring to council.
17. • Maintained 2 radio networks for remote sites
• Old system was a Orbit licensed 900mhz serial radio running at 4800 baud.
• Upgraded radio was Orbit unlicensed spread spectrum 900mhz running at 1 mb.
Old MDS Licensed Radios New MDS Unlicensed Radios
18. • Used AB conversion kits at most existing plant panels to speed up change out and
minimize downtime. Also helped keep cost down from complete rewire.
19. • Increase plant flow to 70 MGD from 53 MGD
• Installed larger primaries 1-4, bar screens, and added secondary effluent lift station
• Replace old, outdated, aging, and failing Process control network at WWTP
• Ran all new fiber ring to each building with new Stratix® switches
• Upgrade all lift station’s, CSO’s, and underground gate structures.
• Including radios and PLC racks
• Replaced with MDS Orbit unlicensed radios running at 1mb
• New cabinets with new CompactLogix® processors
• Separate WWTP and WTP radio communications.
• WWTP is now on unlicensed band while WTP still operates on old licensed radios
• Upgrade plant network and servers
• Updated server to a Stratus Server running Veeam with 8 VM’s
• Add Stratix switch’s to plant network for better data collections and network monitoring in
historian
20. • Increase Productivity and Decrease Downtime
• Add remote troubleshooting and maintenance
• Made as much equipment as possible IP compatible
• Increased radio bandwidth for remote troubleshooting
• Replace aging equipment to reduce downtime and maintenance
• New plant equipment allows it to run smoother with less break downs from aging equipment
• Decrease Overtime/Call outs to reset old equipment
• New remote equipment and PLC’s have cut down almost completely the callouts from faulting
PLC’s or radio failures.
• Replace coding to allow easier troubleshooting.
• New, easier to follow Studio 5000® code has sped up the troubleshooting of any faults for me
with my limited back ground in 5 and 500.
• Have correct programs for PLC’s
• Purchased FactoryTalk® AssetCentre to retain a check in and out process and keep all
construction material in one spot digitally.
21. • Minimize Downtime with Remote site upgrades to maintain sewer operation with minimal discharges
into the river. Also keep safe sewer levels without backing up into basements.
• Worked out a new control strategy that would allow max storage in pipes.
• Set sewer gates to a safe level and set river gates to only allow discharge into river if it reached
control elevation.
• Had floats installed with a switch to allow the LS to work without a PLC to maintain pumping.
• Had radios installed prior to new PLC’s.
• Maintain a safe flow thru plant while bar screens and primaries were being built.
• Split the flow and built bar screens 2 at a time
• Split construction of bar screens.
• Split max plant flow as well to 20 MGD
22. • Maintain 2 SCADA networks during the building process with overlap with minimal downtime.
• Installed new fiber ring to each building. Left old fiber in place until we replaced or upgraded to
new PLC.
• Our old system was 2 ring networks. So we would back feed and bypass on the fiber to keep the
ring intact. Always nice to have extra fiber pairs
• We had 2 servers running, we knew this would make it easier so bought the Stratus server to
make the transition easier.
• Maintain 2 radio networks for remote sites and reduce downtime during transitions.
• New radio allowed for serial and IP to run at same time. So hooked up new radios to old remote
PLC’s and had it feed data to old server. Once new PLC was installed, it was preconfigured for IP.
So we just had to plug CAT5E in and it took off.
• Allowing remote access to integrator to complete work remotely
• We setup remote access for CCI to allow remote work to be done to save the City money and help
expedite any troubleshooting we might need. Worked great for everyone.
23. • Integrate existing equipment and have it be a PlantPAx® Object
• CCI took our existing equipment and made them Objects so all the faceplates looked the same.
Made training on new screens much faster and allowed operators to feel more comfortable with
the new format.
• Bring together existing lift station upgrades into new project.
• With preplanning ahead with a previous lift station upgrade we were able to put the stepping
stones in place for this future project. Allowed us to upgrade panels without wasting money on
PLC until this headworks project. Same went for the radio work. We ran the stations on floats
via a switch until the full upgrade took place to run it via the PLC.
24. • Initial project engineer(not CCI) didn’t understand PlantPAX® to incorporate it into
scope of work
• Main scope of work was already written prior to CCI being brought onboard
• This had cascading effects on OEM screens that were not written for PlantPAX.
• For future projects, this will be the first item addressed in talks.
• Power supply failures
• Lost 4 Stratix switches from power supply failures due to unknown causes.
• All were replaced via warranty from Rockwell Automation. Rockwell sent
out a electrical engineer at no cost to try to figure out the issue.
• Possible causes might have been from electrical fluctuations during
construction but could not be proven. Power conditioning might have
helped eliminated this.
25. • Major flood in 2017 from lightning strike and switchgear failure
• Had no power to plant for 4 to 5 hours.
• Had to run on generators for about 3 weeks and temporary power for 6 months.
• Slowed down the checkout work for CCI since it destroyed equipment that was
part of this project.
• Resulted in about 3 million dollars in damage.
• This resulted in new procedures to be enacted just incase we have issues in
the future.
• Other than these issues, everything went very smoothly thanks to tons of preplanning
that went on because of the CMAR process. Also, being proactive with upcoming
projects to look ahead and make the changes in advance.
26.
27. • Decreased maintenance cost
• Decreased Overtime
• In the first 3 months of operation of 2018 the city saved about 50 hours of overtime with an avg
cost savings of $2,000.
• Decreased downtime plant wide and remotely since upgrades.
• Proper method of maintaining and tracking program changes in PLC. As well as having all files at our
fingertips.
• Now we have a process control standard to allow all future projects to go as smoothly as this.
28. • Complete WTP controls upgrade to make both plants the same – coming soon
• Remote sites as well
• Upgrade SSO(storm sewer overflows) by 2036 – Part of consent decree
• Complete CSO underground basin – Start 2018 for 3 years
• Complete Wet Weather treatment process at WWTP to hit benchmark set by EPA
• Keep working on implementing FactoryTalk® VantagePoint ® to gain data to help
improve the operation of the plant.
• Keep building up FactoryTalk® AssetCentre with documents and pictures from the
project.
• Install ThinManager® plant wide to make access to information faster. Also allow
operators, maintenance, and programmers a faster, more cost effective way to complete
their tasks.
• Look into FactoryTalk® TeamONE™ app