Water Pollution Agency
Monterey Regional Water Pollution Control Agency, a leader in beneficial reuse of domestic wastewater, needed improved regulatory compliance and system reliability at its recycled water facility. After migrating to Plant PAx DCS and FactoryTalk analytics software, the agency realized a 5-year ROI, simplified regulatory reporting, and solved system-wide pressure control issues, saving millions of dollars.
4. Personal Experience
• Five Years of Navy Nuclear Submarine Service
• Nine Years of Wastewater Operations
• Three Years of OT/IT Integrations and Data
Management
• Two Years of PLC Programming
• Three SCADA System Upgrades
• Advanced Water Purification Plant Design
5. MRWPCA
• Who are we?
• Where did we start?
• Planning SCADA Upgrade
• Project 1 – SVRP
• Project 2 – CSIP
• How did we benefit?
• What did we learn?
• Where are we going next?
6. Who Are We?
• Joint Powers Authority
o 11 Member Entities
o 9 Cities Served
o 250,000 People Served
• Collection System
• Regional Treatment Facility
• Salinas Valley Reclamation Project
• Castroville Seawater Intrusion Project
• Salinas River Diversion Facility
• Pure Water Monterey
10. Where Did We Start?
2012
• All PLC-5®’s and SLC™ 505 processors
• Four RSView®32 Projects
o Running on Physical PC’s under the Control Room Desk
o Windows XP Based
o No Backup Power
• 5 Minute Scan Times Across All Radio Networks
o Falsified Com Status
o Irregular Control Ability
• Data Logs and Manual Data Parsing
o SCADA Network Exposed to Internet
o Data Transfer Unreliable
• Custom VB Scripted Popups (Many broken due to Faulty I/O or addressing)
11. Planning
• New SCADA/IT Master Plan in 2013
o Consultant Rated us a 2 out of 10
o Labeled our SCADA System a DA System
o High Risk System
• Rockwell Automation Performed an
Installed Base Evaluation
o Silver and Obsolete Products
• Utilities Supervisor Went to Rockwell Automation TechED
in San Diego
12. Planning
• Multiple Site Visits throughout Northern California
• Visited the Rockwell Automation PlantPAx®
Demonstration Facility and Lab in San Ramon, CA
• Attended Three Day Trainings on:
o FactoryTalk® Historian
o FactoryTalk® View
o FactoryTalk® VantagePoint®
13. Laying The Foundation
• Rebuild Networking and Control Infrastructure
o Pull New Fiber Optic
o Improve Radio Communications
o Upgrade PLC’s and I/O
o Virtualize the System
o Upgrade to FactoryTalk® View SE
o Upgrade to Server 2008 R2
o Network Hardware and Storage
14.
15. Recycled Water SCADA Upgrade
(Project 1 – SVRP)
Five-Phased Approach
1. New Fiber Optic Runs
2. Upgraded PLC-5® to ControlLogix® L73 Processor
3. Conversion of Remote I/O to Logix Platform
4. Direct Conversion of RSView®32 Project to FactoryTalk® View SE
5. Full HMI Upgrade Project Using PlantPAx® & Data Upgrade
16. SVRP PLC Conversion
• Moved Code Directly into L73
• Near Disaster of a Transition
o Promised Operations a Seamless Transition
o Scanner mode resulted in incorrect addressing
o Unable to produce recycled water for 22 hours in June
o Dramatically Increased System Speed
17. SVRP PLC Conversion
Lessons Learned
• Scanner mode results in shifting addresses for
Outputs (Only Scanned the Inputs)
• All Timers need to be adjusted for 1000 ms time
base versus PLC 1 second Time Base
• Never Ever Use the Word Seamless with Operations
o Under Promise & over perform
• Have a Solid Back Out Plan
18. Conversion of Remote I/O
• Performed During the Offseason (Plant Offline)
• New Ethernet/IP Drive I/O reduced installed I/O by
one whole rack
• Speed of the system further increased
• Able to physically move the PLC from the Plant to
the Admin Building
19. Direct HMI Conversion
• RSView®32
o Windows XP
o Physical Workstations in the Control Room
• Rockwell Field Services Built our FactoryTalk® Directory
• FactoryTalk® View SE
o Redundant Virtual Servers
o Increased Reliability and Speed
20. Direct HMI Conversion
Problems
• VB Based Popups
o Rewritten to Work
o Did not take advantage of speed
• Had to Leave RSView®32 Project Running for Data
Capture
• Some System Functionality Was Lost
o Derived Tags
o HMI Tags
o VB Reporting
• Significantly Eroded Operator Trust in the System
21. Full PlantPAx Upgrade
• HMI and PLC Standards in a Box
o Uniform Color Scheme
o Uniform Popups
o Uniform Code
• L73 Allowed Swapping of One System at a Time
o Boolean Controller for toggling between old code control and new code control
o Quickly swapped to insure high reliability
• Modernized Capabilities and Screen Aesthetics
• Fully Utilized PLC Capabilities
• Fully Integrated 525 and 753 Drive Capabilities and Data
22. Full PlantPAx Upgrade
Benefits
• Auto Tune Control Responds Better & Saves Money
o Use Less Chlorine Gas
o Use Less Energy
o Use Less Polymer
• Uniformity reduced training requirements
• PlantPAx® Capabilities for Maintenance and Built-in
Security increased system reliability
28. Polling Master Conversion
• Polling Speeds Decreased from 5 minutes to 30 seconds
• Increased System Reliability
• Data Concentration for FactoryTalk® Historian and
FactoryTalk® VantagePoint®
• Applied Lessons Learned from
SVRP, RTP, & PS Upgrades
29. CSIP RTU Upgrade
• Ubiquity 5ghz Radios
• CompactLogix™ Treated as Remote I/O
• Polling speeds decreased to 400ms
• Robust Coms and Com Monitoring
• Historian Data Provided Real-Time Monitoring
31. How Did We Benefit
• Tighter Faster Chemical Controls = $25,000/Year In Chemical Costs
• PG&E Energy Audit Yielded No Further Energy Savings Suggestions
(System is Fully Optimized)
• Reduced Project, Training, and Maintenance Costs
• Negated Need for Booster Station Controls Upgrade Saving $3 Million
• Automated Regulatory Reporting and Publicly Consumed Data
34. What Did We Learn
• Have A Test Environment If At All Possible
• Have Extra PLC Capacity For Code Conversions
• Back Everything Up Regularly
• Have a Back-Out Plan
• Under Promise and Over Perform
• Trust The Derivative Smoothing
35. What Did We Learn Cont.
• Rockwell Automation Has Great Customer Support
(Crappy Hold Music Though)
• If Possible Complete Planned Ethernet/IP Conversions
Prior to Remote I/O Upgrades
• Be Aware Of DHS Critical Infrastructure Guidelines or
Other Regulations During the Planning Phase
• Training is Key to Stakeholder Buy-In
37. Pure Water Monterey
• Critical Indirect Potable Water Reuse Project For California’s
Water Challenged Central Coast
• Applied All Lessons Learned From Upgrades to Design
• Full PlantPAx® facility From the Ground Up
• All Ethernet/IP and Profibus I/O
• Centralized L75 Redundant Processors
• Reduced Programming Costs Approximately $100k