1. ZDT Group LLC
5460 Sandstone Ct
Cumming, Ga. 30040
P – 770-886-9555
www.zdtgroup.com
By Joe Soroka Copyright 2009
Maximizing Uptime through Predictive Analysis
Operating Mission Critical Facilities in today’s environment means continuing to strive to
meet and exceed our uptime requirements, as our operating budgets continue to shrink.
Taking a holistic approach is needed to track our facility information to accomplish this.
As I have discussed in previous whitepapers and other presentations, uptime is achieved
through a process that I call “RAMPS”. The tier rating of a facility is only the first thing
that affects uptime. You may have heard stories of people who have designed and built
tier IV facilities and have had outages time after time and where a tier I facility has been
operating for over ten years without a single outage. That’s because there is more than
just the design of the facility that affects the facility uptime, and that is “RAMPS”.
Reliability, Availability, Maintainability, Predictability, and Scalability are the keys to
success. Without paying attention to all facets of the facility, your uptime requirements
cannot be realized.
Reliability Center Maintenance (RCM) can be used to help in meeting the demands of a
shrinking operations budget. If you do not properly implement a RCM program it is
called a “LMP” Lack of Maintenance Program not RCM. However, to correctly implement
an RCM program you need to answer 7 questions per SAE JA1011 Evaluation Criteria for
RCM process. These questions are as follows:
1. What is the equipment supposed to do and what is the performance standard?
2. In what ways can it fail to meet the performance standard?
3. What are the events that will lead to failure?
4. What happens when the piece of equipment fails?
5. In what way does each of the failure modes matter to system operations?
6. What systematic task can be performed to prevent the failure?
7. What must be done if a suitable preventative task cannot be implemented?
Prior to implementing an RCM program, one must have a comprehensive Predictive
Analysis program in affect. By understanding the past performance and current operating
conditions of the equipment, we can develop predictions on failure modes for the future.
Tracking more items with a longer duration will result in a greater accuracy in our

2. ZDT Group LLC
5460 Sandstone Ct
Cumming, Ga. 30040
P – 770-886-9555
www.zdtgroup.com
By Joe Soroka Copyright 2009
predictions. This is why it is important to start at the beginning of the project with
design, moving to construction, commissioning and then ongoing operation and
maintenance. The information that is gained in each of these phases needs to be
captured in a manner that allows you to look at specific details across the entire life of the
equipment.
Data gathered without purpose is just that, data. It is important to define how you need
to use this data, and how it can assist you in your operation - from increasing reliability to
reducing operating costs. When in the design phase, the requirements that the facility
needs to operate at are explained in the Basis of Design (BoD) and the Sequence of
Operation (SoO). It is these two documents that need to be fully developed in the design
phase and updated during the life of the facility, for they act as the road map on how the
facility shall operate. With these documents a baseline is established for the performance
of your equipment, and when the site is started up and commissioned it is this baseline
that the facility needs to operate at. During the startup and commissioning phases these
baseline documents are updated so that changes that may have occurred during
construction and startup phases are captured in the updated BoD and SoO documents.
These documents need to be living documents and should be reviewed and updated as
often as required.
The information gathered during the commissioning process is typically never properly
integrated into the operations. Having the data that was discovered during startup and
commissioning is invaluable. Not having this data would be similar to taking all the
photos of your children prior to their high school graduation and locking them up
somewhere. Then years later, when you are sitting down with your son or daughter’s
future spouse, trying to show them your child’s life story, you have to begin with high
school pictures due to lack of data. All of the data collected during birth, startup, and
commissioning is part of the equipment life story. The whole story needs to be looked at
and shared between the parents and the spouse. As the spouse adds new photos to the
album, the story of the person’s life is evolving and you are able to look through the
album and see the entire story.
Predictability is one part of the Uptime RAMPS that often gets ignored. We may have
some trends we look at, and some testing that allows us to do some forecasting, but many
times having a fully predictive program is lacking. The data gathered and collected
during construction and commissioning should be fully integrated into the operation and

3. ZDT Group LLC
5460 Sandstone Ct
Cumming, Ga. 30040
P – 770-886-9555
www.zdtgroup.com
By Joe Soroka Copyright 2009
maintenance program. This data becomes invaluable to you as you develop a
predictability program. Imagine your data center is fully loaded with servers and you are
operating at the design load of 10kW per cabinet. The CFD modeling you did with your
heat load is proving out to be correct and everything is operating correctly. You decide to
leave early for the fishing trip up in the mountains that you had planned for some time
now. As you are enjoying your well-deserved trout fishing you realize how cool it is and
you think “wow it must be hot back in town”. You pull out your cellphone to check the
weather in town and you see there is no cell coverage. You don’t worry, you’re not on call
and you have a good staff at the site, so you go back to fishing. The fish are biting, there
is great weather, and you decide to stay late on Sunday.
When you come home Sunday you are shocked by the amount of voicemails and emails
you missed. It’s the one from your team that really hits you hard in the stomach. “Boss
we not sure where you are and we hope the fishing is good but we just lost the site, the
load dropped, all servers are down”. You immediately turn your car to the site and start
calling everyone. When you arrive you spend the next couple of hours explaining where
you were and trying to figure out what happened.
The next day you review the incident reports and start to put together a detailed
explanation of why your chiller plant failed, so you can complete a failure analysis report.
When the failure occurred, both your primary chiller and redundant chiller failed due to
high head pressure caused by high condenser water temperature. The cooling tower fans
were off and would not start. The Tower Fan VFDs were working and they tried to put
them in hand and still the tower fans would not operate. It took your team some time to
figure out the problem, but in the time it took to correct the issue the data center floor
overheated and servers started to fail. It was the vibration switches that were mounted
on the fans that tripped off. Three of the four tower fans failed and the fourth fan could
not support the load. It took some time to identify the problem but once it was
discovered the vibration sensors were jumped out the fans started and the chillers were
brought back online, but unfortunately not in time to keep the site up.
You remember that you were the only one on the job when the startup and
commissioning occurred, the rest of the team was hired after the site was turned over.
You remember something about those vibration limit switches during startup 3 years ago.
You look in the startup documents and there is no mention of any issue so you make
some calls and track down the person who did the startup. You reach this person and ask

4. ZDT Group LLC
5460 Sandstone Ct
Cumming, Ga. 30040
P – 770-886-9555
www.zdtgroup.com
By Joe Soroka Copyright 2009
him if he remembers what the issue with the vibration switch was, and as luck would
have it he remembers that project. He tells you “Yes, we had a lot of issues with those
switches during the startup, we replaced one and the others were adjusted.”. There is no
mention of the parts that were replaced during startup. If it was noted and was listed in
your system you would have known you had a failed device and you would have looked
into the issues during startup. No matter if a part fails in the factory, during startup, or
during the life of the equipment tracking, those failures and analyzing the nature of the
failure will allow you to predict future issues. During the PM visits since the site was
turned over to Operations, the vibration switches were never part of the maintenance
procedures so they were not tested or adjusted. This lead to the switches tripping prior to
any significant vibration. The Operation team was never trained on the vibration switch
and did not know they were installed or what their function was. Many time equipment
accessories are missed in training and preventative maintenance.
A week later you are at home. You get that dreaded phone call and rush back to site.
This time it’s the UPS system Module A. It has failed and dropped offline, however the
site remains protected by the other modules. You call out your service rep and they find
that the AC filter capacitors failed. Again you sit-down in your office the next morning to
fill out another failure analysis report, and in the process of reviewing past maintenance
records you notice that the AC filter current that is measured and recorded each visit was
about the same - up to the last PM when it was 35% higher from the previous readings. It
was three months ago when those AC Filter Capacitors were telling you it was time to
replace them, but the data was tracked in a manner that couldn’t be analyzed.
That evening at home you start thinking about how much information you were able to
gather from the past documentation that you used to complete your failure analysis
report. Why not use the data to predict what might fail rather than explaining why it did.
Besides, a failure analysis report is just like a predictive analysis report, it’s just after the
fact. So you start to identify your potential issues and figure out a way to capture and
analyze the data.
When preforming startup and ongoing maintenance the testing scripts need to be written
in a switch level detailed fashion. The system that you use should be in a database so you
can analyze the data and set thresholds. For example, do not write a test script that states
“check the oil pressure” and just have a checkbox next to it. First of all, a checkbox is not
sufficient data. You might have checked it but if the oil pressure was 2PSI, I’m sure that

5. ZDT Group LLC
5460 Sandstone Ct
Cumming, Ga. 30040
P – 770-886-9555
www.zdtgroup.com
By Joe Soroka Copyright 2009
your engine will not like that for very long. Your test script should have stated “Record
the oil pressure” and it should list a tolerance level. You inspect the oil pressure, it’s 35
PSI, and you record it. Your system should have a min max tolerance and if it is outside
the acceptable reading, you should get a failed condition and an issue log should be
automatically updated. Also you should have a % changed for a failed condition. This
averages all of the past readings for the oil pressure and results in a failed condition if the
percentage is x% outside of the average reading. Another approach could be rate of
change, a failed condition which occurs if the rate of change is x% from the last reading.
With these three types of automation added to your logging tool, you will start to capture
and analyze items automatically that will allow you to predict failure rates.
With a database as the backend for your commissioning and operations tool, you can
intelligently analyze critical points over a period of time. A set-point can drift over time
and may not be noticed because of the small changes from week to week, but with the
proper automation tools these points can be analyzed and alarms can be generated
automatically. You could have your building automation system run a script every
morning for the critical points that you are trending, and then write the result to a XML
file or a SOAP report that your automation tool could read as input for a daily PM task.
Now, not only is your team performing site inspection and maintenance at the site, but
your Building Automation System is talking directly with your operations automation
tool (or CMMS), and you are gaining valuable data that is being analyzed automatically
and alerting you of potential issues.
Joe Soroka has been working in the mission-critical field for the past 26 years. He has
worked with many clients over the years with conceptual designs, commissioning,
operation and maintenance and failure analysis. Joe has commissioned over 8 million
square feet of mission-critical facilities. He has worked on developing and reviewing
Operations & Maintenance, Training and Safety programs, including Method Operating
Procedures for mission-critical facilities. If you have any questions you may contact Joe
Soroka at joe@zdtgroup.com or visit www.zdtgroup.com