Presentation at Oracle Open World 2004 (Paris)
Showcases the usage of Mobile Access to operators on the Refinaries, helping to improve the operating integrity in processing industries
Good afternoon ladies and gentlemen, thank you for joining us.
Please let me introduce myself; My name is Coos van den Berg and I work for Cap Gemini Ernst & Young in The Netherlands
and my colleague Léon Smiers and I are going to tell you the story of OTTER.
This is a story of how we, by using Oracle technology, successfully closed the management information loop
in the complex world of 7/24 operations in the refineries of Royal Dutch/Shell and by doing so helped reducing the operating cost with double-digit percentages.
We will tell you about the problem with Operations Integrity in the processing industries
We will tell you how we solved this problem by combining the knowledge and experience of three innovative companies
Shell Global Solutions,
Cap Gemini Ernst & Young
And Oracle Corporation
We will give you an insight in the whole process of analyzing, developing and implementing the solution
The first ten minutes I am going to tell you about the business case and its origin.
I will explain the problem and it’s setting.
I will tell you about the goals that were set for this endeavor and the glittering prizes that were won.
Then Léon will take the floor and he will tell you about the hurdles that we had to overcome.
He will take you through the architectural and technical issues.
He will tell you about the functional and technical requirements that had to be met.
What choices were made and which Oracle components were used.
He will warn you for some pitfalls and share his experience.
He will end with a wrap-up of the highlights of this live case.
This all will keep you hopefully entertained for about forty-five minutes and so we will still have some time left for questions and answers.
Let us start with the Business Case
The Business Case originated in the Shell refineries but is applicable to all processing industries.
I will tell you about some common characteristics which lead to the problem definition
I will tell you about the practice-based origin of the problem and its consequences and how this lead to the solution definition.
I will paint a generic picture of our solution of which Léon will tell you a lot more in detail
And I will finish by telling you some of the benefits, both qualitative as well as quantitative, Shell is enjoying since they started using this technology.
The Process industries share three common characteristics in their manufacturing operations set-up
They are all Asset intensive
Which means that all installations consist out of large, expensive and dedicated assets
They work with Long life-cycles of these assets
And they require large investments
They have a Centralized manufacturing process control which is
Typically managed through Distributed Control Systems and panel operators
It produces Real-time information on key process data
But it gives limited information on the actual condition of the manufacturing assets
There are Extensive field operations needed to keep the assets running
These field operations consists of large number of non-manufacturing process related tasks (inspections, checks, measurements)
They cover both preventive and as well as corrective maintenance
And they are typically performed by field operators and maintenance engineers
In general it is Field Operations that are striving for Asset performance and availability and Regulatory compliance which in one phrase we call Operations Integrity.
But let’s take a look in real life and take into consideration that even a modest facility or installation contains a large number of assets which require numerous monitoring and checking tasks and that those tasks have to be performed on a regular basis and most of them with a relative high frequency.
Then we get a more clear picture of our problem:
That is that demonstrating compliance with Operations Integrity will bring about that more than 1.000.000 tasks per year per plant need to be executed and managed
That is quite an undertaking and when we look at today’s practice on the average work floor of the average installation we see that
Only a small portion of these tasks are being executed, That it are often the same tasks
That there is poor recording, And last but not least, there is hardly any analysis
The harsh reality is that this all results in failures that could have been prevented.
And mind you, we are talking about serious failures like
Unplanned shutdowns, or otherwise created poor availability of your operating assets
Upsets and disturbances in your operating process resulting in off spec products
Sub-optimum operations which result in, unnecessary fast degradation of equipment,
catalyst, waste of utilities, etc. Well, to simply sum it up : You loose money and in a big way!
Another reality fact is that by nature of the problem you are not capable to proof
compliance to requirements and regulations of all kind of stakeholders.
Not only company related stakeholders like management and shareholders with a more familiar set of needs and wishes but over the last few years there has been an increasing demand from external stakeholders.
For instance: It is no longer sufficient to pledge allegiance to the good cause of Safety,
Health and Environment; authorities will demand proof of your actions in these fields, and right they are!
And especially in case of incidents one is forced to provide proof of compliance to regulations and is no longer “presumed innocent until proven guilty”.
When we drill down further in this matter we find that the most immediate threats to Operations Integrity start on the shop floor.
The operators on the shop-floor do not have full information on all critical tasks to be performed
The do not know exactly what they should do and how they should do it
The do not know what the effect is of what they do
The do not know what they don’t do, but what should be done
And they do not know what the effect is of what they don’t do
There are several reasons why this happens ...
As I already stated; there is real time data on process information but insufficient real time data on critical equipment also because of missed inspections
The manual handling of measurement and inspection data takes significant time to process
There is a large knowledge drain due to layoffs and this has been going on for almost a decade now
And also a very plain reason; there are often Insufficiently structured plans of work
The end result is that management ...
Does not know what actually happens on the shop-floor
And has no means to implement decisions effectively at the operational level.
So what we are actually saying here is that management is not in control!
On a more aggregate level management has no information on their position in Operations Integrity, either stand-alone or comparative to others
This brings to the challenge at hand
Since it was clear that it was essential to be able to ensure Operations Integrity, it became Shells’ Challenge to create an effective way of monitoring shop floor processes to do so.
As to where it CGEY concerns; it became our task and challenge to
Create a supporting solution that ensures that all operational and technical tasks related to Reliability, Safety, Environment, Product Quality, Maintenance and Utilities are executed effectively, efficiently and timely
Well, that is quite a wide field of responsibilities but for apparent reasons I will give a birds eye view of our solution:
OTTER in a nutshell and I will really try to keep this short.
OTTER brings the information of the back-office systems at the fingertips of the operator.
By combining the equipment information, currently stored in ERP systems and plant control systems,
with all known best practices describing what tasks have to be carried out on the equipment, one is able to create templates of tasks and task lists.
Out of these templates specific tasks and task lists are either automatically generated or manually created
Subsequently, the location information is added to these tasks and a planning is continuously made
enabling the creation of efficient rounds for each operator.
Then the information for the defined round will be downloaded on a handheld computer in the format of tasks and task-lists,
which describe the work to be done by all stakeholders, e.g. field operators, maintenance engineers, EHS inspectors.
After the work is done, measurements, observations and all the other relevant information is
uploaded to close the information loop and provide a solid base for reporting and analysis.
I can not do this shorter than this and as promised will Léon elaborate on all
this.
So when you stay with me for one more slide I will tell you about the glittering
Prices that were won…
Out of the experience we now have we can give you the following numbers on what the impact is of using a solution with this technology in the refinery world of shell.
These figures are based on one of shells’ refineries where they execute with the use of OTTER more than 3,5 million tasks per year!
They have established
The Mechanical availability of the installation improved from 93 to 98% That comes down to 18 more days of production per year
Double digit manpower reduction on the shop-floor, doing the right things on the right time and place leaves a lot more time to be spent on other important tasks
Basic Equipment Care by operators increased from 12 to 24 % , so now they are actually taking care of your investments for a quarter of their working time
Bad oil in pumps reduced from 30 to 5% of pumps
A reduction in operating costs in the range of a 10% - 15%
Increase of margin in the range of a 10% (US$ 40 ct per barrel)
On the level of qualitative benefits I would like to state the following
By really closing the management information loop the use of OTTER enhances
Predictability: By having a clear picture of what has to be done one can predict future capability of facility and operating organization
Agility: One can set direction, mission to achieve new position in time, in income, reputation, HSE
Manageability: On can set and stick to target and milestones for processes in the production cycle
Knowledge Management: One can keep, seek, develop and implement own and others know how
Control: Now one can asses risks, set strategies, effectuate controls and objectives in operating window safeguarding stakeholders interests
Well, now I am going to leave you in the capable hands of Léon who will tell you how it all became reality. Thank you, Léon the floor is yours.
[LÉON]
Thanks Coos for explaining the business case of Otter to us.
My name is Léon Smiers and I work also for CGEY in The Netherlands. I am involved in the Otter project for some 4/5 years now.
I am going to lead you through the rest of the presentation where I will explain what requirements, functional as well as technical, lead to the architecture used, and the implementation of the two main Otter components, the administration and the Operations. For the Operations part I will show you how the operators work on the refinery with their PDA’s, how work is uploaded to the PDA, the work is performed on the PDA and the work results are sent back to the Otter server again. All with the help of Oracle Lite, the 9iAS application server and (of course) the database.
If there are questions during the presentation, don’t be shy to break in.
Within Cap Gemini Ernst & Young we use the Integrated Architecture Framework as a standard for setting up an architecture.
Four aspects areas are investigated, the Business, The information, Information Systems and the Technology Infrastructure. For all of these areas the following questions are investigated
Why are doing it
What are trying to accomplish
How do we want to accomplish it
When
This results in a complete set of Architectural Principles
Coos explained to you the Business and Information part, I will focus on the Information systems and Technological Infrastructure.
We start of with the requirements for the overall architecture, requirements on a high functional level. What and Why are trying to accomplish.
Shell wants to have asset information available all over the world, this requirement contains two dimensions.
First Shell wants to be sure that all the assets are named and used in Otter in the same way. The measurement limits for a certain asset, for example, should be set the same in all refineries.
Secondly Shell wants to be able to monitor the behavior of the refineries and have the possibility to drill down to the level of an asset, and for more then one refinery at the same time. Currently we have the Otter application live on three sites, where Otter is implemented locally. The aim in the near future is to have Otter implemented on more refineries and centrally hosted. For instance one set of host servers for Europe, one for the Americas and one for Asia. Then Shell has the possibility to investigate the behavior of one certain asset for all refineries in Europe.
Another set of important requirement is,
There should be no restrictions on
amount of users in the system,
The administration part should be able to handle hundreds maybe thousands of users at the same time. The administration application is used throughout the whole day.
The workload for the operation part is during work shift handover. In a small timeframe operators are sending data between the PDA and the central server. The system should be able to handle several hundreds of data transfers in a short time, preferably in seconds. When the shift is over the operators want to go home don’t like lengthy handovers, that’s not good for the acceptation of the application.
There should be no restrictions on
amount of refineries, installation in the system, as I just told, the aim is to have Otter hosted for several sites one location. Adding a new refinery should not give a lot of problems.
There should be no restrictions on
Used technologiesThis requirement also contains different dimensions.For instance the Otter application should be able to integrate with back-end systems such as SAP.Another example is, currently for the operations part, an off-line application is used on the PDA. I will explain more about that later in the presentation. We started off two years ago with the development of the PDA’s, at that time, going on the refinery with a wireless PDA was absolutely out of the question, because of security concerning hackers and safety. Now Shell is investigating the possibility of wireless connections on the sites, so maybe in one year time Otter should be able to run wireless. We should choose the platform which can handle such a change of technical Another All
These requirements have one thing in common, we have to allow a grow path in all dimensions for our application
On a high level the Otter application contains two main components.
The Administration part, which contains administration for the
assets, what assets are used on a refinery, what measurement limits have to be set for an asset
the work, what tasks need to be performed on a certain day
setting up users and authorizations
The second aprt of the administration component is the reporting and analysis of measurement data.
The actual work by the operators on the refinery is done with the operation component.
The operators select the work they have to perform,
download the work on the PDA,
record measurements of assets and problem events on the Otter application on the PDA,
and when the work shift is finished send the measurements and notes to the central server.
First I am going to talk about the Administration part and then the operations part where I will explain how to make it possible for the operator to work with a PDA on the refinery.
With 85 entities and 140 modules (Forms, reports, web pages) the Otter application is a medium sized application. The application contains complex entry pages and some of them have the possibility to do uploads from Excel spreadsheets. The amount of data to be processed however is not medium sized .As Coos mentioned earlier, the volume of the data is big, over a 1.000.000 tasks a year which have to be processed.
The main requirements for the Otter application is that the application should be
stable, now and in the future,
Able to support the whole business process
should be able to handle the large volume of data and
extendable to hundreds maybe even thousands of users.
We started off 4 years ago with the combination of Oracle Designer/Developer in client/server mode. Two years ago we moved from Client/Server towards Internet Computing.
But looking at the Administration requirements, there is no need to go to another Development platform for the administration application.
Oracle Designer/Developer is here to stay for a while, and actually it is booming again the last time.
ORACLE FORMS IS AS TRUSTWORTHY
AS YOUR BUSINESS SHOULD BE
Here are the product versions we used for the development and implementation of the Administration part.
We had to use 9iAS Release 1 because Oracle Forms version 6 is only certified against 9iAS Release 1. Let’s go to the next slide to see the implementation of Oracle Forms on the 9iAS.
We use Oracle Forms deployed as a servlet on the JServ java application server, which is part of 9iAS R1. It is a rather simple deployment. First you need to install Oracle 9iAS R1 and Oracle Forms and deploy your application (the forms and report files) in a certain directory. Then you have to change a couple of property files in Forms and on the Jserv Application Server where you let JServ know where to find the Forms servlet and where to find the application files.
After restarting your application server you can start a browser and point to a start HTML file which contains definitions how to start forms as an applet.
Oracle Forms is started from within the browser, which contact the Apache HTTP server. Apache redirect the call to the JServ Java application server, which in turn contact the Forms Listener to let it know that a browser wants to open a Forms session. The Forms session starts a forms process on the server and contacts directly the Browser process. The browser process can be started via native java applets in the Internet Explorer or via Jiniator. The last option is preferred because of caching and a more stable configuration.
From this moment on the communication is performed directly between the Forms Application in the browser and the Forms Listener.
Now we go to the interesting part.
Be able to monitor the refinery, by getting measurement which the operators enter in the Otter application on a PDA . But without making it a difficult task for the operator.
Let’s start of with the technical requirements of the operations work.
The operators go in the field with a lot of equipment, which can add up to some kilos. They have to climb ladders, go in small rooms etc. So the last thing you want to give him is a big heavy equipment like a laptop or even a tablet. So the PDA should be small and light. But also the PDA should be rugged, the operators are not known to be ballerinas and when they let a PDA drop, which happens, the PDA should still function. We even had a case that an operator was on a ladder and let, by accident, drop the PDA from a height of about 20 meter, afterwards it still worked! The refinery contains areas where gasses can occur which even can cause explosions, the operators normally enter those areas with gasmasks on. Since safety and health are huge requirements and regulations on refineries the PDA should be explosion proof and certified by certain authorities.
The PDA market is fast moving market, types, operating systems, memory, possibilities, it all changes rapidly and we should be able to change easily towards a different PDA type.
Wireless is not allowed on the plant itself. There are three reasons for that,
fear of hackers
Safety and
No 100% coverage on the refinery, in closed areas for instance there is no coverage.
At the moment experiments are starting with hotspots on the refineries.
The only place where wireless is allowed is in the central control room, where the operators start and end their shifts.
The last requirement has a big impact on the type of PDA application
Here are the two main type of PDA applications,
On-line and Off-line
An on-line application is directly connected to the central server, the data is retrieved and written from and to the central database. The application is installed centrally.
An off-line stores the data locally, the only connection with the server is to synchronize the data on the PDA and the central server. Because the data is stored locally a local application needs to be developed and deployed.
Since wireless is not an option on the refinery we have to use the Off-line type of application.
Oracle Lite is an off-line type of application which synchronizes data between the central database and a local data storage on for instance a PDA. The reason why we choose Oracle Lite are the following
Oracle Lite supports a variety of PDA types, PocketPC, Win32, Palm, Epoc. Since the PDA market is changing rapidly we want to have a synchronization platform which can handle various PDA types.
The aim of Oracle Lite is to perform the synchronization between the central server and the PDA as quickly as possible. And even for as may users concurrently as possible.
For that reason Oracle Lite is an asynchronous application and split into two parts
First a sending process, which builds up a connection between the central server and the PDA and sends/retrieves as fast as possible data.
Second a process on the PDA and the central server which processes the data which is retrieved.
I’ll will describe the details of this asynchronous behavior later.
An application as defined in Oracle Lite contains a set of rules, concerning
With what PDA type to synchronize data with
What tables from what database need to be synchronized
Rules concerning data synchronization,
fast refresh versus complete refresh,
in case of data changes on the same record on the PDA and the database, who wins.
The possibility to do selective synchronization of data linked to a user
The user management of Oracle Lite contains an authorizations part which enables login facilities and a authorizes usage of an application.
All these qualities of Oracle Lite make it fit for the purpose of the Otter application.
Because the choice is made for an off-line application the work process is as follows
First and operator has to select the work he or she wants to perform
Then the selected work needs to be downloaded to the PDA
The operator can now go on the refinery and performs the work and enters measurement data on the PDA
When the shift is over the operator sends the measurement data from the PDA to the central database and the PDA is reset again.
There are three parts in this work process where a logon is required, during the selection of work, the load of work on the PDA and during the send measurement back from the PDA. Shell required that only one logon should be enough.
We use the following products to implement the work process for the operator.
For the selection of the work we use the combination of Struts/TopLink which we deploy on the 9iAS R2, only the J2EE/Webcache combination is used.
Oracle Lite is used to synchronize all the data between the PDA and the database. On the PDA we use an application developed in C++.
Let’s see how the implementation is performed for all the 4 steps of the work process.
First the selection of work.
On the right hand side we have the database in which we store the actual Otter data and the Oracle Lite repository. We placed the data, we want to synchronize between the PDA and the Otter database, in a separate storage area. The reason for that is that we want to have a clean separation between the real Otter data and the synchronization data.
The Operator starts his (or hers) work in the central control room on the refinery. This is the only place where a wireless connection is allowed.
The selection of the work and the next step ‘load work on the PDA’ is performed here. We choose to perform this work wireless, in order to speed up the work. The other possibility is to put the PDA in a cradle and work from there.
First the operator has to select the work which he wants to do during the shift. From the application on the PDA a browser is started which points to the selection of ‘work’ web application. This is a workflow process in which the operator has to log on, choose one or more plants he wants work, choose a set of tasks and additionally a route. The URL call is picked up by the 9iAS R2 Apache HTTP server. The HTTP server notices that it is a call for a Java web application and redirects the call to the 9iAS R2 Java application server, which is Oracle Containers for Java, also known as OC4J. OC4J executes the applications, which contains the combination of Jakarta Struts and TopLink.
Jakarta Struts is an open source software layer around the Java Servlet technology, which enables easier development for web applications with a sequence of pages to be handled.
TopLink is an Object – Relational mapping tool which enables to map Java objects to objects in the database. Oracle has bought this product over a year ago. I’m very happy with this tool because it simplifies programming a lot. You don’t have to program the mapping, making connections to the database, connection pooling etc.
In the setup of our Otter web application we let TopLink know to use of the DataSource part of the application server. This part takes care of creating connections to the database and connection pooling.
OC4J starts the Otter selection work web application, which retrieves information from the Otter tables and shows for instance the available tasks in the browser on the PDA. In the browser on the PDA the operator is able to select one or more tasks, after pressing OK in the browser these selected tasks are then inserted (also via TopLink) in the storage area in the database. These records contain a reference to the username of the operator.
After finishing the Otter selection work web application, in the Storage area some 10 tables are populated with information concerning the work the operator has to perform.
During the ‘selection of work’ process on the PDA side, the PDA is reset and all existing data is removed from the PDA.
Now the operator has to download the work from the Storage are to the PDA.
As I mentioned in the slide with Orace Lite, Oracle Lite consists of two components
The first process is Webtogo, which builds up the communication between the PDA and the server and sends (as fast as possible) the changed data between the PDA and the database.This process contains a safety valve. If halfway during the synchronization the connection fails between the PDA and the server, no changes are applied to the database and the PDA, and no data is lost.
On the server side two queues are used by webtogo to up-and download data from.
First the IN queue. Webtogo puts all the retrieved data from the PDA in this queue.
Second the OUT queue. Webtogo uses this queue to send data to the PDA.
The second process in Oracle Lite is the Message Generator and Processor (MGP) process, which performs two actions
Moving data from the IN queues to the database, this is called the apply phase
Moving changed data from the database to the OUT queue, this is called the compose phase
The MGP process is a batch process (Java) which is running in the background.
In the previous action the ‘selection of the work’ we have seen that data was inserted into the storage area. This changed data is picked up by the MGP process and put into the OUT queue.
After finishing the selection process in the browser on the PDA, the operator starts the synchronization process from the PDA. This starts a communication process with the webtogo process.
On the PDA no changed data is available, since we started with ‘empty’ database. So webtogo only sends data from the OUT queue to the PDA.
There is one exception in the story I just told, when the refresh type for a table in the Oracle Lite application is set to Complete refresh webtogo retrieves the data directly from the table involved.
When the data is loaded on the PDA, the operator can go to the refinery and perform the work. The application on the PDA contains a workflow part and a possibility for the operator to enter measurements and comments.
During the initial development phase, three development environments were investigated
Visual Basic, C++ and Java.
We choose C++ because of the small footprint on the PDA. The application we use has a size of some 600K.
When the operator has finished the shift he (or she) goes back to the central control room and clicks on the PDA to send the measurement data back to the database. Again webtogo is contacted and a connection is set up between the database and the PDA. Since no changed data is available on the server for the operator only data is send from the PDA to the IN queue. After finishing the webtogo process MGP picks the data from the IN queue and processes this data into the storage area. When this is done automatically a PLSQL procedure is started in the database to process the data from the storage area into the Otter database.
This process caused us some headaches because we had to find out when the MGP process had finished the processing of the data into the storage area for this particular user. Unfortunately there is no trigger or interface in the MGP process so we had to find a way in the Otter Lite repository, and now works ok.
Now that the measurement data is available in the Otter application, reporting and analysis can be performed on that data.
That brings us to the conclusion of the presentation