SlideShare a Scribd company logo
1 of 12
Download to read offline
1
Software and Data Integration Strategy
Alaska Oil Production and Distribution Regulatory System
Business Object and Service Oriented Architecture
Veronica Gardner
July 2016
Contents
Realized Proficiencies ...................................................................................................................................2
Thesis Statement ..........................................................................................................................................2
Economy of Scale within the Oil Industry.....................................................................................................2
Legacy Database Applications.......................................................................................................................2
Integrated System Architecture....................................................................................................................3
Non Tank Vessel Contingency Plan Application............................................................................................3
Decision Support Capability..........................................................................................................................5
Integration Strategy: Service Oriented Architecture ...................................................................................6
Type Mapping: Cross Application Data Usage..............................................................................................7
Entity Types and Open Backend ...................................................................................................................8
Dynamic SQL .................................................................................................................................................9
Business Object Population and Data Access Layer ...................................................................................10
Summary and Project Review.....................................................................................................................11
2
Realized Proficiencies
 Tactical and strategic Decision Support System capabilities.
 Improved data quality, accessibility, and accuracy.
 Reduction of the code footprints and data redundancy.
 Decrease in the number of technical personnel required to support the system.
 Increase the response time for rolling out bug fixes and new software functionality.
 Improve communications by ensuring everyone interacts with the same accurate information.
Thesis Statement
This white paper communicates the software design and implementation strategy used for the system
integration of disparate oil industry regulatory database applications. The integration of both data and
code functionality was accomplished through an infrastructure of standards, autonomous components
and well defined business requirements. The code and SQL snippets, data structures, diagram and
images portrayed within this document are actual working elements of the first phase integration
initiative. Technical discourse encompasses the Service Oriented Architecture (SOA) and ‘System of
Record’ infrastructure with discussion of the backend data structure and data access mechanism. The
Non Tank Vessel Contingency Plan (NTV), is the business application built during the first phase
implementation and is presented in this document to help clarify the information.
Economy of Scale – Oil Industry
The ‘Economy of Scale’ of the oil industry plays a significant role in the feasibility for integrating
disparate information systems and data. Regulating the oil industry of Alaska has a fairly low transaction
frequency when compared to other types of industries. The regulations are governed by law and the
certifications of approval for operations are usually valid for years at a time. The high cost of doing
business in the oil industry also limits the number of competing companies and changes to operating
procedures can generally be expensive so are minimized.
Legacy Environment
The legacy infrastructure encompasses standalone database applications built for a specific regulatory
function. Some of these applications are developed using recent technology such as Asp.Net with MVC
and Entity Framework. Others are legacy MS Access database applications. These isolated applications
and databases often maintain the same business attributes but cross system comparisons are risky if
even feasible to do so. Often times the data value do not match so linking the same attribute (data
column) across systems is problematic. Many of the text field that should contain the same data value
are spelled different and have varying abbreviations. The date attributes often times differed across
systems so investigations were necessary and the accuracy of the systems became uncertain. This
disparity was a real hindrance to a streamlined information system and our potential for correctness
was greatly diminished. Attempting to use a query command to match across applications using
nonstandard business names at minimum required wildcard searches which can be slower with a
greater chance for erroneous data being returned.
3
Integrated System Infrastructure
The infrastructure used for the integration initiative is common throughout the industry and allows for a
secure implementation. Most of the regulatory information is public domain data and is not regulated
under HIPAA or proprietary constraints. The front end applications are outside of the firewall and
accessible by the public. Inside the DMZ resides the reusable services. The services communicate with
the Data Access layer to obtain the data from the data store.
Non Tank Vessel Contingency Plan Application
The Non Tank Vessel Contingency Plan (NTV) is the first regulatory application developed with the new
integrated architecture in mind. This application supports the regulatory requirement that all vessels
with greater than 450 GRT that is not a Barge or Tanker and operates in Alaska waters have a
documented and operational contingency plan. These plans cover numerous vessels that can only
operate in designated agreed upon regions of the state. Core integrated components built for the
integration initiative and used by the NTV application include: the Affiliation Tracking Module (ATM);
Document Management System (DMS), Facility Management System (FMS) and a Common Utilities
component which handles user management and encompasses the automated report generation and
distribution functionality.
The User Interface is not a topic of this technical discussion so only two user interface diagrams are
included. They have been added to provide a little bit of visual into how a user can interact with the
integrated components and services. The dashboard for the NTV Plan application is shown in Figure 1.
This user interface provides easy access to tactical level information and record access is configured per
user login. It is important to note that the data shown for the vessels, affiliations and plans are from
Figure 1 - General System Infrastructure
4
completely independent objects and services. They all exist and operate independently of each other yet
can be associated through business rules and logic of NTV or other regulatory applications. Please be
mindful that all data included in this dissertation are example ‘Test’ data and cannot be taken as true or
accurate information.
Figure 3 depicts the Plan Manager page designed for the maintenance of the contingency plan data. It is
possible to control the data displayed for the various roles and authority of the current user. This page
example represents a manager’s authority which includes viewing and editing of all or any plan data. A
Plan Holder or Response Planning Facilitator (RPF) managing their NTV Plan from the public side of the
system will only see the Plans they have created under a predesignated login. Most of the NTV
contingency plan data is public information and law states the data must be made available for viewing
by the public. The user interface shown in this document is for maintenance of the data and caution is
taken to ensure correct authority for any amendments or action taken on a plan.
Figure 2- Dashboard of NTV Internal Application
5
Decision Support Capability
The Facility, Regulation and Affiliations are autonomous objects available through different service
projects yet related to each other through business logic and associations. This integrated infrastructure
can quickly provide the tactical information necessary for accurate operation forecasting and decision
making. Examples of tactical queries are listed below:
 Which NTV Contingency Plan Holder operates the most vessels in Alaska?
 Which country has the highest fuel capacity operating under an Alaska NTV Plan?
 Get all NTV Contingency Plan Holders that use ‘SEAPRO’ as their cleanup contractor.
 Get all the Vessels that Company One owns and operate in Regions 1, 2 and 3.
 Get all the Vessels where the Plan Holder is different than the vessel owner.
Integrating an oil spill response tracking application with the ATM will add a new dimension to the
reporting capabilities of the infrastructure. The ATM integration is easy with minimal effort. All CRUD
operations are complete, stable and ready for use. A few queries are listed below but this is a small
subset of the potential questions that could be answered by further object integration.
 What is the total spillage of all vessels under a NTV Plan since 2012?
 Tell me which NTV contingency Plan holder is responsible for the most spill incidents.
 List all countries and how many NTV Plans for each. What are the gallons spilled for each?
 What is the maximum oil remediated per Cleanup Contractor? Incident Management Team?
Figure 3 - Plan Management Page
6
Integration Strategy: Service Oriented Architecture
Autonomy, Autonomy, Autonomy
The ‘System of Record’ infrastructure used for the integration initiative promotes the autonomy of both
the data elements and functionality within the system. The authoritative data source is stored in one
physical location and there is one set of functional methods that provide the CRUD operations for that
data. This infrastructure dictates keeping physical control of an object separate from the business
functionality and data rules. Primary and Foreign Key constraints are used to link data tables. These
identifiers can be used to associate and disassociate business objects on the front end but otherwise
have no business value and cannot be directly modified. The logic associations of the objects is
implemented through business rules, functional requirements and the Type Mapping strategy discussed
later in this paper.
To help solve the complex problems of integrating application code across various regulatory entities
and requirements a Service Oriented Architecture (SOA) is central. The software structure is organized
on a functional business perspective so the various business programs can use the built-in and standard
regulatory functionality. The service architecture was designed with a separation of concerns premise to
helps ensure the autonomy of information and business processes. A short summary of the the WCF
Service projects built for the integration effort is provided below.
 Affiliation Tracking Module (ATM): The ATM service project provides all relevant functionality to
create and maintain Affiliation records including the contact, phone, and address data. An
application is easily registered with the ATM which requires that the application identifier along
with the record identifier be input parameters when saving and retrieving affiliation records.
The ATM gives every opportunity to associate and reuse affiliation and contact information
across application configured to use the system. This mechanism works well within the oil
industry because the economy of scale limits the number of interested parties.
 Regulation Services: The Regulation Service Project houses objects and functionality relating to
any regulatory applications. Contingency plans, financial responsibility, oil spill, and
contaminated sites tracking are just a few of those regulations. The first Regulation service
implemented in this phase of the integration is the Non Tank Vessel Contingency Plan. Other
regulatory applications will be systematically integrated according to the priorities of the
business decision makers.
 Facility Services: There are numerous oil industry facilities and operations that are regulated by
the state of Alaska. The list of facility types include but are not limited to vessels, underground
storage tanks, terminal tank farms, production and exploration wells. This list is quite lengthy
but facilities have one thing in common: their operation can cross regulatory boundaries and are
currently tracked in numerous applications and databases.
o The ‘Vessel: System of Record’, is the first Facility service built within the Facility
Services project. The Vessel Services provides all CRUD functionality for Vessel
particulars. Maintaining the relationship to a vessel is the responsibility of the parent or
subscribing application. See the VesselAssociation table in Figure 6 of this document.
 Document Management Services (DMS): The DMS provides the function to store, retrieve, and
display uploaded document relating to a specific application record. The DMS can be used with
7
any application as long as the application identifier and the unique record identifier are
provided and passed into the service methods along with the document itself.
 Common Utility Services: This project is built to provide services for generic utilities whose
functionality is needed in numerous applications and components. The Common Utility service
project includes the automated report generation and distribution for plan activity and changes.
These reports and correspondence documents are systematically uploaded and associated to
the plan records.
Type Mapping: Cross Application Data Usage
A good Decision Support System is dependent on standard data from all relevant heterogeneous
sources. To maintain these data standards it was necessary to manage the usability of data across
varying requirements and applications. A good example is the use of a Vessel Component in varying
business contexts. An NTV Plan has strict data rules that require a regulated Vessel’s details be
populated before legal operation can commence. For marine oil and hazardous spill tracking, the
available details of the vessel may be unknown especially when the vessel is on the bottom of the
ocean. Type Mapping, a form of a declarative language is used to accommodate the varying data rules
across the multiple applications. Figure 4 shows some of the standard Data Types used to integrate data
across the various regulations and applications. Some of these types have a corresponding Type
Mapping which provides the separation of concerns.
Integrated Data Types
Entity Type Entity Identifier: NTV Plan, Spill, Vessel, Oil Production Platform…
Contact Type Primary Contact, 24 Hour Contact, Office Contact…
Affiliation Type Plan Holder, Owner, Operator, Cleanup Contractor, Response Planning Facilitator…
Organization Type Public, Private, Nonprofit, Individual
Facility Type Vessel, Terminal Tank Farm, Production Wells, Exploration Wells…
Vessel Type Fishing, Passenger, Cargo, Chemical…
Plan Status Type Standard operational status – WIP, Active, Terminated…
Status Action Type Used to track actions that change status of a plan: Activate, Terminate, Renew…
Document Type Approval Letter, NTV Application, Certificate for Operation, Reason for Decision…
By mapping the business context to a standard Type list the system can provide application specific
needs yet support the requirement for standardization of data. The Type Mapping mechanism makes it
quick and easy to configure which Type values will be shown in the user interface lists and combo boxes
for any context. This implementation created great advantage for consistency, accuracy and data access
Figure 4 - Dropdown List Types – Representative Only
8
optimization. To Type Map the information it was necessary to ensure autonomy of meaning for each
Type created. The different business functions use different data values for the same Data Type. The
Barge and Tanker Contingency Plans only need their own type, Barge or Tanker. The NTV Plan users
need to see only vessels under their regulation (Cruise Ship, Ferry, Fishing, Chemical…), while a Spill
tracking application interface will need access to the entire Vessel Type list.
The Type Map table that controls which Contact Type can be associated to particular Affiliation Type
within a specified Entity Type (application/facility) context is shown in Figure 5 - Type Mapping Table.
NTV mapping examples include a Primary Contact of an Owner of a Vessel, a Primary Contact of a Plan
Holder of an NTV Plan, or a 24 Hour Contact of a Cleanup Contractor of a NTV Plan. Most of the
regulatory applications (Entity Types) will require and utilize the mapping for a Primary Contact of the
Responsible Party affiliation. These lists are controlled through the column MapTypeID which is 1 for
required or 0 for optional. If the type value is not available for a given Entity Type then that value is not
an option for that business context. The ContactTypeCount column signifies how many times the
‘contact type’ can be used within the current operating context.
Entity Types and Open Backend
Each integrated database is designed to allow CRUD operation for non-complete record sets. The
flexibility of the over all system is greatly enhanced because this open backend architecture keeps
business logic separate from the data storage. Throughout the system the operating pair of EntityTypeID
and EntityID is used extensively for the integration of data and functionality. Technically, the
EntityTypeID is an identifier assigned to a database table but logically, it signifies a business objects, i.e.
a regulation or a particular facility. Currently in use are the EntityTypes of Contingency Plans and the
Facility Vessel. Each of these applications utilize the ATM, DMS, and/or the FMS services by passing in
their own assigned Entity identifier along with the record identifier (EntityID). Any EntityID parameter
used without it’s corresponding Entity Type identifier will be ignored. Figure 6 - NTV Integrated Table
Relationships, is a scaled down version of the table structure of the NTV Plan database integrated with
the Affiliations, Documents and Vessels data stores. Each object controls it’s own CRUD operations but
Figure 5 - Type Mapping Table Structure
9
it’s the applications than govern their own rules to add/update/remove a relationship with other
business objects.
Dynamic SQL
Dynamic SQL is used to help build flexibility into the integrated system by returning datasets for a
variety of criteria. Dynamic SQL is a little slower due to the compilation so thought and caution was
taken during the implementation to ensure quick response from the data store. The stored procedures
never apply business logic and always use the identity columns in the relational joins among the tables.
Care was taken using this implementation to ensure proper use of returned datasets. Data returned
within a transactional context (CRUD functionality) generally need far less data records returned as
compared to usage of data records for reporting purposes. The stored procedure NTVPlan_SELECT is
shown in Figure 7 and is a simple representation how the dynamic SQL is implemented.
Figure 6 - NTV Plan Integrated Table Relationships
10
CREATE PROCEDURE [dbo].[NTVPlan_SELECT]
@NTVPlanID INT = NULL,
@PlanNumber varchar(10) = null,
@PlanStatusID int = null,
@EntityTypeID int = null
AS
BEGIN
DECLARE @WhereClause varchar(max)
SET @WhereClause = ''
IF(@NTVPlanID > 0)
BEGIN
SET @WhereClause = N'WHERE p.NTVPlanID = ' + CAST(@NTVPlanID AS varchar(20))
END
ELSE
BEGIN
IF (@PlanNumber != '')
BEGIN
SELECT @PlanNumber = RTrim(LTrim(@PlanNumber))
SET @WhereClause = N'WHERE p.PlanNumber = ' + '''' + @PlanNumber + ''''
END
ELSE
BEGIN
IF(@PlanStatusID > 0)
BEGIN
SET @WhereClause = N'WHERE p.PlanStatusID = ' + CAST(@PlanStatusID AS varchar(20))
END
IF(@EntityTypeID > 0)
BEGIN
IF(@WhereClause = '')
BEGIN
SET @WhereClause = N'WHERE p.EntityTypeID = ' + CAST(@EntityTypeID AS varchar(20))
END
ELSE
BEGIN
SET @WhereClause = @WhereClause + N'AND p.EntityTypeID = ' + CAST(@EntityTypeID AS varchar(20))
END
END
END
END
EXEC ('SELECT p.NTVPlanid, p.PlanNumber, ps.PlanStatusType, pt.EntityType, p.PlanActivationDate,
p.PlanExpirationDate, p.VesselInAlaskaDate, p.ApplicantTypeID,
p.PlanStatusID, p.UserID, p.EntityTypeID,sde.UserName,
p.RPFTypeID, p.ApplicationSubmittedByID, p.ApplicationCertifiedById, p.CreateDate, p.ModifyDate
FROM DB1.NTVPlan p inner join DB1.PlanStatusType ps on p.PlanStatusID = ps.PlanStatusTypeID
LEFT JOIN DB2.dbo.EntityType pt on p.EntityTypeID = pt.EntityTypeID
LEFT JOIN DB5.dbo.Users user on user.UserID = p.UserID ' +
@WhereClause
Business Object Population and Data Access Layer
All data in the business objects are maintained via that objects CRUD methods. This includes the data
access and conversions of the List<T> child objects a component may have. Each access method is
autonomous along with each object conversion method. The data access layer receives record result
sets in either data table or data set format. The database result sets are converted to the corresponding
object structure and all business rules and processing thereafter is applied to that object. Figure 8 is the
code snippet for converting an NTVPlan data table result set from the NTVPlan_SELECT method into the
NTVPlan object structure. The List<t> child properties of a NTV plan (i.e. List<VesselAssociation>,
List<Affiliation>) have their own conversion and data access methods therefore are not included in the
conversion method for the plan itself. There is no business logic within these conversion methods other
than the annullable attributes.
Figure 7 Dynamic SQL within the Stored Procedures
11
List<models.NTVPlan> NTVPlanList = new List<models.NTVPlan>
(
NTVPlanDataStoreResults.AsEnumerable().Select(O => new models.NTVPlan
{
NTVPlanID = O.Field<int>("NTVPlanID"),
PlanNumber = O.Field<string>("PlanNumber"),
EntityTypeID = O.Field<int>("EntityTypeID"),
EntityType = O.Field<string>("EntityType"),
PlanStatusID = O.Field<int>("PlanStatusID"),
PlanStatus = O.Field<string>("PlanStatusType"),
PlanActivationDate = O.Field<DateTime?>("PlanActivationDate"),
PlanExpirationDate = O.Field<DateTime?>("PlanExpirationDate"),
VesselInAlaskaDate = O.Field<DateTime?>("VesselInAlaskaDate"),
PlanHolderTypeID = O.Field<int?>("ApplicantTypeID"),
UserID = O.Field<int>("UserID"),
CreateDate = O.Field<DateTime?>("CreateDate"),
ModifyDate = O.Field<DateTime?>("ModifyDate")
}));
Summary and Project Review
The strategy used to integrate data and functionality across the various applications has been extremely
successful. The integration eliminates redundant code, improves data accuracy, and promotes DDS
capabilities. The SOA service projects and components so far include the Affiliation Tracking Module, the
Document Management System, the Facilities Services (Vessel system of record), and the Common
Utilities services. The NTV Plan functionality within the Regulation Services is the first of the regulatory
service structures to be implemented. The ATM is one of the more complex integrated component
therefore is the thesis of the next paper covering this integration initiative.
All of the technical discourse, code, SQL snippets, diagram, and images within these documents were
designed and implemented for the integration initiative and the development of the NTV Plan
application. The initiative took three developers dedicating at minimum 50% of their time approximately
a year and a half to complete the first phase development. It was the first time the team worked
together on a project and all members had other duties assigned throughout the duration of the
initiative.
Software Implementations and Tools that Worked Well
 Separation of Concerns greatly improves the flexibility for adding features and functionality.
 Open backend infrastructure fully supports the services reuse across varying business rules.
 The Types and Type Mappings Declarative Markup Language provides the foundation for much
of the decision support capabilities and improves the front end user experience.
 SOA architecture is key to this integration initiative and is the corner stone of the
implementation.
Figure 8 Convert NTV Plan Data Access Results to NTV Plan Business Object
12
 Coding and Naming Standards significantly improve enhancement capabilities, troubleshooting
efforts, and the ability for developers to understand and maneuverer the system.
Improvements and Considerations
 Needed more developers (or less work) to take the time to design the NTVPlan and NTV
Certificate tables as a more generic implementation that could accommodate other regulations.
Note: NTV Plans are not governed through the same regulations as all other contingency plans.
 Some say SQL IDENTITY columns should not be used for primary / foreign key implementation.
Would it be better to use some other type of identifier? These Identity columns make
development of the backend so easy.
 Some database programmers do not like the lack of database referential integrity for the
relationship between the Plan, Affiliation, Document and Facility databases.
 Would have liked to create the service and model layout that fully accommodates a Restful
architecture. No team expert on Restful and too little time to prove a good ROI for its
implementation.

More Related Content

Viewers also liked

Viewers also liked (12)

Sefer Yetsirah
Sefer YetsirahSefer Yetsirah
Sefer Yetsirah
 
PNC_SpotMeCash-3
PNC_SpotMeCash-3PNC_SpotMeCash-3
PNC_SpotMeCash-3
 
Alteryx Technical Overview
Alteryx Technical OverviewAlteryx Technical Overview
Alteryx Technical Overview
 
BROCHURE
BROCHUREBROCHURE
BROCHURE
 
Derecho concursal
Derecho concursal Derecho concursal
Derecho concursal
 
Redes sociales que te ayudarán a desarrollarte profesionalmente
Redes sociales que te ayudarán a desarrollarte profesionalmenteRedes sociales que te ayudarán a desarrollarte profesionalmente
Redes sociales que te ayudarán a desarrollarte profesionalmente
 
home-secrets-presentation
home-secrets-presentationhome-secrets-presentation
home-secrets-presentation
 
Prederechogalohuatatoca
PrederechogalohuatatocaPrederechogalohuatatoca
Prederechogalohuatatoca
 
hs-media-kit
hs-media-kiths-media-kit
hs-media-kit
 
Concepto sgalohuatatoca
Concepto sgalohuatatocaConcepto sgalohuatatoca
Concepto sgalohuatatoca
 
Psalm138-14Project
Psalm138-14ProjectPsalm138-14Project
Psalm138-14Project
 
datareport_feb_rv2[1]-2
datareport_feb_rv2[1]-2datareport_feb_rv2[1]-2
datareport_feb_rv2[1]-2
 

Similar to SoftwareIntegrationStrategy

Delphix modernization whitepaper
Delphix  modernization whitepaperDelphix  modernization whitepaper
Delphix modernization whitepaperFranco_Dagosto
 
NetWrix Change Reporter Suite - Product Review by Don Jones
NetWrix Change Reporter Suite - Product Review by Don JonesNetWrix Change Reporter Suite - Product Review by Don Jones
NetWrix Change Reporter Suite - Product Review by Don JonesNetwrix Corporation
 
NIST Definition for Cloud Computing
NIST Definition for Cloud ComputingNIST Definition for Cloud Computing
NIST Definition for Cloud ComputingAjay Ohri
 
NIST 2011 Cloud Computing definitions
NIST 2011 Cloud Computing definitionsNIST 2011 Cloud Computing definitions
NIST 2011 Cloud Computing definitionsi-SCOOP
 
Controlling SOA in Support of Operational Improvement (ISPE PE Vol 31 No 4 - ...
Controlling SOA in Support of Operational Improvement (ISPE PE Vol 31 No 4 - ...Controlling SOA in Support of Operational Improvement (ISPE PE Vol 31 No 4 - ...
Controlling SOA in Support of Operational Improvement (ISPE PE Vol 31 No 4 - ...David Stokes
 
Cloud Based Development for Medical Pharmacy and DCA Application
Cloud Based Development for Medical Pharmacy and DCA ApplicationCloud Based Development for Medical Pharmacy and DCA Application
Cloud Based Development for Medical Pharmacy and DCA ApplicationIRJET Journal
 
Xd planning guide - storage best practices
Xd   planning guide - storage best practicesXd   planning guide - storage best practices
Xd planning guide - storage best practicesNuno Alves
 
Business Continuity for Mission Critical Applications
Business Continuity for Mission Critical ApplicationsBusiness Continuity for Mission Critical Applications
Business Continuity for Mission Critical ApplicationsDataCore Software
 
Supplier Base Optimization White Paper
Supplier Base Optimization White PaperSupplier Base Optimization White Paper
Supplier Base Optimization White PaperAnn Rose Myladoor
 
Management High-level overview of the OMG Data Distribution Service (DDS)
Management High-level overview of the OMG Data Distribution Service (DDS)Management High-level overview of the OMG Data Distribution Service (DDS)
Management High-level overview of the OMG Data Distribution Service (DDS)Gerardo Pardo-Castellote
 
IRJET- In-House File Tracking System
IRJET-  	  In-House File Tracking SystemIRJET-  	  In-House File Tracking System
IRJET- In-House File Tracking SystemIRJET Journal
 
M.S. Dissertation in Salesforce on Force.com
M.S. Dissertation in Salesforce on Force.comM.S. Dissertation in Salesforce on Force.com
M.S. Dissertation in Salesforce on Force.comArun Somu Panneerselvam
 
Continuous Integration and Continuous Deployment Pipeline with Apprenda on ON...
Continuous Integration and Continuous Deployment Pipeline with Apprenda on ON...Continuous Integration and Continuous Deployment Pipeline with Apprenda on ON...
Continuous Integration and Continuous Deployment Pipeline with Apprenda on ON...Shrivatsa Upadhye
 
Joe Graziano – Challenge 2 Design Solution (Part 1)
Joe Graziano – Challenge 2 Design Solution (Part 1)Joe Graziano – Challenge 2 Design Solution (Part 1)
Joe Graziano – Challenge 2 Design Solution (Part 1)tovmug
 

Similar to SoftwareIntegrationStrategy (20)

Delphix modernization whitepaper
Delphix  modernization whitepaperDelphix  modernization whitepaper
Delphix modernization whitepaper
 
Observability
ObservabilityObservability
Observability
 
NetWrix Change Reporter Suite - Product Review by Don Jones
NetWrix Change Reporter Suite - Product Review by Don JonesNetWrix Change Reporter Suite - Product Review by Don Jones
NetWrix Change Reporter Suite - Product Review by Don Jones
 
NIST Definition for Cloud Computing
NIST Definition for Cloud ComputingNIST Definition for Cloud Computing
NIST Definition for Cloud Computing
 
NIST 2011 Cloud Computing definitions
NIST 2011 Cloud Computing definitionsNIST 2011 Cloud Computing definitions
NIST 2011 Cloud Computing definitions
 
Nist cloud comp
Nist cloud compNist cloud comp
Nist cloud comp
 
Controlling SOA in Support of Operational Improvement (ISPE PE Vol 31 No 4 - ...
Controlling SOA in Support of Operational Improvement (ISPE PE Vol 31 No 4 - ...Controlling SOA in Support of Operational Improvement (ISPE PE Vol 31 No 4 - ...
Controlling SOA in Support of Operational Improvement (ISPE PE Vol 31 No 4 - ...
 
Cloud Based Development for Medical Pharmacy and DCA Application
Cloud Based Development for Medical Pharmacy and DCA ApplicationCloud Based Development for Medical Pharmacy and DCA Application
Cloud Based Development for Medical Pharmacy and DCA Application
 
Data warehouse system
Data warehouse systemData warehouse system
Data warehouse system
 
Xd planning guide - storage best practices
Xd   planning guide - storage best practicesXd   planning guide - storage best practices
Xd planning guide - storage best practices
 
project plan
project planproject plan
project plan
 
Business Continuity for Mission Critical Applications
Business Continuity for Mission Critical ApplicationsBusiness Continuity for Mission Critical Applications
Business Continuity for Mission Critical Applications
 
Supplier Base Optimization White Paper
Supplier Base Optimization White PaperSupplier Base Optimization White Paper
Supplier Base Optimization White Paper
 
Management High-level overview of the OMG Data Distribution Service (DDS)
Management High-level overview of the OMG Data Distribution Service (DDS)Management High-level overview of the OMG Data Distribution Service (DDS)
Management High-level overview of the OMG Data Distribution Service (DDS)
 
Performance testing wreaking balls
Performance testing wreaking ballsPerformance testing wreaking balls
Performance testing wreaking balls
 
IRJET- In-House File Tracking System
IRJET-  	  In-House File Tracking SystemIRJET-  	  In-House File Tracking System
IRJET- In-House File Tracking System
 
M.S. Dissertation in Salesforce on Force.com
M.S. Dissertation in Salesforce on Force.comM.S. Dissertation in Salesforce on Force.com
M.S. Dissertation in Salesforce on Force.com
 
Continuous Integration and Continuous Deployment Pipeline with Apprenda on ON...
Continuous Integration and Continuous Deployment Pipeline with Apprenda on ON...Continuous Integration and Continuous Deployment Pipeline with Apprenda on ON...
Continuous Integration and Continuous Deployment Pipeline with Apprenda on ON...
 
Joe Graziano – Challenge 2 Design Solution (Part 1)
Joe Graziano – Challenge 2 Design Solution (Part 1)Joe Graziano – Challenge 2 Design Solution (Part 1)
Joe Graziano – Challenge 2 Design Solution (Part 1)
 
DDS Best Practices
DDS Best PracticesDDS Best Practices
DDS Best Practices
 

SoftwareIntegrationStrategy

  • 1. 1 Software and Data Integration Strategy Alaska Oil Production and Distribution Regulatory System Business Object and Service Oriented Architecture Veronica Gardner July 2016 Contents Realized Proficiencies ...................................................................................................................................2 Thesis Statement ..........................................................................................................................................2 Economy of Scale within the Oil Industry.....................................................................................................2 Legacy Database Applications.......................................................................................................................2 Integrated System Architecture....................................................................................................................3 Non Tank Vessel Contingency Plan Application............................................................................................3 Decision Support Capability..........................................................................................................................5 Integration Strategy: Service Oriented Architecture ...................................................................................6 Type Mapping: Cross Application Data Usage..............................................................................................7 Entity Types and Open Backend ...................................................................................................................8 Dynamic SQL .................................................................................................................................................9 Business Object Population and Data Access Layer ...................................................................................10 Summary and Project Review.....................................................................................................................11
  • 2. 2 Realized Proficiencies  Tactical and strategic Decision Support System capabilities.  Improved data quality, accessibility, and accuracy.  Reduction of the code footprints and data redundancy.  Decrease in the number of technical personnel required to support the system.  Increase the response time for rolling out bug fixes and new software functionality.  Improve communications by ensuring everyone interacts with the same accurate information. Thesis Statement This white paper communicates the software design and implementation strategy used for the system integration of disparate oil industry regulatory database applications. The integration of both data and code functionality was accomplished through an infrastructure of standards, autonomous components and well defined business requirements. The code and SQL snippets, data structures, diagram and images portrayed within this document are actual working elements of the first phase integration initiative. Technical discourse encompasses the Service Oriented Architecture (SOA) and ‘System of Record’ infrastructure with discussion of the backend data structure and data access mechanism. The Non Tank Vessel Contingency Plan (NTV), is the business application built during the first phase implementation and is presented in this document to help clarify the information. Economy of Scale – Oil Industry The ‘Economy of Scale’ of the oil industry plays a significant role in the feasibility for integrating disparate information systems and data. Regulating the oil industry of Alaska has a fairly low transaction frequency when compared to other types of industries. The regulations are governed by law and the certifications of approval for operations are usually valid for years at a time. The high cost of doing business in the oil industry also limits the number of competing companies and changes to operating procedures can generally be expensive so are minimized. Legacy Environment The legacy infrastructure encompasses standalone database applications built for a specific regulatory function. Some of these applications are developed using recent technology such as Asp.Net with MVC and Entity Framework. Others are legacy MS Access database applications. These isolated applications and databases often maintain the same business attributes but cross system comparisons are risky if even feasible to do so. Often times the data value do not match so linking the same attribute (data column) across systems is problematic. Many of the text field that should contain the same data value are spelled different and have varying abbreviations. The date attributes often times differed across systems so investigations were necessary and the accuracy of the systems became uncertain. This disparity was a real hindrance to a streamlined information system and our potential for correctness was greatly diminished. Attempting to use a query command to match across applications using nonstandard business names at minimum required wildcard searches which can be slower with a greater chance for erroneous data being returned.
  • 3. 3 Integrated System Infrastructure The infrastructure used for the integration initiative is common throughout the industry and allows for a secure implementation. Most of the regulatory information is public domain data and is not regulated under HIPAA or proprietary constraints. The front end applications are outside of the firewall and accessible by the public. Inside the DMZ resides the reusable services. The services communicate with the Data Access layer to obtain the data from the data store. Non Tank Vessel Contingency Plan Application The Non Tank Vessel Contingency Plan (NTV) is the first regulatory application developed with the new integrated architecture in mind. This application supports the regulatory requirement that all vessels with greater than 450 GRT that is not a Barge or Tanker and operates in Alaska waters have a documented and operational contingency plan. These plans cover numerous vessels that can only operate in designated agreed upon regions of the state. Core integrated components built for the integration initiative and used by the NTV application include: the Affiliation Tracking Module (ATM); Document Management System (DMS), Facility Management System (FMS) and a Common Utilities component which handles user management and encompasses the automated report generation and distribution functionality. The User Interface is not a topic of this technical discussion so only two user interface diagrams are included. They have been added to provide a little bit of visual into how a user can interact with the integrated components and services. The dashboard for the NTV Plan application is shown in Figure 1. This user interface provides easy access to tactical level information and record access is configured per user login. It is important to note that the data shown for the vessels, affiliations and plans are from Figure 1 - General System Infrastructure
  • 4. 4 completely independent objects and services. They all exist and operate independently of each other yet can be associated through business rules and logic of NTV or other regulatory applications. Please be mindful that all data included in this dissertation are example ‘Test’ data and cannot be taken as true or accurate information. Figure 3 depicts the Plan Manager page designed for the maintenance of the contingency plan data. It is possible to control the data displayed for the various roles and authority of the current user. This page example represents a manager’s authority which includes viewing and editing of all or any plan data. A Plan Holder or Response Planning Facilitator (RPF) managing their NTV Plan from the public side of the system will only see the Plans they have created under a predesignated login. Most of the NTV contingency plan data is public information and law states the data must be made available for viewing by the public. The user interface shown in this document is for maintenance of the data and caution is taken to ensure correct authority for any amendments or action taken on a plan. Figure 2- Dashboard of NTV Internal Application
  • 5. 5 Decision Support Capability The Facility, Regulation and Affiliations are autonomous objects available through different service projects yet related to each other through business logic and associations. This integrated infrastructure can quickly provide the tactical information necessary for accurate operation forecasting and decision making. Examples of tactical queries are listed below:  Which NTV Contingency Plan Holder operates the most vessels in Alaska?  Which country has the highest fuel capacity operating under an Alaska NTV Plan?  Get all NTV Contingency Plan Holders that use ‘SEAPRO’ as their cleanup contractor.  Get all the Vessels that Company One owns and operate in Regions 1, 2 and 3.  Get all the Vessels where the Plan Holder is different than the vessel owner. Integrating an oil spill response tracking application with the ATM will add a new dimension to the reporting capabilities of the infrastructure. The ATM integration is easy with minimal effort. All CRUD operations are complete, stable and ready for use. A few queries are listed below but this is a small subset of the potential questions that could be answered by further object integration.  What is the total spillage of all vessels under a NTV Plan since 2012?  Tell me which NTV contingency Plan holder is responsible for the most spill incidents.  List all countries and how many NTV Plans for each. What are the gallons spilled for each?  What is the maximum oil remediated per Cleanup Contractor? Incident Management Team? Figure 3 - Plan Management Page
  • 6. 6 Integration Strategy: Service Oriented Architecture Autonomy, Autonomy, Autonomy The ‘System of Record’ infrastructure used for the integration initiative promotes the autonomy of both the data elements and functionality within the system. The authoritative data source is stored in one physical location and there is one set of functional methods that provide the CRUD operations for that data. This infrastructure dictates keeping physical control of an object separate from the business functionality and data rules. Primary and Foreign Key constraints are used to link data tables. These identifiers can be used to associate and disassociate business objects on the front end but otherwise have no business value and cannot be directly modified. The logic associations of the objects is implemented through business rules, functional requirements and the Type Mapping strategy discussed later in this paper. To help solve the complex problems of integrating application code across various regulatory entities and requirements a Service Oriented Architecture (SOA) is central. The software structure is organized on a functional business perspective so the various business programs can use the built-in and standard regulatory functionality. The service architecture was designed with a separation of concerns premise to helps ensure the autonomy of information and business processes. A short summary of the the WCF Service projects built for the integration effort is provided below.  Affiliation Tracking Module (ATM): The ATM service project provides all relevant functionality to create and maintain Affiliation records including the contact, phone, and address data. An application is easily registered with the ATM which requires that the application identifier along with the record identifier be input parameters when saving and retrieving affiliation records. The ATM gives every opportunity to associate and reuse affiliation and contact information across application configured to use the system. This mechanism works well within the oil industry because the economy of scale limits the number of interested parties.  Regulation Services: The Regulation Service Project houses objects and functionality relating to any regulatory applications. Contingency plans, financial responsibility, oil spill, and contaminated sites tracking are just a few of those regulations. The first Regulation service implemented in this phase of the integration is the Non Tank Vessel Contingency Plan. Other regulatory applications will be systematically integrated according to the priorities of the business decision makers.  Facility Services: There are numerous oil industry facilities and operations that are regulated by the state of Alaska. The list of facility types include but are not limited to vessels, underground storage tanks, terminal tank farms, production and exploration wells. This list is quite lengthy but facilities have one thing in common: their operation can cross regulatory boundaries and are currently tracked in numerous applications and databases. o The ‘Vessel: System of Record’, is the first Facility service built within the Facility Services project. The Vessel Services provides all CRUD functionality for Vessel particulars. Maintaining the relationship to a vessel is the responsibility of the parent or subscribing application. See the VesselAssociation table in Figure 6 of this document.  Document Management Services (DMS): The DMS provides the function to store, retrieve, and display uploaded document relating to a specific application record. The DMS can be used with
  • 7. 7 any application as long as the application identifier and the unique record identifier are provided and passed into the service methods along with the document itself.  Common Utility Services: This project is built to provide services for generic utilities whose functionality is needed in numerous applications and components. The Common Utility service project includes the automated report generation and distribution for plan activity and changes. These reports and correspondence documents are systematically uploaded and associated to the plan records. Type Mapping: Cross Application Data Usage A good Decision Support System is dependent on standard data from all relevant heterogeneous sources. To maintain these data standards it was necessary to manage the usability of data across varying requirements and applications. A good example is the use of a Vessel Component in varying business contexts. An NTV Plan has strict data rules that require a regulated Vessel’s details be populated before legal operation can commence. For marine oil and hazardous spill tracking, the available details of the vessel may be unknown especially when the vessel is on the bottom of the ocean. Type Mapping, a form of a declarative language is used to accommodate the varying data rules across the multiple applications. Figure 4 shows some of the standard Data Types used to integrate data across the various regulations and applications. Some of these types have a corresponding Type Mapping which provides the separation of concerns. Integrated Data Types Entity Type Entity Identifier: NTV Plan, Spill, Vessel, Oil Production Platform… Contact Type Primary Contact, 24 Hour Contact, Office Contact… Affiliation Type Plan Holder, Owner, Operator, Cleanup Contractor, Response Planning Facilitator… Organization Type Public, Private, Nonprofit, Individual Facility Type Vessel, Terminal Tank Farm, Production Wells, Exploration Wells… Vessel Type Fishing, Passenger, Cargo, Chemical… Plan Status Type Standard operational status – WIP, Active, Terminated… Status Action Type Used to track actions that change status of a plan: Activate, Terminate, Renew… Document Type Approval Letter, NTV Application, Certificate for Operation, Reason for Decision… By mapping the business context to a standard Type list the system can provide application specific needs yet support the requirement for standardization of data. The Type Mapping mechanism makes it quick and easy to configure which Type values will be shown in the user interface lists and combo boxes for any context. This implementation created great advantage for consistency, accuracy and data access Figure 4 - Dropdown List Types – Representative Only
  • 8. 8 optimization. To Type Map the information it was necessary to ensure autonomy of meaning for each Type created. The different business functions use different data values for the same Data Type. The Barge and Tanker Contingency Plans only need their own type, Barge or Tanker. The NTV Plan users need to see only vessels under their regulation (Cruise Ship, Ferry, Fishing, Chemical…), while a Spill tracking application interface will need access to the entire Vessel Type list. The Type Map table that controls which Contact Type can be associated to particular Affiliation Type within a specified Entity Type (application/facility) context is shown in Figure 5 - Type Mapping Table. NTV mapping examples include a Primary Contact of an Owner of a Vessel, a Primary Contact of a Plan Holder of an NTV Plan, or a 24 Hour Contact of a Cleanup Contractor of a NTV Plan. Most of the regulatory applications (Entity Types) will require and utilize the mapping for a Primary Contact of the Responsible Party affiliation. These lists are controlled through the column MapTypeID which is 1 for required or 0 for optional. If the type value is not available for a given Entity Type then that value is not an option for that business context. The ContactTypeCount column signifies how many times the ‘contact type’ can be used within the current operating context. Entity Types and Open Backend Each integrated database is designed to allow CRUD operation for non-complete record sets. The flexibility of the over all system is greatly enhanced because this open backend architecture keeps business logic separate from the data storage. Throughout the system the operating pair of EntityTypeID and EntityID is used extensively for the integration of data and functionality. Technically, the EntityTypeID is an identifier assigned to a database table but logically, it signifies a business objects, i.e. a regulation or a particular facility. Currently in use are the EntityTypes of Contingency Plans and the Facility Vessel. Each of these applications utilize the ATM, DMS, and/or the FMS services by passing in their own assigned Entity identifier along with the record identifier (EntityID). Any EntityID parameter used without it’s corresponding Entity Type identifier will be ignored. Figure 6 - NTV Integrated Table Relationships, is a scaled down version of the table structure of the NTV Plan database integrated with the Affiliations, Documents and Vessels data stores. Each object controls it’s own CRUD operations but Figure 5 - Type Mapping Table Structure
  • 9. 9 it’s the applications than govern their own rules to add/update/remove a relationship with other business objects. Dynamic SQL Dynamic SQL is used to help build flexibility into the integrated system by returning datasets for a variety of criteria. Dynamic SQL is a little slower due to the compilation so thought and caution was taken during the implementation to ensure quick response from the data store. The stored procedures never apply business logic and always use the identity columns in the relational joins among the tables. Care was taken using this implementation to ensure proper use of returned datasets. Data returned within a transactional context (CRUD functionality) generally need far less data records returned as compared to usage of data records for reporting purposes. The stored procedure NTVPlan_SELECT is shown in Figure 7 and is a simple representation how the dynamic SQL is implemented. Figure 6 - NTV Plan Integrated Table Relationships
  • 10. 10 CREATE PROCEDURE [dbo].[NTVPlan_SELECT] @NTVPlanID INT = NULL, @PlanNumber varchar(10) = null, @PlanStatusID int = null, @EntityTypeID int = null AS BEGIN DECLARE @WhereClause varchar(max) SET @WhereClause = '' IF(@NTVPlanID > 0) BEGIN SET @WhereClause = N'WHERE p.NTVPlanID = ' + CAST(@NTVPlanID AS varchar(20)) END ELSE BEGIN IF (@PlanNumber != '') BEGIN SELECT @PlanNumber = RTrim(LTrim(@PlanNumber)) SET @WhereClause = N'WHERE p.PlanNumber = ' + '''' + @PlanNumber + '''' END ELSE BEGIN IF(@PlanStatusID > 0) BEGIN SET @WhereClause = N'WHERE p.PlanStatusID = ' + CAST(@PlanStatusID AS varchar(20)) END IF(@EntityTypeID > 0) BEGIN IF(@WhereClause = '') BEGIN SET @WhereClause = N'WHERE p.EntityTypeID = ' + CAST(@EntityTypeID AS varchar(20)) END ELSE BEGIN SET @WhereClause = @WhereClause + N'AND p.EntityTypeID = ' + CAST(@EntityTypeID AS varchar(20)) END END END END EXEC ('SELECT p.NTVPlanid, p.PlanNumber, ps.PlanStatusType, pt.EntityType, p.PlanActivationDate, p.PlanExpirationDate, p.VesselInAlaskaDate, p.ApplicantTypeID, p.PlanStatusID, p.UserID, p.EntityTypeID,sde.UserName, p.RPFTypeID, p.ApplicationSubmittedByID, p.ApplicationCertifiedById, p.CreateDate, p.ModifyDate FROM DB1.NTVPlan p inner join DB1.PlanStatusType ps on p.PlanStatusID = ps.PlanStatusTypeID LEFT JOIN DB2.dbo.EntityType pt on p.EntityTypeID = pt.EntityTypeID LEFT JOIN DB5.dbo.Users user on user.UserID = p.UserID ' + @WhereClause Business Object Population and Data Access Layer All data in the business objects are maintained via that objects CRUD methods. This includes the data access and conversions of the List<T> child objects a component may have. Each access method is autonomous along with each object conversion method. The data access layer receives record result sets in either data table or data set format. The database result sets are converted to the corresponding object structure and all business rules and processing thereafter is applied to that object. Figure 8 is the code snippet for converting an NTVPlan data table result set from the NTVPlan_SELECT method into the NTVPlan object structure. The List<t> child properties of a NTV plan (i.e. List<VesselAssociation>, List<Affiliation>) have their own conversion and data access methods therefore are not included in the conversion method for the plan itself. There is no business logic within these conversion methods other than the annullable attributes. Figure 7 Dynamic SQL within the Stored Procedures
  • 11. 11 List<models.NTVPlan> NTVPlanList = new List<models.NTVPlan> ( NTVPlanDataStoreResults.AsEnumerable().Select(O => new models.NTVPlan { NTVPlanID = O.Field<int>("NTVPlanID"), PlanNumber = O.Field<string>("PlanNumber"), EntityTypeID = O.Field<int>("EntityTypeID"), EntityType = O.Field<string>("EntityType"), PlanStatusID = O.Field<int>("PlanStatusID"), PlanStatus = O.Field<string>("PlanStatusType"), PlanActivationDate = O.Field<DateTime?>("PlanActivationDate"), PlanExpirationDate = O.Field<DateTime?>("PlanExpirationDate"), VesselInAlaskaDate = O.Field<DateTime?>("VesselInAlaskaDate"), PlanHolderTypeID = O.Field<int?>("ApplicantTypeID"), UserID = O.Field<int>("UserID"), CreateDate = O.Field<DateTime?>("CreateDate"), ModifyDate = O.Field<DateTime?>("ModifyDate") })); Summary and Project Review The strategy used to integrate data and functionality across the various applications has been extremely successful. The integration eliminates redundant code, improves data accuracy, and promotes DDS capabilities. The SOA service projects and components so far include the Affiliation Tracking Module, the Document Management System, the Facilities Services (Vessel system of record), and the Common Utilities services. The NTV Plan functionality within the Regulation Services is the first of the regulatory service structures to be implemented. The ATM is one of the more complex integrated component therefore is the thesis of the next paper covering this integration initiative. All of the technical discourse, code, SQL snippets, diagram, and images within these documents were designed and implemented for the integration initiative and the development of the NTV Plan application. The initiative took three developers dedicating at minimum 50% of their time approximately a year and a half to complete the first phase development. It was the first time the team worked together on a project and all members had other duties assigned throughout the duration of the initiative. Software Implementations and Tools that Worked Well  Separation of Concerns greatly improves the flexibility for adding features and functionality.  Open backend infrastructure fully supports the services reuse across varying business rules.  The Types and Type Mappings Declarative Markup Language provides the foundation for much of the decision support capabilities and improves the front end user experience.  SOA architecture is key to this integration initiative and is the corner stone of the implementation. Figure 8 Convert NTV Plan Data Access Results to NTV Plan Business Object
  • 12. 12  Coding and Naming Standards significantly improve enhancement capabilities, troubleshooting efforts, and the ability for developers to understand and maneuverer the system. Improvements and Considerations  Needed more developers (or less work) to take the time to design the NTVPlan and NTV Certificate tables as a more generic implementation that could accommodate other regulations. Note: NTV Plans are not governed through the same regulations as all other contingency plans.  Some say SQL IDENTITY columns should not be used for primary / foreign key implementation. Would it be better to use some other type of identifier? These Identity columns make development of the backend so easy.  Some database programmers do not like the lack of database referential integrity for the relationship between the Plan, Affiliation, Document and Facility databases.  Would have liked to create the service and model layout that fully accommodates a Restful architecture. No team expert on Restful and too little time to prove a good ROI for its implementation.