Your SlideShare is downloading. ×
A Metadata Architecture For Enterprise-Wide Data Sharing
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

A Metadata Architecture For Enterprise-Wide Data Sharing

1,016
views

Published on


0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,016
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
70
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Title Chart: Self-explanatory
  • Relatively self-explanatory. Icons represent complex networks of both functionally oriented and organizationally oriented systems of independent AISs sharing data item Values among each other. The entire system involves somewhere between 3000 – 5000 application systems.
  • This is just one example from the logistics domain of a cross-organizational and cross functional network of application systems That share data among themselves and also feed a joint system with logistics data necessary for command and control This example Involves about 73 critical applications and 50+ contributing systems.
  • The Defense Data Dictionary System (DDDS) is a repository that was designed for the purpose of capturing and storing the metadata that describes approved DoD standardized data elements. The concept being that these SDEs are the enterprise level data elements that should be employed by DoD application systems in order to be interoperable with each other.
  • Self-explanatory.
  • Self-explanatory
  • Self-explanatory but should emphasize that this architectural concept is not intended to be a solution that will fix every impediment to interoperable data sharing, but it will fix those related to data element naming problems.
  • These are the four layers of “data element design” that have to be accomplished and accommodated anytime we are trying to insure interoperability among disparate data information systems. Whether they realize it or not, these four layers of design are almost always accomplished by information system designers/developers every time they begin to either create a new data information system or to make major modifications to an existing system. Although, in most cases it is not done (or documented!) in any formal or structured process resulting in poor quality designs, and that is one reason many new systems often fail to meet business functional requirements. Historically, system developers have been notorious for giving short shrift to considering thorough requirements analysis and data architecture needs before beginning to write code.
  • The first level (layer) of abstraction in designing an architecture for an information system data schema. Some functional area data modeling working group teams created and proposed standard data elements at this level of abstraction that were approved for inclusion in the DDDS as DoD standardized data elements.
  • The second level of data architecture design which takes the previous Layer 1 design and represents it in a specific technology schema such as an SQL schema or COBOL copybook file description, etc. Notice that there is no specific relationship defined between entity at Layer 1 and Table at Layer 2. At the technology “implemented” layer a design might include entities from multiple “functional area” models from Layer 1. So that if I were representing a logistics model of inventory data element requirements in Layer 2 and some of the data elements I require have to do with cost and price amounts, I might source them from an entity from the the financial management domain at Layer one rather than recreating them in my logistics functional area model, i.e., reuse.
  • This design level is the instantiation of the technology based schema into a particular vendor’s implementation of the schema as an operational DBMS. If the technology based schema was a standard SQL schema, then it is a virtual certainty that each vendors SQL DBMS schema that is based on the Layer 2 SQL standard schema will be modified in some manner since most vendors will add their own unique extensions/modifications to the SQL standard in order to improve the performance of their DBMS or to create some functionality that the SQL standard may not support. These extensions/modifications almost always make seamless and transparent data sharing either very difficult or impossible between DBMSs from different vendors that are based on the same Layer 2 SQL schema.
  • Finally this level represents the design of an application systems internal view schema taken from the DBMS for internal processing in the application system. This layer assumes that the application does not directly access the data resident in the DBMS database. All application system data manipulation operations that will result in modifying data item values in the DBMS database in some fashion (create, update, delete) must be processed through the DBMS and conform to DBMS business rules for data integrity. This is a very important information system design principle that exists to protect the accuracy and integrity of process data in the DBMS database.
  • Now into this “design environment” we will introduce the concept of a metadata repository (a special kind of database) whose purpose, or function, is to capture and store the characteristics, or the “metadata”, that defines and describes the data elements that are created as a product of this data architecture design framework. But we are not interested in just any kind of data element. Instead we we are only interested in capturing metadata for “Enterprise” type data elements– that is data elements whose data item values are always interpreted by any sending system and any receiving system in the enterprise to mean the same thing and represented and processed in the same way. This is the kind of metadata repository the DDDS was intended to be.
  • The process for identifying and creating “enterprise” level data elements for populating the DDDS is defined in the DoD 8320 series of policies and procedures. It basically begins at the “bottom” with data modeling working groups of functional subject matter experts from a particular functional/operational process led by an “experienced” data modeler. The goal of these groups is to develop high level, conceptual, logical models of the data elements employed by their operations/processes and to develop the metadata that define and describes those data elements. A major objective of this effort is to surface and identify data elements that represent concepts that are shared (mean the same thing across the entire enterprise). These data elements are then proposed for registration in the DDDS as enterprise level data elements– so called DoD standard data elements (SDEs). The DDDS then is intended to be a repository that is populated with data elements that represent common enterprise level data element concepts– one named enterprise level data element for each unique enterprise level data element concept. If more than one named data element for a unique enterprise data element concept were permitted to exist in the DDDS, then this would represent redundancy and ambiguity with respect to the data element concept that they represent. This is exactly what has happened in the DDDS. Over time, as different independent functional area data modeling groups created data elements, there was no consistency among them in how they represented and named the data elements they created, defined and named. Some groups took a physical system view approach and used application view/field names, others focused on A DBMS schema and its column names as the basis for their models, and so on. And because as we have seen in the process of architecting a design for the data elements of an information system that the same data element concept can be represented by different attribute names, schema column names, DBMS column names, and view column/field names, the consequence is that the 17000+ SDEs in the DDDS contain many redundancies and ambiguity because they are mostly representative of attributes in functional area models, columns in schemas, DBMS columns in DBMS table structures, and view columns in application table structures.
  • Self-explanatory
  • The missing layer
  • Transcript

    • 1. A Metadata Architecture For Enterprise-Wide Data Sharing
    • 2. Same Data Requirements Different Functional Needs, Same Descriptions, Different Names Department of Defense (DoD) Data Interoperability Challenge Logistics Components/ Services Medical Procurement Finance Transportation Personnel Command & Control
    • 3. GATES MTMC AMS CMOS TCACCIS G081 BROKER GOPAX ADANS JALIS DoD Joint Applications GCCS & GCSS (IDEs) CLAS PENTAGON CLAS PACOM CLAS USFK JTAV CLAS JFCOM CLAS EUCOM CLAS CENTCOM UNCLAS PENTAGON UNCLAS PACOM Server UNCLAS USFK JTAV Server UNCLAS JFCOM Servers UNCLAS EUCOM Server UNCLAS CENTCOM (SOUTHCOM SOCOM) Server TRANSCOM GTN DLA USA COALITION USN USMC USAF MILSEALIFT Commercial JMAR DARPA GSA ISSE GUARD ISSE GUARD USCG DoD TARGET DATA SHARING ENVIRONMENT A Logistics Perspective ATAC MRP II GCCS = Global Command & Control System GCSS = Global Combat Support System Essential AIS Contributing AIS Legend: MIMMS ATLAS II CAMS AMMRL IMRL NAOMIS ATAC NSIPS NALDA II U2 SAUDI ARABIA QATAR KUWAIT UAE MEXICO JAPAN FRANCE UK CANADA DVD Prime Vendors (personnel?) ARAMIS CMIS SUPPLY MGMT SCS CRIM CAIMS GDSS WPS IBS CFM DITTS RFT-E RFT-K COP CSE GCCS - C2 MC-TFS LIF DSS CC SS AWRDS APS SAILS SAMS TAMMIS MEDSUP ALP JL ACTD DOD CAV TAMMIS MEDASM MEDSILS UDR VMI NAV NAC MEDLOG DBSS MUFFIN SNAP/SUADPS/FIMAR FUELS - NEURS CAIMS ATAC MANPERS SBSS CAS A GO81 AFEMS DO35 MAARS II MPS (BIC) SCS SASSY FIMAR CASEMIS ISRAEL OAS ASIAN SEATO NATO TCAIMS II SARSS SAAS-MOD ROAMS TAPDB TAMMIS ATAV (LIDB) WARS AMMIS ARMS PMIS ATEMS SPS/SDW SAMMS FLIS JECPO URD DAAS LOTS IC3 DISA TAPE TAPE TAPE Freight Links SCCR CAIMS
    • 4. The DDDS: The Current DoD Repository of DoD Standardized Data Elements Intended to be the DoD Repository of Data Elements to Support DoD Enterprise-wide Interoperable Data Sharing METADATA REPOSITORY Defense Data Dictionary System (DoD Standardized Data Elements {SDE}) 17000+ SDEs
    • 5. PART I: The Problem
    • 6. Current Problems 1. Incorrect data architecture abstraction level for representing Enterprise Level Data Elements for interoperable data sharing 2. Numerous redundant representations of Standardized Data Elements (SDEs) ( DIFFERENT NAMES – SAME DEFINITION ) 3. Incomplete, non-existent and/or, non-current SDE metadata 4. Inadequate categories of SDE metadata 5. Inadequate support / enforcement of data administration processes for data management
    • 7. A Data Element Name and Data Element Definition Refresher: Two Data Element Metadata Attributes
      • Data Element Name is a label given to establish Data Element identity
      • Data Element Definition is a description providing complete,
      • unambiguous meaning represented by a Data Element
      • Name and Definition together provide the semantic context for the
      • data item values represented by a Data Element
      • Some key concepts/principles/facts about Data Element Naming and Definition:
        • NAME and DEFINITION are inseparable
        • NAME is a unique identifier for DEFINITION
        • A NAME can be viewed as a kind of very short DEFINITION
        • In order of precedence, DEFINITION creation should always
        • precede NAME creation
        • Impossible to correctly NAME a data element with precision without a DEFINITION
        • NAME and DEFINITION are at the core of the data integration / sharing process
        • Many data sharing issues arise from “bad” data element NAMING and
        • DEFINITION practices
    • 8. A Proposed Metadata Architecture for Shared Enterprise Data Elements
      • Focuses on solutions for:
        • Problem 1 – Incorrect data architecture abstraction level
        • Problem 2 – Differently named data elements for the same
        • data element concept
        • Problem 4 - Inadequate categories of standardized data element
        • metadata
      • Not a solution for every kind of impediment to
      • interoperable data sharing
      • Problems 3 and 5 require quality improvements in
      • process execution and in data management governance
      • and will be addressed in Part II.
      • Will begin by looking more closely at the
      • fundamental metadata architecture levels…….
    • 9.
      • Specified Context Data Model Layer
        • Roughly analogous to high level conceptual Entity-Relationship (E-R) data
        • models of functional area/business domain, or Community of Interest (COI)
        • data depicting the structure and relationships of entities and their attributes.
      • Implemented Technology Data Model Layer
        • Database schemas represented in a particular technology (SQL, COBOL, etc)
        • based on fully attributed 3 rd normal form logical data models
      • Operational (Vendor) DBMS Data Model Layer
        • Roughly analogous to a physical data model representing a particular
        • vendor’s version of a technology based schema, i.e., Oracle SQL DBMS vs
        • IBM DB2 SQL DBMS vs Sybase SQL DBMS, etc, etc
      • Business Application View Data Model Layer
        • Represents the application system access interface to a DBMS which
        • preserves the separation and integrity of the database data from systems
        • that operate on and manipulate the data
      Design Layers of a Business Information System Database Architecture
    • 10.
      • May be represented in one, a combination of, or all of the following views:
        • Entity w/attributes; no key designations; un-normalized; unresolved many – to – many relationships
        • Key based entities; un-normalized; un-resolved many – to – many relationships
        • Fully attributed; un-normalized; resolved or un-resolved many – to – many relationships
        • Fully attributed; 3 rd normal form; developed sub-typing; resolved many – to manys
      • Current DDDS / DDA functional / subject area domains map to domains represented by
      • designated DoD Functional Data Administrators (FDAds):
        • Logistics DUSD(L)
        • Personnel USD(P&R)
        • Comptroller USD(C)
        • Health Affairs ASD(HA)
        • Etc, etc
      • Each DoD “Subject Area” DAd should have a “specified” data model that represents the data element
      • structures and relationships of functional area data element requirements for their respective functional area
      • or community of interest.
      Layer 1
      • Specified Context (Community of Interest) Data Model Layer
      ATTRIBUTE ENTITY SUBJECT “ PERSONNEL” FUNCTIONAL AREA DEPENDENT DATA MODEL TEMPLATES ATTRIBUTE ENTITY SUBJECT “ LOGISTICS” FUNCTIONAL AREA DEPENDENT DATA MODEL TEMPLATES Design Layers for a Business Information System Database Architecture ATTRIBUTE ENTITY SUBJECT “ FINANCIAL” FUNCTIONAL AREA DEPENDENT DATA MODEL TEMPLATES “ SPECIFIED” DATA MODEL LAYER (DoD FDAd Domain)
    • 11. Layer 1 Layer 2
      • 3 RD Normal form ERD logical data model
      • Represented in a technology dependent data architecture schema
      • Technology driven / constrained data element naming
      • Subject area entity and attribute templates are deployed into schema tables and
      • columns that must conform to a particular chosen technology such as COBOL or SQL.
      • Layer 1 attribute metadata is inherited by Layer 2 columns
      • The relationship between Layer 1 and Layer 2 is one – to – many. That is to say that
      • any attribute from Layer 1 may be deployed as a column into many Layer 2 schemas.
      • Implemented Technology Data Model Layer
      TECHNOLGY DEPENDENT (e.g.,IDMS,) FULLY ATTRIBUTED, LOGICAL DATA MODEL COLUMN TABLE SCHEMA TECHNOLGY DEPENDENT (e.g.,COBOL,) FULLY ATTRIBUTED, LOGICAL DATA MODEL COLUMN TABLE SCHEMA TECHNOLGY DEPENDENT (e.g.,SQL,) FULLY ATTRIBUTED, LOGICAL DATA MODEL “ IMPLEMENTED” DATA MODEL LAYER (Data Architect / Modelers Domain) COLUMN TABLE SCHEMA Design Layers for a Business Information System Database Architecture ATTRIBUTE ENTITY SUBJECT FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES “ SPECIFIED” DATA MODEL LAYER (DoD COI DAd Domain)
    • 12. Layer 1 Layer 2 Layer 3
      • Roughly analogous to a physical data model
      • Vendor’s versions of particular technology based schema such as SQL,
      • i.e., Oracle SQL DBMS vs Informix SQL DBMS, vs Sybase SQL DBMS, etc, etc.
      • Data element naming is bound by vendor’s implemented DBMS business rules
      • for a particular technology based schema.
      • Again, the relationship between Layer 2 and Layer 3 is one – to – many. That is to say that
      • any column from a Layer 2 schema may be deployed as a DBMS column in many Layer 3 DBMSs.
      ATTRIBUTE ENTITY SUBJECT FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES “ SPECIFIED” DATA MODEL LAYER (DoD COI DAd Domain)
      • Operational Vendor DBMS Data Model Layer
      TECHNOLGY DEPENDENT (SQL, COBOL, ETC), FULLY ATTRIBUTED, LOGICAL DATA MODEL “ IMPLEMENTED” DATA MODEL LAYER (Data Architect / Modelers Domain) COLUMN TABLE SCHEMA DBMS COLUMN DBMS TABLE DBMS SCHEMA DBMS DEPENDENT (e.g., Sybase) DBMS DATA MODEL DBMS COLUMN DBMS TABLE DBMS SCHEMA DBMS DEPENDENT (e.g., DB2) DBMS DATA MODEL DBMS COLUMN DBMS TABLE DBMS SCHEMA DBMS DEPENDENT (e.g., Oracle) DBMS DATA MODEL “ OPERATIONAL” DATA MODEL LAYER (Domain of Database Administrators (DBAs)) Design Layers for a Business Information System Database Architecture
    • 13. Layer 4
      • Data element naming in conformance with functional area common business language terms
      • Finally, with respect to a single DBMS, the relationship between Layers 3 and 4 is also one– to – many
      • from 3 to 4. That is, a DBMS column may be deployed as view columns in many applications
      • that may interface with a particular DBMS.
      Business Application View Data Model Layer ATTRIBUTE ENTITY SUBJECT FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES “ SPECIFIED” DATA DATA MODEL LAYER (DoD COI DAd Domain) Layer 1 TECHNOLGY DEPENDENT (SQL, COBOL, ETC), FULLY ATTRIBUTED, LOGICAL DATA MODEL “ IMPLEMENTED” DATA MODEL LAYER (Data Architect / Modelers Domain) COLUMN TABLE SCHEMA Layer 2 DBMS COLUMN DBMS TABLE DBMS SCHEMA DBMS DEPENDENT (Oracle, DB2, Sybase, etc) DBMS DATA MODEL “ OPERATIONAL” DATA MODEL LAYER (Domain of Database Administrators (DBAs)) Layer 3 APPLICATION VIEW COLUMN APPLICATION VIEW TABLE BUSINESS INFORMATION SYSTEM BUSINESS APPLICATION SYSTEM VIEW DATA MODEL (Command & Control) APPLICATION VIEW COLUMN APPLICATION VIEW TABLE BUSINESS INFORMATION SYSTEM BUSINESS APPLICATION SYSTEM “ VIEW” DATA MODEL (Personnel App) APPLICATION VIEW COLUMN APPLICATION VIEW TABLE BUSINESS INFORMATION SYSTEM BUSINESS APPLICATION SYSTEM “ VIEW” DATA MODEL (Logistics App) (Domain of Application System Managers (SMs and/or PMs) Design Layers for a Business Information System Database Architecture
    • 14. Enterprise Registry of Standardized Data Elements (SDE) To Represent Common Enterprise Data Element Concepts Enterprise Data Element Concept “A” Effective Result : Four differently named versions, or, representations of the Enterprise common Data Element Concept, “A”, that will exist in the Registry as Standardized Data Elements. Redundancy and ambiguity is the consequence. The Problem: Sourcing Enterprise “Context Independent” Data Element Standards from Enterprise “Context Dependent” Data and Information Systems and Databases Enterprise Registry of Standardized Shared Data Elements: The DDDS Layer 1 Person Given Name Layer 2 Salesman First Name Layer 4 Name Layer 3 EFN Standalone Database #1 Enterprise Common Data Element Concept “A” Layer 1 Sailor Given Name Layer 2 Sailor First Name Layer 4 First Name Layer 3 Sail_Frst_Nm Standalone Database #2 Enterprise Common Data Element Concept “A” Layer 1 Employee Given Name Layer 2 Employee First Name Layer 4 Given Name Layer 3 Emp_Gv_Nam Standalone Database #4 Enterprise Common Data Element Concept “A” SDE “A”: Person Given Name SDE “A”: Employee First Name SDE “A”: Sail_Frst_Nm SDE “A”: Legal First Name Standalone Database #3 Enterprise Common Data Element Concept “A” Layer 1 Legal Given Name Layer 2 Authoritative First Name Layer 4 Legal First Name Layer 3 Lg_Auth_Frst_Nm
    • 15. Layer 1
      • Database System Architecture Design Steps
        • Specified Context Data Model Layer
        • Implemented Technology Data Model Layer
        • Operational Vendor DBMS Data Model Layer
        • Business Application View Data Model Layer
      Layer 2 Layer 3 Layer 4 The DDDS Repository Intended to represent DoD globally shared enterprise standard data elements. Thus, the repository should contain only one named data element standard for each unique enterprise level data element concept. ATTRIBUTE ENTITY SUBJECT FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES “ SPECIFIED” DATA MODEL LAYER (DoD COI DAd Domain) TECHNOLGY DEPENDENT (SQL, COBOL, ETC), FULLY ATTRIBUTED, LOGICAL DATA MODEL “ IMPLEMENTED” DATA MODELLAYER (Data Architect / Modelers Domain) COLUMN TABLE SCHEMA DBMS COLUMN DBMS TABLE DBMS SCHEMA DBMS DEPENDENT (Oracle, DB2, Sybase, etc) DBMS DATA MODEL “ OPERATIONAL” DATA MODEL LAYER (Domain of Database Administrators (DBAs)) APPLICATION VIEW COLUMN APPLICATION VIEW TABLE BUSINESS INFORMATION SYSTEM BUSINESS APPLICATION SYSTEM “ VIEW” DATA MODEL (Domain of Application System Managers (SMs and/or PMs) METADATA REPOSITORY Defense Data Dictionary System (DoD Standardized Data Elements) 17000+ SDEs Design Layers for a Business Information System Database Architecture
    • 16. Layer 1
      • Database System Architecture Design Steps
        • Specified Context Data Model Layer
        • Implemented Technology Data Model Layer
        • Operational Vendor DBMS Data Model Layer
        • Application View Data Model Layer
      Layer 2 Layer 3 Layer 4 METADATA REPOSITORY Defense Data Dictionary System The DDDS Repository Intended to represent DoD globally shared enterprise standard data elements. Thus, the repository should contain only one named data element standard for each unique enterprise level data element concept. In reality, the repository contains many cases of differently named data elements that represent the same data element concept. The result is uncontrolled redundancy and ambiguity incapable of supporting seamless and interoperable data sharing. A source for DDDS SDEs A source for DDDS SDEs A source for DDDS SDEs A source for DDDS SDEs One GIGANTIC semantic mess 17000+ SDEs ATTRIBUTE ENTITY SUBJECT FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES “ SPECIFIED” DATA MODEL LAYER (DoD COI DAd Domain) TECHNOLGY DEPENDENT (SQL, COBOL, ETC), FULLY ATTRIBUTED, LOGICAL DATA MODEL “ IMPLEMENTED” DATA MODEL LAYER (Data Architect / Modelers Domain) COLUMN TABLE SCHEMA DBMS COLUMN DBMS TABLE DBMS SCHEMA DBMS DEPENDENT (Oracle, DB2, Sybase, etc) DBMS DATA MODEL “ OPERATIONAL” DATA MODEL LAYER (Domain of Database Administrators (DBAs)) APPLICATION VIEW COLUMN APPLICATION VIEW TABLE BUSINESS INFORMATION SYSTEM BUSINESS APPLICATION SYSTEM “ VIEW” DATA MODEL (Domain of Application System Managers (SMs and/or PMs) Design Layers for a Business Information System Database Architecture
    • 17. … .But, Where’s the Beef ?? .…the Enterprise Data Element “Beef” ?? The DDDS “ Big Bun”
    • 18. Design Layers for a Business Information System Data Architecture DATA ELEMENT CONCEPT STRUCTURE DATA ELEMENT CONCEPT ISO 11179 BUSINESS CONTEXT INDEPENDENT DATA ELEMENT REPRESENTATION Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements) (Domain of DoD Data Administration) Layer 0
      • The Enterprise Data Element Layer
      • ISO 11179 Naming and Definition
      • Context Independent Data Elements
        • Uniform Naming
        • Uniform Semantics
        • Uniform Value Domains
      DATA ELEMENT CONCEPT STRUCTURE TYPE DATA ELEMENT CONCEPT CONCEPT STRUCTURE CONCEPT STRUCTURE TYPE CONCEPTUAL VALUE DOMAIN CONCEPTUAL VALUE DOMAIN STRUCTURE CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE VALUE DOMAIN VALUE DOMAIN STRUCTURE VALUE DOMAIN STRUCTURE TYPE
    • 19. Layer 1 Layer 2 Layer 3 Layer 4 Design Layers for a Business Information System Data Architecture One gigantic semantic mess- redundancies & ambiguities DDDS Source DDDS Source DDDS Source DDDS Source DATA ELEMENT CONCEPT STRUCTURE DATA ELEMENT CONCEPT CONCEPTUAL VALUE DOMAIN ISO 11179 BUSINESS CONTEXT INDEPENDENT DATA ELEMENT REPRESENTATION Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements) (Domain of DoD Data Administration) Layer 0 ATTRIBUTE ENTITY SUBJECT FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES “ SPECIFIED” DATA MODEL LAYER (DoD FDAd Domain) TECHNOLGY DEPENDENT (SQL, COBOL, ETC), FULLY ATTRIBUTED, LOGICAL DATA MODEL “ IMPLEMENTED” DATA MODEL LAYER (Data Architects / Modelers Domain) COLUMN TABLE SCHEMA DBMS COLUMN DBMS TABLE DBMS SCHEMA DBMS DEPENDENT (Oracle, DB2, Sybase, etc) DBMS DATA MODEL “ OPERATIONAL” DATA MODEL LAYER (Domain of Database Administrators (DBAs)) APPLICATION VIEW COLUMN APPLICATION VIEW TABLE BUSINESS INFORMATION SYSTEM BUSINESS APPLICATION SYSTEM “ VIEW” DATA MODEL LAYER (Domain of Application System Managers (SMs and/or PMs) The DDDS: 17000+ SDEs DATA ELEMENT CONCEPT STRUCTURE TYPE DATA ELEMENT CONCEPT CONCEPT STRUCTURE CONCEPT STRUCTURE TYPE CONCEPTUAL VALUE DOMAIN STRUCTURE CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE VALUE DOMAIN VALUE DOMAIN STRUCTURE VALUE DOMAIN STRUCTURE TYPE
    • 20. CONCEPT CONCEPT STRUCTURE CONCEPT STRUCTURE TYPE VALUE DOMAIN VALUE DOMAIN STRUCTURE VALUE DOMAIN STRUCTURE TYPE DATA ELEMENT CONCEPT STRUCTURE TYPE DATA ELEMENT CONCEPT STRUCTURE DATA ELEMENT CONCEPT ATTRIBUTE COLUMN DBMS COLUMN ENTITY SUBJECT TABLE SCHEMA DBMS TABLE DBMS SCHEMA ISO 11179 BUSINESS CONTEXT INDEPENDENT DATA ELEMENT REPRESENTATION FUNCTIONALLY DEPENDENT & TECHNOLOGY INDEPENDENT DATA MODEL TEMPLATES ATTRIBUTE  INHERITS DATA ELEMENT “ SPECIFIED” DATA MODEL (DoD FDAd Domain) TECHNOLGY DEPENDENT & DBMS INDEPENDENT MODEL / SCHEMA COLUMN  INHERITS ATTRIBUTE “ IMPLEMENTED” DATA MODEL) (Data Architects / Modelers Domain) DBMS DEPENDENT & APPLICATION VIEW INDEPENDENT DBMS COLUMN (Oracle, DB2, etc) INHERITS  COLUMN “ OPERATIONAL” DATA MODEL (Domain of Database Administrators (DBAs)) APPLICATION VIEWS OF DBMS TABLES & COLUMNS “ VIEW” DATA MODEL (Domain of Application System Managers (SMs and/or PMs) Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements) (Domain of DoD Data Administration) META MODEL ARCHITECTURE SUPPORTING ENTERPRISE WIDE SHARED DATA CONCEPTUAL VALUE DOMAIN CONCEPTUAL VALUE DOMAIN STRUCTURE CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE DATA ELEMENT VIEW VIEW COLUMN BUSINESS INFORMATION APPLICATION SYSTEM VIEW COLUMN STRUCTURE VIEW COLUMN STRUCTURE PROCESS VIEW COLUMN STRUCTURE TYPE Data sharing occurs at the “operational and application” view layers. Made possible through the relationships between all layers represented by metadata in a repository that enables relating syntax, structure, and semantics from any layer to a common ISO 11179 standard representation. METADATA REPOSITORY Implemented Model Operational DBMS Application View Specified Model ISO 11179
    • 21. Layer 1 Layer 2 Layer 3 Layer 4 Design Layers for a Business Information System Data Architecture DATA ELEMENT CONCEPT STRUCTURE DATA ELEMENT CONCEPT ISO 11179 BUSINESS CONTEXT INDEPENDENT DATA ELEMENT REPRESENTATION Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements) (Domain of DoD Data Administration) Layer 0 ATTRIBUTE ENTITY SUBJECT FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES “ SPECIFIED” DATA MODEL LAYER (DoD FDAd Domain) TECHNOLGY DEPENDENT (SQL, COBOL, ETC), FULLY ATTRIBUTED, LOGICAL DATA MODEL “ IMPLEMENTED” DATA MODEL LAYER (Data Modelers Domain) COLUMN TABLE SCHEMA DBMS COLUMN DBMS TABLE DBMS SCHEMA DBMS DEPENDENT (Oracle, DB2, Sybase, etc) DBMS DATA MODEL “ OPERATIONAL” DATA MODEL LAYER (Domain of Database Administrators (DBAs)) APPLICATION VIEW COLUMN APPLICATION VIEW TABLE BUSINESS INFORMATION SYSTEM BUSINESS APPLICATION SYSTEM “ VIEW” DATA MODEL LAYER (Domain of Application System Managers (SMs and/or PMs) DATA ELEMENT CONCEPT STRUCTURE TYPE DATA ELEMENT CONCEPT CONCEPT STRUCTURE CONCEPT STRUCTURE TYPE CONCEPTUAL VALUE DOMAIN CONCEPTUAL VALUE DOMAIN STRUCTURE CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE VALUE DOMAIN VALUE DOMAIN STRUCTURE VALUE DOMAIN STRUCTURE TYPE DoD CORE DATA ELEMENT METADATA REPOSITORY Implemented Model Layer Operational DBMS Layer Application View Layer Specified Model Layer ISO 11179 Model Layer
    • 22. CONCEPT CONCEPT STRUCTURE CONCEPT STRUCTURE TYPE VALUE DOMAIN VALUE DOMAIN STRUCTURE VALUE DOMAIN STRUCTURE TYPE DATA ELEMENT CONCEPT STRUCTURE TYPE DATA ELEMENT CONCEPT STRUCTURE DATA ELEMENT CONCEPT ATTRIBUTE COLUMN DBMS COLUMN ENTITY SUBJECT TABLE SCHEMA DBMS TABLE DBMS SCHEMA ISO 11179 BUSINESS CONTEXT INDEPENDENT DATA ELEMENT REPRESENTATION FUNCTIONALLY DEPENDENT & TECHNOLOGY INDEPENDENT DATA MODEL TEMPLATES ATTRIBUTE  INHERITS DATA ELEMENT “ SPECIFIED” DATA MODEL (DoD FDAd Domain) TECHNOLGY DEPENDENT & DBMS INDEPENDENT MODEL / SCHEMA COLUMN  INHERITS ATTRIBUTE “ IMPLEMENTED” DATA MODEL) (Data Architects / Modelers Domain) DBMS DEPENDENT & APPLICATION VIEW INDEPENDENT DBMS COLUMN (Oracle, DB2, etc) INHERITS  COLUMN “ OPERATIONAL” DATA MODEL (Domain of Database Administrators (DBAs)) APPLICATION VIEWS OF DBMS TABLES & COLUMNS “ VIEW” DATA MODEL (Domain of Application System Managers (SMs and/or PMs) Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements) (Domain of DoD Data Administration) META MODEL ARCHITECTURE SUPPORTING ENTERPRISE WIDE SHARED DATA CONCEPTUAL VALUE DOMAIN CONCEPTUAL VALUE DOMAIN STRUCTURE CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE DATA ELEMENT VIEW VIEW COLUMN BUSINESS INFORMATION APPLICATION SYSTEM VIEW COLUMN STRUCTURE VIEW COLUMN STRUCTURE PROCESS VIEW COLUMN STRUCTURE TYPE Data sharing occurs at the “operational and application” view layers. Made possible through the relationships between all layers represented by metadata in a repository that enables relating syntax, structure, and semantics from any layer to a common ISO 11179 standard representation. METADATA REPOSITORY Implemented Model Operational DBMS Application View Specified Model ISO 11179
    • 23. Supply Item Resource Quantity Supply Item Resource Quantity Supply Item Resource Quantity Supply Item Resource Quantity Functional/Organizational Context Dependent “ Specified” Model Army Logistics Management Navy Logistics Management Technology Dependent “ Implemented” Model ANSI SQL ANSI SQL Vendor Dependent SQL DBMS “ Operational” Model “ Oracle” DBMS “ Sybase” DBMS Business Application Information System (AIS) “ View” Model Navy UADPS (AIS) Army SAMS (AIS) Concepts Data Element Concept Value Domain Data Element View Column Names DBMS Column Names SQL Column Names Attribute Names Business Fact Semantic Template Name Metadata Repository Architecture of Related Representations of DoD Enterprise Shared Data Elements in Support of Data and Information Sharing The quantity of each type of Federal Supply System materiel item contained in an identifiable inventory of materiel objects. Data Element Definition: Additional Data Element Structural Metadata: Supply Item Resource Quantity Materiel Resource Physical Item Balance Quantity Physical Measure Conceptual Value Domain Data type characteristics, local definition, enumerated values ( if specific ), etc. ISO 11179 Context Inde pendent Data Element Representation Meta Model An Optimal Application of an ISO 11179 Based Data Element Architecture for Resolving Disparate Representations of Shared Enterprise Data Elements Supply Item Resource Quantity Supply Item Resource Quantity Supply Item Resource Quantity Supply Item Resource Quantity METADATA REPOSITORY Defense Data Dictionary System (DoD Standardized Data Elements) 17000+ SDEs
    • 24. The “Optimal” Solution
    • 25. Mat_Itm_Inv_Qt Materiel Inventory Quantity Materiel Item Inventory Quantity Mat_Inv_Qty Materiel Unit Inventory Quantity Supply Unit Quantity Stocked Materiel Quantity Ships Stores Quantity Functional/Organizational Context Dependent “ Specified” Model Army Logistics Management Navy Logistics Management Technology Dependent “ Implemented” Model ANSI SQL ANSI SQL Vendor Dependent SQL DBMS “ Operational” Model “ Oracle” DBMS “ Sybase” DBMS Business Application Information System (AIS) “ View” Model Navy UADPS (AIS) Army SAMS (AIS) Concepts Data Element Concept Value Domain Data Element CORE METADATA REPOSITORY Implemented Model Operational DBMS Application View Specified Model ISO 11179 Model View Column Names DBMS Column Names SQL Column Names Attribute Names Business Fact Semantic Template Name Metadata Repository Architecture of Related Representations of DoD Enterprise Shared Data Elements in Support of Data and Information Sharing The quantity of each type of Federal Supply System materiel item contained in an identifiable inventory of materiel objects. Data Element Definition: Additional Data Element Structural Metadata: Supply Item Resource Quantity Materiel Resource Physical Item Balance Quantity Physical Measure Conceptual Value Domain Data type characteristics, local definition, enumerated values ( if specific ), etc. ISO 11179 Context Inde pendent Data Element Representation Meta Model Example Logistics Application of an ISO 11179 Based Data Element Architecture for Relating Disparate Representations of Shared Enterprise Data Elements
    • 26. Sail_Rat_Cde Soldier Rank Code Sailor Rating Code Sold_Rnk_Cd Unit Member Rank Code Squad Member Rank Code Crew Member Rating Code Launch Team Member Rating Code Functional/Organizational Context Dependent “ Specified” Model Army Personnel Management Navy Personnel Management Technology Dependent “ Implemented” Model ANSI SQL ANSI SQL Vendor Dependent SQL DBMS “ Operational” Model “ Oracle” DBMS “ Sybase” DBMS Business Application Information System (AIS) “ View” Model Navy (AIS) Army (AIS) Concepts Data Element Concept Value Domain Data Element CORE METADATA REPOSITORY Implemented Model Operational DBMS Application View Specified Model ISO 11179 Model View Column Names DBMS Column Names SQL Column Names Attribute Names Business Fact Semantic Template Name Metadata Repository Architecture of Related Representations of DoD Enterprise Shared Data Elements in Support of Data and Information Sharing The code that represents the level of authority and responsibility occupied by Person in a hierarchy of levels ranging from most superior to most subordinate in which each level is subordinate to levels above and superior to levels below. Data Element Definition: Additional Data Element Structural Metadata: Person Grade Code Human Resource Personnel Classifi- cation Grade Code Personnel Ranking Measure Conceptual Value Domain Data type characteristics, etc. ISO 11179 Context Inde pendent Data Element Representation Meta Model Example Personnel Application of an ISO 11179 Based Data Element Architecture for Relating Disparate Representations of Shared Enterprise Data Elements
    • 27. Part II: Implementing the Metadata Architecture For Enterprise-wide Data Sharing in a Legacy System Environment
    • 28. Table of Contents
      • A Realistic and Practical Approach to Achieve the Ideal Solution
      • How Do We Get to the Ideal?
      • Find Our Metadata
      • Perform Smart Meta -Data Mining
      • Find the Right Starting Layer
      • Reverse Engineer to Build the Upper Layers
      • Overall Forward Engineering Process
      • Process Statistics
      • Lessons Learned
    • 29. CONCEPT CONCEPT STRUCTURE CONCEPT STRUCTURE TYPE VALUE DOMAIN VALUE DOMAIN STRUCTURE VALUE DOMAIN STRUCTURE TYPE DATA ELEMENT CONCEPT STRUCTURE TYPE DATA ELEMENT CONCEPT STRUCTURE DATA ELEMENT CONCEPT ATTRIBUTE COLUMN DBMS COLUMN ENTITY SUBJECT TABLE SCHEMA DBMS TABLE DBMS SCHEMA ISO 11179 BUSINESS CONTEXT INDEPENDENT DATA ELEMENT REPRESENTATION FUNCTIONALLY DEPENDENT & TECHNOLOGY INDEPENDENT DATA MODEL TEMPLATES ATTRIBUTE  INHERITS DATA ELEMENT “ SPECIFIED” DATA MODEL (DoD FDAd Domain) TECHNOLGY DEPENDENT & DBMS INDEPENDENT MODEL / SCHEMA COLUMN  INHERITS ATTRIBUTE “ IMPLEMENTED” DATA MODEL) (Data Architects / Modelers Domain) DBMS DEPENDENT & APPLICATION VIEW INDEPENDENT DBMS COLUMN (Oracle, DB2, etc) INHERITS  COLUMN “ OPERATIONAL” DATA MODEL (Domain of Database Administrators (DBAs)) APPLICATION VIEWS OF DBMS TABLES & COLUMNS “ VIEW” DATA MODEL (Domain of Application System Managers (SMs and/or PMs) Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements) (Domain of DoD Data Administration) 1.0 META MODEL ARCHITECTURE SUPPORTING ENTERPRISE WIDE SHARED DATA CONCEPTUAL VALUE DOMAIN CONCEPTUAL VALUE DOMAIN STRUCTURE CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE DATA ELEMENT VIEW VIEW COLUMN BUSINESS INFORMATION APPLICATION SYSTEM VIEW COLUMN STRUCTURE VIEW COLUMN STRUCTURE PROCESS VIEW COLUMN STRUCTURE TYPE Data sharing occurs at the “operational and application” view layers. Made possible through the relationships between all layers represented by metadata in a repository that enables relating syntax, structure, and semantics from any layer to a common ISO 11179 standard representation. METADATA REPOSITORY Implemented Model Operational DBMS Application View Specified Model ISO 11179
    • 30. Sail_Rat_Cde Soldier Rank Code Sailor Rating Code Sold_Rnk_Cd Unit Member Rank Code Squad Member Rank Code Crew Member Rating Code Launch Team Member Rating Code Functional/Organizational Context Dependent “ Specified” Model Army Personnel Management Navy Personnel Management Technology Dependent “ Implemented” Model ANSI SQL ANSI SQL Vendor Dependent SQL DBMS “ Operational” Model “ Oracle” DBMS “ Sybase” DBMS Business Application Information System (AIS) “ View” Model Navy (AIS) Army (AIS) Concepts Data Element Concept Value Domain Data Element CORE METADATA REPOSITORY Implemented Model Operational DBMS Application View Specified Model ISO 11179 Model View Column Names DBMS Column Names SQL Column Names Attribute Names Business Fact Semantic Template Name Metadata Repository Architecture of Related Representations of DoD Enterprise Shared Data Elements in Support of Data and Information Sharing The code that represents the level of authority and responsibility occupied by Person in a hierarchy of levels ranging from most superior to most subordinate in which each level is subordinate to levels above and superior to levels below. Data Element Definition: Additional Data Element Structural Metadata: Person Grade Code Human Resource Personnel Classifi- cation Grade Code Personnel Ranking Measure Conceptual Value Domain Data type characteristics, etc. ISO 11179 Context Inde pendent Data Element Representation Meta Model 1.1 Example Personnel Application of an ISO 11179 Based Data Element Architecture for Relating Disparate Representations of Shared Enterprise Data Elements
    • 31. 2. How Do We Get to the Ideal? (or the least un-ideal)
      • Find our metadata
      • Perform smart metadata mining
      • Pick the right starting layer
      • Reverse engineer to build the upper layers
      • Forward engineer to build standard-data
      • based applications
    • 32. 3. Find Our Metadata
      • Existing schemas within running applications as
      • that’s the only place where data-truth resides
      • Extract Cobol FDs within running applications
      • for the same truth reason
      • Finally, research metadata libraries like ERwin models
    • 33. 3.1 Where We Started
      • DoD had 493 (Erwin) data models that were
      • developed in the 1990s. There were 5709 tables
      • and 16921 columns in these tables.
      • We did not inventory each DoD Agency, but the key
      • investigator (Hank Lavender) is very much aware of
      • what, where, and how much all the schemas overlapped.
      • This effort was to “prove the process”. We will soon
      • start real Enterprise-wide data sharing projects.
    • 34. 4. Perform Smart Meta -Data Mining
      • Pick backbone and rib-cage (HR, Finance, Inventory
      • Customer Management, Sales) Applications
      • Pick the most commonly used schemas across the
      • enterprise that support the backbone and rib-cage
      • applications
      • Pick the subset of schemas that have the most
      • commonly used tables
      • (note: commonly used is different from exactly the same as…)
      • Make Where-Used and Frequency-Used Matrices
    • 35. 4.1 Where Used & Frequency Matrices IDM Data Model Counts Data Model Description Tables Columns Relationships C-03 Budgets & Currency 56 178 53 C3-12 Command & Control 28 276 65 ES-07 Environmental Hazards 41 464 46 ES-08 Environmental Projects 28 185 40 LG-06 Transportation Operations 19 91 26 LG-23 Materiel Documentation 36 272 51 LG-28 Materiel Characteristics 45 225 48 PR-22 Training & Instruction 20 135 23 PR-31 Person Characteristics 36 118 41 Totals 309 1944 * 393 * 542 Unique Data Element Concepts Basic Types and Populations
    • 36. 4.1 (cont) Subject Areas Use Across IDM Schemas SDM IDM Schemas Subject Areas C-03 C3-12 ES-07 ES-08 LG-06 LG-23 LG-28 PR-22 PR-31 Environmental Management X X X Health Management X X X X Logistics Management X X X X Logistics Operations X X X X X Logistics Planning X X X Materiel Maintenance X X Materiel Management X X X X X X X Transportation Operations X X X Property Management X X X Personnel Management X X X X X X Management Administration X X X
    • 37. 4.1 (cont) Where Used & Frequency Matrices Organization X X X X X X X Person X X X X X X Country X X X X X Location X X Task X X Facility X X Plan X X Guidance X X Geolocation X X X SDM Subject Area Entity C-03 C3-12 ES-07 ES-08 LG-06 LG-23 LG-28 PR-22 PR-31 Logistics Management Personnel Management Logistics Operations Logistics Operations Management Administration Property Management Logistics Planning Environmental Management Property Management IDM Schemas & Tables Frequency of Use Matrix
    • 38. 5. Find the Right Starting Layer Data Modeling Layer Description Start Here ? Data Elements Context independent business fact semantic templates NO, these are not database models and have no context Specified Data Model Technology independent data model templates NO, as these are just templates and not database models Implemented Data Model DBMS independent database data models and hosts for database object classes OK- if you have “Erwin” like data models that can be researched, tabulated, and extracted via Excel or SQL DDL Operational Data Model DBMS dependent and Operating System specific YES! This is the best as it matches the reality of operating databases and applications View Data Model Database application specific SQL views No, as this is too application-use specific, and not data model centric.
    • 39. 6. Reverse Engineer to Build The Upper Layers
      • Import to appropriate layer
      • Promote to higher data modeling layer
      • Re-engineer the Specified Data Model layer
      • Analyze to discover the Data Elements
      • Build Data Element Model Metadata layer
    • 40. 6.1 Import to Appropriate Layer IDM Importing SQL DDL
    • 41. 6.1 Import to Appropriate Layer IDM Tables IDM
    • 42. 6.2 Promote to Higher Data Modeling Layer Promote IDM to SDM
    • 43. 6.2 Key Promotion Issues Key Difference Between Subject-Entity-Attribute vs Schema-Table-Column Model of a subject area. Intellectual boundaries, not data processing boundaries. Not a conceptual version of a logical database. It’s a subject based data model template. Define once, use many times, differently in IDM models. Subject-Entity-Attribute (SDM) Model of a database schema that may involve attributes from multiple entities in one table, or attributes of entities across multiple tables. Intended to be implemented within a DBMS as an operational database. Schema-Table-Column (IDM) Subject-Schema Not related. This would then mean Transformational Relationship. Entity-Table Not related. This would mean Transformational Relationship Attribute-Column Yes, Related. This allows define once, use many times modeling.
    • 44.
      • Assign Entities to different Subjects
      • Reassign Entities to within Entities (sub-typing)
      • Reassign Attribute’s Semantics
      • Conform Attribute Names to Subject Area Scope
      • Reassign Attributes to different Entities
      • Reassign Attributes to different Data Elements
      6.3 Re-engineer the Specified Data Model Reassign Entity to Subject SDM
    • 45. 6.3 Re-engineering the Specified Data Model Reassign Entity to Subject Assign Attribute Meta Category Values Reassign Attribute to Data Element Reassign Attribute to Entity BASIC PROCESSES SDM
    • 46. 6.4 Reallocate Foreign Keys to Encapsulate Subject’s Entities
      • For Each Subject Area:
      • Make a List of Entities
      • Make a Subject Area Based E-R Model Diagram
      • Delete Unnecessary Foreign Keys from Existing Entities
      • Make New Foreign Keys Where Needed
      • Export to E-R Diagrammer to Verify Result
      • Recycle if Necessary
      SDM
    • 47. SDM 6.4 (cont) Reallocate Foreign Keys to Encapsulate Subject’s Entities Validate/Create SDM Foreign Keys
    • 48. SDM 6.4 (cont) Reallocate Foreign Keys to Encapsulate Subject’s Entities Modify SDM Foreign Keys
    • 49. 6.5 Discover the Data Element Promote SDM Attributes to Data Elements
    • 50. 6.6 Build Data Element Model Level Metadata
    • 51. 6.7 Data Element, Attribute, Column Differences Key Differences Among Data Element, Attribute, Column Characteristic Data Element Attribute Column Context Reason for existence Frequency of use Example Stand-alone independent business fact template. A characteristic of an entity that exists within the context of a subject. A characteristic of a table that exists within the context of a schema. Source of semantics and common general meaning for classes of attributes and columns. Source of value based refinement of the intent of the entity. The set of all attributes fully define an instance of an entity. Source of value based refinement of the intent of the table. The set of all columns fully define an instance of a table. Defined once within the context of an entity. source of business facts across one or more columns within one or more tables. Define once, use many times to provide semantics to attributes or columns. Defined once within the context of a table. Identifier Asset Identifier Person Identifier Customer Identifier
      • Invoice Line Item
      • Part Number
      • Salesman Identifier
      • Customer Identifier
    • 52. 7.0 Overall Forward Engineering Process
      • Import from higher level to lower level
      • Map IDM to ODM legacy schemas to
      • preserve existing systems environment
      • and/or
      • Generate new ODM schemas to replace
      • legacy systems
      • SQL Views can support legacy names or
      • new names
      • Generate Application
    • 53. 7.1 Import From Higher Level To Lower Level Subject Area Data Model to Implemented Data Model
      • Start Metabase IDM
      • Make the Target Schema
      • Pick an SDM Subject
      • Select the Root Entity
      • Create the Data Model Entity Tree
      • Perform Import (from SDM to IDM)
      • “ Prune” Schema-Table Set to Just Those Needed
      • “ Prune” Table-Column Set to Just Those Needed
      • Move Columns Among Tables as Needed
      • Import Next SDM Model and Perform “Pruning” Steps
      • Mapping to New IDM from SDM Preserved-----Of Course!
    • 54. 7.1 (cont) Import From Higher Level To Lower Level Subject Area Data Model to Implemented Data Model Import SDM Entities to IDM Tables
    • 55. Implemented Data Model to Operational Data Model
      • Start Metabase ODM
      • Make the Target DBMS Schema
      • Pick an IDM Schema
      • Select the Root Table
      • Create the Data Model Table Tree
      • Perform Import (from IDM to ODM)
      • “ Prune” DBMS Schema-DBMS Table Set to Just Those Needed
      • “ Prune” DBMS Table-DBMS Column Set to Just Those Needed
      • Move DBMS Columns Among DBMS Tables as Needed
      • Import Next IDM Model and Perform “Pruning” Steps
      • Mapping to New ODM from IDM Preserved-----Of Course!
      7.1 (cont) Import From Higher Level To Lower Level
    • 56. Implemented Data Model to Operational Data Model 7.1 (cont) Import From Higher Level To Lower Level Import IDM Schema Tables to ODM DBMS Tables
    • 57. ODM 7.2 Generate SQL Generate DBMS SQL DDL
    • 58. 7.3 Generate Application ODM
    • 59.     8. Process Statistics Task Name Notes Effort Metric 1 Find the Right Starting Point With MS/Access based repository of data models. Close to about 100 models 2 days 2 Import to appropriate layer Had to fix a number of data modeling errors in source CASE tool 40 hours for 10+ data models 3 Promote to Higher Data Modeling Layer Required several cycles of distilling subjects 40 hours for 10+ models 4 Re-Engineer Had to re-engineer Fkeys, rename entities and some attributes. Also had to reconnect new attributes to “old columns.” 240 hours for 10+ models
    • 60. Task Name Notes Effort Metric 5 Abstract to Data Element Required review of each attribute, and creation of MCVs, etc. 0.25 hours per attribute for 1000 attributes. So, 250 hours. 6 Build Data Element Model Level Metadata Required generation of higher level concepts, value domains, etc. 80 hours 7 Import from higher level to lower level Required design of new data models for new databases from data model templates, and/or just re-mapping to existing models 8 hours per model for 10 existing models and for 2 new models. Thus, 100 hours 8 Generate SQL Required specification of data types and lengths 20 columns per hour for 80 hours. 9 Input to and then Generate Application Export of one Model from IDM 1 hour to export, 1 hour to generate 1 st cut application 8. (cont) Process Statistics
    • 61. 9. Lessons Learned
      • It can be done. However, it is not a walk in the park!
      • It requires clear understanding of separation of Data Models. Data Element
      • from Specified DMs, from Implemented DMs, and from Operational DMs.
      • These are NOT transformations (conceptual to logical to physical). These
      • are different data models.
      • Subject Matter Experts are Essential, Critical, and Absolutely Necessary.
      • It’s not top down. It’s bottom-up. But once built, use it top-down.
      • You must have a metadata repository and data modeling tool that works
      • at the enterprise level, and not just at the database or data model level.
      • We made changes to the metadata repository system along the way. So,
      • being able to change the meta model, entry and update and reports, is essential.
      • Given that Entity reuse for just these ODS models was about 4x, the value
      • for the data model template reuse in data warehouses and data marts is
      • incalculable.
    • 62. THANK YOU Hank Lavender Senior Information Engineer 1310 Braddock Place Alexandria, Virginia 22314-1648 Phone: +1.703.836.5900 Fax: +1.703.836.8691 Email: [email_address] WWWeb: <http://www.amerind.com> Michael M. Gorman Founder and President 2008 Althea Lane Bowie, Maryland 20716-1518 Phone: +1.301.249.1142 Fax: +1.301.249.8955 Email: [email_address] WWWeb: <http://www.wiscorp.com> n I c
    • 63. Back-ups
    • 64. DoD CORE DATA ELEMENT METADATA REPOSITORY Implemented Model Layer Operational DBMS Layer Application View Layer Specified Model Layer ISO 11179 Model Layer DoD Enterprise Interoperability Metadata Repository Core DoD Enterprise Data Element Metadata Repository Business Rule Metadata Application Metadata Data Movement Metadata Data Management Metadata Proposed DoD Metadata Repository Complete Representations of Data Element Metadata Data Element Metadata Relationships to Multiple Categories of Metadata Today The Future The Payoff Seamless & Transparent Information Interoperability
    • 65. Mat_Inv_Qty Functional/Organizational Context Dependent “ Specified” Model Army Logistics Management Navy Logistics Management Technology Dependent “ Implemented” Model ANSI SQL ANSI SQL Vendor Dependent SQL DBMS “ Operational” Model “ Oracle” DBMS “ Sybase” DBMS Business Application Information System (AIS) “ View” Model Navy UADPS (AIS) Army SAMS (AIS) View Column Names DBMS Column Names SQL Column Names Attribute Names The quantity of each type of Federal Supply System materiel item contained in an identifiable inventory of materiel objects. Data Element Definition: Additional Data Element Structural Metadata: Data type characteristics, local definition, numerated values ( if specific), etc. The Current DoD Architecture for Defining Standard Data Element Representations of Shared Enterprise Data Elements METADATA REPOSITORY Defense Data Dictionary System (DoD Standardized Data Elements) 16000+ SDEs Business Rule: Only one named representation is permitted to exist in the repository as an Enterprise SDE. (SDE Access Name) Mat_Itm_Inv_Qt Materiel Inventory Quantity Materiel Item Inventory Quantity Materiel Unit Inventory Quantity Supply Unit Quantity Stocked Materiel Quantity Ships Stores Quantity
    • 66. SQL DBMS ODM LAYER XML Schema Info Tables OUTPUT ODM LAYER HUMAN RESOURCES DATABASE Output – XML Wrapped Metadata Output Schema Information Feedback DoD Global Information Grid (GIG) METABASE REPOSITORY Implemented Data Model (IDM) Specified Data Model (SDM) ISO 11179 Data Element Templates Operational DBMS Model (ODM)