Master Data Management From Assessment, Design up to operation Ali BELCAID – Managing Consultant
Master Data Management : An Overview Information is a PriorityQuality and actionable information is fundamental to deliver many business strategies. Enterprise Operations Enterprise Information Management & Capabilities Management & Capabilities MDM is the glue that Solutions M Solutions blends operational and • ERP • BI/DW information • CRM D • BPM management solutions • Supply Chain M • Portals Operational MDM Analytical MDM
Master Data Management : An Overview MDM Requires Both IT and BusinessMDM is a component that promotes process efficiency, simplicity, and data quality,improving the value IT brings to business. Impact of MDM Initiative on Business and IT Global Business Master Data IT Management Enterprise Centralized, Efficient Data • Avoid data redundancies Wide Storage • Assure data consistency Consistency Cost • Centralize data Improved System Effectiveness distribution (one source) Integration Reliable • Provide unique identifier Minimal Data Analytics and • Create global hierarchies Conversions Reporting and attributes
Master Data Management : An Overview MDM Implementation Styles Implementation Style Description Third party suppliers and managers of domain specificExternal Databases master data(Service Provider) Examples: database marketing, government service bureaus Master information file/database, system of record (SOR)Persistent Operational data store, active data warehouse(Database) Relational DBMS + extract-transform-load (ETL) + data quality (DQ) Metadata layer + distributed query (e.g., EII)Registry Enterprise application integration (e.g., EAI), distributed(Virtual) system Portal Ability to fine-tune performance and availability by alteringComposite amount of master data persisted(Hybrid) XML, web services, service-oriented architecture (SOA)
Master Data Management An Overview “Persistent” Master Data Repository (Illustrative Scenario)This Customer Data Integration (CDI) solution architecture illustrates how process and technology worktogether through a centralized “persistent” master data repositoryOperational Systems Master Data Management Business Process Initiate Evaluate Approve Initiate Entry/Update Request Request Entry/UpdateSystem Owners Data Stewards Business Analysts Customer Master Data Repository Workflow Business Rules Mapping Rules Catalogue/Index Automated Entry Updates Customer Care Extract Transform Load SAP Care Enterprise Application Integration Reporting DIM Data Mart CRM (ETL) Siebel Operational DIM Campaign DIM Data Store Management (ODS) Customer Reporting Contract FACT Data Mart Negotiations IMS Customer Financial Aggregate DIM Consolidation DIM Financial Monthly End (EAI) DIM Data Mart Reporting PBMS Close Enterprise Warehouse Data Marts DATA INFORMATION
Approach to MDM Implementation Business Assessment & Technology Selection Current State Future State Develop RoadmapActivities Organization & Current State Governance Details Process & Implementation Quick scan Gap Analysis Methodology Roadmap Requirements Technology SelectionDeliverables Project Initiation Current State : Data Mgt. Future State : Data Mgt. Prioritization Gap Analysis Current Sate : Organization Future Sate : Organization Roadmap Current State : Architecture Future State : Architecture 4 to 6 weeks varies with scope
Approach to MDM Implementation Business Assessment & Technology Selection Topic What to Do ? What to Deliver ? • Maturity Assessment • Business Direction, objectives, … • Scope of the Project : Which MDM shouldQuick Scan • Engagement Management (project be implemented ? management, change management, quality management and risk management ) • workshops with key client stakeholders toBusiness Requirement Analysis identify business issues with master data (Data quality, Duplicity, Incoherence, …) • workshops with key client stakeholders to identify technology issues that could delayTechnical Requirement Analysis the delivery of accurate and reliable master data to consumers (multi-systems, duplicity, • MDM Finding & Assessment non-synchronization, …) • based on business and technical findings andGap Analysis requirements • that consists of business, technology andFuture State Recommendations data architectureRoadmap Definition • for attaining the future state • Roadmap Definition & Planning
Approach to MDM Implementation Business Assessment & Technology Selection Maturity Assessment
Approach to MDM Implementation MDM Implementation Framework Continues Implementation Phases This part is done once(Part of the Assessment Phase) Design Kick Off of the MDM Initiative Build Technology Roadmap & Business Assessment & Foundation Requirement Software Selection Activities Integrate Begin next Operate Iteration Define & Validate the data governance & operating model Implement the data governance & Operating model
Approach to MDM Implementation Roadmap & Foundation ActivitiesThe Roadmap provides the detailed requirements and solution definition Meta Data Managementthat applies to the continuous implementation. It has the followingobjectives: Master Data Master Data Master Data Modelling Migration Integration Refine strategic business requirements to a detailed level for iterative design Master Data Master Data Establish standards and develop solutions to common problems Re-engineering Profiling Define the development and delivery environments Detailed planning for this cycle of the implementation Master Data ArchitectureThe Roadmap can be summarized as providing the Plan, the SolutionRequirements and the Solution Definition for the continuous Major deliverables and points to be addressed when setting up the roadmap & foundation activities :implementation part. Detailed Project RoadmapFoundation Activities focus on aspects of each of the streams of Testing and Deployment plansdevelopment. These activities are : Detailed Information Modeling Detailed Migration Plan (historical Data) Recommended process and system changes for Meta Data Management improved Data Governance Data Modelling Identification of root causes leading to Data Governance Data Migration issues Data Integration Data Governance Metrics Quantitative Data Investigation Data Reengineering Improved Data Quality Data Profiling Create/Revise Solution Architecture Data Solution Architecture Ensure the availability of Software Development Environment
Approach to MDM Implementation MDM Work streams Design Build Integrate OperateMDM Program ManagementChange/Issue ManagementOperations Management Meta Data Master Data Master Data Master Data Management Modelling Migration IntegrationTraining and Support Master Data Master Data Master Data Re-engineering Profiling Architecture Iteration
Approach to MDM Implementation Meta Data ManagementSignificant metadata artifacts are produced related to data definition, business rules, transformation logic and data quality. This informationshould be stored in a metadata repository; getting this repository in place from the early stages of the MDM project. Model Management is the capability to manage Versioning of metadata provides the ability for looking back into structures and processes used to describe the metadata history to gain a more comprehensive understanding of the in a system. current state Metadata Integration capability provides a basic ability to Configuration Management is a fundamental process for build metadata flows into and out of a managed metadata developing metadata. It is the role that process and governance environment. plays in the development and operations of a managed metadata environment. Identity Matching as a foundation capability ensures consistent and accurate reuse of metadata. a system must have the ability Model Query provides the fundamental ability for publication of to identify metadata uniquely so that the metadata may be metadata. Its capabilities form the foundation of providing reused, validated and versioned within the managed metadata Metadata Reporting Packages environment. Metadata Access Control is a capability for providing a control Validation capabilities ensure the quality and consistency of layer over metadata models. Metadata can often be sensitive metadata flowing through the managed metadata environment information that should have restrictive controls to prevent unauthorized access
Approach to MDM Implementation Master Data ModellingThe data modeling process is used as an intermediary data store to bring data together from multiple systems ina hub fashion. This data store provides a common, integrated model where data may undergo significant re-engineering. Design Logical Master Implement Physical Data Model Master Data Model Input: Input: Conceptual Data Model Logical Data Model Data Specification Standards Solution Architecture Data Modeling Standards Data Specification Standards Data Security Standards Data Modeling Standards Detailed Business Requirements for Data Security Standards each Iteration Detailed Business Requirements for each iteration Output: Output: Logical Data Model Physical Data Model Database Definition Language (DDL) Scripts Sizing Estimates
Approach to MDM Implementation Master Data IntegrationDependencies: Data Integration is one of the Foundation Capabilities of MDM Development. It provides a mechanism Metadata Management for bringing together information from a number of distributed systems by interfacing into sources, Data Profiling Data Re-engineering providing a capability to transform data between the systems, enforcing business rules and being able Data Modeling to load data into a different types of target areas. Data Migration ETL ETL ETL flows & Logical Design Physical Design jobs Testing Input: Input: Input: Business requirements ETL Logical Design Test scenarios Designed Process Flow Solution Architecture Data Sampling Source & Target interfaces Data Specification Standards load dependencies and integration with Data Modeling Standards Output: metadata processes Data Security Standards Tested flows and jobs Source & Target Data models Output: Output: ETL flows and Jobs ETL Logical Design
Approach to MDM Implementation Master Data Re-engineeringData Re-Engineering is a term used to describe a number of related functions for standardizing data to a common format,correcting data quality issues, removing duplicate information/building linkages between records that did not exist previously, orenriching data with supplementary information. Data standardization brings data into a common format for In the Data Matching and Consolidation task, data is migrating into target environment. It addresses problems associated with other records to identify matching related to: sets. Matching records can then either be consolidated to remove duplications or linked to Redundant domain values another to form new associations. Formatting problems Non-atomic data from complex fields Embedded meaning in data Data Enrichment provide an organisation’s internal data with data from external sources like : Data Correction typically addresses problems related to: Personal data such as date-of-birth and gender codes Geographical data Missing data Postal Data, such as Delivery Point Identifiers (DPID) Value issues due to range problems Demographic information Value issues related non-unique fields Economic data Temporal or state issues World event information Name and address data that can be referenced against existing reference sets
Approach to MDM Implementation Master Data ProfilingData Profiling focuses on conducting an assessment of actual data and data structures. It helps provide the following: Identifies data quality issues - measurements are taken against a number of dimensions, to help identify issues at the individual attribute level, at the table-level and between tables. Captures metadata. Identifies business rules – The next step is to perform the data mapping. Data profiling will assist in gaining an understanding of the data held in the system and in identifying business rules for handling the data. This will feed into the future data mapping exercise. Assesses the source system data to satisfy the business requirements. The focus is on gaining a very detailed understanding of the source data that will feed the MDM target system, to ensure that the quality level is sufficient to meet the requirements. Perform Table Perform Multi Perform Column Finalize Data Profiling Tables Profiling Profiling Quality Report (Analyze Data (Analyze redundancy (Analysis of single (Signoff of Data across rows in and referential or complex field) Quality Report) single table) integrity issues) Major Deliverables Data Quality Assessment Report (per Source System) Data Quality Metrics updated to Metadata Repository Mapping Rules and Business Rules updated to Metadata Repository
Approach to MDM Implementation Master Data Profiling1.Column Input: 3.Multi-Table Input: Profiling Profiling Completion of Table Profiling Information Requirements for column-level data Information Requirements for multi-table level data analysis analysis Relevant data extracts Relevant data extracts Output: Output: Completion of Multi-Table Profiling Redundancy Analysis will identify: Completion of Column Profiling Potential relationships with fields in other tables Understanding all the fields and document their Redundant data between tables descriptions in the profiling tool Potential referential integrity issues eg. Completion of the relevant sections of the Data Quality Identification of orphans records Assessment Report Completion of the relevant sections of the Data Quality Updates to metadata repository Assessment Report Updates to metadata repository 2.Table Input: Profiling 4. Quality Input: Completion of Column Profiling Report Completion of Column Profiling Information Requirements for table-level data analysis Completion of Table Profiling Relevant data extracts Completion of Multi-Table Profiling Output: Output: Completion of the Data Quality Assessment Report Completion of Table Profiling Understand all the fields and document their descriptions in the profiling tool Primary keys for each table Completion of the relevant sections of the Data Quality Assessment Report Updates to metadata repository
Approach to MDM Implementation Master Data MigrationAn MDM program will typically involve a migration of historical data across systems, into or through a centralizedhub. This is where many of the data quality issues are resolved in a progressive fashion before operationalizingsome of these rule-sets for the ongoing implementation. Prod Target 7 Data Producers (ERP, CRM, Test Legacy, …) Target 6 Data Integration Migration Staging Integrated Data Store 5 Transformations • Attribute Scan • Common Data Model 1 • Tables Scan • Detailed Data • Assessment • Apply Re-engineering rules • Reporting 4 2 Data Profiling Data Data Re- Integration engineering Master Data Modelling Metadata Management 3
Approach to MDM Implementation Master Data MigrationThe key activities in the MDM migration process include:1. Extraction of data from producers (ERP, CRM, Legacy systems, …) into a staging area.2. The data in the staging area will be profiled to measure down columns, across rows and between tables. This information will be used to determine which business rules and transformations need to be invoked early in the process.3. Metadata such as data mapping rules will begin to be established at this time. Data Standards will be agreed to and invoked at this stage in preparation for data movement. All source attributes will be mapped into the target attributes within the metadata management environment.4. All agreed to transformations and standardizations required to move the data into the staging area for testing and production are implemented. The data is moved into the Integrated Data Store.5. Data Profiling is done again and measured against the agreed upon move success criteria for all steps up to this point. Additional data standardizations are performed in to assist in the data matching and generally measure data quality against agreed upon criteria. After the standardizations the rules for which records can not or should not be moved are applied. It expected that this step will require considerable analysis.6. This step involves the actual move of the data into either the testing environment7. Data is loaded into the production system where some further data quality cleanup may be required. Production Verification Testing is conducted, which should also include functional testing of features that are environment specific. After testing is complete, the system is activated as a live production system.
Approach to MDM Implementation Master Data ArchitectureThe Master Data Architecture defines in detail the Solution Architecture for the MDM environment. The Solution Architecture providesthe overall technology solution for a specific increment and ties together the overall approach. Define ETL conceptual Design Define SDLC conceptual Design Define Security conceptual Design - List of sources - Testing Strategy - Security Standards - List of targets - SDLC Procedures -Security Requirements - Testing Plans for Applications & Infrastructure - Major Transformations Deliverables: -Deployment Plan - Estimate volumes • Security Implementation Document - Timing Deliverables: • SDLC procedures document Deliverables: • Testing Plan Define Infrastructure Management • ETL Design architecture • Deployment Plan conceptual Design • ETL Implementation Software -Backup & Recovery -Archiving Documents -Controlling & Monitoring • ETL Technical architecture document - Environments (dev, test, prod) setup Define Metadata Management conceptual Design Deliverables: - Business definition of the data • Configuration Management Document - Physical data models Define Data Quality Processes - Data Re-Engineering metadata - Data model -Data Quality metadata formulas used to MDM Software Implementation -Profiling derive data -Software Implementation Planning - Re-engineering - Parameterization/Configuration Deliverables: - Software Testing and deployment Deliverables: • Metadata Design architecture • Data Quality Design architecture • Metadata Implementation Software Deliverables: Documents • Software Installation and Configuration • Data Quality Implementation • Metadata Technical architecture Document Software Documents document • Data Quality Technical architecture document Master Data Solution Architecture
Approach to MDM Implementation Prototyping the ArchitecturePrototyping the architecture helps to :• test some of the major technology risk areas for the proposed MDM Solution Architecture• gain a better understanding of how the solution will work before moving into a more formalized design process.• Prototyping the proposed solution should provide an end-to-end approach that includes each of the major components of the architecture.
Approach to MDM Implementation MDM - Key Lessons LearnedIn an MDM implementation, there are some key lessons learned that should be consideredwhen initiating an MDM program. Key Lessons Joint business and IT team Make the case for change Data as a common good Think big but start small Measure and communicate success Processes first, technology last Business ownership of data Roles and responsibilities Data cleanliness and migration Communicate, communicate, communicate !
Knowledge, is quite simply question of sharing. http://intelligenteenterprise.blogspot.com/ http://www.linkedin.com/in/albel