Transcript of "Moving towards Enterprise Master Data"
Moving towardsEnterprise Data Architecture Kevin Messer, Enterprise Data Architect Standard Register
About . . .• Standard Register (HQ: Dayton, Ohio; NYSE: SR) helps the world‟s leading organizations build and protect their reputations through expert management and precise execution of critical communications > Secure, confidential print > Multichannel marketing > Industrial labels and logos > Print outsourcing > Fraud prevention solutions > eLearning > MSRA-resistant armbands > NFC-enabled pokenHEALTH > www.standardregister.com• Kevin Messer (Presenter) > SR Enterprise Data Architect > Prior: Lockheed, Accenture, NCR > www.linkedin.com/in/kevinmesser Moving towards Enterprise Data Architecture 2
Background: Improved Data Management• New initiatives > Complex integrations > Heterogeneous source systems > How to avoid data duplication? > Reference sources for data quality?• Systems documentation challenges > Multiple repositories > Variety of documentation tools > Some documentation ≠ data > Budget and resource constraints• Options? Moving towards Enterprise Data Architecture 3
Master Data Management?• Brief survey of products > Informatica, Microsoft, Oracle > Software AG, Talend, Teradata > Etc, etc, etc.• Many Challenges > Large upfront cost (>$1M) > Hidden costs - Internal staffing, training - Process and culture changes - How to measure Return on Investment and Total Costs? - Where to start?• Lower cost alternatives? > Start small, grow scope incrementally > Every plan needs an architecture . . . Moving towards Enterprise Data Architecture 4
Simple Master Data Architecture Data structured for reporting and analysis, based on Reference, Enterprise DataThis simple Master, BU datadiagram forMaster Data Business Unit variations of MasterArchitecture is Data, used for specific markets Business Unit Data and transactionseasy for people toremember and Data consistently used by thewas the basis for business: customer; product Master Datathe next steps,starting with External data used by reference:metadata. . . zip codes; area codes; state; city; Reference Data country Data about data: source, formats, business rules Metadata Moving towards Enterprise Data Architecture 5
Teradata Meta Data Services (MDS)• Part of Teradata Utility Pack• Already licensed, so essentially “free” > No client or seat licenses• Very low-cost implementation > Windows client utility for administration > Microsoft IIS Server (we used laptops) > Web browser interface (MetaSurf) for users• Did I mention it was FREE? Moving towards Enterprise Data Architecture 6
Teradata MDS Architecture• MDS Repository is a MDS Utilities Custom BI Tools Teradata database Applications• Versions 13 to V2R6.0• API supports C++, MDS APIs XML Bridge Microsoft COM, XML, Java, and Excel Auto DIM MDS Engine• ERwin® Data Modeler r7 Update includes a wizard• CWM XMI for Metadata Interchange using other Teradata tools• Supports storing MDS metadata from other Repository DBMS Diagram from “Discover Teradata Meta Data Services” by Rudy Ezquerro, Copyright 2007 by Teradata Corporation. Moving towards Enterprise Data Architecture 7
Building a “Master Data Catalog”• Started “proof of concept” effort in early 2011• Objectives: > Use MDS as a central “catalog” of all database metadata - Capture 100% of Teradata database schemas - Capture 100% of SQL-compliant non-Teradata sources - Explore approaches for non-SQL databases > Evaluate use of MDS as central catalog/portal for both technical and business process documentation• The slides that follow describe the steps we took and the outcomes Moving towards Enterprise Data Architecture 8
First Steps: Teradata Metadata• Capturing from Teradata > We used QA databases > Using MetaManager, connect to Teradata > Select the databases to load• This approach worked without difficulty Moving towards Enterprise Data Architecture 9
Second Step: Importing non-Teradata Metadata• Our original plan: > Using ERwin, reverse-engineer the databases > Export schemas from Erwin as XML (CWM XMI) files > Use MetaBridge to import the XMI• Software compatibility issues between Erwin and our version of MDS emerged, forcing a different approach Moving towards Enterprise Data Architecture 10
Alternate approach to non-Teradata Metadata• The alternate approach used Excel > Using the MDS API Interfaces• Microsoft Excel Macros are included with MDS Development Kit sample files• Macros include loading descriptions for: > Databases, tables, columns, views > Business attributes, entities, subject areas > Macros can be modified• We worked with tools created by Dan Riehle Moving towards Enterprise Data Architecture 11
Examples• Oracle > Created a query joining All_Tab_Columns and all_tables system views• SQL Server > Used INFORMATION_Schema.columns system view• Supra II (mainframe) > Flatten the DDL export• Export the DDL information to Excel format Moving towards Enterprise Data Architecture 12
Important Tip• MetaSurf (web browser viewer of MDS) only displays Teradata data types• When exporting DDL from non-Teradata databases, data types must be converted• Example > VARCHAR converted to a code of „CV‟ to display* *Note: Teradata column type abbreviations are found in the Teradata RDBMS Data Dictionary manual. Moving towards Enterprise Data Architecture 13
MetaSurf Examples• Example views of metadata from an Oracle database Moving towards Enterprise Data Architecture 14
Lessons Learned about Teradata MDS• MDS is a powerful, flexible tool for > Quickly capturing database technical information > Providing this database information centrally to - Database Administrators, ETL and Interface Programmers, Support Staff > Searching across multiple databases for similar tables, columns, attributes, etc• MDS is also extensible, allowing it to serve as a portal to information related to databases > Data Model diagrams > Business Process diagrams and descriptions > Links to documentation stored in other repositories like SharePoint, Subversion, or the internet• MDS provides a low-cost way to incrementally implement Enterprise Data Architecture and Master Data Management Moving towards Enterprise Data Architecture 15
Next Steps• Plans include: > Implementing MDS in production - Staffing - Support - Governance > Position MDS as the company‟s Master Data Catalog - Single site for documentation about data and systems - Content to include links to documents in other repositories Moving towards Enterprise Data Architecture 16
Questions? ? Kevin Messer Dan Riehle Dan@GetReals.com www.getreals.com Moving towards Enterprise Data Architecture 17
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.