Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DQS & MDS in SQL Server 2016

Do you lose precious time due to data quality problems?
Do you need to integrate data from multiples sources and provide an integrated view of your customer or product attributes to other systems?
SQL Server 2016 Data Quality and Master Data Services can help you.

  • Be the first to comment

DQS & MDS in SQL Server 2016

  1. 1. Data Quality Services and Master Data Services in SQL Server 2016 2017-03-15
  2. 2. Agenda • Introduction : 5 min. • Reality check, do you have these issues? • What is master data management? • Master data management in SQL Server 2016 : 15 min. • Data Quality Services (DQS) • Master Data Services (MDS) • What’s new in SQL Server 2016 for MDS? • Data Quality Services demo : 15 min. • Cleansing & Matching data using Excel MDS Add-in • Cleansing data using SSIS • Master Data Services demo: 20 min. • Creating model, entities, attributes and business rules • Load data using SSIS
  3. 3. 33, Prince Street, Suite 284, Montreal (Quebec) H3C 2M7 Sébastien Notebaert Senior Consultant sebastien.notebaert@exia.ca 514-973-6108 Microsoft Data Platform expert with 15+ years experience in deploying data warehousing, analytics, data integration and MDM solutions.
  4. 4. • Services : • Projets BI • Conseil BI • Fondée en 2010 • 35 experts • Clientèle : • Grande entreprise • PMEs • Services : • Infra Cloud (Azure) • Solutions Cloud • Nouvelle pratique • Issue de l’acquisition GTO • 2 experts, recrutement actif • Clientèle : • Grande entreprise • PMEs • Services gérés • Cloud privé • Cloud Public • Fondée en 2006 • 20 experts • Clientèle : • PMEs • Domaine Médical • Produits : • Solution BI Financier • Solution processus budgétaire • Reporting ERO • Fondée en 2010 • 5 experts • Clientèle : • PMEs Who we are…
  5. 5. Reality check: do you have these issues? • Do you have instances of invalid data impacting business processes? • Do you wish your business users could manage data themselves such as Customer and Product? • Do you have IT resources spending time investigating data quality issues and/or applying data fixes? • Do you have the need for consolidation and distribution of data to other systems? • Do you have an environment of heterogeneous systems which all could benefit from a single view of domain data such as Customer or Product? -> You need master data management!
  6. 6. What is master data management? • The technology, tools, and processes required to create and maintain consistent and accurate master data • Master data is a set of data objects that are at the center of business activities like: Customers, Products, Cost Centers, Locations, … • Master data is not transactional data like a sale or inventory movement • To succeed an MDM initiative must include business processes, people AND data
  7. 7. MDM Architecture Style Style Description Which System is Master Update Source Systems Comple xity REGISTRY Master data is not consolidated, but is maintained as a set of “stub” records mapped to attributes stored in the source systems. There is little data movement other than create or delete global stub records in the registry. The golden record is assembled dynamically using complex distributed queries. The upside is that a real time central reference can be made with little or no infrastructure investment. The downside is that without central governance of the data the golden record isn’t highly reliable. Source system Not applicable Data stays in source systems 1 CONSOLIDATED Master data is consolidated from multiple source systems into a physical golden record for downstream consumption; however, any updates made to the master data are not returned to the original sources. Consolidated MDM hubs are quick and inexpensive to set up, and offer a big return by enabling reliable enterprise-wide reporting. Data movement is typically tolerant of inter-day latency and managed through inexpensive batch processes, but it’s a one- way street from source systems. Source system No 2
  8. 8. MDM Architecture Style (continued) Style Description Which System is Master Update Source Systems Comple xity COEXISTENCE Just like consolidated, coexistence harmonizes multiple sources into a physical golden record for downstream consumption. Coexistence adds the important step of updating the source systems. These requirements are typically high latency and can usually be met at an acceptable time and cost through bi-directional batch processes. Coexistence is a natural evolution of the consolidated architecture with the added benefit of linking centrally governed data back to the source systems. Interfacing with complex data sources, such as ERP systems, can become a costly drawback. Source system Yes 3 TRANSACTIO OR CENTRALIZED While this approach also creates a physical golden record, the key differentiator from the consolidated and coexistence styles is the MDM hub extends enterprise governance over the source systems. This introduces shorter latency requirements that are typically addressed with a combination of web services integration and an authoring application that bridges centralized and application- specific governance needs. In most cases, this is a big step up in cost, complexity and implementation time. The pay-off is more comprehensive governance in real-time; however, too many complex sources can push cost and complexity beyond the value created by the architecture. MDM system Yes 4
  9. 9. Master Data Management Entity Examples People Things Places Abstract Customers Vendors Sales People Employees Partners Patients Products Business Units Bill of Materials Parts Storage Bins Equipment Locations Stores Wells Power Lines Geo Areas Warehouses Accounts Warranties Time Metrics Securities Contracts
  10. 10. Master Data Management Solution Areas Data Quality Improve Efficiency Compliance Retain Customers Merger & Acquisition Cross Reference Golden Records • Human typographical errors; incomplete information; spreadsheet data management • Mergers and consolidation; ERP implementations, consolidation or migration • New purposes for old data; retire old applications such as mainframe applications • Single point of data maintenance; BI reporting • Different types of customer accounts • Accurate view of data by implementing MDM and DQ • Single point of data maintenance • Cross sell and upsell • Tracking spends by customer • Province and federal legislation • Single view of customer spend, channels, cross sell and upsell • Cross reference of same customers across multiple systems • Survivorship of best consolidated data across multiple systems • Single view of anything that has attributes that can be matched • Cleanup of source systems with business rules and golden records pushed back • Merging chart of accounts; consolidate financial reporting • Single view of product • Single view of customers
  11. 11. Master Data Management in SQL Server 2016
  12. 12. Master Data Management in SQL 2016
  13. 13. MDS & DQS in a Consolidated MDM Architecture
  14. 14. Data Quality Services
  15. 15. Data Quality Services • SQL Server Data Quality Services (DQS) is a knowledge- driven data quality tool. • DQS enables you to build a knowledge base and use it to perform a variety of critical data quality tasks including: correction, enrichment, standardization, and de-duplication of your data. • DQS is like a spell checker for your data • DQS was shipped in SQL Server 2012 (ent. & BI ed.)
  16. 16. Data Quality Services (continued) • DQS consists of Data Quality Server and Data Quality Client, both of which are installed as part of SQL Server 2016. • Data Quality Server is a SQL Server instance feature that consists of three SQL Server catalogs with data-quality functionality and storage
  17. 17. Data Quality Services Building Blocks • DQS building blocks • Knowledge base • Domains • Knowledge discovery • Matching policy • Data Quality projects • Cleansing project • Matching (deduplication) project
  18. 18. Using Data Quality Services Create Domains Knowledge Discovery Clean Data Match Data • Create a domain like product brand or category • Set it properties • Set it values • Set it rules • Set its term based relations • Train your domains by loading data • Accept / reject / enrich / correct your domains • Clean data sets using your trained knowledge base • Match and deduplicate your data using a matching policy
  19. 19. Data Quality Services Features & Tasks Knowledge Bases and Domains •Building a KB •Importing and Exporting Knowledge •Managing a domain •Managing a composite domain Data Quality Projects •Create a Data Quality project •Open, Unlock, … a DQ project •Open an SSIS Project in DQ client Data Cleansing •Cleanse Data Using DQS •Cleanse Data in a Composite Domain Source: https://msdn.microsoft.com/en-us/library/hh213042.aspx Data Matching •Create a matching policy •Run a Matching Project Reference Data Services in DQS •Configure DQS to use reference data •Attach domain to reference data •Cleanse data using reference data Data Profiling and Notifications DQS Administration DQS Security -> Topics in bold will be covered during the demo
  20. 20. Master Data Services
  21. 21. Master Data Services • SQL Server Master Data Services is an Master Data Management (MDM) tool to define and manage non-transactional lists of data, with the goal of compiling maintainable and validated master lists for the enterprise. • An MDM project generally includes an evaluation and restructuring of internal business processes along with the implementation of MDM technology. The result of a successful MDM solution is reliable, centralized data that can be analyzed, resulting in better business decisions. • MDS was shipped in SQL Server 2008 R2 (datacenter & ent. ed.)
  22. 22. Master Data Services (continued) Master Data Services has the following components: • Master Data Services Configuration Manager, a tool you use to create and configure MDS databases and web applications. • Master Data Manager, a web application you use to perform administrative tasks (like creating a model or business rule), and that users access to update data. • MDSModelDeploy.exe, a tool you use to create packages of your model objects and data so you can deploy them to other environments. • Master Data Services web service (WCF), which developers can use to extend or develop custom solutions for Master Data Services. • Master Data Services Add-in for Excel, which you use to manage data and create new entities and attributes.
  23. 23. Master Data Services Building Blocks • The model is the most fundamental object in an MDS solution • Models are the containers that encapsulate all other MDS objects (i.e. entities, attributes and business rules)
  24. 24. Interacting with Master Data Services • Master Data Manager Web User Interface • MDS Web Services API (WCF API) • Subscription views via T-SQL • Staging stored procedures • MDS Add-in for Excel SSIS package that calls MDS stored procedures ->
  25. 25. Master Data Manager Web Application Work with master data Work with models, entities, attributes, business rules
  26. 26. Master Data Services Excel Add-in • The Excel Add-in for MDS offers users functions that can be found in the Master Data Manager web app • Users can update and view master data, as well as modify or create entities • A major benefit of the Excel Add-in is the ability to quickly bulk load data into MDS • The Excel Add-in provides users the ability to use Data Quality Services to clean data before it moves into MDS
  27. 27. Using Master Data Services Create Entities Load Members Validate Review Version • Create entities and attributes to hold your master data • Define business rules to ensure quality • Load members using the staging process, Excel add-in or the API • Validate the model’s data loaded to ensure it complies to all business rules • Correct and enrich data that fails validation • Version your model so that subscribing apps access only valid data
  28. 28. Massive improvements to performance and scalability 15x MDS: What’s new in SQL Server 2016?
  29. 29. Transaction log retention Configurable settings for retaining the MDS transaction history table to enable automatic truncation. New member retention mode. Display name for each object Gives more control over the names displayed for a given object – including the Code and Name attributes. Entity Sync Modeling and Management MDS: What’s new in SQL Server 2016?
  30. 30. Type 2 subscription views See attribute change history in an easy to consume format. Custom and compound Indexes Add a custom index on commonly used attributes to improve overall performance. Business rules supporting custom SQL Scripts One level approval workflows Modeling and Management MDS: What’s new in SQL Server 2016?
  31. 31. Granular security permissions Allows permissions to be set around read, write, create, and delete. Multiple administrator roles Support for Super User and Model Admin roles allows for multiple system administrators, and model level admins. MDS: What’s new in SQL Server 2016?
  32. 32. User Experience Improvements MDS: What’s new in SQL Server 2016? -> Silverlight dependency implies using IE, no Chrome or Safari support
  33. 33. Master Data Services Features & Tasks Create Structures to Contain Data •Models •Entities •Custom Index •Attributes •Domain-Based Attributes •Attribute Groups Maintain Master Data •MDS Add-in for Microsoft Excel •Members •Business Rules •Transactions •Annotations •Hierarchies Source: https://msdn.microsoft.com/en-us/library/hh231022.aspx Improve Data Quality •Validation •Versions •Notifications •Security Move Data •Importing Data •Exporting Data •Deploying Models Develop a Custom Application •Developer's Guide •Microsoft.MasterDataService Namespace -> Topics in bold will be covered during the demo
  34. 34. Demo
  35. 35. References Master Data Services • Microsoft Master Data Services in SQL Server 2012 by James Serra: http://bit.ly/QW6kpQ • Master Data Overview: https://msdn.microsoft.com/en-us/library/ff487003.aspx • Master Data Features and Tasks: https://msdn.microsoft.com/en-us/library/hh231022.aspx • Master Data Developer’s Guide: https://msdn.microsoft.com/en-us/library/hh230994.aspx • What’s New in MDS 2016 : https://msdn.microsoft.com/en-us/library/ff929136.aspx Data Quality Services • Data Quality Services: https://msdn.microsoft.com/en-us/library/ff877925.aspx • DQS Matching Transform for SSIS: https://ssisdqsmatching.codeplex.com/ • MS DQS Blog using DQS Matching Transform: https://blogs.msdn.microsoft.com/dqs/2013/06/25/automating-the-data-matching-process-in- sql-server-data-quality-services-dqs/
  36. 36. We are recruiting! PROFILES BI and Analytics Architect Data Architect Microsoft BI Developper Analytics and Data Visuzlization Specialist 2 TECHNOLOGIES Microsoft BI (SSAS, SSIS, SSRS) Power BI Tableau MDM SQL Server recrutement@exia.ca
  37. 37. THANK YOU FOR YOUR TIME! THANK YOU TO THE DATA SCIENCE INSTITUTE FOR THIS OPPORTUNITY! DRIVE SAFELY ON YOUR WAY HOME! ☺ 33, Prince Street, Suite 284, Montreal (Quebec) H3C 2M7

    Be the first to comment

    Login to see the comments

  • JehadSenan

    Jun. 5, 2018
  • Destrogiro

    Dec. 13, 2018
  • masakitakeda2

    Jun. 22, 2019

Do you lose precious time due to data quality problems? Do you need to integrate data from multiples sources and provide an integrated view of your customer or product attributes to other systems? SQL Server 2016 Data Quality and Master Data Services can help you.

Views

Total views

8,763

On Slideshare

0

From embeds

0

Number of embeds

45

Actions

Downloads

391

Shares

0

Comments

0

Likes

3

×