The document provides an overview of Data Quality Services (DQS) and Master Data Services (MDS) in SQL Server 2016. It discusses the key components and features of DQS for cleansing and matching data, and MDS for defining master data structures and maintaining master data. The document also outlines the agenda for a presentation on DQS and MDS, including demos of using DQS to cleanse data and MDS to create models, entities, and load data.

1. Data Quality Services and Master Data
Services in SQL Server 2016
2017-03-15

2. Agenda
• Introduction : 5 min.
• Reality check, do you have these issues?
• What is master data management?
• Master data management in SQL Server 2016 : 15 min.
• Data Quality Services (DQS)
• Master Data Services (MDS)
• What’s new in SQL Server 2016 for MDS?
• Data Quality Services demo : 15 min.
• Cleansing & Matching data using Excel MDS Add-in
• Cleansing data using SSIS
• Master Data Services demo: 20 min.
• Creating model, entities, attributes and business rules
• Load data using SSIS

3. 33, Prince Street, Suite 284, Montreal (Quebec) H3C 2M7
Sébastien Notebaert
Senior Consultant
sebastien.notebaert@exia.ca
514-973-6108
Microsoft Data Platform expert with
15+ years experience in deploying data
warehousing, analytics, data
integration and MDM solutions.

4. • Services :
• Projets BI
• Conseil BI
• Fondée en 2010
• 35 experts
• Clientèle :
• Grande entreprise
• PMEs
• Services :
• Infra Cloud (Azure)
• Solutions Cloud
• Nouvelle pratique
• Issue de l’acquisition
GTO
• 2 experts, recrutement
actif
• Clientèle :
• Grande entreprise
• PMEs
• Services gérés
• Cloud privé
• Cloud Public
• Fondée en 2006
• 20 experts
• Clientèle :
• PMEs
• Domaine Médical
• Produits :
• Solution BI
Financier
• Solution processus
budgétaire
• Reporting ERO
• Fondée en 2010
• 5 experts
• Clientèle :
• PMEs
Who we are…

5. Reality check: do you have these issues?
• Do you have instances of invalid data impacting business processes?
• Do you wish your business users could manage data themselves such
as Customer and Product?
• Do you have IT resources spending time investigating data quality
issues and/or applying data fixes?
• Do you have the need for consolidation and distribution of data to
other systems?
• Do you have an environment of heterogeneous systems which all
could benefit from a single view of domain data such as Customer or
Product?
-> You need master data management!

6. What is master data management?
• The technology, tools, and processes required to create and
maintain consistent and accurate master data
• Master data is a set of data objects that are at the center of
business activities like: Customers, Products, Cost Centers,
Locations, …
• Master data is not transactional data like a sale or inventory
movement
• To succeed an MDM initiative must include business processes,
people AND data

7. MDM Architecture Style
Style Description Which
System is
Master
Update
Source
Systems
Comple
xity
REGISTRY Master data is not consolidated, but is maintained as a set of “stub”
records mapped to attributes stored in the source systems. There
is little data movement other than create or delete global stub
records in the registry. The golden record is assembled dynamically
using complex distributed queries. The upside is that a real time
central reference can be made with little or no infrastructure
investment. The downside is that without central governance of the
data the golden record isn’t highly reliable.
Source
system
Not
applicable
Data stays
in source
systems
1
CONSOLIDATED Master data is consolidated from multiple source systems into a
physical golden record for downstream consumption; however, any
updates made to the master data are not returned to the original
sources. Consolidated MDM hubs are quick and inexpensive to set
up, and offer a big return by enabling reliable enterprise-wide
reporting. Data movement is typically tolerant of inter-day latency
and managed through inexpensive batch processes, but it’s a one-
way street from source systems.
Source
system
No 2

8. MDM Architecture Style (continued)
Style Description Which
System is
Master
Update
Source
Systems
Comple
xity
COEXISTENCE Just like consolidated, coexistence harmonizes multiple sources into a
physical golden record for downstream consumption. Coexistence
adds the important step of updating the source systems. These
requirements are typically high latency and can usually be met at an
acceptable time and cost through bi-directional batch processes.
Coexistence is a natural evolution of the consolidated architecture
with the added benefit of linking centrally governed data back to the
source systems. Interfacing with complex data sources, such as ERP
systems, can become a costly drawback.
Source
system
Yes 3
TRANSACTIO
OR
CENTRALIZED
While this approach also creates a physical golden record, the key
differentiator from the consolidated and coexistence styles is the
MDM hub extends enterprise governance over the source systems.
This introduces shorter latency requirements that are typically
addressed with a combination of web services integration and an
authoring application that bridges centralized and application-
specific governance needs. In most cases, this is a big step up in cost,
complexity and implementation time. The pay-off is more
comprehensive governance in real-time; however, too many complex
sources can push cost and complexity beyond the value created by
the architecture.
MDM
system
Yes 4

9. Master Data Management Entity Examples
People Things Places Abstract
Customers
Vendors
Sales People
Employees
Partners
Patients
Products
Business Units
Bill of Materials
Parts
Storage Bins
Equipment
Locations
Stores
Wells
Power Lines
Geo Areas
Warehouses
Accounts
Warranties
Time
Metrics
Securities
Contracts

10. Master Data Management Solution Areas
Data Quality
Improve Efficiency
Compliance
Retain Customers
Merger & Acquisition
Cross Reference
Golden Records
• Human typographical errors; incomplete information; spreadsheet data management
• Mergers and consolidation; ERP implementations, consolidation or migration
• New purposes for old data; retire old applications such as mainframe applications
• Single point of data maintenance; BI reporting
• Different types of customer accounts
• Accurate view of data by implementing MDM and DQ
• Single point of data maintenance
• Cross sell and upsell
• Tracking spends by customer
• Province and federal legislation
• Single view of customer spend, channels, cross sell and upsell
• Cross reference of same customers across multiple systems
• Survivorship of best consolidated data across multiple systems
• Single view of anything that has attributes that can be matched
• Cleanup of source systems with business rules and golden records pushed back
• Merging chart of accounts; consolidate financial reporting
• Single view of product
• Single view of customers

11. Master Data Management in SQL
Server 2016

12. Master Data Management in SQL 2016

13. MDS & DQS in a Consolidated MDM Architecture

14. Data Quality Services

15. Data Quality Services
• SQL Server Data Quality Services (DQS) is a knowledge-
driven data quality tool.
• DQS enables you to build a knowledge base and use it
to perform a variety of critical data quality tasks
including: correction, enrichment, standardization, and
de-duplication of your data.
• DQS is like a spell checker for your data
• DQS was shipped in SQL Server 2012 (ent. & BI ed.)

16. Data Quality Services (continued)
• DQS consists of Data Quality Server and Data Quality Client,
both of which are installed as part of SQL Server 2016.
• Data Quality Server is a SQL Server instance feature that consists
of three SQL Server catalogs with data-quality functionality and
storage

17. Data Quality Services Building Blocks
• DQS building blocks
• Knowledge base
• Domains
• Knowledge discovery
• Matching policy
• Data Quality projects
• Cleansing project
• Matching (deduplication)
project

18. Using Data Quality Services
Create Domains
Knowledge
Discovery
Clean Data Match Data
• Create a domain
like product
brand or
category
• Set it properties
• Set it values
• Set it rules
• Set its term
based relations
• Train your
domains by
loading data
• Accept / reject /
enrich / correct
your domains
• Clean data sets
using your
trained
knowledge base
• Match and
deduplicate
your data using
a matching
policy

19. Data Quality Services Features & Tasks
Knowledge Bases and Domains
•Building a KB
•Importing and Exporting Knowledge
•Managing a domain
•Managing a composite domain
Data Quality Projects
•Create a Data Quality project
•Open, Unlock, … a DQ project
•Open an SSIS Project in DQ client
Data Cleansing
•Cleanse Data Using DQS
•Cleanse Data in a Composite Domain
Source:
https://msdn.microsoft.com/en-us/library/hh213042.aspx
Data Matching
•Create a matching policy
•Run a Matching Project
Reference Data Services in DQS
•Configure DQS to use reference data
•Attach domain to reference data
•Cleanse data using reference data
Data Profiling and Notifications
DQS Administration
DQS Security
-> Topics in bold will be covered during the demo

20. Master Data Services

21. Master Data Services
• SQL Server Master Data Services is an Master Data Management
(MDM) tool to define and manage non-transactional lists of
data, with the goal of compiling maintainable and validated
master lists for the enterprise.
• An MDM project generally includes an evaluation and
restructuring of internal business processes along with the
implementation of MDM technology. The result of a successful
MDM solution is reliable, centralized data that can be analyzed,
resulting in better business decisions.
• MDS was shipped in SQL Server 2008 R2 (datacenter & ent. ed.)

22. Master Data Services (continued)
Master Data Services has the following components:
• Master Data Services Configuration Manager, a tool you use to create and
configure MDS databases and web applications.
• Master Data Manager, a web application you use to perform administrative
tasks (like creating a model or business rule), and that users access to
update data.
• MDSModelDeploy.exe, a tool you use to create packages of your model
objects and data so you can deploy them to other environments.
• Master Data Services web service (WCF), which developers can use to
extend or develop custom solutions for Master Data Services.
• Master Data Services Add-in for Excel, which you use to manage data and
create new entities and attributes.

23. Master Data Services Building Blocks
• The model is the most fundamental object in an MDS solution
• Models are the containers that encapsulate all other MDS objects (i.e. entities, attributes and business rules)

24. Interacting with Master Data Services
• Master Data Manager Web User Interface
• MDS Web Services API (WCF API)
• Subscription views via T-SQL
• Staging stored procedures
• MDS Add-in for Excel
SSIS package that calls
MDS stored procedures ->

25. Master Data Manager Web Application
Work with master data
Work with models, entities,
attributes, business rules

26. Master Data Services Excel Add-in
• The Excel Add-in for MDS offers users functions that can be found in the Master Data
Manager web app
• Users can update and view master data, as well as modify or create entities
• A major benefit of the Excel Add-in is the ability to quickly bulk load data into MDS
• The Excel Add-in provides users the ability to use Data Quality Services to clean data
before it moves into MDS

27. Using Master Data Services
Create Entities Load Members Validate Review Version
• Create
entities and
attributes to
hold your
master data
• Define
business
rules to
ensure
quality
• Load
members
using the
staging
process,
Excel add-in
or the API
• Validate the
model’s data
loaded to
ensure it
complies to
all business
rules
• Correct and
enrich data
that fails
validation
• Version your
model so
that
subscribing
apps access
only valid
data

28. Massive improvements to performance and scalability
15x
MDS: What’s new in SQL Server 2016?

29. Transaction log retention
Configurable settings for retaining the MDS
transaction history table to enable automatic
truncation. New member retention mode.
Display name for each object
Gives more control over the names displayed for a
given object – including the Code and Name
attributes.
Entity Sync
Modeling and
Management
MDS: What’s new in SQL Server 2016?

30. Type 2 subscription views
See attribute change history in an easy to consume
format.
Custom and compound Indexes
Add a custom index on commonly used attributes
to improve overall performance.
Business rules supporting custom
SQL Scripts
One level approval workflows
Modeling and
Management
MDS: What’s new in SQL Server 2016?

31. Granular security permissions
Allows permissions to be set around read, write,
create, and delete.
Multiple administrator roles
Support for Super User and Model Admin roles
allows for multiple system administrators, and
model level admins.
MDS: What’s new in SQL Server 2016?

32. User Experience Improvements
MDS: What’s new in SQL Server 2016?
-> Silverlight dependency implies using IE, no Chrome or Safari support

33. Master Data Services Features & Tasks
Create Structures to Contain Data
•Models
•Entities
•Custom Index
•Attributes
•Domain-Based Attributes
•Attribute Groups
Maintain Master Data
•MDS Add-in for Microsoft Excel
•Members
•Business Rules
•Transactions
•Annotations
•Hierarchies
Source:
https://msdn.microsoft.com/en-us/library/hh231022.aspx
Improve Data Quality
•Validation
•Versions
•Notifications
•Security
Move Data
•Importing Data
•Exporting Data
•Deploying Models
Develop a Custom Application
•Developer's Guide
•Microsoft.MasterDataService
Namespace
-> Topics in bold will be covered during the demo

34. Demo

35. References
Master Data Services
• Microsoft Master Data Services in SQL Server 2012 by James Serra: http://bit.ly/QW6kpQ
• Master Data Overview: https://msdn.microsoft.com/en-us/library/ff487003.aspx
• Master Data Features and Tasks: https://msdn.microsoft.com/en-us/library/hh231022.aspx
• Master Data Developer’s Guide: https://msdn.microsoft.com/en-us/library/hh230994.aspx
• What’s New in MDS 2016 : https://msdn.microsoft.com/en-us/library/ff929136.aspx
Data Quality Services
• Data Quality Services: https://msdn.microsoft.com/en-us/library/ff877925.aspx
• DQS Matching Transform for SSIS: https://ssisdqsmatching.codeplex.com/
• MS DQS Blog using DQS Matching Transform:
https://blogs.msdn.microsoft.com/dqs/2013/06/25/automating-the-data-matching-process-in-
sql-server-data-quality-services-dqs/

36. We are recruiting!
PROFILES
BI and Analytics Architect
Data Architect
Microsoft BI Developper
Analytics and Data Visuzlization Specialist
2
TECHNOLOGIES
Microsoft BI (SSAS, SSIS, SSRS)
Power BI
Tableau
MDM
SQL Server
recrutement@exia.ca

37. THANK YOU FOR YOUR TIME!
THANK YOU TO THE DATA SCIENCE
INSTITUTE FOR THIS OPPORTUNITY!
DRIVE SAFELY ON YOUR WAY HOME!
☺
33, Prince Street, Suite 284, Montreal (Quebec) H3C 2M7