Quality Assurance for Data with
Data Quality Services (MS SQL 12)
Dmitriy Romanov
Itera Consulting, Kiev
Dmitriy Romanov
dmitriy.romanov@gmail.com

Areas of expertise:
Test Automation for various projects in:
Business Intellige...
Agenda
• Intro
– Data Quality – what it is about ?
– Data Quality in Business Intelligence projects
– Tools selection

• D...
Typical information flow
Data Quality Components
Validity

Accuracy

Completeness

DATA
QUALITY
Timeliness

Consistency

Integrity
Data Quality Issues
Before QA :

After QA :
Data Quality: What is it?
Business intelligence (BI) is a set of
methodologies, processes, and technologies
that transform...
Data Quality: Tools selection
PROS Custom

Tools

• Variety of technologies
• Flexibility
• Accuracy
CONS
• Higher Compete...
Gartner Magic Quadrant for BI platforms

ABILITY TO EXECUTE

CHALLENGERS

NICHE PLAYERS
COMPLETENESS OF VISION

LEADERS

V...
Data Quality: tasks
Data Quality Services (DQS) is a Knowledge-Driven data quality
solution enabling data stewards to easi...
Knowledge
Management

Build

Connect

Knowledge
Base

Use
DQ Projects
DQS Structure
Azure Market Place

DQ Clients

DQ Server

RD Services API
(Browse, Set,
Validate…)

3rd Party
/ Internal

R...
DQS Usage
Design

Run

Knowledge
Base

Activity
Monitoring

Monitor

SSIS Package
Values/Rules
Source

Reference Data
Defi...
Real Examples
Business Case – Source Data Quality Assurance
Source
Data

Screen

Confirm

Load

Oracle

DB2

DQS

csv

DQ
Reports

KDVH
How DQS could help QA Engineer ?
• In general it allows to bring closer things Data
Analytics usually deal with
• Helps to...
Тестирование данных с помощью Data Quality Services (MS SQL 12)
Upcoming SlideShare
Loading in …5
×

Тестирование данных с помощью Data Quality Services (MS SQL 12)

1,406 views

Published on

Презентация доклада Дмитрия Романова на конференции SQADays-14, Львов 8-9 ноября 2013

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,406
On SlideShare
0
From Embeds
0
Number of Embeds
216
Actions
Shares
0
Downloads
18
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Тестирование данных с помощью Data Quality Services (MS SQL 12)

  1. 1. Quality Assurance for Data with Data Quality Services (MS SQL 12) Dmitriy Romanov Itera Consulting, Kiev
  2. 2. Dmitriy Romanov dmitriy.romanov@gmail.com Areas of expertise: Test Automation for various projects in: Business Intelligence RIA Billing systems
  3. 3. Agenda • Intro – Data Quality – what it is about ? – Data Quality in Business Intelligence projects – Tools selection • Data Quality Services – Structure – Project component – Data Quality routine • Conclusions
  4. 4. Typical information flow
  5. 5. Data Quality Components Validity Accuracy Completeness DATA QUALITY Timeliness Consistency Integrity
  6. 6. Data Quality Issues Before QA : After QA :
  7. 7. Data Quality: What is it? Business intelligence (BI) is a set of methodologies, processes, and technologies that transform raw data into meaningful and useful information for business purposes. Data Quality – represents the degree to which Data is suitable for business usages
  8. 8. Data Quality: Tools selection PROS Custom Tools • Variety of technologies • Flexibility • Accuracy CONS • Higher Competence level in business area / tech. stack • Lots of development efforts rd-party PROS 3 software • Established methods, standards, algorithms • Open / Expandable / Reusable • Lower entry level for newcomers CONS • • Scalability / performance issues Limitations
  9. 9. Gartner Magic Quadrant for BI platforms ABILITY TO EXECUTE CHALLENGERS NICHE PLAYERS COMPLETENESS OF VISION LEADERS VISIONARIES
  10. 10. Data Quality: tasks Data Quality Services (DQS) is a Knowledge-Driven data quality solution enabling data stewards to easily improve the quality of their data Cleansing Matching Profiling Monitoring
  11. 11. Knowledge Management Build Connect Knowledge Base Use DQ Projects
  12. 12. DQS Structure Azure Market Place DQ Clients DQ Server RD Services API (Browse, Set, Validate…) 3rd Party / Internal Reference Data API (Browse, Get, Update…) DQ Engine DQS User Interface SSIS DQ Component Knowledge Discovery DQ Projects Store DQ Active Projects Data Profiling & Exploration Cleansing Matching Common Knowledge Store Data Domains Reference Data Knowledge Base Store Published KBs
  13. 13. DQS Usage Design Run Knowledge Base Activity Monitoring Monitor SSIS Package Values/Rules Source Reference Data Definition DQS Cleansing Component Destination Interactive Cleansing Project Review & Manage
  14. 14. Real Examples
  15. 15. Business Case – Source Data Quality Assurance Source Data Screen Confirm Load Oracle DB2 DQS csv DQ Reports KDVH
  16. 16. How DQS could help QA Engineer ? • In general it allows to bring closer things Data Analytics usually deal with • Helps to understand underlaying data better • Introduce measurement and manageability to DQ matters • Increase re-use/decrease re-work • Open and extendable proposal of new standard to store and treat Knowledge Bases on iterative basis

×