Using Data Quality tools in Data Migration Steve Tuck, 1 st October 2009

A brief history of Steve Tuck 18 years experience of data migrations & data quality Worked with leading technologies & companies

18 years experience of data migrations & data quality

Worked with leading technologies & companies

About Datanomic Data Management Software Specialist Founded 2001 Acquired Tranato 2005 Blue Chip Customer Base Global Deployments Global Partner Network

Data Management Software Specialist

Founded 2001

Acquired Tranato 2005

Blue Chip Customer Base

Global Deployments

Global Partner Network

Using DQ tools in Data Migrations Moving data Where do things go wrong? How can data quality tools help? How do they fit with ETL? Q & A

Moving data

Where do things go wrong?

How can data quality tools help?

How do they fit with ETL?

Q & A

Moving data Delivering the right data To the right place At the right time

Delivering the right data

To the right place

At the right time

Moving data Delivering the right data To the right place At the right time “ Most data migration projects fail to come in on-time and in-budget” Dylan Jones, Founder of DataMigrationPro.com and DataQualityPro.com, May 2009 “ 84% of data migration projects fail to meet expectations 64% deliver late 37% come in over budget” Philip Howard, Bloor Research (from Data Migration in the Global 2000, Sept. 2007)

Delivering the right data

To the right place

At the right time

Where do things go wrong? Little attention to data contents Some data is already broken Some gets lost or damaged in transit The target system needs data that is fit-for-purpose… This is a BUSINESS issue, not a question for IT alone

Little attention to data contents

Some data is already broken

Some gets lost or damaged in transit

The target system needs data that is fit-for-purpose…

This is a BUSINESS issue, not a question for IT alone

How can data quality tools help?

Understand the data Using data profiling to discover potential issues

Using data profiling to discover potential issues

Understand the data And collaborate with subject matter experts from the business

And collaborate with subject matter experts from the business

Understand the data Validate business critical information with targeted rules Screen out dummy & default values Identify missing and misfielded data Spot minimum state issues Refine migration data mappings

Validate business critical information with targeted rules

Screen out dummy & default values

Identify missing and misfielded data

Spot minimum state issues

Refine migration data mappings

Improve the data Build rules from the data, not the documentation From simple LUTs to semantic parsing

Build rules from the data, not the documentation

From simple LUTs to semantic parsing

Improve the data Clean and match logical entities to remove duplicates

Clean and match logical entities to remove duplicates

Control the data Provide ongoing insight as data is prepared for migration and throughout the migration event Set quality thresholds as well as ones based on the volume of data moved

Provide ongoing insight as data is prepared for migration and throughout the migration event

Set quality thresholds as well as ones based on the volume of data moved

Data quality in data migrations

Summary Data quality tools ensure that the target system is loaded with fit-for-purpose data This requires collaboration between IT and business experts Rules are commonly developed iteratively DQ tools can also be used to monitor and maintain data standards in the target

Data quality tools ensure that the target system is loaded with fit-for-purpose data

This requires collaboration between IT and business experts

Rules are commonly developed iteratively

DQ tools can also be used to monitor and maintain data standards in the target

Questions?

Steve Tuck Chief Strategy Officer +44 1223 228 435 [email_address] www.datanomic.com