This document provides tips and best practices for cleaning up and maintaining a clean database. It discusses identifying dirty data through data mapping and assessment. Techniques presented for scrubbing data include using standards to prevent errors, spell check, function keys, security groups and search/replace functions. SQL scripts are demonstrated for updating object names, migrating text entries and monitoring changes through audit trails and usage reports. The importance of ongoing monitoring and creating a culture that values data quality is emphasized.

1. How Clean is your
Database?
Data scrubbing for all skill sets
Chad Petrovay
Data Conversion Specialist
Gallery Systems

2. What is dirty data?

3. The Data Warehousing Institute
Dirty data is estimated to cost
U.S. businesses more than
$600 billion each year.

4. What fields do you have that
contain dirty data?

5. What resources do you have to
scrub your dirty data?

6. What is your personal skill level?
1. Power User 2. Administrator 3. SQL Expert

7. PREVENTION
“An ounce of prevention is worth a pound of cure”
– Benjamin Franklin

8. Name this country.

9. US
U.S.
USA
U.S.A.
U.S. of A.
United States
America
United States of America

10. Standards
• Establish the rules for
data entry
• Conceptualize terms
and authority values
Prevents
• Data entry errors
• Formatting errors
• Inconsistency
• Creativity

11. www.asmallchange.net/how-clean-up-database
Everyone at your organization
should be trained to know how
to use your database...

12. Spell Check
Uses the Spelling and
Grammar engine in
Microsoft Office.
Prevents:
• Typographical errors
• Misspellings
• Punctuation errors
• Grammatical errors

13. Function Keys
Reduces keystrokes
when entering
repeated text.
Prevents:
• Typographical errors
• Misspellings
• Punctuation errors
• Grammatical errors
• Formatting errors

14. Security Groups
• If your institution does
not use a field, then
restrict permissions
Prevents
• Populating obsolete fields
• Creativity

15. ASSESSMENT
“Great discoveries and improvements
invariably involve the cooperation of many minds.”
– Alexander Graham Bell

16. We need a list of the
distinct values in a field

17. Distinct Values
Is field a list of distinct values in Query Assistant?

18. Distinct Values
In a List View? Export into Excel; create a Pivot Table.

19. Distinct Values
In a List View? Export into Excel; create a Pivot Table.

20. Distinct Values
Database Configuration » Manage » Tables/Columns

21. Distinct Values
Right-click on the field » Frequency

22. Distinct Values
A SQL query will return all
records, including:
• Departments you
cannot see
• Template records
SELECT
DISTINCT ObjectName
FROM Objects
SELECT
ObjectName, COUNT(*)
FROM Objects
GROUP BY ObjectName
[HAVING COUNT(*) = 1]
[HAVING COUNT(*) > 1]

23. We need to find records that
use a different string format

24. Formatting
Query using wildcards:
single character (?) or multi-character (*)
Format TMS Search
(646) 733-2239 “(???) ???-????”
646.733.2239 *???.???.????*
+44 (0)207379 8188 +*
(510) 652-8950 ext 223 “* ext*”
Chad M. “* ?.”
Cheryl & Edward *&*
Cheryl and Edward “* and *”

25. Formatting
Query using wildcards:
single character (_) or multi-character (%)
Format TMS Search
(646) 733-2239 ( _ _ _ ) _ _ _ - _ _ _ _
646.733.2239 % _ _ _ . _ _ _ . _ _ _ _ %
+44 (0)207379 8188 +%
(510) 652-8950 ext. 223 % ext%
Chad M. % _.
Cheryl & Edward %&%
Cheryl and Edward % and %

26. Some of our fields are
too long or too short

27. Data Length
Database Configuration » Manage » Tables/Columns

28. Data Length
CREATE PROCEDURE [dbo].[DM_Cache_Integer4] AS
UPDATE ObjContext
SET ObjContext.Integer4 = DATALENGTH(Objects.Description)
FROM ObjContext
INNER JOIN Objects
ON ObjContext.ObjectID = Objects.ObjectID
WHERE
DATALENGTH(Objects.Description) <> ObjContext.Integer4
UPDATE ObjContext
SET Integer4 = 0
WHERE
Integer4 IS NULL
GO
Execute Stored Procedure as a Job

29. Data Length
Add field to Advance Query group

30. PLANNING
“To achieve great things, two things are needed:
a plan, and not quite enough time.”
– Leonard Bernstein

31. Human Capital
Human capital is
essential for any data
scrubbing project.
• Colleagues
• Interns
• Volunteers

32. Project Management
• Record projects
• Plan future projects
• Track progress
• Provide metrics for
administration

33. Data Map
Cheat sheet for scrubbers

34. DATA SCRUBBING
“It seems as if an age of genius
must be succeeded by an age of endeavor;
riot and extravagance by cleanliness and hard work.”
– Virginia Woolf

35. * Results may vary. May void hardware warranty.

36. Search and Replace
Maintenance » Database » Search and Replace

37. We have duplication
in our Constituent records

38. Constituent Merge Tool
Plugin; Select the constituents you want to merge.

39. Constituent Merge Tool
Review and modify fields and select joined records.

40. We want to update Object
Names based on a Package

41. Updating with SQL

42. Updating with SQL
-- SCRIPT FOR UPDATING OBJECT NAMES FROM AN OBJECT PACKAGE
DECLARE @ObjectName varchar(256), @login varchar(32), @numberRoot varchar(32), @objPackage
varchar(64), @id int, @old varchar(256)
DECLARE @tempTable TABLE(ObjectID int, Processed tinyint)
-- NEW OBJECT NAME
SET @ObjectName = ‘containers’
-- OBJECT PACKAGE NAME
SET @objPackage = ‘SQL update containers’
SET @login = 'chad.petrovay‘
-- DO NOT EDIT SCRIPT BELOW
SQL for updating ObjectName; including in Audit Trail

43. Updating with SQL
INSERT INTO @tempTable
SELECT ObjectID, 0
FROM ObjPkgList OPL
INNER JOIN ObjectPackages OP
ON OPL.ObjectPackageID = OP.ObjectPackageID
AND OP.Name = @objPackage
WHILE EXISTS ( SELECT * FROM @tempTable WHERE Processed = 0 )
BEGIN
SELECT @id = MIN(ObjectID) FROM @tempTable WHERE Processed = 0
SELECT @old = ObjectName FROM Objects WHERE ObjectID = @id
UPDATE Objects SET ObjectName = @ObjectName WHERE ObjectID = @id
INSERT INTO AuditTrail (ObjectID, ModuleID, TableName, ColumnName, OldValue, NewValue,
LoginID) VALUES (@id, '1', 'Objects', 'ObjectName', @old, @ObjectName, @login)
UPDATE @tempTable SET Processed = 1 WHERE ObjectID = @id
END
SQL for updating ObjectName; including in Audit Trail

44. Updating with SQL

45. Updating with SQL

46. We want to migrate all
Text Entries of a particular type
into a new context

47. Updating with SQL

48. Updating with SQL

49. Updating with SQL
-- SCRIPT FOR MIGRATING A TEXT ENTRY
DECLARE @OldContext INT, @NewContext INT, @TextTypeID INT
SET @OldContext = 108
SET @NewContext = 89
SET @TextTypeID = 165
-- DO NOT EDIT SCRIPT BELOW
UPDATE TextEntries SET TableID = @NewContext
WHERE TableID = @OldContext AND TextTypeID = @TextTypeID
UPDATE TextTypes SET TableID = @NewContext
WHERE TableID = @OldContext AND TextTypeID = @TextTypeID
SQL for changing the context of a Text Entry

50. Updating with SQL

51. Updating with SQL
• ALWAYS backup your database
before trying a SQL script
• If possible, try the SQL script in a
sandbox environment first
• Modifications made using SQL may
not be supported by GS

52. MONITORING
“We’re just going to be watching and monitoring.”
– John Huber

53. Saved Queries
Save time by saving your periodic review queries.

54. Audit Trail
• Proactively monitor
changes to the
database to quickly
resolve and fix errors.
• Identify “culprits” who
may require additional
training.

55. Usage Report
Record counts associated with each authority value.

56. Alerts
Notify data quality team
when common data entry
errors occur.

57. FINAL THOUGHTS
“When every physical and mental resources is focused,
one's power to solve a problem multiplies tremendously.”
– Norman Vincent Peale

58. www.asmallchange.net/how-clean-up-database
Create a culture that values data.
If you do not use the database or
are not a champion for accurate
data then your staff & co-workers
will not be either.

59. The 7th Rule of the Data Scrub
Data scrubbing
goes on as long as it has to.

60. Q&A
Chad M Petrovay
chad@gallerysystems.com