Data quality begins at the source. There are many ways "bad" data can get into a transactional system. This presentation describes some of those paths and offers tips for avoiding the pitfalls.

1. Stop Bad Data in its
OLTP Tracks
Denise McInerney
Silicon Valley SQL Server Users Group
June 19, 2012
Copyright © 2012 Denise McInerney

2. Agenda
Topics
 What is “bad” data?
 Table design
 NULL pitfalls
 Constraints: friend or foe?
 Hardcoding
 Transactions
 Garbage in
 Testing
 Change Management & Controls
Silicon Valley SQL Server Users Group
June 19, 2012 2Copyright © 2012 Denise McInerney

3. Who am I?
• SQL Server DBA since 1998
 Web-based OLTP applications
 Focus on design and
performance tuning
• Staff DBA Intuit
• Director, PASS Board
• Virtual Chapters
• Women in Tech VC
• Contact me:
 denise.mcinerney@sqlpass.org
 denise_mcinerney@intuit.com
 @denisemc06
Copyright © 2012 Denise McInerney 3
Silicon Valley SQL Server Users Group
June 19, 2012

4. My Framework
 DBA = data steward
 Backups, security—what about quality?
 DBA <> Developer
 Don‟t count on ETL for everything
Silicon Valley SQL Server Users Group
June 19, 2012 4Copyright © 2012 Denise McInerney

5. Define “bad” data
 Inaccurate
 Missing
 Misleading
 Causes bugs
 Breaks reports
 Can compromise security
 Impacts customers
 Impacts business
Silicon Valley SQL Server Users Group
June 19, 2012 5Copyright © 2012 Denise McInerney

6. Design
 Have one—even in an “agile” shop
 It‟s a system, not a collection of tables
 Don‟t be lazy
 Bad design invites bad data
Silicon Valley SQL Server Users Group
June 19, 2012 6Copyright © 2012 Denise McInerney

7. Design
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 7

8. Bad Design Examples
Silicon Valley SQL Server Users Group
June 19, 2012 8Copyright © 2012 Denise McInerney

9. Pitfalls of NULL
 What is NULL?
 Missing data
 Unknown in comparisons
 NULL <> „‟ or “”
 NULL <> 0
 NULL can be valid
 But application can‟t handle it
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 9

10. NULL Examples
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 10

11. Constraints: Your Frenemies
 CHECK
 INSERT/UPDATE fails if condition evaluates to
FALSE
 Comparisons to NULL result in unknown
 Can use to enforce logic, business rules
 DEFAULT
 Can prevent some bugs
 Won‟t solve all your problems
 Can be “forgotten”
 100% of records have default value
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 11

12. Constraint Examples
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 12

13. Hardcoding
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 13

14. Transactions
 Understand them
 Don‟t assume others understand them
 BEGIN TRAN…COMMIT TRAN is not
enough
 Failed INSERT…the silent killer
Silicon Valley SQL Server Users Group
June 19, 2012 14Copyright © 2012 Denise McInerney

15. Transactions
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 15

16. Garbage In
 Many ways things go wrong
 Concatenation
 Recycled identifiers
 Duplicates
 Honor the schema
 Be consistent
 Fix it on the way in
Silicon Valley SQL Server Users Group
June 19, 2012 16Copyright © 2012 Denise McInerney

17. Garbage In
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 17

18. Testing
 Database is not exempt
 Not just the happy path
 Look at the data!
 People
 Machines
Silicon Valley SQL Server Users Group
June 19, 2012 18Copyright © 2012 Denise McInerney

19. Change Control
 Lots of ways data gets IN
 Code—application & SQL
 Version control & Management
 Scripts
 GUI (!)
 People
 Everyone takes short cuts
 Every check-in potential data bug
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 19

20. There is no “I” in “Bad Data”
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 20

21. Pessimism
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 21

22. Vigilance
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 22

23. Summary
 DBA‟s job to guard data quality
 Design, Plan, Process, Controls
 Engage Others
 Keep your guard up
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 23

24. Q & A
Silicon Valley SQL Server Users Group
June 19, 2012 Copyright © 2012 Denise McInerney 24