Resurrecting the Prodigal Son - Data Quality    “ Rise from Ashes: Battle of Data Quality Testing”
Speakers <ul><li>Bhoomika Goyal </li></ul><ul><ul><li>Working @ Microsoft for over an year </li></ul></ul><ul><ul><li>Engi...
Horror Story <ul><li>  Loss: $ 125 million  </li></ul><ul><li>  Reason: Discrepancy   between the two  </li></ul><ul><li> ...
Bad, Bad, Bad Data Quality www.Test2008.in Erroneous Mailing hit  $611 billion  for US  businesses in 2002
DQ is not my problem? Think Again !!!!!  www.Test2008.in
DQ Hot Candidates www.Test2008.in Data Movement Migrations Backups Restore Import Export Data Warehousing Business Intelli...
DQ  Ishikawa Diagram www.Test2008.in Bad  Decisions  (Loss $ & Customers) DQ Reqmts not documented  Lack of white box test...
www.Test2008.in Data Quality DQ is an  indicator  that tells about the health of the DATA
www.Test2008.in GOOD Data Quality DQ is  good  if data is fit to use for  decision making
www.Test2008.in Data Quality Testing <ul><ul><li>Involves  validating ,  monitoring  &  reporting  various attributes of D...
DQ Checks www.Test2008.in Row Counts Consistency Referential Integrity Redundancy Usability Completeness Domain Integrity ...
Row Count Check www.Test2008.in
Completeness Check www.Test2008.in
Among Voters seen Dead People www.Test2008.in US General Election: 4,755 deceased  people voted
Consistency Check www.Test2008.in
A One-House, $400 Million Bubble Goes Pop www.Test2008.in $1,21, 000 overvalued at $ 400 million Govt. Expected $8 million...
Accuracy Check www.Test2008.in
Validity Check www.Test2008.in
CD Mail Fraud <ul><li>Man received 22,260  CDs at discounted price by making  each address different enough </li></ul>www....
Redundancy Check www.Test2008.in
Referential Integrity Check www.Test2008.in
Domain Integrity Check www.Test2008.in
Timeliness www.Test2008.in
How do we test DQ? www.Test2008.in DQ Rule Engine Metadata Results Create Procedure RowCount (SrcTbl, TgtTbl) Begin Declar...
You can’t improve what you can’t measure www.Test2008.in Threshold Time 5 % 10 % 100 % Data Quality Red: BAD DQ Yellow: Wa...
DQ Testing is your friend !!! <ul><li>High Data (Test) Coverage </li></ul><ul><li>Automation (Manual Effort Reduction) </l...
References <ul><li>http://www.dataqualitysolutions.com/data/index.shtml </li></ul><ul><li>http://searchdatamanagement.tech...
<ul><li>Thank you. </li></ul><ul><li>[email_address]   </li></ul><ul><li>[email_address] </li></ul>www.Test2008.in
Upcoming SlideShare
Loading in …5
×

Test2008 Resurrecting The Prodigal Son Data Quality (http://www.geektester.blogspot.com)

1,316 views

Published on

http://www.geektester.blogspot.com/

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,316
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
34
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Test2008 Resurrecting The Prodigal Son Data Quality (http://www.geektester.blogspot.com)

    1. 1. Resurrecting the Prodigal Son - Data Quality “ Rise from Ashes: Battle of Data Quality Testing”
    2. 2. Speakers <ul><li>Bhoomika Goyal </li></ul><ul><ul><li>Working @ Microsoft for over an year </li></ul></ul><ul><ul><li>Engineer from Mumbai </li></ul></ul><ul><ul><li>Loves playing Chess, Solving Puzzles and Reading </li></ul></ul><ul><li>Raj </li></ul><ul><li> </li></ul><ul><ul><li>W orking @ Microsoft Business Intelligence COE </li></ul></ul><ul><ul><li>5.5 + years of Testing experience </li></ul></ul><ul><ul><li>Loves watching movies, reading suspense thrillers & playing cricket </li></ul></ul><ul><ul><li>Passion - Testing ( http://www.itest.co.nr ) </li></ul></ul>www.Test2008.in
    3. 3. Horror Story <ul><li> Loss: $ 125 million </li></ul><ul><li> Reason: Discrepancy between the two </li></ul><ul><li> measures (rocket thrusts to newtons) </li></ul><ul><li>NASA Mars Climate Orbiter spacecraft LOST </li></ul>www.Test2008.in
    4. 4. Bad, Bad, Bad Data Quality www.Test2008.in Erroneous Mailing hit $611 billion for US businesses in 2002
    5. 5. DQ is not my problem? Think Again !!!!! www.Test2008.in
    6. 6. DQ Hot Candidates www.Test2008.in Data Movement Migrations Backups Restore Import Export Data Warehousing Business Intelligence OLTP OLAP CRM ERP
    7. 7. DQ Ishikawa Diagram www.Test2008.in Bad Decisions (Loss $ & Customers) DQ Reqmts not documented Lack of white box testing Data is dynamic CRM & ERPs Implementations Mergers / Take Over
    8. 8. www.Test2008.in Data Quality DQ is an indicator that tells about the health of the DATA
    9. 9. www.Test2008.in GOOD Data Quality DQ is good if data is fit to use for decision making
    10. 10. www.Test2008.in Data Quality Testing <ul><ul><li>Involves validating , monitoring & reporting various attributes of Data </li></ul></ul><ul><ul><li>like </li></ul></ul><ul><ul><li>accuracy , validity , timeliness etc </li></ul></ul>
    11. 11. DQ Checks www.Test2008.in Row Counts Consistency Referential Integrity Redundancy Usability Completeness Domain Integrity Timeliness Accuracy Validity
    12. 12. Row Count Check www.Test2008.in
    13. 13. Completeness Check www.Test2008.in
    14. 14. Among Voters seen Dead People www.Test2008.in US General Election: 4,755 deceased people voted
    15. 15. Consistency Check www.Test2008.in
    16. 16. A One-House, $400 Million Bubble Goes Pop www.Test2008.in $1,21, 000 overvalued at $ 400 million Govt. Expected $8 million as Tax Revenue
    17. 17. Accuracy Check www.Test2008.in
    18. 18. Validity Check www.Test2008.in
    19. 19. CD Mail Fraud <ul><li>Man received 22,260 CDs at discounted price by making each address different enough </li></ul>www.Test2008.in David Loshin 123 Main Street Any town, NY 11787 David Loshin 123 Main Street, Near Wal-Mart Any town, NY 11787
    20. 20. Redundancy Check www.Test2008.in
    21. 21. Referential Integrity Check www.Test2008.in
    22. 22. Domain Integrity Check www.Test2008.in
    23. 23. Timeliness www.Test2008.in
    24. 24. How do we test DQ? www.Test2008.in DQ Rule Engine Metadata Results Create Procedure RowCount (SrcTbl, TgtTbl) Begin Declare SRC, TGT Integer Select SRC = Count(*) from SrcTbl Select TGT = Count(*) from TgtTbl) If SRC = TGT Then Return “PASS” Else Return SRC – TGT End If End Metadata Results Row Count Logic Duplicate Logic Create Procedure Duplicate(Tbl) Begin Declare Dup Integer Select Dup = Count of Select * from Tbl GroupBy <<ColumnList>> Having count(*) > 1 If Dup = 0 Then Return “PASS” Else Return Dup End If End End Rule Tbl1 Tbl2 RC Emp Emp RI Emp Dept DC HR HR Rule Result Comment RC Pass - RI Fail 10 DC Pass -
    25. 25. You can’t improve what you can’t measure www.Test2008.in Threshold Time 5 % 10 % 100 % Data Quality Red: BAD DQ Yellow: Watch it Green: Good DQ
    26. 26. DQ Testing is your friend !!! <ul><li>High Data (Test) Coverage </li></ul><ul><li>Automation (Manual Effort Reduction) </li></ul><ul><li>High confidence about your data </li></ul><ul><li>Accurate Decisions </li></ul>www.Test2008.in
    27. 27. References <ul><li>http://www.dataqualitysolutions.com/data/index.shtml </li></ul><ul><li>http://searchdatamanagement.techtarget.com/generic/0,295582,sid91_gci1251808,00.html </li></ul><ul><li>http://en.wikipedia.org/wiki/Effect_of_Hurricane_Katrina_on_New_Orleans </li></ul>www.Test2008.in
    28. 28. <ul><li>Thank you. </li></ul><ul><li>[email_address] </li></ul><ul><li>[email_address] </li></ul>www.Test2008.in

    ×