Big Data Profiling
Upcoming SlideShare
Loading in...5
×
 

Big Data Profiling

on

  • 351 views

Data profiling comprises a broad range of methods to efficiently analyze a given data set. In a typical scenario, which mirrors the capabilities of commercial data profiling tools, tables of a ...

Data profiling comprises a broad range of methods to efficiently analyze a given data set. In a typical scenario, which mirrors the capabilities of commercial data profiling tools, tables of a relational database are scanned to derive metadata, such as data types and value patterns, completeness and uniqueness of columns, keys and foreign keys, and occasionally functional dependencies and association rules. Individual research projects have proposed several additional profiling tasks, such as the discovery of inclusion dependencies or conditional functional dependencies.

Data profiling deserves a fresh look for two reasons: First, the area itself is neither established nor defined in any principled way, despite significant research activity on individual parts in the past. Second, current data profiling techniques hardly scale beyond what can only be called small data. Finally, more and more data beyond the traditional relational databases are being created and beg to be profiled. The talk proposes new research directions and challenges, including interactive and incremental profiling and profiling heterogeneous and non-relational data.

Speaker: Felix Naumann studied mathematics, economy, and computer sciences at the University of Technology in Berlin. After receiving his diploma (MA) in 1997 he joined the graduate school "Distributed Information Systems" at Humboldt University of Berlin. He completed his PhD thesis on "Quality-driven Query Answering" in 2000. In 2001 and 2002 he worked at the IBM Almaden Research Center on topics around data integration. From 2003 - 2006 he was assistant professor for information integration at the Humboldt-University of Berlin. Since then he holds the chair for information systems at the Hasso Plattner Institute at the University of Potsdam in Germany.

Statistics

Views

Total Views
351
Views on SlideShare
351
Embed Views
0

Actions

Likes
0
Downloads
17
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Big Data Profiling Big Data Profiling Presentation Transcript

    • Big Data Profiling Fribourg May 2014 Felix Naumann
    • The Hasso Plattner Institute ■ Founded in 1998 as a Public Private Partnership ■ Hasso Plattner, co-founder of SAP, endowed over 200 Mio. Euro. ■ Adjoined with the University of Potsdam ■ 500 students □ BA, MA, PhD 2 ■ Enterprise Platform and Integration Concepts ■ Internet Technologies and Systems ■ Human Computer Interaction ■ Computer Graphics Systems ■ Operating Systems and Middleware ■ Business Process Technology ■ Software Architecture ■ Information Systems ■ System Engineering and Modeling ■ School of Design Thinking Felix Naumann | Data Profiling | CUSO 2014
    • Research Topics ■ Data Profiling and Analytics ■ Data Quality and Data Cleansing ■ Similarity Search and ETL Management ■ Knowledge Discovery and Text Extraction ■ (Linked) Open Data Integration ■ For more information on research topics and on teaching, please see http://www.hpi.uni-potsdam.de/naumann/home.html 3 Felix Naumann | Data Profiling | CUSO 2014
    • Profiling in Spreadsheets Felix Naumann | Data Profiling | CUSO 2014 4
    • Felix Naumann | Data Profiling | CUSO 2014 5
    • Felix Naumann | Data Profiling | CUSO 2014 6
    • Felix Naumann | Data Profiling | CUSO 2014 7
    • Felix Naumann | Data Profiling | CUSO 2014 8
    • Felix Naumann | Data Profiling | CUSO 2014 9
    • Felix Naumann | Data Profiling | CUSO 2014 10
    • Many interesting questions remain ■ What are possible keys and foreign keys? □ Phone □ firstname, lastname, street ■ Are there any functional dependencies? □ zip -> city □ race -> voting behavior ■ Which columns correlate? □ county and first name □ DoB and last name ■ What are frequent patterns in a column? □ ddddd □ dd aaaa St Felix Naumann | Data Profiling | CUSO 2014 11
    • Definition Data Profiling ■ Data profiling is the process of examining the data available in an existing data source [...] and collecting statistics and information about that data. Wikipedia 09/2013 ■ Data profiling refers to the activity of creating small but informative summaries of a database. Ted Johnson, Encyclopedia of Database Systems ■ A fixed set of data profiling tasks / results Felix Naumann | Data Profiling | CUSO 2014 12
    • „Big“ Data Profiling or How big is „Big“? Data profiling = measuring the „Vs“ ■ Volume □ Row counts, etc. ■ Velocity □ Temporal profiling ■ Variability □ How difficult to integrate and analyse ■ Veracity □ How good is it? ■ … Felix Naumann | Data Profiling | CUSO 2014 13 Big Data Volume Velocity Variety Veracity Viscosity Virality
    • Use Cases for Profiling ■ Query optimization □ Counts and histograms ■ Data cleansing □ Patterns, rules, and violations ■ Data integration □ Cross-DB inclusion dependencies ■ Scientific data management □ Handle new datasets ■ Data inspection, analytics, and mining □ Profiling as preparation to decide on models and questions ■ Database reverse engineering ■ Data profiling as preparation for any other data management task Felix Naumann | Data Profiling | CUSO 2014 14
    • Classification of Traditional Profiling Tasks Felix Naumann | Data Profiling | CUSO 2014 15 Dataprofiling Single column Cardinalities Patterns and data types Value distributions Multiple columns Uniqueness Key discovery Conditional Partial Inclusion dependencies Foreign key discovery Conditional Partial Functional dependencies Conditional Partial
    • Single-column vs. multi-column ■ Single column profiling □ Most basic form of data profiling □ Often part of the basic statistics gathered by DBMS □ Discovery complexity: Number of values/rows ■ Multicolumn profiling □ Discover joint properties □ Discover dependencies □ Discovery complexity: Number of columns and number of values Felix Naumann | Data Profiling | CUSO 2014 16
    • Scalable profiling ■ Scalability in number of rows ■ Scalability in number of columns □ “Small” table with 100 columns: 2100 – 1 = 1,267,650,600,228,229,401,496,703,205,375 = 1.3 nonillion column combinations ◊ Impossible to check or even enumerate ■ Possible solutions □ Scale up: More RAM, faster CPUs ◊ Expensive □ Scale in: More cores ◊ More complex (threading) □ Scale out: More machines ◊ Communication overhead □ Intelligent enumeration and aggressive pruning Felix Naumann | Data Profiling | CUSO 2014 17
    • Challenges of (Big) Data Profiling Felix Naumann | Data Profiling | CUSO 2014 18 ■ Computational complexity □ Number of rows □ Number of columns (and column combinations) ■ Large solution space ■ New data types (beyond strings and numbers) ■ New data models (beyond relational): RDF, XML, etc. ■ New requirements □ User-oriented □ Interactive □ Streaming data
    • Agenda 19 ■ Basic statistics ■ Functional dependencies ■ Keys and foreign keys ■ Data profiling tools ■ Advanced profiling Felix Naumann | Data Profiling | CUSO 2014
    • Cardinalities, distributions, and patterns Category Task Description Cardinalities num-rows Number of rows value length Measurements of value lengths (min, max, median, and average) null values Number or percentage of null values distinct Number of distinct values; aka “cardinality” uniqueness Number of distinct values divided by number of rows Value distributions histogram Frequency histograms (equi-width, equi-depth, etc.) constancy Frequency of most frequent value divided by number of rows quartiles Three points that divide the (numeric) values into four equal groups soundex Distribution of soundex codes first digit Distribution of first digit in numeric values; to check Benford's law Patterns, data types, and domains basic type Generic data type: numeric, alphabetic, date, time data type Concrete DBMS-specific data type: varchar, timestamp, etc. decimals Maximum number of decimal places in numeric values precision Maximum number of digits in numeric values patterns Histogram of value patterns (Aa9…) data class Semantic, generic data type: code, indicator, text, date/time, quantity, identifier, etc. domain Classification of semantic domain: credit card, first name, city, phenotype, etc. Felix Naumann | Data Profiling | CUSO 2014 20
    • Data types and value patterns ■ String vs. number ■ String vs. number vs. date ■ Categorical vs. continuous ■ SQL data types □ CHAR, INT, DECIMAL, TIMESTAMP, BIT, CLOB, … ■ Domains □ VARCHAR(12) vs. VARCHAR (13) ■ XML data types □ More fine grained ■ Regular expressions (d{3})-(d{3})-(d{4})-(d+) ■ Semantic domains □ Adress, phone, email, first name Felix Naumann | Data Profiling | CUSO 2014 21 Increasingsemantics
    • An Aside: Benford Law Frequency (“first digit law”) ■ Statement about the distribution of first digits d in (many) naturally occurring numbers: □ 𝑃 𝑑 = 𝑙𝑜𝑔10 𝑑 + 1 − 𝑙𝑜𝑔10 𝑑 = 𝑙𝑜𝑔10 1 + 1 𝑑 □ Holds if log(x) is uniformly distributed Felix Naumann | Data Profiling | CUSO 2014 22 0 20 40 1 2 3 4 5 6 7 8 9
    • Examples for Benford‘s Law ■ Surface areas of 335 rivers ■ Sizes of 3259 US populations ■ 104 physical constants ■ 1800 molecular weights ■ 5000 entries from a mathematical handbook ■ 308 numbers contained in an issue of Reader's Digest ■ Street addresses of the first 342 persons listed in American Men of Science Felix Naumann | Data Profiling | CUSO 2014 23 Heights of the 60 tallest structures http://en.wikipedia.org/wiki/List_of_tallest_buildings_and_structures_in_the_world# Tallest_structure_by_category
    • Agenda 24 ■ Basic statistics ■ Functional dependencies ■ Keys and foreign keys ■ Data profiling tools ■ Advanced profiling Felix Naumann | Data Profiling | CUSO 2014
    • Naive Discovery Approach ■ Functional dependency „X → A“: whenever two records have the same X values, they also have the same A values. ■ Given relation R, detect all minimal, non-trivial FDs X → A. ■ For each column combination X □ For each pair of tuples (t1,t2) ◊ If t1[XA] = t2[XA] and t1[A]  t2[A]: Break ■ Complexity □ Exponential in number of attributes □ times number of rows squared Felix Naumann | Data Profiling | CUSO 2014 25
    • Tane – General Idea [HKPT99] ■ Two elements of approach 1. Reduce column combinations through pruning ◊ Reasoning over FDs 2. Reduce tuple sets through partitioning ◊ Partition tuple IDs according to attribute values ◊ Level-wise increase of size of attribute set ● Consider sets of tuples whose values agree on that set Felix Naumann | Data Profiling | CUSO 2014 26
    • Discovery strategy ■ Bottom up traversal through lattice □  only minimal dependencies □ Pruning □ Re-use results from previous level ■ For a set X, test all XA → A, AX □  only non-trivial dependencies □ Test on efficient data structure Felix Naumann | Data Profiling | CUSO 2014 27 A B C D AB ACAD BC BD CD ABC ABD ACD BCD ABCD
    • Functional Dependencies: State of the Art Felix Naumann | Data Profiling | CUSO 2014 28
    • Partial and conditional dependencies ■ Partial dependency: dependencies that do not perfectly hold □ For all but 10 of the tuples □ Only for 90% of the tuples □ Only for 1% of the tuples ■ Partiality also for patterns, types, uniques, and other constraints ■ Given a partial dependencies: For which part does it hold? □ Expressed as a condition over the attributes of the relation ■ Problems: □ Infinite possibilities of conditions □ Interestingness: ◊ Many distinct values: less interesting ◊ Few distinct values: surprising condition – high coverage ■ Useful for □ Integration: cross-source condition inclusion dependency Felix Naumann | Data Profiling | CUSO 2014 29
    • Agenda 30 ■ Basic statistics ■ Functional dependencies ■ Keys and foreign keys ■ Data profiling tools ■ Advanced profiling Felix Naumann | Data Profiling | CUSO 2014
    • Uniqueness, keys, and foreign keys ■ Uniqueness and keys □ Unique column: Only unique values □ Unique column combination: Only unique value combinations ◊ Minimality: No subset is unique □ Key candidate: No null values ◊ Uniqueness and non-null in one instance does not imply key: Only human can specify keys (and foreign keys) ■ Inclusion dependencies and foreign keys □ A  B: All values in A are also present in B □ A1,…,Ai  B1,…,Bi: All value comb. in A1,…,Ai are also present in B1,…,Bi □ Prerequisite for foreign key ◊ Across relations and across databases ◊ Again: Discovery on a given instance, only user can specify for schema Felix Naumann | Data Profiling | CUSO 2014 31
    • Uniqueness and keys ■ Unique column □ Only unique values ■ Unique column combination □ Only unique value combinations □ Minimality: No subset is unique ■ Uniques: {A, AB, AC, BC, ABC} ■ Minimal uniques: {A, BC} ■ (Maximal) Non-uniques: {B, C} Felix Naumann | Data Profiling | CUSO 2014 32 A B C a 1 x b 2 x c 2 y
    • Null values ■ Null values have a wide range of interpretations. □ Unknown (date of birth) □ Non-applicable (driver license number for kids) □ Undefined (result of integration/outer join) ■ What are minimal uniques for the following data set? ■ Primary key {A}; Some unusual uniques: {C} and {CD} ■ Distinct: {A, BC} but not {CD} Felix Naumann | Data Profiling | CUSO 2014 33 A B C D a 1 x 1 b 2 y 2 c 3 z 5 d 3 ⊥ 5 e ⊥ ⊥ 5
    • Pruning effect of a pair Felix Naumann | Data Profiling | CUSO 2014 34 A B C D E AB AC AD AE BC BD BE CD CE DE ABC ABDABE ACD ACEADE BCD BCE BDE CDE ABCDABCE ABDE ACDE BCDE ABCDE minimal unique unique
    • Pruning with uniques ■ Pruning: inferring the type of a combination without actual verification ■ If A is unique, supersets must be unique ■ Finding a unique column prunes half of the lattice □ Remove column from initial data set and restart ■ Finding a unique column pair removes a quarter of the lattice □ In general, the lattice over the combination is removed ■ The pruning power of a combination is reduced by prior findings □ AB prunes a quarter □ BC additionally prunes only one eighth □ ABC already pruned one eights Felix Naumann | Data Profiling | CUSO 2014 35
    • Pruning both ways Felix Naumann | Data Profiling | CUSO 2014 36 A B C D E AB AC AD AE BC BD BE CD CE DE ABC ABDABE ACD ACEADE BCD BCE BDE CDE ABCDABCE ABDE ACDE BCDE ABCDE minimal unique unique maximal non-unique non-unique
    • TPCH – Uniques and Non-Uniques Felix Naumann | Data Profiling | CUSO 2014 37 non-uniqueunique 8 columns 9 columns 10 columns
    • Unique Column Combination Discovery ■ DUCC □ Basic idea: random walk through lattice □ Pick random superset if current combination is non-unique □ Pick random subset otherwise □ Lazy prune with previously visited nodes Felix Naumann | Data Profiling | CUSO 2014 38 Row-basedColumn-based Hybrid Gordian [SBHR06] Apriori [GW99] HCA [AN11] DUCC [HQA+14] SWAN [AQN14]
    • A B C D E AB AC AD AE BC BD BE CD CE DE ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE ABCD ABCE ABDE ACDE BCDE ABCDE ABCD ABC ABCE ABD ABDE AB ACD CD ACD BCD CDE Minimum unique column combination candidate Minimum unique column combination Maximum non-unique column combination candidate Maximum non-unique column combination Pruned Visited nodes: 10 out of 26 Felix Naumann | Data Profiling | CUSO 2014 39
    • Scaling the number of columns ■ NCVoter, 100k rows Felix Naumann | Data Profiling | CUSO 2014 40
    • Scaling the number of rows ■ NCVoter, 15 columns Felix Naumann | Data Profiling | CUSO 2014 41
    • Analysis of DUCC ■ Runtime mainly depends on size of solution set ■ Worst case: solution set in the middle Felix Naumann | Data Profiling | CUSO 2014 42
    • Uniques and non-uniques in NC-voter data ■ A minimal unique: voter_reg_num, zip_code, race_code ■ A maximal non-unique: voter_reg_num, status_cd, voter_status_desc, reason_cd, voter_status_reason_desc, absent_ind, name_prefx_cd, name_sufx_cd, half_code, street_dir, street_type_cd, street_sufx_cd, unit_designator, unit_num, state_cd, mail_addr2, mail_addr3, mail_addr4, mail_state, area_cd, phone_num, full_phone_number, drivers_lic, race_code, race_desc, ethnic_code, ethnic_desc, party_cd, party_desc, sex_code, sex, birth_place, precinct_abbrv, precinct_desc, municipality_abbrv, municipality_desc, ward_abbrv, ward_desc, cong_dist_abbrv, cong_dist_desc, super_court_abbrv, super_court_desc, judic_dist_abbrv, judic_dist_desc, nc_senate_abbrv, nc_senate_desc, nc_house_abbrv, nc_house_desc, county_commiss_abbrv, county_commiss_desc, township_abbrv, township_desc, school_dist_abbrv, school_dist_desc, fire_dist_abbrv, fire_dist_desc, water_dist_abbrv, water_dist_desc, sewer_dist_abbrv, sewer_dist_desc, sanit_dist_abbrv, sanit_dist_desc, rescue_dist_abbrv, rescue_dist_desc, munic_dist_abbrv, munic_dist_desc, dist_1_abbrv, dist_1_desc, dist_2_abbrv, dist_2_desc, confidential_ind, age, vtd_abbrv, vtd_desc Felix Naumann | Data Profiling | CUSO 2014 43
    • Dynamic Data: Challenges ■ Inserts may create new duplicate combinations □ Minimal uniques (mUCs) might become non-unique □ Maximal non-uniques (mNUCs) might lose maximality ■ Deletes remove duplicate value combinations □ NUCs might get unique □ mUCs might lose minimality ■ Idea □ Leverage the knowledge of previously discovered mUCs and mNUCs □ Create appropriate indices Felix Naumann | Data Profiling | CUSO 2014 44
    • SWAN Architecture [AQN14] Felix Naumann | Data Profiling | CUSO 2014 45 SW AN Database (input dataset) Repository (MUCS and MNUCS) Inserts Handler Uniqueness Checker Deletes Handler Duplicate Checker deletesinserts MUCS-indexdata-index duplicate-index inserts/deletes inserts/deletes update
    • Scaling the Number of Columns ■ 100k rows and 10k inserts Felix Naumann | Data Profiling | CUSO 2014 46 0.2$ 0.9$ 1$ 10$ 100$ 1000$ 10000$ 100000$ 10$ 20$ 30$ 40$ 50$ 60$ Executiontime(s) Number of columns Ducc Gordian-Inc Swan
    • ■ TPCH with 16 columns and 5 million rows ■ Swan/Ducc combination is able to process larger datasets than Ducc on a static dataset Stressing the Number of Inserts Felix Naumann | Data Profiling | CUSO 2014 47 0" 2000" 4000" 6000" 8000" 10000" 12000" 10%" 20%" 30%" 40%" 50%" 60%" 70%" 80%" 90%" 100%" Executiontime(s) Insert size wrt. initial dataset size Ducc Swan
    • Next steps ■ Finding primary keys □ Uniqueness is necessary criteria □ No null values □ Include other features ◊ Name includes “id”, number of columns ■ Partial uniques □ 99.9% of the data unique □ Useful to detect data errors □ Gordian, HCA, and DUCC can be easily modified ■ Incremental discovery Felix Naumann | Data Profiling | CUSO 2014 48
    • Inclusion Dependencies: Definition ■ INDs involve more than one relation. ■ Let D be a relational schema and let I be an instance of D. ■ R[A1, …, An] denotes projection of I on attributes A1, … An, of relation R: R[A1, …, An] = πA1, …, An(R) ■ IND  = R[A1, …, An]  S[B1, …, Bn], where R, S are (possibly identical) relations of D. □ Projection on R and S must have same number of attributes. ■ An instance I of D satisfies  if I(R)[A1, …, An]  I(S)[B1, …, Bn] ■ Values of R: “dependent values” ■ Values of S: “referenced values” Felix Naumann | Data Profiling | CUSO 2014 49
    • IND types ■ Unary INDs □ INDs on single attributes: R[A]  S[B] ■ n-ary INDs □ INDs on multiple attributes: R[X]  S[Y] ■ Partial INDs □ IND R[A]  S[B] is satisfied for x% of all tuples in R □ IND R[A]  S[B] is satisfied for all but x tuples in R ■ Approximate INDs □ IND R[A]  S[B] is satisfied with probability p. □ Based on sampling or other heuristics Felix Naumann | Data Profiling | CUSO 2014 50
    • Motivation for IND discovery ■ General insight into data ■ Detect unknown foreign keys ■ Example □ PDB: Protein Data Bank □ OpenMMS provides relational schema ◊ Parses protein and nucleic acid macromolecular structure data from the standard mmCIF format. □ 175 tables with primary key constraints □ 2705 attributes □ But: Not a single foreign key constraint! Felix Naumann | Data Profiling | CUSO 2014 51
    • Motivation for IND discovery ■ Ensembl – genome database □ shipped as MySQL dump files □ more than 200 tables □ Not a single foreign key constraint! ■ Why are FKs missing? □ Lack of support for checking foreign key constraints in the host system ◊ Example: Oracle did not support FKs up to v6 □ Fear that checking such constraints would impede database performance □ Lack of database knowledge within the development team Felix Naumann | Data Profiling | CUSO 2014 52
    • Felix Naumann | Data Profiling | CUSO 2014 53 SPIDER: Single Pass Inclusion DEpendency Recognition [BLNT07] ■ Main ideas □ Test all IND-candidate pairs in parallel. □ Read attribute values only once. □ Stop test of an IND-candidate after first counter-example. □ Reduce number of value comparisons by specialized data structure. □ No need to build inverted index. ■ Two steps: □ Sort and distinct all attribute‘s values and write them to disk ◊ For each attribute: SELECT DISTINCT A FROM R ORDER BY A □ Test all IND candidate pairs in parallel
    • SPIDER by example ■ In each step: Intersect „attributes to process“ with each refs list of previous step Felix Naumann | Data Profiling | CUSO 2014 54 attributes A, B, C A B C s s t t t x y y y z attributes to process dep A refs dep B refs dep C refs Init B,C A,C A,B Step 1 A,C C A,C A Step 2 A,B,C C A,C A Step 3 A  A,C A Step 4 A,B,C  A,C A Step 5 C  A,C 
    • Problem: Automatic Determination of Foreign Keys ■ Given □ Relational schema □ Database instance of that schema □ Complete set of (observed) inclusion dependencies ◊ Attributes A and B with R[A]  S[B] (in short A  B) ■ Find □ All foreign key constraints: attributes A and B with A  B ■ Difficulty □ Foreign keys are not intrinsic to data, but defined by humans □ Discover semantics ■ Machine learning approach based on syntactic features [RAB+09] Felix Naumann | Data Profiling | CUSO 2014 55
    • Features ■ DependentAndReferenced □ Counts how often the dependent attribute A appears as referenced attribute in the set of all INDs. □ Usually, a foreign key is not also a primary key that is referenced as foreign key by other tables. ■ MultiDependent □ Counts how often A appears as dependent attribute in the set of all INDs. □ If s(A) is contained in the set of values of many other attributes, the likelihood for each of these INDs being a FK is decreased. ■ MultiReferenced □ Counts how often B appears as referenced attribute in the set of all INDs. □ Often, primary keys are referenced by more than one foreign key. Felix Naumann | Data Profiling | CUSO 2014 56 A a B a b ? C a D a A a B a b ? C a D a A a B a b ? C a D a
    • Features ■ DistinctDependentValues □ The cardinality of s(A). □ Usually, attributes that are foreign keys contain at least some different values. ■ ValueLengthDiff □ Difference between the average value length (as string) in s(A) and s(B). □ Usually, average length of the values is similar whenever foreign keys reference a non-biased sample of the primary keys. Felix Naumann | Data Profiling | CUSO 2014 57 A a a a a a B a b c d e ? A abab abab abab c d B abab b c d e ?
    • Features ■ Coverage □ The ratio of values in s(B) that are covered by s(A) compared to all values in s(B). □ Usually, foreign keys cover a considerable number of primary key values. ◊ 60% of FK-attribute values cover all ref-values ◊ Each covers at least 10% ■ OutOfRange □ Percentage of values in s(B) that are not within [ min(s(A)), max(s(A)) ]. □ Usually, the dependent values should be evenly distributed over the referenced values. □ Mostly, less than 5% of values outside of range ■ TableSizeRatio □ Ratio of number of tuples in A and number of tuples in B. □ Usually in life sciences databases, table sizes do not differ wildly Felix Naumann | Data Profiling | CUSO 2014 58 A b c b c B a b c d e f g ?
    • Features ■ ColumnName □ Similarity between name(A) and name(B), also considering the name of the table of which B is an attribute. ■ TypicalNameSuffix □ Checks whether name(A) ends with a substring that indicates a foreign key. □ „id“, „key“, and „nr“ Felix Naumann | Data Profiling | CUSO 2014 59 FILMTEXTE.FILMTEXTTYPNR  FILMTEXTTYPEN.FILMTEXTTYPNR CUSTOMER.C_NATIONKEY  NATION.N_NATIONKEY SG_SEQFEATURE.ENT_OID  SG_COMMENT.ENT_OID COURSE.STUDENT  STUDENT.ID SG_BIOENTRY.TAX_OID  SG_TAXON.OID
    • Agenda 60 ■ Basic statistics ■ Functional dependencies ■ Keys and foreign keys ■ Data profiling tools ■ Advanced profiling Felix Naumann | Data Profiling | CUSO 2014
    • Tools have very long feature lists Felix Naumann | Data Profiling | CUSO 2014 61 ■ Num rows ■ Min value length ■ Median value length ■ Max value length ■ Avg value length ■ Precision of numeric values ■ Scale of numeric values ■ Quartiles ■ Basic data types ■ Num distinct values ("cardinality") ■ Percentage null values ■ Data class and data type ■ Uniqueness and constancy ■ Single-column frequency histogram ■ Multi-column frequency histogram ■ Pattern discovery (Aa9) ■ Soundex frequencies ■ Benford Law Frequency ■ Single column primary key discovery ■ Multi-column primary key discovery ■ Single column IND discovery ■ Inclusion percentage ■ Single-column FK discovery ■ Multi-column IND discovery ■ Multi-column FK discovery ■ Value overlap (cross domain analysis) ■ Single-column FD discovery ■ Multi-column FD discovery ■ Text profiling
    • Oracle Data Profiling and Quality Control Center Felix Naumann | Data Profiling | CUSO 2014 62
    • Screenshots from IBM Information Analyzer Felix Naumann | Data Profiling | CUSO 2014 63
    • Typical Shortcomings of Tools (and methods from research) ■ Usability □ Complex to configure □ Results complex to view and interpret ■ Scalability □ Main-memory based □ SQL based ■ Efficiency □ Coffee, Lunch, Overnight ■ Functionality □ Restricted to simplest tasks □ Restricted to individual columns or small column sets ◊ “Realistic” key candidates vs. further use-cases □ „Checking“ vs. „discovery“ ■ Interpretation of profiling results Felix Naumann | Data Profiling | CUSO 2014 64 That‘s the big one
    • Metanome – Profiling your Datanome Felix Naumann | Data Profiling | CUSO 2014 65  Algorithm execution  Result management  Algorithm configuration  Result presentation Configuration Measurements SPIDER jar DUCC jar SWAN jar txt xml csv DB2 DB2 MySQL Results
    • Agenda 66 ■ Basic statistics ■ Functional dependencies ■ Keys and foreign keys ■ Data profiling tools ■ Advanced profiling Felix Naumann | Data Profiling | CUSO 2014
    • Online Profiling ■ Profiling is long procedure □ Boring for developers □ Expensive for machines (I/O and CPU) ■ Challenge: Display intermediate results □ … of improving/converging accuracy □ Allows early abort of profiling run ■ Gear algorithms toward that goal □ Allow intermediate output □ Enable early output: “progressive” profiling Felix Naumann | Data Profiling | CUSO 2014 67
    • Incremental Profiling ■ Data is dynamic □ Insert (batch or tuple-based) □ Updates □ Deletes ■ Problem: Keep profiling results up-to-date… □ … without re-profiling the entire data set. □ Easy examples: SUM, MIN, MAX, COUNT, AVG □ Difficult examples: MEDIAN, uniqueness (see earlier slides), dependencies Felix Naumann | Data Profiling | CUSO 2014 68
    • Piggyback Profiling ■ Goal: Determine metadata for query results ■ Challenge: With as little query processing overhead as possible □ Baseline: Run second SQL query □ Piggybacking: profile along query plan (using base statistics) Felix Naumann | Data Profiling | CUSO 2014 69
    • Profiling for Integration ■ Profile multiple sources simultaneously ■ Schema matching/mapping □ What constitutes the “difficulty” of matching/mapping? ■ Duplicate detection □ Estimate data overlap □ Estimate fusion effort ■ Create measures to estimate integration (and cleansing) effort □ Schema and data overlap □ Severity of heterogeneity Felix Naumann | Data Profiling | CUSO 2014 70
    • Profiling new Types of Data ■ Traditional data profiling: Single table or multiple tables ■ More and more data in other models □ XML / nested relational / JSON □ RDF triples □ Textual data: Blogs, Tweets, News □ Multimedia data ■ Different models offer new dimensions to profile □ XML: Nestedness, measures at different nesting levels □ RDF: Graph structure, in- and outdegrees □ Multimedia: Color, video-length, volume, etc. □ Text: Sentiment, sentence structure, complexity, and other linguistic measures Felix Naumann | Data Profiling | CUSO 2014 71
    • Average Sentence Length Felix Naumann | Data Profiling | CUSO 2014 75 „Literature Fingerprinting: A New Method for Visual Literary Analysis” by Daniel A. Keim and Daniela Oelke
    • Hapax Legomena Felix Naumann | Data Profiling | CUSO 2014 76 „Literature Fingerprinting: A New Method for Visual Literary Analysis” by Daniel A. Keim and Daniela Oelke
    • News Statistics Felix Naumann | Data Profiling | CUSO 2014 77 Master thesis Matthias Kohnen
    • Summary 78 ■ Basic statistics ■ Functional dependencies ■ Keys and foreign keys ■ Data profiling tools ■ Advanced profiling Felix Naumann | Data Profiling | CUSO 2014
    • Summary Felix Naumann | Data Profiling | CUSO 2014 79 Data Profiling Single source Single column Cardinalities Uniqueness and keys Patterns and data types Distributions Multiple columns Uniqueness and keys Inclusion and foreign key dep. Functional dependencies Conditional and approximate dep. Multiple sources Topical overlap Topic discovery Topical clustering Schematic overlap Schema matching Cross-schema dependencies Data overlap Duplicate detection Record linkage