Data fingerprinting

Copyright © 2018 Talentica Software (I) Pvt Ltd. All rights reserved.
Fingerprinting Latent
Structure in Data
MRITYUNJAY KUMAR & GUNTUR RAVINDRA
TECHNOLOGY EXCELLENCE GROUP
TALENTICA SOFTWARE
PRESENTED AT DAIR (DATA ANALYTICS AND INTELLIGENCE RESEARCH ,INDIAN INSTITUTE OF TECHNOLOGY, DELHI)

Agenda
 Challenge with building data-driven algorithms
 Small-data
 Introduction to data fingerprinting
 Two problem statements
 Solving a Question complexity problem
 Solving an Image recognition problem
 Fingerprinting the structure in data
 Extracting structure
 Representing structure as a signature
 Other complex problems

What is data fingerprinting
 A method to represent a block of data as an entity
 Applications: Easy validation, proof of originality, tamper detection, DLP
 Classical techniques
 Bloom filters, cryptographic hashes
 Main issues with fingerprinting
 Do not capture data semantics
 Large number of fingerprints  complexity

Two Problems

Recognizing question complexity

Recognizing structural deformation in
cells
Data source: https://www.kaggle.com/c/data-science-bowl-2018/data

Data-driven algorithms with Small-
Data
 Need for problem-specific data
 Rule-based approaches
 Rule-based approaches are easy to implement
 Not all data characteristics can be captured as rules
 Does not automatically adapt to the data
 Machine learning approach
 ML approaches need large amounts of data
 Generic models and open-source data are not suitable for application-specific
needs
 Can build complex structures and designs

Architecting a solution
• Knowledge has a latent structure
• Sequence, Geometry
• There can be a hierarchies of structures
• convert structure to a computational representation
• Objective: context of application
capabilities Influences computational
representation

Problem Formulation
A set of elements : images, questions, Text messages
An objective
A subset of structures relevant to an objective
How do we define and how do we find
Transformation of elements into a structure and hence
a computational entity
A human in the loop

Structures in Data
How many buses are plying in Mumbai on a route originating at Dadar and ending at Vashi?
How many students are in the class?

Structures in Data
Intensity Projections
Oriented gradients

Problem Formulation
For computational ease we make
A function that maps a structure to vector
The inverse of the function results in one of
many structures
a binary bit-vector
Goal is to find so as to satisfy the constraints
This is a constrained optimization formulation

Solution : Optimization formulation
 Based on the problem formulation
 We have an optimization formulation that has an inverse that results in the
variable itself or a subset of variables
 A related function is a neural auto-encoder
 Solution boils down to
 Training an auto-encoder with one class of data
 Recognizing data class involves
 Data clustering
 Human intelligence/visual inspection to mark clusters
 Data in clusters used to train the auto-encoder

Recognition : Cell Structure

Recognition : Question Complexity
How much can the SP alter income tax in Scotland?
What is stage 1 in the life of a bill?
Who is the President of Egypt?
Why do some people purposely resist officers of the law?
Why is the need for acceptance of punishment needed?
Why would one plead guilty to a crime involving civil disobedience?
Why is giving a defiant speech sometimes more harmful for the individual?
Why did Harvard end its early admission program?

 The auto-encoder output has distortions
 Detect the distortion
 Quantify the distortion
Solution : Recognition

Building Complexity
 Incremental addition of data classes
 Using stacking
 Unique binary code injected in each
stacked layer
 Collapse stacked layers into a
classification model  redeploy

Data Type Test
Cases
True
Positive
False
Positive
True False
Negative
With classes
like in
training data
1781 1774 NA NA 7
With classes
not like in
training data
8789 NA 13 8776 NA

Summary
 A large number of applications are still small-data applications
 Data has latent structure
 Extraction is objective based and data specific
 We can harness data-hungry algorithms for small-data applications
 Use structures instead of raw data
 Auto-encoders are powerful tools
 Build incremental complexity

Data fingerprinting

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Similar to Data fingerprinting

Similar to Data fingerprinting (20)

Recently uploaded

Recently uploaded (20)

Data fingerprinting

Editor's Notes