Much of the focus on big data has been on the problem of processing very large sources. There is an equally hard problem of how to normalize, integrate, and transform the data from many sources into the format required to run large-scale anal- ysis and visualization tools. We have previously developed an approach to semi-automatically mapping diverse sources into a shared domain ontology so that they can be quickly com- bined. In this paper we describe our approach to building and executing integration and restructuring plans to support analysis and visualization tools on very large and diverse datasets.

1. Semantics for
Big Data Integration and Analysis
Craig Knoblock and Pedro Szekely
University of Southern California

2. Dimensions of Big Data

3. Our Focus:
Karma
Variety
source types
formats
representations
Big Karma
Volume
cloud execution
Velocity
streaming sources
Exploit Semantics of Data to Support
Big Data Integration

4. Karma
Hierarchical
Sources
Services
Model
Karma
Tabular
Sources
Database
CSV
KML
…
Interactive tool for rapidly extracting, cleaning,
transforming, and publishing data
4Craig Knoblock

5. The Secret Sauce

6. Karma Models
Model
Karma
Meaning of the data in each column
Relationships among data columns
+=
6Craig Knoblock

7. Semantic Types:
Meaning of Data in Columns
Semantic vs Syntactic Types
String String String String StringDate
Not useful for information integration

8. Semantic Types:
Meaning of Data in Columns
Semanticvs Syntactic Types
Name
of
Person
Nationality
of
Entity
Location
of
Event
Type
of
Event
Description
of
Event
Date
of
Event

9. Person Entity Event Event Event Event
Semantic Types
Defined Using an Ontology
Ontology
Name Nationality Location Type DescriptionDate
PropertiesClasses

10. Person Entity Event Event Event Event
Karma Learns the
Semantic Types
Name Nationality Location Type DescriptionDate
1. User specifies them once
2. Karma learns features to recognize them
3. Next time Karma sees similar data it automatically proposes semantic types

11. Organization
Person
Event
Relationships Specified in Terms
of Classes and Properties
Name Nationality
Location Type DescriptionDate
member
perpetrator

12. Organization
Person
Event
Karma Automatically Infers
Relationships
Name Nationality
Location Type DescriptionDate
member
perpetrator
1. Karma
automatically finds
relationships using
the ontology
2. When proposed
relationships are
incorrect, the user
adjusts them

13. Alignment
1. load
sample
2. demonstrate
transformations
3. model

14. Alignment
1. load
sample
2. demonstrate
transformations
3. model

15. Alignment
1. load
sample
2. demonstrate
transformations
3. model

16. Alignment
1. load
sample
2. demonstrate
transformations
3. model

17. Batch Execution
1. load data 2. apply model 3. export

18. Batch Execution
dataset A model A RDF A+
dataset B model B RDF B+
dataset C model C RDF C+
Triple Store
(RDF)
select name, location
of people associated with FARC

19. BigKarma

20. select name, location
of people associated with FARC
{ organization: FARC
last-name: Cano
location: Putumayo
}
JSON
FARC Reyez Vaupes
FARC Cano Bogota
Relational
Org Name Location
The Big Idea

21. JSON XML
Relational CSV
XML Excel
XML Excel( ) > Select > Project
Very large datasets, streaming
The Big Idea

22. Karma Alignment
1. load
sample
2. demonstrate
transformations
3. model

23. 1. load
sample
2. demonstrate
transformations
3. model
4. combine
models
5. join, union,
select, project
6. export
workflow
BigKarma Alignment

24. Person
Org…
Event Person Organization
Model Combination
+ +
Person
Event
Organization
=

25. BigKarma Cloud Execution
1. load
multiple
datasets
2. apply
combined
model
3. export
transformed
results
CHALLENGE 1
Materialize and transform only the data needed for
the results
CHALLENGE 2
Use a Column Store rather than RDF to integrate
the data

26. BigKarma Streaming
1. stream
multiple
datasets
2. apply
combined
model
3. export
transformed
results
CHALLENGE 1
Modeling streaming sources
CHALLENGE 2
Buffering to support joins

27. BigKarma Analysis

28. Pedro Szekely and Craig KnoblockUniversity of Southern California
http://isi.edu/integration/karma