This document discusses methods for distributed stream consistency checking against a conceptual model. It presents the problem of ensuring streaming data complies with an ontology model while dealing with noise and large volumes. Two methods - NTM and LN - are proposed and evaluated. The LN method models the negative inclusion axioms in the ontology as a pipeline of bolts, reducing the load on individual bolts compared to NTM and improving performance up to 300%. Future work is discussed around more expressive languages, inconsistency repair, and implementation on other stream processing engines.

1. Distributed Stream Consistency Checking
Shen Gao, Daniele Dell’Aglio, Jeff Z. Pan and Abraham Bernstein
Cáceres, Spain, 08.06.2018
Carlo Bernaschina (presenter)

2. Problem setting
ICWE, 08.06.2018Distributed Stream Consistency Checking2/25
 Real time processing of huge volumes of dynamic data
 Smart cities
 News
 Knowledge graph

3. The problem of noise
ICWE, 08.06.2018Distributed Stream Consistency Checking
 Streaming data are often noisy
 Broken sensors
 Malicious data injection
 Measurement errors
 How to cope with noise?
 Machine learning and numerical analyses to cope with noise in
time series
 When streams are complex (as Web streams), we want to
ensure that they are compliant to a (non-trivial)
conceptual model
3/25

4. Research question
How to assess the consistency of streams w.r.t. a
fixed and known a-priori conceptual model?
ICWE, 08.06.2018Distributed Stream Consistency Checking4/25

5. Towards a solution
ICWE, 08.06.2018Distributed Stream Consistency Checking
 How to model the stream consistency check problem?
5/25

6. How to model the conceptual model?
ICWE, 08.06.2018Distributed Stream Consistency Checking
 DL-Litecore
 The set of PIs and NIs composes a TBoxT
Person
Student Employee
Faculty Admin
Positive Inclusion (PI)
PhD student
Person
Organization
DJ
Negative Inclusion (NI)
6/25

7. How to model the data?
ICWE, 08.06.2018Distributed Stream Consistency Checking
 ABox axioms associate:
 Individuals to classes
 Shen is a
 University of Zurich is a
 Individuals to other individuals
 Shen attends the University of Zurich
 Inconsistencies arise when the ontology (TBox + ABox)
contains contraditions
 Daniele is a
 Daniele is a
 disjoint
PhD student
University
PhD student
University
PhD student University
7/25

8. How to model the data stream?
ICWE, 08.06.2018Distributed Stream Consistency Checking
 Ontology stream
 One staticTBox
 A sequence of time-annotated
ABoxes with the updates
 Sliding window over the
ontology stream
 Captures a recent set of events
A1
A3
A5
{ Shen is a }
3
5
1
t
PhD student
{ Jeff is a
Daniele is a }
Employee
Student
{ Avi is a }PhD student
TBoxPerson
Student Employee
Faculty AdminPhD student
Organiz.
Univers. High school
DJ
8/25

9. The stream consistency check problem
ICWE, 08.06.2018Distributed Stream Consistency Checking
 Given an ontology stream,
we want to check if it is
consistent w.r.t. a sliding
window of a fixed size
 At each time instant, we
want to check if the events
captured by the sliding
window are consistent
 TheTBox and the current
window content compose
an ontology
A1
A3
A5
{ Shen is a }
3
5
1
t
PhD student
{ Jeff is a
Daniele is a }
University
Student
{ Jeff is a }PhD student
TBoxPerson
Student Employee
Faculty AdminPhD student
Organiz.
Univers. High school
DJ
9/25

10. Towards a solution
ICWE, 08.06.2018Distributed Stream Consistency Checking
 How to model the stream consistency check problem?
 Description logics, ontology streams
 How to cope with a huge amount of streaming data?
10/25

11. Scalability
ICWE, 08.06.2018Distributed Stream Consistency Checking
How to cope with the problem when the data volume is
big?
 Sliding windows
 The content of the window may still be too large to be
processed online
 Distribution of the stream consistency checking process
 We build our solution on top of a Distributed Stream
Processing Engine (DSPE)
 We adopt the Storm terminology to introduce the main
concepts, but they are common to other DSPEs
11/25

12. DSPE concepts
ICWE, 08.06.2018Distributed Stream Consistency Checking
S B1 B2
B1 B2S
B1
B1 B2
Logical topology
Physical topology
BoltsSpout
Node 1
Node 2
Node 3
Tuples
12/25

13. Towards a solution
ICWE, 08.06.2018Distributed Stream Consistency Checking
 How to model the stream consistency check problem?
 Description logics, ontology streams
 How to cope with a huge amount of streaming data?
 Distributed stream processing engines
 How to perform stream consistency checking over DSPE?
13/25

14. The NI closure
ICWE, 08.06.2018Distributed Stream Consistency Checking
 Given theTBox T, it is possible to compute all the
possible Negative Inclusion axioms
 The set of all the possible NI axioms is named NI closure
Person
Student Employee
Faculty AdminPhD student
DJ Organization
University Company
14/25

15. B1
The NIs Topology Method (NTM)
ICWE, 08.06.2018Distributed Stream Consistency Checking
 The resulting topology is the following
 A bolt evaluates when the disjoint axioms in the NI
closure are satisfied
 Each axiom is encoded as a conjunction operation
S B1
15/25
Daniele is a Person
Inconsistency
Daniele is a University
o1
Shen is a Company
Inconsistency
Shen is a Student
o2

16. Improving NTM
ICWE, 08.06.2018Distributed Stream Consistency Checking
 Drawback of NTM
 The NI closure size can be exponential to the size of theTBox
 The bolt B1 becomes the bottleneck of the topology
 Introduction of inference operations to reduce the number
of conjunction operations
16/25
o
Daniele is a Student Daniele is a Person
S B1

17. Improving NTM - intuition
ICWE, 08.06.2018Distributed Stream Consistency Checking
Person
Student Employee
DJ Organization
University Company
9 NIs
S B1 B2
Student -> Person
Employee -> Person
Company -> Organization
University -> Organization
S B1
1 NI
17/25

18. The Pipeline Topology Method (LN)
ICWE, 08.06.2018Distributed Stream Consistency Checking
DJ(Person,Publication)
DJ(Student,Publication)
DJ(Student,Employee)
DJ(Article,Student)
DJ(Person,Organization)
...
Computes the
NI closure
DJ(Person,Publication)
DJ(Student,Employee)
DJ(Person,Organization)
...
Identifies the
essential NIs
Groups and
orders the
essential NIs
18/25

19. The Pipeline Topology Method (LN) cont’d
ICWE, 08.06.2018Distributed Stream Consistency Checking
Groups are
assigned to bolts
This step has a
major impact on
performance!
Less NIs w.r.t. NTM
19/25

20. Towards a solution
ICWE, 08.06.2018Distributed Stream Consistency Checking
 How to model the stream consistency check problem?
 Description logics, ontology streams
 How to cope with a huge amount of streaming data?
 Distributed stream processing engines
 How to perform stream consistency checking over DSPE?
 NTM, LN
 How to they perform?
20/25

21. Setup
ICWE, 08.06.2018Distributed Stream Consistency Checking
 Ontologies
 LUBM
 56 PIs, 70 NIs
 NPD
 332 PIs, 51 Nis
 Six machines
 128GB ram
 2 E5-2680 v2 processors (10 cores per processor)
 Twitter Heron 0.14.3
21/25

22. Comparing NTM and LN
ICWE, 08.06.2018Distributed Stream Consistency Checking
S B1 B2
LN-x:
x NI groups
Half of the
nodes assigned
to check
consistency
Similar results
LN-2 outperforms
NTM up to 139% The load on the first
node increases
22/25

23. Investigating the results
ICWE, 08.06.2018Distributed Stream Consistency Checking
LN LN LN
LN
LN
LN
LN
LN
LN LN
LN LN LN LN LN LN
LN
LN LN LN
NTM
23/25

24. Conclusions
ICWE, 08.06.2018Distributed Stream Consistency Checking
 It is possible to perform consistency checking over high
volumes of data streams
 We developed two methods (NTM and LN) and studied
their performance
 More than 14 million tuples/minute
 LN can outperform NTM up to 300%
 What’s next
 Towards more expressive ontological languages
 Repairing inconsistencies
 Implementation and testing over other DPSEs
24/25

25. Thank you! Questions?
Distributed Stream Consistency Checking
Shen Gao, Daniele Dell’Aglio, Jeff Z. Pan,Abraham Bernstein
ICWE, 08.06.2018Distributed Stream Consistency Checking25/25