J.C. Freytag & Fabian Fier, Humboldt University Berlin, presented at the 2016 HPCC Systems Engineering Summit Community Day. Finding similar objects in large datasets is an important database operation. The operation is used in applications like plagiarism detection, document clustering, or duplicate removal. With increasing dataset sizes, this problem cannot be solved canonically anymore. With straightforward approaches, the runtime becomes very large even for moderately large datasets and despite using distributed systems. We give an overview on finding similarity on text data and discuss scalable solution approaches. In our research, we experimentally compare algorithmic approaches in order to optimize the runtime. We show that is a complex problem due to many involved parameters such as data properties like skew, runtime parameters, or implementation details. We give insights to our practical findings when comparing implementations on Hadoop with implementations on HPCC Systems. This talk aims at practitioners as well as theoreticians who are interested in similarity search, text processing, and scalable algorithmic approaches that are inspired by MapReduce and are adaptable to HPCC Systems. J.C. Freytag Johann-Christoph Freytag is currently full professor for Databases and Information Systems (DBIS) at the Computer Science Department of the Humboldt-Universität zu Berlin, Germany. Before joining the department in 1994, he was a research staff member at the IBM Almaden Research Center (1985-1987), a researcher at the European Computer-Industry-Research Centre (ECRC, in Munich, Germany, 1987-1989), and the head of Digital's Database Technology Center (also in Munich, 1990-1993). He holds a Ph.D. in Applied Mathematics/Computer Science from Harvard University, MA. Prof. Freytag's research interests include all aspects of query processing and query optimization in object-relational database systems, new developments in the database area (such as semi-structured data, data quality), privacy in database systems, and applying database technology to applications such as GIS, genomics, and bioinformatics/life science. In the last years he received the IBM Faculty Award four times for collaborative work in the areas of databases, middleware, and bioinformatics/life science. Fabian Fier Fabian Fier is PhD student at the research group of Prof. Johann-Christoph Freytag. He holds a diploma in computer science from Humboldt-Universität zu Berlin. His research interest is similarity search on web-scale data. He uses techniques from textual similarity joins on Big Data and adapts them to similarity search.

1. •

3. Copying in suspect text (not reference text) with no reference Grayscale color
Significant copying: multiple identical sentences in a paragraph 6
some copying: ≥5 contiguous words within a sentence of ≥ 5 words 5
near copying: multiple (≥ 3) exact phrases (≥ 3 words) overlap in contiguous
sentences in a paragraph
4
similarity 1: several (<3) exact phrases (≥ 3 words) overlap in contiguous
sentences in a paragraph or more than 9 words in sequence
3
similarity 2: many (≥5) of the exact same words (excluding function words)
in contiguous sentences in the same paragraph
2
similarity 2: topic overlap with facts and standard phrases 1
Standard words or phrases 0
General knowledge / facts 0

6. •
•
• 𝑠𝑖𝑚 𝑟, 𝑠 ↦ [0; 1]
• 𝜃
• {𝑟𝜖𝑅|𝑠𝑖𝑚 𝑟, 𝑠 ≥ 𝜃}

7. 𝑠𝑖𝑚 𝑟, 𝑠 =
|𝑟∩𝑠|
|𝑟∪𝑠|
=
3
8

8. 𝑅 𝜃
𝑟 𝜖 𝑅
sim 𝑟, 𝑠 ≥ 𝜃:

9. •
•
• 𝑟, 𝑠 𝜖 𝑅 × 𝑅 𝑠𝑖𝑚 𝑟, 𝑠 ≥ 𝜃
•
•
•
•

10. •
•
•
•

11. •
{ 𝑡𝑒𝑟𝑚, {𝑑𝑜𝑐𝑢𝑚𝑒𝑛𝑡} }
•
𝑡𝑒𝑟𝑚, {𝑑𝑜𝑐𝑢𝑚𝑒𝑛𝑡}
(𝑟, 𝑠)
sim r, s ≥ 𝜃:

12. prefixJaccard 𝑟, 𝜃 = 𝑟 − 𝑓𝑙𝑜𝑜𝑟 𝜃 ∗ 𝑟 + 1
prefixJaccard 𝑟, 0.5
prefixJaccard 𝑠, 0.5

13. •

14. Runtime
Data
System
Parameters
Algorithm
Parameters
Implemen-
tation
Problem
Statement
Fairness?

15. •
•
•
•
•
•
•

16. •
•
•
•
•
•
•

20. •
ε = 1 − 𝜃