This document discusses the work of Antiplagiat Research, which tackles challenging natural language processing and plagiarism detection problems. It outlines their focus on cross-language plagiarism detection, machine-generated text detection, and intrinsic plagiarism detection. It also describes Antiplagiat Research's collaboration opportunities and their participation in evaluating plagiarism detection algorithms through workshops like Dialogue Evaluation.

1. Antiplagiat Research
Rita Kuznetsova, Oleg Bakhteev, Alexey Romanov
12.11.2016 AINL FRUCT’16 1 / 29

2. Outline
Intro
Cross-Language Plagiarism Detection
Machine-Generated Text Detection
Intrinsic Plagiarism Detection
Collaboration
12.11.2016 AINL FRUCT’16 2 / 29

3. What’s Anti-Plagiat JSC
Anti-Plagiat System
• Detects text reuse in any language and for any popular
file type
• Discovers cheating
Few numbers
• Over 500 universities
• 140 M sources in search databases
• 25 M text checked per year
12.11.2016 AINL FRUCT’16 3 / 29

4. What’s Antiplagiat Research?
Antiplagiat Research tackles the most challenging problems
in the area of natural language processing and plagiarism
detection.
• Development of advancing technology
• Propagation of scientific thought
• Unity of young talents from leading institutions
— Moscow Phystech (MIPT)
— Computing Centre of RAS
— Moscow State University
12.11.2016 AINL FRUCT’16 4 / 29

5. History of the Project
• Oct ’14: launch of the project by Antiplagiat JSC
• Aug ’15: first conference participation
• Nov ’15: comprehensive study on machine-generated text
detection in real-world data
• Apr ’16: PAN 2016 participation (Top-1 in 2 tracks of
Author Diarization task)
• Jul ’16: development of cross-language plagiarism
detection tool powered by state-of-the-art techniques
. . . and great growth opportunities
12.11.2016 AINL FRUCT’16 5 / 29

6. Areas of Interest
• Cross-Language Plagiarism
• Paraphrase Detection
• Machine-Generated Text Detection
• Automatic Text Categorization
• Intelligent Search and Topic Search
• Author Diarization
• Smart Evaluation of Research Papers
12.11.2016 AINL FRUCT’16 6 / 29

7. Problems in Focus
12.11.2016 AINL FRUCT’16 7 / 29

8. Types of Text Reuse
Text reuse can be classified into several categories:
• copying text ”as is”
• text reuse with paraphrasing
— Mr.Dursley always sat with his back to the window in his
office on the ninth floor.
— Mr.Dursley always propped his back on the glass window on
the ninth floor of the office.
• cross-language plagiarism
— A cat was sitting on the table.
— На столе сидела кошка.
12.11.2016 AINL FRUCT’16 8 / 29

9. Cross-Language Plagiarism Problem
The problem has ancient origins and still remains topical...
12.11.2016 AINL FRUCT’16 9 / 29

10. Cross-Language Plagiarism Problem
The problem has ancient origins and still remains topical...
12.11.2016 AINL FRUCT’16 10 / 29

11. Cross-Language Plagiarism Problem
Problem
• A large proportion of texts contain reused fragments from
another language.
• The problem of cross-lingual textual similarity in the case
of Russian being one of the languages in a pair is poorly
known.
• The majority of methods that involve machine translation
stage, generates texts that differ too much from the
sources of plagiarism.
Our goal
Develop a method for cross-lingual (Russian and English) text
reuse detection that based on the monolingual approach.
12.11.2016 AINL FRUCT’16 11 / 29

12. Cross-Language Plagiarism Detection Tool
• Explicit Semantic Analysis for Cross-Language Retrieval in Case of
Russian-English Translation — RuSSIR 2015
• A Monolingual Approach to Detection of Text Reuse in Russian-English
Collection — AINL-ISMW FRUCT 2015
• Candidate Document Retrieval for Cross-Lingual Plagiarism Detection — IDP
2016
12.11.2016 AINL FRUCT’16 12 / 29

13. Cross-Language Plagiarism Detection - main stages
• Given: English document collection and suspicious
Russian document
• The first stage:
— Find candidate documents, which possibly contain reused text
from the suspicious document, in the collection.
— Rank these documents according to their relevance values.
• The second stage:
— Split the suspicious document and candidate documents into
segments.
— Compare with each other.
12.11.2016 AINL FRUCT’16 13 / 29

14. Machine-Generated Text Detection Problem
• Problem is not new, tools for paper generation have been
available for 10 years already
• Past research on generated papers discovered a hundred
of them in IEEE, Elsevier, Springer journals (2009 and
later)
Task
Distinguish machine-generated papers from authentic
documents automatically.
Key assumption
Most of papers are generated with one of several popular tools.
12.11.2016 AINL FRUCT’16 14 / 29

15. Machine-Generated Text Detection Problem
Today you can write a paper on a given topic with one click!
SCIgen - An Automatic CS Paper Generator
12.11.2016 AINL FRUCT’16 15 / 29

16. Machine-Generated Text Detection Problem
Today you can write a paper on a given topic with one click!
Mathgen: Randomly generated math papers
12.11.2016 AINL FRUCT’16 16 / 29

17. Machine-Generated Text Detection in Real-World
Data
Automatic detection of gibberish papers should:
• deal with big data (millions of papers in real-world
collections),
• be applicable for the Russian language,
• capture texts prepared with various generation tools,
• also detect machine-translated text chunks containing
grammatical errors.
Our findings:
• Исследование коллекции eLIBRARY.RU на наличие искусственных и
ненаучных текстов — SCIENCE ONLINE 2016
12.11.2016 AINL FRUCT’16 17 / 29

18. eLIBRARY.RU
• Search a collection of scientific papers of eLIBRARY.RU
for machine-generated and non-scientific papers
• Classification task
— Machine-generated vs. human-written texts
— Scientific papers vs. fiction texts
• Text features:syntactic and lexical
• Results
— We did’t find any machine-generated texts like «Korchevatel»
in the collection of eLIBRARY.RU
— We found: anniversary congratulations, business news,
interviews, bibliographies, memorials, etc.
12.11.2016 AINL FRUCT’16 18 / 29

19. “Fly, pie, to the oven”. Non-scientific paper in a
scientific journal on baking bread
12.11.2016 AINL FRUCT’16 19 / 29

20. Machine-Translated Text Detection
• Recent advances in the field of statistical machine
translation (SMT) lead to high availability of SMT
systems on the Web.
• Student reports, term works and theses lack proper
analysis by their tutors.
• It is very tempting to find relevant information in English,
automatically translate it into Russian, and paste it into
the paper “as is”!
• Machine-translated texts often contain grammatical errors
or inappropriate words:
— First individuals in the system take the maximum number of
contacts for any parameter combination.
— Первые лица в системе взять максимальное количество
контактов для любой комбинации параметров.
12.11.2016 AINL FRUCT’16 20 / 29

21. Solution design for MT detection
• Let’s estimate the likelihood that a sentence is
machine-translated, according to several language models
(LMs). . .
— Lexical 2,3-gram LMs trained on authentic texts
— Lexical 2,3-gram LMs trained on machine-translated texts
— POS tag 2,3-gram LMs trained on authentic texts
— POS tag 2,3-gram LMs trained on machine-translated texts
— word2vec (skip-gram and CBOW) models trained on
authentic texts
• . . . and use these estimates as features for classification
task. 2 * 4 + 2 = 10 features in total
• The classifier is trained on a mixed labeled sample of
authentic and machine-translated sentences.
Our findings:
• Machine-Translated Text Detection in a Collection of Russian Scientific
Papers — Dialogue 2016
12.11.2016 AINL FRUCT’16 21 / 29

22. Intrinsic Plagiarism Detection Problem
IPD Task
Detecting the plagiarized parts of given document by analyzing
the writing style.
Main Challenges
• No external collection
• No further possibilities to uncover plagiarism besides
detecting suspicious text parts which significantly differ
from the rest of the document
• Even if suspicious text parts are found, there is still no
guarantee that these parts are truly plagiarized
12.11.2016 AINL FRUCT’16 22 / 29

23. PAN @ CLEF 2016
PAN: Uncovering Plagiarism, Authorship and Social Software
Misuse
• Held since 2007
• Offers:
— Large-scale corpora for EPD and IPD algorithms
— Performance measure scheme
12.11.2016 AINL FRUCT’16 23 / 29

24. PAN Tasks
1. Intrinsic plagiarism detection.
1.1 There exists one main author who wrote at least 70% of the
text.
1.2 Up to the other 30% may be written by other authors.
2. Diarization with a given number (n) of authors.
2.1 There are (n) of authors, no main author
2.2 Each author may have contributed to an arbitrary extent.
3. Diarization with an unknown number of authors.
3.1 No information about how many authors contributed to the
document.
12.11.2016 AINL FRUCT’16 24 / 29

25. Solving the Problem
Common scheme involves several stages:
• text segmentation (sentences, blocks, paragraphs etc.),
• map each segment to the feature space,
• outlier detection (or clustering for author diarization).
• Methods for Intrinsic Plagiarism Detection and Author Diarization—Notebook
for PAN at CLEF 2016. In CLEF 2016 Evaluation Labs and Workshop –
Working Notes Papers, 5-8 September, ´Evora, Portugal, September 2016.
CEUR-WS.org. ISSN 1613-0073.
12.11.2016 AINL FRUCT’16 25 / 29

26. Collaboration Opportunities
12.11.2016 AINL FRUCT’16 26 / 29

27. Research Collaboration
Opportunities for research collaboration include:
• Joint non-profit studies
• Custom research
• Consulting and mentorship
• Joint laboratories (joint & grant financing)
• Internship opportunities
• Thesis research
12.11.2016 AINL FRUCT’16 27 / 29

28. Dialogue Evaluation’17 - Plagiarism Detection
The PlagEvalRus workshop
Focused on evaluation of Russian-specific plagiarism detection
algorithms. The workshops emphasize on external plagiarism
detection in scientific texts (academic plagiarism).
With support of:
• PAN
• Dialogue conference
• CyberLeninka
www.dialog-21.ru/evaluation/2017/plageval/
12.11.2016 AINL FRUCT’16 28 / 29

29. Thanks for you attention!
Questions / Comments?
12.11.2016 AINL FRUCT’16 29 / 29