SlideShare a Scribd company logo

Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection

The rise of language models such as BERT allows for high-quality text paraphrasing. This is a problem to academic integrity, as it is difficult to differentiate between original and machine-generated content. We propose a benchmark consisting of paraphrased articles using recent language models relying on the Transformer architecture. Our contribution fosters future research of paraphrase detection systems as it offers a large collection of aligned original and paraphrased documents, a study regarding its structure, classification experiments with state-of-the-art systems, and we make our findings publicly available.

1 of 12
Download to read offline
Are Neural Language Models Good
Plagiarists? A Benchmark for
Neural Paraphrase Detection
1
September 2021, JCDL, Virtual Event
Authors: Jan Philip Wahle, Terry Ruas, Norman Meuschke, Bela
Gipp
Agenda/Outline
1. Motivation
2. Overview
3. Method
4. Benchmark Details
5. Experiments
6. Conclusion & Future Work
3
Motivation
Plagiarists increasingly use automated paraphrasing tools to trick
plagiarism detection software.
Problem:
• Text-matching software fails if ≈20% of words are replaced [1].
• Neural language models can mimic human writing.
• No benchmark exists to test neural language model plagiarism.
Solution:
• A benchmark to train/test detection of neural paraphrasing.
4
[1] J. P. Wahle, T. Ruas, T. Foltynek, N. Meuschke, and B. Gipp, “Identifying Machine-Paraphrased
Plagiarism,” arXiv:2103.11909 [cs], Jan. 2021.
Overview
5
Paragraphs from
arXiv, Wikipedia, theses
Transformer models
BERT, RoBERTa, Longformer
Paraphrased documents
Mask words
randomly
Predict
replacement
candidates
fastText+SVM,
BERT, RoBERTa, Longformer
Detection of
plagiarism
A mathematically rigorous approach to quantum field theory based on
operator algebras is called an algebraic quantum field theory...
Method - Example
6
<mask> mathematically rigorous approach to quantum field theory
based <mask> operator <mask> is called an algebraic <mask> field
theory...
The mathematically rigorous approach to quantum field theory based
upon operator equations is called an algebraic Quantum field theory...
Input
Models
Output
Sources arXiv, Wikipedia, student theses
Paraphrasing Models BERT, RoBERTa, Longformer
Masking probabilities [0.15, 0.2, 0.3, 0.4, 0.5]
Original Examples 163 735
Paraphrased Examples 2 455 725
Benchmark Details
7
Ad

Recommended

How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 

More Related Content

Recently uploaded

Quality safety and legislations of cosmetics.pptx
Quality safety and legislations of cosmetics.pptxQuality safety and legislations of cosmetics.pptx
Quality safety and legislations of cosmetics.pptxDeviSky1
 
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Nan Yang Academy of Sciences
 
PROSTHETIC FEET description and its types
PROSTHETIC FEET description and its typesPROSTHETIC FEET description and its types
PROSTHETIC FEET description and its typeseshasmalik27
 
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...Abhinav S
 
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdf
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdfCW 2 - Frustrated Lewis Pair - Molly winterbottom.pdf
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdfMollyWinterbottom
 
Grade 8, Quarter 3.pdf lesson plan third
Grade 8, Quarter 3.pdf lesson plan thirdGrade 8, Quarter 3.pdf lesson plan third
Grade 8, Quarter 3.pdf lesson plan thirdgmail227828
 
Hypertension in Children and Adolescents
Hypertension in Children and AdolescentsHypertension in Children and Adolescents
Hypertension in Children and AdolescentsTristanBabaylan1
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsPeter Coles
 
Thornyissue testing of slideshow for website
Thornyissue testing of slideshow for websiteThornyissue testing of slideshow for website
Thornyissue testing of slideshow for websitesuelcarter1
 
The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...Advanced-Concepts-Team
 
ROLES OF MICROBES IN BIOCONTROL BY ANKIT CHOUDHARY.ppsx
ROLES OF MICROBES IN BIOCONTROL BY ANKIT CHOUDHARY.ppsxROLES OF MICROBES IN BIOCONTROL BY ANKIT CHOUDHARY.ppsx
ROLES OF MICROBES IN BIOCONTROL BY ANKIT CHOUDHARY.ppsxAnkitChoudhary955647
 
Kavita Punekar: Illuminating Minds and Igniting Passion in Science Education
Kavita Punekar: Illuminating Minds and Igniting Passion in Science EducationKavita Punekar: Illuminating Minds and Igniting Passion in Science Education
Kavita Punekar: Illuminating Minds and Igniting Passion in Science Educationdsnow9802
 
Open Access Publishing and the Open Journal of Astrophysics
Open Access Publishing and the Open Journal of AstrophysicsOpen Access Publishing and the Open Journal of Astrophysics
Open Access Publishing and the Open Journal of AstrophysicsPeter Coles
 
Duchenne Muscular Dystrophy or DMD .pptx
Duchenne Muscular Dystrophy or DMD .pptxDuchenne Muscular Dystrophy or DMD .pptx
Duchenne Muscular Dystrophy or DMD .pptxNavanidhan.M
 
Analytical Coursework - Molly Winterbottom.pdf
Analytical Coursework - Molly Winterbottom.pdfAnalytical Coursework - Molly Winterbottom.pdf
Analytical Coursework - Molly Winterbottom.pdfMollyWinterbottom
 
From Leaf to Lab: Uncovering the Molecular Mysteries of Cannabis
From Leaf to Lab: Uncovering the Molecular Mysteries of CannabisFrom Leaf to Lab: Uncovering the Molecular Mysteries of Cannabis
From Leaf to Lab: Uncovering the Molecular Mysteries of CannabisMarkus Roggen
 
Construction of Magic Squares by Swapping Rows and Columns.pdf
Construction of Magic Squares by Swapping Rows and Columns.pdfConstruction of Magic Squares by Swapping Rows and Columns.pdf
Construction of Magic Squares by Swapping Rows and Columns.pdfLossian Barbosa Bacelar Miranda
 
Volatile Oils-Introduction for pharmacy students and graduates
Volatile Oils-Introduction for pharmacy students and graduatesVolatile Oils-Introduction for pharmacy students and graduates
Volatile Oils-Introduction for pharmacy students and graduatesAhmed Metwaly
 

Recently uploaded (20)

Quality safety and legislations of cosmetics.pptx
Quality safety and legislations of cosmetics.pptxQuality safety and legislations of cosmetics.pptx
Quality safety and legislations of cosmetics.pptx
 
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
 
PROSTHETIC FEET description and its types
PROSTHETIC FEET description and its typesPROSTHETIC FEET description and its types
PROSTHETIC FEET description and its types
 
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
 
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdf
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdfCW 2 - Frustrated Lewis Pair - Molly winterbottom.pdf
CW 2 - Frustrated Lewis Pair - Molly winterbottom.pdf
 
Grade 8, Quarter 3.pdf lesson plan third
Grade 8, Quarter 3.pdf lesson plan thirdGrade 8, Quarter 3.pdf lesson plan third
Grade 8, Quarter 3.pdf lesson plan third
 
Research methods in ethnobotany- Exploring Traditional Wisdom
Research methods in ethnobotany- Exploring Traditional WisdomResearch methods in ethnobotany- Exploring Traditional Wisdom
Research methods in ethnobotany- Exploring Traditional Wisdom
 
Hypertension in Children and Adolescents
Hypertension in Children and AdolescentsHypertension in Children and Adolescents
Hypertension in Children and Adolescents
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
 
Thornyissue testing of slideshow for website
Thornyissue testing of slideshow for websiteThornyissue testing of slideshow for website
Thornyissue testing of slideshow for website
 
The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...
 
ROLES OF MICROBES IN BIOCONTROL BY ANKIT CHOUDHARY.ppsx
ROLES OF MICROBES IN BIOCONTROL BY ANKIT CHOUDHARY.ppsxROLES OF MICROBES IN BIOCONTROL BY ANKIT CHOUDHARY.ppsx
ROLES OF MICROBES IN BIOCONTROL BY ANKIT CHOUDHARY.ppsx
 
ALL the evidence webinar: Appraising and using evidence about community conte...
ALL the evidence webinar: Appraising and using evidence about community conte...ALL the evidence webinar: Appraising and using evidence about community conte...
ALL the evidence webinar: Appraising and using evidence about community conte...
 
Kavita Punekar: Illuminating Minds and Igniting Passion in Science Education
Kavita Punekar: Illuminating Minds and Igniting Passion in Science EducationKavita Punekar: Illuminating Minds and Igniting Passion in Science Education
Kavita Punekar: Illuminating Minds and Igniting Passion in Science Education
 
Open Access Publishing and the Open Journal of Astrophysics
Open Access Publishing and the Open Journal of AstrophysicsOpen Access Publishing and the Open Journal of Astrophysics
Open Access Publishing and the Open Journal of Astrophysics
 
Duchenne Muscular Dystrophy or DMD .pptx
Duchenne Muscular Dystrophy or DMD .pptxDuchenne Muscular Dystrophy or DMD .pptx
Duchenne Muscular Dystrophy or DMD .pptx
 
Analytical Coursework - Molly Winterbottom.pdf
Analytical Coursework - Molly Winterbottom.pdfAnalytical Coursework - Molly Winterbottom.pdf
Analytical Coursework - Molly Winterbottom.pdf
 
From Leaf to Lab: Uncovering the Molecular Mysteries of Cannabis
From Leaf to Lab: Uncovering the Molecular Mysteries of CannabisFrom Leaf to Lab: Uncovering the Molecular Mysteries of Cannabis
From Leaf to Lab: Uncovering the Molecular Mysteries of Cannabis
 
Construction of Magic Squares by Swapping Rows and Columns.pdf
Construction of Magic Squares by Swapping Rows and Columns.pdfConstruction of Magic Squares by Swapping Rows and Columns.pdf
Construction of Magic Squares by Swapping Rows and Columns.pdf
 
Volatile Oils-Introduction for pharmacy students and graduates
Volatile Oils-Introduction for pharmacy students and graduatesVolatile Oils-Introduction for pharmacy students and graduates
Volatile Oils-Introduction for pharmacy students and graduates
 

Featured

Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at WorkGetSmarter
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...DevGAMM Conference
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationErica Santiago
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellSaba Software
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming LanguageSimplilearn
 
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...Palo Alto Software
 
9 Tips for a Work-free Vacation
9 Tips for a Work-free Vacation9 Tips for a Work-free Vacation
9 Tips for a Work-free VacationWeekdone.com
 
I Rock Therefore I Am. 20 Legendary Quotes from Prince
I Rock Therefore I Am. 20 Legendary Quotes from PrinceI Rock Therefore I Am. 20 Legendary Quotes from Prince
I Rock Therefore I Am. 20 Legendary Quotes from PrinceEmpowered Presentations
 
How to Map Your Future
How to Map Your FutureHow to Map Your Future
How to Map Your FutureSlideShop.com
 
Beyond Pride: Making Digital Marketing & SEO Authentically LGBTQ+ Inclusive -...
Beyond Pride: Making Digital Marketing & SEO Authentically LGBTQ+ Inclusive -...Beyond Pride: Making Digital Marketing & SEO Authentically LGBTQ+ Inclusive -...
Beyond Pride: Making Digital Marketing & SEO Authentically LGBTQ+ Inclusive -...AccuraCast
 
Read with Pride | LGBTQ+ Reads
Read with Pride | LGBTQ+ ReadsRead with Pride | LGBTQ+ Reads
Read with Pride | LGBTQ+ ReadsKayla Martin-Gant
 
Exploring ChatGPT for Effective Teaching and Learning.pptx
Exploring ChatGPT for Effective Teaching and Learning.pptxExploring ChatGPT for Effective Teaching and Learning.pptx
Exploring ChatGPT for Effective Teaching and Learning.pptxStan Skrabut, Ed.D.
 
How to train your robot (with Deep Reinforcement Learning)
How to train your robot (with Deep Reinforcement Learning)How to train your robot (with Deep Reinforcement Learning)
How to train your robot (with Deep Reinforcement Learning)Lucas García, PhD
 

Featured (20)

Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy Presentation
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming Language
 
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
 
9 Tips for a Work-free Vacation
9 Tips for a Work-free Vacation9 Tips for a Work-free Vacation
9 Tips for a Work-free Vacation
 
I Rock Therefore I Am. 20 Legendary Quotes from Prince
I Rock Therefore I Am. 20 Legendary Quotes from PrinceI Rock Therefore I Am. 20 Legendary Quotes from Prince
I Rock Therefore I Am. 20 Legendary Quotes from Prince
 
How to Map Your Future
How to Map Your FutureHow to Map Your Future
How to Map Your Future
 
Beyond Pride: Making Digital Marketing & SEO Authentically LGBTQ+ Inclusive -...
Beyond Pride: Making Digital Marketing & SEO Authentically LGBTQ+ Inclusive -...Beyond Pride: Making Digital Marketing & SEO Authentically LGBTQ+ Inclusive -...
Beyond Pride: Making Digital Marketing & SEO Authentically LGBTQ+ Inclusive -...
 
Read with Pride | LGBTQ+ Reads
Read with Pride | LGBTQ+ ReadsRead with Pride | LGBTQ+ Reads
Read with Pride | LGBTQ+ Reads
 
Exploring ChatGPT for Effective Teaching and Learning.pptx
Exploring ChatGPT for Effective Teaching and Learning.pptxExploring ChatGPT for Effective Teaching and Learning.pptx
Exploring ChatGPT for Effective Teaching and Learning.pptx
 
How to train your robot (with Deep Reinforcement Learning)
How to train your robot (with Deep Reinforcement Learning)How to train your robot (with Deep Reinforcement Learning)
How to train your robot (with Deep Reinforcement Learning)
 

Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection

  • 1. Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection 1 September 2021, JCDL, Virtual Event Authors: Jan Philip Wahle, Terry Ruas, Norman Meuschke, Bela Gipp
  • 2. Agenda/Outline 1. Motivation 2. Overview 3. Method 4. Benchmark Details 5. Experiments 6. Conclusion & Future Work 3
  • 3. Motivation Plagiarists increasingly use automated paraphrasing tools to trick plagiarism detection software. Problem: • Text-matching software fails if ≈20% of words are replaced [1]. • Neural language models can mimic human writing. • No benchmark exists to test neural language model plagiarism. Solution: • A benchmark to train/test detection of neural paraphrasing. 4 [1] J. P. Wahle, T. Ruas, T. Foltynek, N. Meuschke, and B. Gipp, “Identifying Machine-Paraphrased Plagiarism,” arXiv:2103.11909 [cs], Jan. 2021.
  • 4. Overview 5 Paragraphs from arXiv, Wikipedia, theses Transformer models BERT, RoBERTa, Longformer Paraphrased documents Mask words randomly Predict replacement candidates fastText+SVM, BERT, RoBERTa, Longformer Detection of plagiarism
  • 5. A mathematically rigorous approach to quantum field theory based on operator algebras is called an algebraic quantum field theory... Method - Example 6 <mask> mathematically rigorous approach to quantum field theory based <mask> operator <mask> is called an algebraic <mask> field theory... The mathematically rigorous approach to quantum field theory based upon operator equations is called an algebraic Quantum field theory... Input Models Output
  • 6. Sources arXiv, Wikipedia, student theses Paraphrasing Models BERT, RoBERTa, Longformer Masking probabilities [0.15, 0.2, 0.3, 0.4, 0.5] Original Examples 163 735 Paraphrased Examples 2 455 725 Benchmark Details 7
  • 7. Experiments 8 We perform three experiments to answer the questions: 1. Does the meaning of paraphrased paragraphs remain the same? ○ Experiment #1: Test the semantic similarity of paragraphs 2. How many words can be changed without compromising paraphrasing quality? ○ Experiment #2: Test different masking probabilities 3. Are neural language models good plagiarists? ○ Experiment #3: Test detection performance
  • 8. Experiment #1: Semantic Similarity 9 • 30 paragraphs • BERT-embeddings • 2D reduction (t-SNE)
  • 9. Experiment #2: Optimal Masking Probability 10 • Detection scores of fastText+SVM • Masking probabilities from [0.15, …, 0.5]
  • 10. Experiment #3: Detection Performance 11 • Baseline: fastText + SVM • Models detecting each others plagiarism • Model detect its own plagiarism best.
  • 11. Conclusion & Future Work • Neural paraphrasing increases the difficulty to spot plagiarism. • Our benchmark [3] fosters tests for this new form of plagiarism. • Early experiments confirm that models used for plagiarism are typically best in detecting their own plagiarism. • Extending the benchmark with additional auto-encoding models 12 [3] https://zenodo.org/record/4621403 - Terms and conditions apply for the source code.
  • 12. Thank you for your attention! 14 Do you have any questions? Image by undraw.co