By Carolina Scarton

1. Detecting and Verifying Online Rumours
Carolina Scarton
c.scarton@Sheffield.ac.uk
NILC – ICMC – USP, 15/05/2019
1WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

2. What is an online rumour?
2WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297
Information that is unverified at the time of posting
Recipe:
• Social platform (e.g. Twitter, Facebook)
• Easy access from everywhere
• No moderation
• Share what you hear/see, etc. (emerging rumours)
• Updates on breaking news (emerging rumours)
• Some interest -- Obama is muslim (long standing rumours)
New challenge: closed-access communication platforms (e.g. WhatsApp)

3. Examples
True rumour: "10 people dead in Charlie Hebdo according to witnesses”
False rumour: "GERMAN NEWS REPORT: Co-Pilot of Germanwings Airbus
Was MUSLIM CONVERT …’Hero of Islamic State’?”
Unverified rumour: "Police in Ferguson claimed that Mike Brown had been
involved in a robbery"
3WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

4. But why should we care?
Social media is a good and powerful resource:
• News gathering 🡪 eyewitnesses
• Public opinion 🡪 elections
But only if the information is TRUE
New data science tools able to:
• detect a rumour
• track a detected rumour
• analyse individual stances
• verify it
• support content verification professionals
4WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

5. Rumour detection and verification
5WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

6. Detecting a rumour
First step in rumour analysis
• Can be omitted if the rumour is already known!
• Binary classification
Can be identified through reactions from “skeptical users”
• If a post has a considerable amount of enquiring tweets 🡪 information is rumourous
Datasets from PHEME project
• Includes a collection of 1,972 rumours and 3,830 non-rumours
• Associated with five breaking news stories
6WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

7. Tracking a rumour
Second step in rumour analysis
• Previous work uses already identified rumours 🡪 no studies for emerging rumours
• Can involve network monitoring
Widely used dataset for rumour tracking (Qazvinian et al., 2011)
• Includes over 10,000 tweets, associated with 5 different rumours
• Each tweet annotated for relevance towards the rumour as related or unrelated
7WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

8. Stance classification
Third step in rumour analysis
• Determining the type of orientation that each individual post expresses towards the
disputed veracity of a rumour
• Relevant in the context of social media where unverified reports are continually being
posted and discussed
• Can guide veracity classification
8WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

9. Stance classification
9WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

10. Stance classification
PHEME dataset
10WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

11. Stance classification
RumourEval 2019 shared task
• Task A: stance classification (Comment/Support/Deny/Query)
11WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297
Task A
Macro F
System A 0.6187
System B 0.6067
System X 0.5776

12. Veracity classification
Final step in rumour analysis
• Determine the veracity of an entire rumour
• Binary or three-class problem (true/false/unverified)
• Potentially less useful for professionals
RumourEval 2019 shared task
• Three-class: true/false/unverified
12WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297
Task B
Macro F RMSE
System X 0.5765 0.6078
System Y 0.2856 0.7642
System Z 0.2620 0.8012

13. WeVerify project
13WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

14. WeVerify
Cross-modal and cross-platform approach
• Evidence from text, images, and videos; deep fakes
Overcome effort fragmentation
• A blockchain-based, authoritative database of already debunked fake items
Verification made simple
• A digital companion for citizens and an advanced version for journalists
Look beyond the content
• Analysis of disinformation source, spread strategies, and community networks
Build a community, not just technology
• Pilot with large stakeholder communities
14WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

15. WeVerify
Database of known fakes
• Expected to store previous debunked content
• Useful for professionals, non-professionals and data scientists
• Main challenges include technology (scalability) and ethics (right to be forgotten)
15WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

16. WeVerify
Tools for supporting professionals dealing with content verification
• Journalists, human rights activists, etc.
Support professional debunking (instead of performing the debunking)
• Inaccurate results of verification tools are more harmful than helpful!
Claim identification and stance classification may be the most useful
• Challenge on how to show the information to the user
16WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

17. WeVerify
Digital Companion
• Empower citizens to verify content
• A guide instead of a classifier
• Start point: InVID plugin
https://www.invid-project.eu/
17WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

18. WeVerify
18WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

19. Challenges on rumour detection and verification
Scarcity of data
Several levels of veracity (a claim can be “half true”, “inaccurate”)
Ethical approval 🡪 social data is a kind of personal data
Requirements for data storage and usage
Difficulty to reach closed channels (e.g. WhatsApp)
19WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297

20. Thank you for your attention!
20
More about WeVerify:
www.weverify.eu
https://twitter.com/WeVerify
WeVerify project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825297
Opportunity for collaboration through
The SoBigData project invites researchers and
professionals to apply to participate in Short-Term
Scientific Missions (STSMs) to carry forward their own big
data projects.
These opportunities are offered as part of SoBigData's
Transnational Access (TNA) activities and applications can
be submitted at any time.