Social Media Verification Challenges, Approaches and Applications
•
0 likes•670 views
As grassroots and social media-based journalism becomes more widespread, the need to verify information coming from such channels becomes imperative. The objective of this talk is to explore the challenges involved in social media computational verification to automatically classify unreliable media content as fake or real. After presenting a generic conceptual architecture, there will be a focus on tweets around big events linking to images (fake or real) of which the reliability could be verified by independent online sources. The REVEALr platform will be demonstrated, a scalable and efficient content-based media crawling and indexing framework featuring a novel and resilient near-duplicate detection approach and intelligent content- and context-based aggregation capabilities (e.g. clustering, named entity extraction)
Recommended
Recommended
More Related Content
Similar to Social Media Verification Challenges, Approaches and Applications
Similar to Social Media Verification Challenges, Approaches and Applications (20)
More from Yiannis Kompatsiaris
More from Yiannis Kompatsiaris (17)
Recently uploaded
Recently uploaded (16)
Social Media Verification Challenges, Approaches and Applications
- 2. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Overview • Introduc.on – Mo.va.on – Challenges • Social Media in News and Journalism • The problem of verifica.on • Approaches – Context extrac.on from Web and Social Media – Image Forensics – Computa.onal verifica.on • Demos - Resources • Conclusions 2
- 9. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on 9 Social Networks as Real-Life Sensors • Social Networks is a data source with an extremely dynamic nature that reflects events and the evolu.on of community focus (user’s interests) • Huge smartphones and mobile devices penetra2on provides real-.me and loca.on-based user feedback • Transform individually rare but collec2vely frequent media to meaningful topics, events, points of interest, emo.onal states and social connec.ons • Present in an efficient way for a variety of applica.ons (news, marke.ng, science, health, entertainment)
- 10. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on 10 Real-life Social Networks • Social networks have emergent proper2es. Emergent proper.es are new aVributes of a whole that arise from the interac.on and interconnec.on of the parts • Emo.ons, Health, Sexual rela.onships depend on our connec2ons (e.g. number of them) and on our posi2on - structure in the social graph • Central – Hub • Outlier • Transi.vity (connec.ons between friends)
- 18. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Policy – Licensing – Legal challenges • Fragmented access to data – Separate wrappers/APIs for each source (TwiVer, Facebook, etc.) – Different data collec.on/crawling policies • Limita.ons imposed by API providers (“Walled Gardens”) • Full access to data impossible or extremely expensive (e.g. see data licensing plans for GNIP and DataSit) • Non-transparent data access prac.ces (e.g. access is provided to an organiza.on/person if they have a contact in TwiVer) • Constant change of model and ToS of social APIs – No backwards compa.bility, addi.onal development costs • Ephemeral nature of content • Social search results oten lead to removed content à inconsistent and unreliable referencing • User Privacy & Purpose of use • Fuzzy regulatory framework regarding mining user-contributed data 18
- 20. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on 20 Source: GeVy Images “It’s really hard to find the nuggets of useful stuff in an ocean of content” (BBC) “Things that aren’t relevant crowd out the content you are looking for” (MSN) “The filters aren’t configurable enough” (CNN)
- 25. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on The Problem • Everyone can easily publish content on the Web • Content can be easily repurposed and manipulated • Not only for fun but also for propaganda • News outlets are compe.ng for views and clicks à Pressure for airing stories very quickly leaves very liVle room for verifica.on. à Very oten, even well- reputed news providers fall for fake news content. • Mul.ple tools and services available for individual tasks à complex verifica.on process Very hard and 2me consuming to check the veracity of Web mul2media 25
- 26. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Image verifica2on: tools of the trade • Metadata analysis – E.g. do the dates/loca.ons match? Is the image already copyrighted? By whom? • Context Extrac.on from Web and Social Networks – Reverse image search using e.g. Google or TinEye – Clustering • Has the image been posted elsewhere? Does it originate from a different context? • Supervised machine learning for automa.c classifica.on – Exploi.ng paVerns of usage, content, linking of fake/real content • Content analysis (forensics) for tampering localiza.on – Most commonly, Error Level Analysis (ELA)
- 28. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Media REVEALr • Developed within the REVEAL project: hVp://revealproject.eu/ • Framework for collec.ng, indexing and browsing mul.media content from the Web and social media • Support for verifica.on: – Near-duplicate detec.on against an indexed collec.on – Clustering of social media posts by visual similarity à compara.ve view of the same incident – Aggrega.on and visualiza.on of Named En..es around an incident 28
- 30. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Named En2ty Detec2on • Brevity and noisy nature of text in social media poses a serious challenge • Employed solu.on: – Pre-processing: tokeniza.on, user men.on resolu.on, text cleaning – Stanford NER + user men.on resolu.on – Regular expressions to remove special characters and symbols (e.g., #, @, URLs, etc.) 30
- 31. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Visual Indexing • Content-based image retrieval to solve Near- Duplicate Search (NDS) problem • Based on local descriptors (SURF), aggrega.on (VLAD), dimensionality reduc.on (PCA), quan.za.on (PQ) and indexing (IVFADC) • State-of-the-art visual similarity search – High precision/recall – Very efficient and scalable implementa2on (search many millions of images in a few msec, maintain full index in memory using ~1GB/10M images) 31
- 34. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Classifier Details • Random Forest used as base classifier • Cost Sensi.ve meta-classifier to penalize misclassifica.on of True Posi.ves • Challenge due to Class Imbalance (overlay descriptors << useful image content descriptors) – Cost Sensi.ve meta-classifier performs over-sampling of minority class to balance the training set • Training set created by collec.ng images with overlays (e.g., memes) from the Web and manually annota.ng them (selec.ng areas w. fonts/overlays) 34
- 35. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Mining: Clustering and Aggrega2on • Visual aggrega.on – DBSCAN on the visual feature representa.on (PCA-reduced VLAD vectors) – Element (tweet) selected based on the largest amount of keywords (expected to result in more informa.on) • En.ty aggrega.on – NER on individual items – En.ty categoriza.on (àPersons, Loca.on, Organiza.ons) – En.ty ranking based on frequency of occurrence 35
- 40. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Evalua2on: NER • Manual annota.on of 400 tweets from the SNOW Data Challenge dataset (Papadopoulos et al., 2014) • Measure: Accuracy à instance is considered correct when both en.ty and type are correctly iden.fied • Three compe.ng solu.ons: – Base Stanford NER (S-NER) – S-NER + Extensions/Post-processing (S-NER+) – Ellogon library (hVp://www.ellogon.org) 40
- 41. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Evalua2on: NDS • Benchmark Datasets – Holidays: 1,491 images, 500 queries (Jegou et al., 2008) – Oxford: 5,063 images, 55 queries (Philbin et al., 2008) – Paris: 6,412 images, 55 queries (Philbin et al., 2008) • Accuracy: mean Average Precision (mAP) 41
- 49. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Computa2onal Verifica2on in Social Media • Create a computa$onal verifica$on framework to classify tweets with unreliable media content. • Events used for experimenta.on 49 Fake images posted during Hurricane Sandy natural disaster Fake images posted during Boston Marathon bombings
- 50. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Goals/Contribu2ons • Dis.nguish between fake and real content shared on TwiVer using a supervised approach • Provide closer to reality es.mates of automa.c verifica.on performance • Explore methodological issues with respect to evalua.ng classifier performance • Create reusable resources – Fake (and real) tweets (incl. images) corpus – Open-source implementa.on 50
- 51. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Methodology • Corpus Crea.on – Topsy API – Near-duplicate image detec.on • Feature Extrac.on – Content-based features – User-based features – Link-based features • Classifier Building & Evalua.on – Cross-valida.on – Independent photo sets – Cross-dataset training 51
- 52. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Corpus Crea2on • Define a set of keywords K around an event of interest. • Use Topsy API (keyword-based search) and keep only tweets containing images T. • Using independent online sources, define a set of fake images IF and a set of real ones IR. • Select TC ⊂ T of tweets that contain any of the images in IF or IR. • Use near-duplicate visual search (VLAD+SURF) to extend TC with tweets that contain near-duplicate images. • Manually check that the returned near-duplicates indeed correspond to the images of IF or IR. 52
- 53. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Features (verifica2on handbook) 53 # User Features 1 Username 2 Number of friends 3 Number of followers 4 Number of followers/number of friends 5 Number of .mes the user was listed 6 If the user’s status contains URL 7 If the user is verified or not # Content Features 1 Length of the tweet 2 Number of words 3 Number of exclama.on marks 4 Number of quota.on marks 5 Contains emo.con (happy/sad) 6 Number of uppercase characters 7 Number of hashtags 8 Number of men.ons 9 Number of pronouns 10 Number of URLs 11 Number of sen.ment words 12 Number of retweets 13 Readability1 # Link-based features 1 Web Of Trust score (WOT)2 2 In-degree and harmonic centrali.es3 3 Alexa rankings4 1 Flesch reading ease method to compute a score in [0,100] range, 0 hard- to-read and 100 easy-to-read text 2 A metric for how trustworthy a website is, based on user ra$ngs 3 Rankings computed based on the Web graph 4 Alexa rankings, which evaluate the frequency of visits on various websites
- 57. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Cross-dataset Training-Tes2ng • In the most unfavourable case, the dataset used for training should refer to a different event than the one used for tes.ng. • Simulates real-world scenario of a breaking story, where no prior informa.on is available to news professionals. • Variants: – Different event, same domain – Different event, different domain (very challenging!) 57
- 58. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Evalua2on • Datasets – Hurricane Sandy – Boston Marathon bombings • Evalua.on of two sets of features (content/ user) • Evalua.on of different classifier se‚ngs 58
- 60. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Dataset – Boston Marathon Bombings 60 The bombings occurred on 15 April, 2013 during the Boston Marathon when two pressure cooker bombs exploded at 2:49 pm EDT, killing three people and injuring an es.mated 264 others. Hashtags Boston Marathon #bostonMarathon Boston bombings #bostonbombings Boston suspect #bostonSuspect manhunt #manhunt watertown #watertown Tsarnaev #Tsarnaev 4chan #4chan Sunil Tripathi #prayForBoston
- 61. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Dataset Sta2s2cs 61 Tweets with other image URLs 343939 Tweets with fake images 10758 Tweets with real images 3540 Hurricane Sandy Boston Marathon Tweets with other image URLs 112449 Tweets with fake images 281 Tweets with real images 460 Tweets with fake images 1% Tweets with other image URLs Tweets with fake images Tweets with real images Tweets with other image URLs 3% 1% 96% Tweets with fake images Tweets with real images Tweets with other image URLs
- 62. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Predic2on accuracy (1) 62 0. 10. 20. 30. 40. 50. 60. 70. 80. 90. 100. Total User Content J48 decision tree 0. 10. 20. 30. 40. 50. 60. 70. 80. 90. 100. Total User Content KStar 0. 10. 20. 30. 40. 50. 60. 70. 80. 90. 100. Total User Content Random Forest Boston Marathon Hurricane Sandy • 10-fold cross valida.on results using different classifiers ~80%
- 63. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Predic2on accuracy (2) • Results using different training and tes.ng set from the Hurricane Sandy dataset 63 0. 25. 50. 75. 100. Total User Content Random Forest Kstar J48 decision tree • Results using Hurricane Sandy for training and Boston Marathon for tes.ng 0. 10. 20. 30. 40. 50. 60. 70. 80. 90. 100. Total User Content Random Forest Kstar J48 decision tree ~75% ~58% separate classifiers might be built for certain types of incidents
- 64. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Sample Results 64 • Real tweet My friend's sister's Trampolene in Long Island. #HurricaneSandy Classified as real • Real tweet 23rd street repost from @wendybarton #hurricanesandy #nyc Classified as fake • Fake tweet Sharks in people's front yard #hurricane #sandy #bringing #sharks #newyork #crazy hZp://t.co/PVewUIE1 Classified as fake • Fake tweet Statue of Liberty + crushing waves. hZp://t.co/7F93HuHV #hurricaneparty #sandy Classified as real
- 66. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Reusable results • Computa2onal verifica2on – Dataset: hVps://github.com/MKLab-ITI/image-verifica.on-corpus – Code: hVps://github.com/socialsensor/computa.onal-verifica.on • The Wild Web Tampered Image Dataset – 80 confirmed digital forgeries, 10,870 images, Ground truth binary masks – Dataset: hVps://mklab.i..gr/project/wild-web-tampered-image-dataset • The Deutsche Welle Tampered Image Dataset – 6 original images, 3 image sources, 7 different modified versions – Surprisingly tough to crack using the state-of-the-art – Dataset: hVps://revealproject.eu/the-deutsche-welle-image-forensics-dataset/ • Open-source projects (Apache License v2): hVps://github.com/socialsensor – Data collec.on (stream-manager, storm-focused-crawler) – Indexing (framework-client, mul.media-indexing) – Mining (topic-detec.on, mul.media-analysis, community-evolu.on- analysis, social-event-detec.on) 66
- 67. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Contribu2ons • Dr. Symeon Papadopoulos – Social network analysis, social media content mining and mul.media indexing and retrieval – hVp://mklab.i..gr/people/papadop – TwiVer: @sympap • Dr. Zampoglou Markos – Web mul.media verifica.on, image forensics for verifica.on – markzampoglou@i..gr • Boididou Chris.na – computa.onal approaches for verifica.on – boididou@i..gr 67
- 69. 3rd interna*onal conference on Internet Science INSCI 2016 Social Media Verifica*on Conclusions • Social media data useful in many applica.ons – From confirming exis.ng and known correla.ons to predic.on and decision-making • Many challenges exist – Data availability (infrastructure, policies) – Personal data value (legal, ethical) – Real-.me and scalable approaches – Fusion of various modali.es (Content, social, temporal, loca.on) • Verifica.on requires contribu.on from various disciplines – Content Analy.cs – Machine Learning – Network Analysis – Psychology – Social Sciences (paVerns of presenta.on, sharing) – Visualiza.on 69