Data Cleaning for social media knowledge extraction
•Download as PPTX, PDF•
1 like•1,617 views
Social media platforms let users share their opinions through textual or multimedia content. In many settings, this becomes a valuable source of knowledge that can be exploited for specific business objectives. Brands and companies often ask to monitor social media as sources for understanding the stance, opinion, and sentiment of their customers, audience and potential audience. This is crucial for them because it let them understand the trends and future commercial and marketing opportunities. However, all this relies on a solid and reliable data collection phase, that grants that all the analyses, extractions and predictions are applied on clean, solid and focused data. Indeed, the typical topic-based collection of social media content performed through keyword-based search typically entails very noisy results. We recently implemented a simple study aiming at cleaning the data collected from social content, within specific domains or related to given topics of interest. We propose a basic method for data cleaning and removal of off-topic content based on supervised machine learning techniques, i.e. classification, over data collected from social media platforms based on keywords regarding a specific topic. We define a general method for this and then we validate it through an experiment of data extraction from Twitter, with respect to a set of famous cultural institutions in Italy, including theaters, museums, and other venues. For this case, we collaborated with domain experts to label the dataset, and then we evaluated and compared the performance of classifiers that are trained with different feature extraction strategies.
Recommended
Recommended
More Related Content
What's hot
What's hot (18)
Similar to Data Cleaning for social media knowledge extraction
Similar to Data Cleaning for social media knowledge extraction (20)
More from Marco Brambilla
More from Marco Brambilla (20)
Recently uploaded
Recently uploaded (13)
Data Cleaning for social media knowledge extraction
- 1. Emre Calisir, Marco Brambilla KDWEB2018, Cáceres, Spain The Problem of Data Cleaning for Knowledge Extraction from Social Media
- 2. Knowledge Extraction from Social Media is a Need
- 3. Keyword or hash-tag based filtering is insufficient
- 4. Is it possible to extract a sub-selection of content items if and only if they are actually relevant to the topic or context of interest ?
- 5. Examples to Related Studies 1. Earthquake Alarm System, Sakaki et al. Proc. of the 19th int.conference on WWW, 2010 2. Detection of influenza-like illnesses Culotta. Proc. of the 1st workshop on social media analytics, Washington, D.C, 2010 3. Discovering health topics, Paul & Dredze, PLoS ONE 9, e103408, 2014 4. Detection of prescription medication abuse, Sarker et al. Drug Safety, 2016 5. Tracking baseball and fashion topics, Lin et al. KDD, 2011 6. Event detection system, Kunneman & Bosch, BNAIC, 2014 7. Credibility of trend-topic hashtag usage Castillo et al, Proc. of the 20th int. Conf. on WWW. ACM, Hyderabad, USA, 2011 8. Non-relevant tweet filtering, Hajjem & Latiri, Procedia computer science, 112, 2017
- 6. Supervised learning trained on annotated data could help us
- 8. Proposed Data Cleaning Method for Knowledge Extraction
- 10. Best #Hotel Deals in #Pompei #HotelDegliAmiciPompei starting at EUR99.60 https://t.co/5DxkKn4o69 Pompei Hero Pliny the Elder May Have Been Found 2000 Years Later https://t.co/PyR2rP1Xpe #2017Rewind #archeology #history #Pompei #rome #RomanEmpire Non-Relevant Tweet Relevant Tweet
- 11. 4 feature extraction strategies evaluated N-grams (unigrams, bigrams, trigrams) Word2Vec Word2Vec + additional tweet features Dimensionality Reduction with PCA
- 12. Annotated Data 726 tweets. Contains tweet having specific hashtags and keywords related to Pompei, Colosseo and Teatro Alla Scala The data contains 50% relevant and 50% non relevant.
- 13. Model 1: Text transformation to ngrams # of [unigrams, bigrams, trigrams] : [494,287,228] vocabulary size: 1009 words Example:
- 14. Model 2: Text transformation to word2vec • Word2vec dimension is selected as 25. • Word2vec vocabulary is built with 12K unlabeled tweets • Preprocessing operations before building word2vec model • Convert to lowercase, • Discard • Web links • Words with character size < 3 • Stopwords are eliminated before model building.
- 15. Model 2: Text transformation to word2vec
- 16. Model 2: Text transformation to word2vec
- 17. Model 3: word2vec + Additional Features Tweet Author Full text: #nuovi #corsi #inglese #settembre #pompei #chiamaci per #info https://t.co/QRrXlMC0g1 Number of Friends: 4 Number of Followers: 9 Number of Lists: 15 Number of Favourited Tweets: 0 Language: en Number of Tweets: 4220 Source: PostPickr Verified Account: False Number of Favorited: 0 Geo Enabled: False Number of Retweets: 0 Default Profile: False Example:
- 18. Model 4: PCA applied on Model 3
- 19. Model 1 2 3 4 Accuracy 0.84 0.81 0.82 0.83 Precision 0.84 0.78 0.83 0.84 Recall 0.83 0.86 0.8 0.81 F1 0.83 0.82 0.81 0.82 Model 1: ngrams Model 2: word2vec Model 3: word2vec + additional features Model 4: PCA applied on Model 3 10fold Cross-Validated Results
- 20. Conclusions Supervised Machine Learning techniques could help to obtain topic relevant social media data
- 21. Collecting more data to build larger Word2Vec Vocabulary New Use Cases Challenges ahead
- 22. THANKS! QUESTIONS? Emre Calisir, Marco Brambilla The Problem of Data Cleaning for Knowledge Extraction from Social Media Marco Brambilla @marcobrambi marco.brambilla@polimi.it http://datascience.deib.polimi.it http://home.deib.polimi.it/marcobrambi
Editor's Notes
- 500 million tweets each day. 100 million active daily people. People tend to share every part of their lives, opinions... Every sector wants to extract knowledge from social media: banking, health, tourism...
- Data is created by humans. Very open to errors. Words can have multiple meanings. Rule-based filtering systems fail. (we mean by Rule-based systems systems that collect tweets just using queries) They bring related and unrelated content together.
- Our research question.
- All of them are based on Twitter data Various use cases, various machine learning techniques Supervised (1 to 7) SVM, Multinomial Bayes, Logistic Regression, Decision Trees, Bayes Networks, Unsupervised (Last one) Latent Dirichlet Allocation We benefit from these studies. Early Alert System. Technique: SVM classifier. Features: keywords and the context of target-event words. Detection of Illnesses. Technique: Log Regr. Bag-of-Words Filter-out Non-Relevant Content. Technique: Log. Regr. Unigrams-Bigrams-Trigrams. Very similar to first model in our study. Discovery of patterns of abuse of specific medications. Topic tracking on tweet streams. Again unigrams Discard noisy data for an event detection system based on Twitter stream data. Assesment of credibility of trend-topics, to check if they are really related with the topic of used hash-tag, or spam. They tried SVM, decision trees and Bayes. Unsupervised technique. Pooling method using Information Retrieval and Latent Dirichlet Allocation.
- Annotated dataset, labeled as Relevant or Non-Relevant with the topic. We train SVM based model with labeled data. We predict the new unlabeled data. We selected Support Vector Machines with Linear Kernel. The recommended approach when there is sparse vector features.
- Our basic approach to obtain a relevant dataset. We applied with Twitter but is also applicable to any kind of social media data.
- And a more detailed illustration to general flow. Initially we build our model. And then we make the prediction. Finally we have a clean dataset.
- Pompei, Colosseo and Teatro Alla Scala
- Imagine that we have a social media monitoring tool. We want to track tweets related historical value of Pompei. How to do it? Label relevant and non-relevant data. Who will label it? Subject experts
- Subject experts are labeled the tweets as relevant and non relevant. A random guess could predict with 0.5 accuracy.
- Ngrams is a widely used technique in text classification. It is an example to how we transform a tweet to feature vector. Also we applied Tf-idf to increase accuracy. It is a best-practive in ngrams usage.
- Default word2vec dimension is 100. However, our dataset is limited. We performed analysis and achieved better results by representing with 25 features. Larger the word2vec vocabulary, better creating the semantic relations.
- Top 3 Similarities showed for specific words. Inside parantheses, there is vectorial distance btw words. For a given tweet, we calculate the average value of its word vectors.
- Another illustration of trained word2vec model. This graph is generated with few words to show you just the vectorial representations of words.
- Feature Extraction Strategy Vector features: Word2Vec Numerical features: MinMaxScaler Categorical features: OneHotEncoder Dimension of features after transformation: dim(Word2Vec) = 25 -- dim(numerical features) = 7 -- dim(one-hot-encoding) = 68 -- Total number of features = 125
- The target is to analyze impact of dimensionality reduction. It could improve model accuracy. Graphic: Desired dimension < 40 the model is underfitting Desired dimension > 40 the model is overfitting We selected dimension size : 40
- We preferred cross-validation because size of our labeled dataset is limited. (726 tweets) All the classification models are succesful. Dotted line shows random guess. (50% relevant and 50% non relevant tweets exist inside data) Model 1 has best performance. Model 4 has second best performance. If we have a larger word2vec vocabulary, word2vec could have better accuracy then ngrams. ROC curve and AUC scores prove the performance of our model.
- We addressed our research question. We proved that text classsification is a very convenient way to obtain topic relevant data.
- We still a way to go for building a more accurate system. We now create a larger dataset. And we will find out new use cases, also open datasets to compare our results with the literature. Also, we will try other algorithms, not only SVM. Also, we can bring external data by importing content from given weblinks.