Recommended
More Related Content
Similar to ICWE 2017: Recruiting from the network
Similar to ICWE 2017: Recruiting from the network (20)
Recently uploaded
Recently uploaded (20)
ICWE 2017: Recruiting from the network
- 1. 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton 1 Recruiting from the network: discovering Twitter users who can help combat Zika epidemics Paolo Missier1, Callum McClean1, Jonathan Carlton1, Diego Cedrim2, Leonardo Silva2, Alessandro Garcia2, Alexandre Plastino3, and Alexander Romanovsky1 1 School of Computing Science, Newcastle University, UK 2 PUC-Rio, Rio de Janerio, Brazil 3 Universidad Federal Fluminense, Niteròi, Brazil 17th International Conference on Web Engineering Rome, June 2017
- 2. 2 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton Motivation, Goals, and Hypothesis • Motivation: • Mosquito-borne epidemics frequently occur and preventions programs have no been effective. • Long-term goal: • Leverage real-time, social media streams to increase effectiveness and speed of health campaigns. • Hypothesis: • Social sensors and their signals can be detected from real-time social media streams. • Through continuous analysis of the Twitter stream, it is possible to automatically identify them
- 3. 3 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton Challenges/Problems • Discovery of relevant tweets requires a long harvest period. • On average, relevant tweets make up 10% of a typical harvest. • Sparsity of users and tweets • Most users contribute a single tweet • Lack of meaningful connections and content • Needle in haystack problem
- 7. 7 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton User Ranking Metrics • TwitterRank [1] • Topological approach that takes into account the topical similarity between users and network structure to determine a score. • Topic Focus • 𝑇𝐹 𝑢 = 𝑅 𝐾(𝑢) 𝑇 𝐾(𝑢) • TF(u) is the topic focus for a user u, TK(u) is the number of tweets contributed by u, RK(u) is the number of relevant tweets in TK(u). • Overall Focus • 𝑂𝐹 𝑢 = 𝑅 𝐾(𝑢) 𝑇(𝑢) • OF(u) is the overall focus for a user u, T(u) is the total number of tweets posted by u during the harvest period.
- 8. 8 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton Experiments • Classifier trained on 10,000 labelled examples • Collection of tweets 278,351 from September – December 2016, 15,124 are classified as relevant.
- 9. 9 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton Results: TwitterRank • Limited success; but notable spread between top and bottom ranks. • Requires meaningful connections to exist and a lack of ground truth.
- 10. 10 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton Results: Topic Focus • Suggested correlation between a high TwitterRank and high Topic Focus • More than 10 users have a Topic Focus = 100
- 11. 11 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton Results: Overall Focus • Users here also have a high Topic Focus • FlorzinhaSimoes and pelotelefone also rank high in other lists
- 12. 12 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton Fragment of candidate users graph
- 13. 13 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton Conclusions • Lessons learnt • Ineffectiveness of TwitterRank • Non-topological metrics aid in producing results • Research in progress • Continually experimenting with larger data sets • Engagement with candidate users
- 14. 14 17thInternationalConferenceonWebEngineering Rome,June2017 J.Carlton [1] Weng, Jianshu, Ee-Peng Lim, Jing Jiang, and Qi He. "Twitterrank: finding topic-sensitive influential twitterers." In Proceedings of the third ACM international conference on Web search and data mining, pp. 261-270. ACM, 2010. [2] Missier, Paolo, Callum McClean, Jonathan Carlton, Diego Cedrim, Leonardo Silva, Alessandro Garcia, Alexandre Plastino, and Alexander Romanovsky. "Recruiting from the network: discovering Twitter users who can help combat Zika epidemics." arXiv preprint arXiv:1703.03928 (2017). [3] Missier, Paolo, Alexander Romanovsky, Tudor Miu, Atinder Pal, Michael Daniilakis, Alessandro Garcia, Diego Cedrim, and Leonardo da Silva Sousa. "Tracking Dengue Epidemics using Twitter Content Classification and Topic Modelling." In International Conference on Web Engineering, pp. 80-92. Springer International Publishing, 2016. [4] Chawla, Nitesh V., Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. "SMOTE: synthetic minority over-sampling technique." Journal of artificial intelligence research 16 (2002): 321-357. References
Editor's Notes
- Motivation Mosquito-borne epidemics are becoming more frequent in tropical and subtropical regions of the world. Prevention programs have not been particularly effective. Example: The Brazilian Health System requires that health agents report each Zika case, however, it takes several days to process and publish. Goals Social media provides, potentially, a faster vehicle for information than traditional channels; such as in-person data collection or surveys. Hypothesis We investigate into the kind of social sensors (people who stand out for their quality and relevance of their contribution) signals that can be detected from real-time social media streams. Ranking these social sensors will provide target users for the health authorities.
- Long Harvest Period Finding a sufficient number of relevant tweets per user requires a long harvest period. A 3-month harvest in 2016 yielded 13,000 relevant tweets (roughly 10% of the entire harvest) with most users (who have tweets classified as relevant) only contributing one single relevant tweet Sparsity of Users Our adapted single-topic version of Twitter Rank requires both meaningful connections and content to exist to be effective and efficient. When applying it to a sparse data set yields little or no interesting results.
- Combination of content-based automated classification of tweets, to isolate relevant signals from noisy chatter, and ranking users who author such content. Left side is offline and right side is online.
- A keyword harvester listens to the live Twitter stream. Keywords are from sample tweets inspected by domain experts. Pre-processing removes social media-type communication. POS tagging and lemmatisation are also performed. Re-sampling: to boost relevant samples, we added an extra 600 annotated relevant examples then applied over-sampling using the SMOTE algorithm. This boosted samples from 12.1% to 24.3%. Model: Using a Random Forest classifier with SMOTE over-sampling, no attribute selection, and 1,2,3-grams we achieved an accuracy of 84.1
- Both the tweets and pre-processing is done in the same way as the offline method. The classifier assigns either Relevant, News, or Noise label depending on the tweet content. News and Noise are no longer considered, relevant tweets move forward. Non-topological is performed instantly and results are ready to be pushed to the portal. Before Twitter Rank is performed, the followers of Relevant contributing users are fetched from Twitter. Can produce a bottle neck, especially if a large amount of followers need to be collected.
- Twitter Rank We adapted the original version to only consider a single topic rather than multiple. Also replaced some metrics to better suit our needs Detailed in the long version of the paper I’m presenting. Topic Focus The proportion of tweets in the harvest that are Relevant for a specific user. Overall Focus The proportion of Relevant tweets in the harvest vs the total number of tweets posted by the user in the harvest period. These are interesting as they provide a comparative metric for the Twitter Rank approach, without taking the network structure into account.
- Classifier was trained using 10,000 manually annotated examples. Table shows various combinations of N-grams and Multinomial Naïve Bayes vs Random Forest. Best overall accuracy across all configurations is 84.1% from Random Forest with 100 trees, 1,2,3-grams, no attribute selection and SMOTE-based boosting. From 278,000 tweets we could only harvest 15,000 Relevant tweets, giving us a small amount to work with. The image is an example of a Relevant classified tweet. The table shows the sparsity of contributions by users, long single-tweet tail giving us a lot of candidate users – 13,228.
- Very small values are given to users, the original paper does not provide any reference figures. Notable spread between the top and bottom ranks (150%). The results are questionable as the approach only yields interesting values when a user has at least some of its followers as candidate users. We find our candidate users have very few connections amongst each other.
- Note that SeizeTheHeaven appears in both top-10’s for Twitter Rank and Topic Focus All top 10 Twitter Rank users appear as top 30 Topic Focus users, suggesting that a high Twitter Rank may correlate with a high Topic Focus There are more than 10 users that have a Topic Focus of 100.
- Overall Focus users also have a high Topic Focus and they rank within the top-20 Two users here rank highly in other lists: FlorzinhaSimoes and pelotelefone.
- Green nodes = top 10 Twitter Rank Blue nodes = top 10 Twitter Rank and Overall Focus Red nodes = top 10 Twitter Rank and Topic Focus
- Ineffectiveness Given the sparsity of contributors and their limited connections in the social graph, it’s not surprising that TwitterRank – which relies on these – is not particularly effective. Non-topological In comparison, these appear to be effective. Going forward We are experimenting with larger data sets that are continually being harvested from Twitter. High season has just passed, so a larger corpus of relevant tweets is hoped for. Rather than past-tense analysis of the users, we’re discussing directly engaging with the users that are identified when relevant content is posted.