Eswc2013 audience short


Published on

Published in: Technology, Sports
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Solet‘sstartbyexploringwhatistheproblemwearetryingtoadresswiththiswork? So take a minutetoreadthose 2 sample tweetsandthinkabouthowyouwould tag them. Whataretopicsofthosetweets? You will noticeitisoftenprettyhardtopredictthetopicsof a tweetwithouthavingfurthercontextualinformation. So in thisexamplethesolutionismusicandfashion. Why do theauthors not providemorecontext? Makethemessageasshortaspossibleandasinformtiveasrequiredandtheyknow/estimatetheiraudience. But whatdoesthatmeaniftheymaketheirinformationaas informative asrequired? This impliesthatthey must estimatewhatthereaudienceknows.Theaudienceofthesecondauthorhere will knowwhois Rafael Cennamo. The audienceofthefirstauthor will knowthatsheis a singer. This examplenicelyshowsthattherearecertainpeoplewhohavetherightbackgroundknowledgetomake sense out ofthesparseinformation.
  • So thisworkbuilds on thisintuitionthatifonehasaccesstothebackgroundknowledgeoftheaudiencethiswouldhelptointerpretthemeaningoftweetsandannotatethemwithsemantics. Andtherefore, we hypothesize that semantic technologies which aim to enrich social media with semantics would benefit from incorporating the background knowledge of the audience.Concretlyweexplorethefollowing 2 researchquestions
  • Toadresstheseresearchquestionsweconduct a messageclassificationtofigure out howwellcantheaudiencepredictthehashtagsofstream‘smessages. Thatmeansweusedtweetsoftheaudienceofhashtagstreamsastrainingsamplesandthehashtagitselfaslabels. Wetestedhowwelltheaudience-informedclassifiercanpredictthehashtagoffuturemessagescomparedto a classifiertrained on messagesofrandomaudiences.
  • Wetrained different classificationmodelsandfoundthat linear SVM worksbestandthereforereportthisresultshere. Toestimatethe BK foreachhashtagstreamwecompare different audienceandcontentselectionapproachesand different methodsforestimatingthe BK.
  • As a datasetwecreated a diverse sample ofhahstags. Weusedthe 8 categoriesofhashtagswhichhavebeenidentifiedby Romero and Kleinberg in theirpreviousworkanddrawfromeachcategory 10 hashtags (weonlyusedhashtagswhichare still active).We bias our random sample towards active hashtag streams by re-sampling hashtags for which we found less than 1,000 messages when crawling (4. March 2012). For those categories for which we could not find 10 hashtags which had more than 1,000 messages (games and celebrity), we select the most active hashtags per category (i.e., the hashtags for which we found the most messages). Since two hashtags #bsb and #mj appeared in the sample of two different categories, we ended up having a sample of 78 different hashtags (see Table 1).
  • Hereyoucanseesome sample hashtagsforeachcategorywhichareinluded in our sample. Weendeduphaving 78 hashtagcatgeories.
  • After having selected a diverse sample of hashtags we started the crawling process. We crawled the data for each hastag at 3 different time points. First we retrieved hashtag-stream tweets that were authored within the last week. Next we extracted the authors of those tweets and crawled their SN. Finally, we also crawled the most recent 3,200 tweets (or less if less were available) of all users who belong either to the top hundred authors or audience users of each hashtag stream. We ranked authors by the number of tweets they published within the hashtag stream and audience users by the number of authors they are friends with. We repeated this process three times.
  • To estimate the audience of a hashtag stream, we rank the friends (or followers) of the stream's authors by the number of authors they are related with. In this example, the hashtag stream #football has four authors. User B is a friend of all four authors of the stream and is therefore most likely to be exposed to the messages of the stream and to be able to interpret them. Consequently, user B receives the highest rank. User C is a friend of two authors and receives the second highest rank. The user with the lowest rank (user A) is only the friend of one author of the stream.
  • After havingselectedtheaudienceusers per hashtagstream, we also need a waytoselectthecontentsourcewhichwewanttouselaterforestimatingtheaudience‘s BK. Wetested 4 different approacheshere.
  • Finnaly, after havingselectedthe top audienceusersandtheircontentof a hashtagstreamwe also need a waytoabstractthecontentandrepresentit in form offeatures
  • Toanswerour 1st RQ wecombine different audienceselectionand BK estimationstrategiesandtrain a linear SVM usingmessagesauthoredbytheaudienceat t0 orbeforeastrainingsamples. Weextractfeaturesfromthosemessagesbyusingtfidfandtopicmodel. Wetesttheperformanceoftheclassifiersusingfuturehashtagstreammessages.
  • Ok let‘slookattheresultsfromthisstudy. First notethatsincewehave 78 classes a randomguesserwouldachieve a verylowperformance. Weusedas a baseline a classifiertrainedwith a randomaudience. Thatmeanswerandomlyswappedaudiencesandhashtagstreamsbeforetraining. Thereforewedistroyedtherelationbetweentrainingdataandlabels. A baselineclassifiercanachieve an F1 score of 0.01. Whenusingthe real audienceof a hashtagstream (the top 10 friends) andtheirmostrecentlypublishedmessagesweachive a performanceof 0.25. This issignificvantlybetter.
  • So I showedyoutheoverallperformance so far, but we also lookedattheperformanceof individual categoriesofhashtagsusingthebestmethod (mostrecentmessagesauthoredbythe top audience). Onecanseefromthisthatthereisone negative outlier–idioms. Sincethosearethehashtagswhichare not representingsemanticconceptsweare not so worriedaboutthat.
  • Since we observed that the audience of certain hashtag streams is more useful than for others we wanted to know what types of streams tend to have useful audiences. Therefore we performed a correlation analysis btw… we don’t want to claim that there is a causal relation but we repeated our exp at different time points and observed the same correlations.
  • Toquantifystructuralpropertiesofsocialstreamsweuse a coupleofstaticanddynamicstreammeasures.
  • Across all categories, the overlapmeasures show the highest positive correlation with the F1-scores. This indicates that streams which..
  • The only significant coverage measure is the conversational coverage measure. It indicates that the audiences of conversational streams are better in interpreting the meaning of a stream's messages. This suggests that it is not only important that a community exists around a stream, but also that the community is communicative.
  • All entropy measures show significant negative correlations with the F1-Scores. This shows that the more focused the author-, follower-, followee- and/or friend-distribution of a stream is (i.e., lower entropy), the higher the F1-Scores of an audience-based classification model are. The entropy measures the randomness of a random variable. For example, the author-entropy describes how random the set of authors around a hashtag stream is formed – i.e., how well one can predict who will author the next message in this hashtag stream. The friend-entropy describes how random the friends of hashtag stream's authors are – i.e., how well one can predict who will be a friend of most hashtag stream's authors. Our results suggest that streams tend to have a better audience if their authors and author's followers, followees and friends are less random.
  • The KL divergences of the author-, follower-, and followee-distributions show a significant negative correlation with the F1-Scores. The author distribution of a hashtag stream reveals how many messages each author contributed. The follower distribution reveals how many authors a follower is followiing. The Followee distribution reveals how many authors are following a folllowee.This indicates that the more stable the author, follower and followee distribution is over time, the better the audience of a stream. If for example the followee distribution of a stream changes heavily over time, authors are shifting their social focus. If the author distribution of a stream has a high KL divergence, this indicates that the sets of authors of streams are changing over time.
  • To summerize our work provides initial empirical evidence for the fact that… And we also found that certain stream characteritiscs are strongly correlated with the usefulness of the audience. We do not know if this is a causal relation but we repeated our experiment on a later time point and found compareable results.
  • Currentlyand in thenearfuturewework on comparingtheutilityof .... Andwe also wanttoexploitourresultsfordeveloping a newhashtagrecommendationalgorithm.
  • To understand whether the structure of a stream has an effect on the usefulness of its audience for interpreting the meaning of its messages, we perform a correlation analysis and investigate to what extent the ability of an audience to interpret the meaning of messages correlates with structural stream properties introduced in Section Structural Stream Measures.
  • Eswc2013 audience short

    1. 1. The Wisdom of the Audience: An Empirical Study ofSocial Semantics in Twitter StreamsClaudia Wagner, Philipp Singer, Lisa Posch and Markus Strohmaier10th Extended Semantic Web Conference, Montpellier, 29.5.2013
    2. 2. Problem#music#fashionAuthors make their messages as informative as required but do not providemore information than necessary (Maxim of Quantity by Grice (1975))[src:]
    3. 3. Research Questions3RQ 1: To what extent is the background knowledge of audiences useful foranalyzing the semantics of social media messages?RQ 2: What are the characteristics of an audience which possesses usefulbackground knowledge for interpreting the meaning of a streams messagesand which types of streams tend to have useful audiences?[scr:]
    4. 4. MethodologyMessage Classification TaskUse hashtags as ground truthLaniado and Mika (2010) showed that around half of all hashtags canbe associated with Freebase conceptsCompare real audience with random audience - how well can anaudience predict the hashtag of a tweet?The audience which is better in guessing the hashtag of a Twittermessage is better in interpreting the meaning of the messageNull hypothesis: If the audience of a stream does not possessmore knowledge about the semantics of the streams messagesthan a randomly selected baseline audience, the results fromboth classification models should not differ significantly4
    5. 5. MethodologyTrain different multiclass classifiers on the backgroundknowledge of the audienceLogistic Regression, Stochastic Gradient Descent, Multinomial NaiveBayes and Linear SVMCompare different approaches for estimating thebackground knowledgeDifferent audience and content selection approachesDifferent methods for estimating the background knowledgeTest how well each model can predict the hashtag offuture messagesWeighted Macro F15
    6. 6. DatasetDiverse sample of hashtagsRomero et al. (2011) identified eight categories ofhashtags on a large data samplecelebrity, games, idioms, movies/TV, music, political, sports, andtechnologyWe randomly draw from each category tenhashtags which were still in use6
    7. 7. Dataset7Technology Idioms Sports Politics#blackbery,#iphone, #google#omgfacts,#factsaboutme,#iwish#football, #nfl,#yankees#climate, #iran,#teapartyGames Music Celebrity Movies#gaming,#mafiawars,#wow#lastfm,#eurovision,#nowplaying#bsb,#michaeljackson,#rogis#avatar, #tv,#glennbeck
    8. 8. Dataset8t0 t1 t23/4/2012 4/1/2012 4/29/2012streamtweetscrawl ofsocialstructurestreamtweetscrawl ofsocialstructurestreamtweetscrawl ofsocialstructure1 weekcrawl of audiencetweetscrawl of audiencetweetscrawl of audiencetweetst1 t2 t3Stream Tweets 94,634 94,984 95,105Stream Authors 53,593 54,099 53,750Friends 7,312,792 7,896,758 8,390,143Audience Tweets 29,144,641 29,126,487 28,513,876
    9. 9. Audience SelectionABCAuthorsAudienceRank123StreamTeam bc tryouts tomo#footballWhat we learned thisweek: Chelsea areworking in reverseand Avram is coming#football #soccerWeekend pleeeeasehurrrrry #sanmarcos#footballHoly #ProBowl Imspent for the rest ofthe day. #footballFifa warns Indonesiato clean up its footballor face sanctions#Indonesia #Football
    10. 10. Background KnowledgeContent SelectionRecentThe most recent messages authored by theaudience usersTop Links (plain and enriched)the messages authored by the audience whichcontain one of the top links of that audienceTop Tagsthe messages authored by the audience whichcontain one of the top hashtags of that audience10
    11. 11. Background KnowlegdeRepresentationPreprocessing: remove stopwords, twittersyntax, stemmingRepresent background knowledge of the audiencevia the most likely topics or most important wordsof their messagesBag of Words: TF and TFIDFTopic Models: LDA11
    12. 12. Empirical EvaluationRQ 1: To what extent does the backgroundknowledge of the audience support the semanticannotation of individual messages?Combine audience selection and backgroundknowledge estimation approaches to generatesemantic features of the messages authored by anaudienceTraining data on audience’s messages crawled at t0Test model using messages of the hashtag streamscrawled at t112
    13. 13. Results13F1 (TF-IDF) F1 (LDA)Random Guessing 1/78 1/78Baseline (random audience) 0.01 0.01F1 (TF-IDF) F1 (LDA)Random Guessing 1/78 1/78Baseline (random audience) 0.01 0.01Audience - recent 0.25 0.23F1 (TF-IDF) F1 (LDA)Random Guessing 1/78 1/78Baseline (random audience) 0.01 0.01Audience – recent 0.25 0.23Audience – top links enriched 0.13 0.10Audience – top links plain 0.12 0.10F1 (TF-IDF) F1 (LDA)Random Guessing 1/78 1/78Baseline (random audience) 0.01 0.01Audience – recent 0.25 0.23Audience – top links enriched 0.13 0.10Audience – top links plain 0.12 0.10Audience – top tags 0.24 0.21The audience of a hashtag stream contains knowledgewhich is useful for predicting the hashtags of futuremessages
    14. 14. Results14F1 (TF-IDF) F1 (LDA)Random Guessing 1/78 1/78Baseline (random audience) 0.01 0.01F1 (TF-IDF) F1 (LDA)Random Guessing 1/78 1/78Baseline (random audience) 0.01 0.01Audience - recent 0.25 0.23F1 (TF-IDF) F1 (LDA)Random Guessing 1/78 1/78Baseline (random audience) 0.01 0.01Audience – recent 0.25 0.23Audience – top links enriched 0.13 0.10Audience – top links plain 0.12 0.10F1 (TF-IDF) F1 (LDA)celebrity 0.17 0.15games 0.25 0.22idioms 0.09 0.05movies 0.22 0.18music 0.23 0.18political 0.36 0.33sports 0.45 0.42technology 0.22 0.22
    15. 15. Empirical EvaluationRQ 2: What are the characteristics of anaudience which possesses useful backgroundknowledge for interpreting the meaning of astreams messages and which types of streamstend to have useful audiences?Correlation analysis between the ability of anaudience to interpret the meaning ofmessages and structural properties of thestream15
    16. 16. Structural Stream PropertiesStatic MeasuresCoverage: informational, hashtag, retweet andconversational extent of a streamEntropy: randomness of a streams authors and theirfollowers, followees and friendsOverlap: overlap between authors and followers,authors and followees and authors and friendsDynamic MeasuresKL divergence between the author-, the follower-, andthe friend-distributions of a stream at different timepoints16
    17. 17. Stat. Significant SpearmanRank Correlation (p<0.05)17F1 (TF-IDF) F1 (LDA)Overlap Author-Follower 0.675 0.655Overlap Author-Followee 0.642 0.628Overlap Author-Friend 0.612 0.602Streams which are produced and consumed by acommunity of users who are tightly interconnected tend tohave a useful audience.A useful audience possesses background knowledge whichhelps interpreting the meaning of messages.
    18. 18. Stat. Significant SpearmanRank Correlation (p<0.05)18F1 (TF-IDF) F1 (LDA)Conversation Coverage 0.256 0.256Conversational streams tend to have a usefulaudience.
    19. 19. Stat. Significant SpearmanRank Correlation (p<0.05)19F1 (TF-IDF) F1 (LDA)Entropy Author Distribution -0.270 -0.400Entropy Friend Distribution -0.307 -Entropy Follower Distribution -0.400 -0.319Entropy Followee Distribution -0.401 -0.368Streams which are produced and consumed by afocused set of authors, followers, followees andfriends tend to have a useful audience.
    20. 20. Stat. Significant SpearmanRank Correlation (p<0.05)20F1 (TF-IDF) F1 (LDA)KL Follower Distribution -0.281 -KL Followee Distribution -0.343 -0.302KL Author Distribution -0.359 -0.307Socially stable streams tend to have an audiencewhich is good in interpreting the meaning of astreams messages.
    21. 21. Summary & ConclusionsThe audience of a social stream possesses knowledge whichmay indeed help to interpret the meaning of a streamsmessagesBut not all streams have similar useful audiencesThe audience of a social stream seems to be most useful ifthe stream is created and consumed by a stable, focused andcommunicative community – i.e., a group of users who areinterconnected and have few core users to whom almosteveryone is connectedWe do not know if those relations are causal but we gotsimilar results when repeating our experiments on t1 and t221
    22. 22. Current and Future WorkCompare the utility of ontological knowledge withaudience background knowledge for the hashtagprediction taskAlgorithmic exploitation of our resultsHybrid hashtag recommendation algorithmStructural stream measures may inform weighting (how muchcan we count on the audience?)Differentiate between social and topical hashtagsUser-centric algorithms work only for active users who usedhashtags beforeAn audience-integrated approach only requires an active audience22
    23. 23. ReferencesGrice, H. P. (1975). Logic and conversation. In Speech acts, 3, 41–58. NewYork: Academic Press.Laniado, D., & Mika, P. (2010). Making sense of twitter. In Proceedings ofthe 9th international semantic web conference (pp. 470-485). Shanghai,China.Romero, D. M., Meeder, B., & Kleinberg, J. (2011). Differences in theme-chanics of information diffusion across topics: idioms, political hashtags,and complex contagion on twitter. In Proceedings of the 20th internationalconference on world wide web (pp. 695–704). Hyderabad, India.24
    24. 24. Experimental Setupsrc: YOUclaudia.wagner@joanneum.at[src:]