Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Detection of Football Spoilers on Twitter

124 views

Published on

This is a presentation slide in CollabTech2018.

  • Be the first to comment

  • Be the first to like this

Detection of Football Spoilers on Twitter

  1. 1. 1
  2. 2. SNSs SNSs are widely prevalent as a communication tool. 2
  3. 3. SNSs SNSs are widely prevalent as a communication tool. 3 For example Message exchange, browsing a friend's doing now and so on.
  4. 4. Sharing Events in Real-time 4
  5. 5. Sharing Events in Real-time 5
  6. 6. Sharing Events in Real-time 6
  7. 7. Spoiler Problem Recording 7 I have to go to work!!
  8. 8. Spoiler Problem Going back to home... 8 I wonder what Sam is doing now.
  9. 9. Spoiler Problem Spoiler!! 9 Lando Calrissian is return!!!!! Obi-Wan Kenobi is appear!!! Optimistic people want to hear the bad news first, while pessimists ask for the good. Realists just start drinking. I don’t have a dirty mind, I have a sexy imagination.
  10. 10. Spoiler Problem Sports is very shocking. 10 Not the best result we expected. Bélgica 0-0 Japón Anything could happen. Tonight is the night. I’ll keep my fingers crossed for Belgian always look down Japanese. This is kalma.
  11. 11. Example(2) By another interested event 11 The festival seems interesting!
  12. 12. Example(2) They will easily encounter spoilers. 12
  13. 13. Example(3) By push notifications 13 @you My baby has come!!. Tapped!
  14. 14. Example(3) Spoiler... 14
  15. 15. Example(4) Recording 15 I'll go to bed now. VS 3:00 AM in Japan
  16. 16. Example(4) In the next morning, half asleep... 16
  17. 17. Example(4) Spoiler!! 17 Not the best result we expected. Bélgica 0-0 Japón Anything could happen. Tonight is the night. I’ll keep my fingers crossed for Belgian always look down Japanese. This is kalma.
  18. 18. Example(5) Because of time difference 18
  19. 19. Example(5) An example of time difference spoiler. 19
  20. 20. Solution for Spoiler Problem People sometimes stay away from the Internet and choose a self imposed isolation. 20
  21. 21. Solution for Spoiler Problem People sometimes stay away from the Internet and choose a self imposed isolation. It affects the maintenance of relationships with their friends. 21
  22. 22. Solution for Spoiler Problem People sometimes stay away from the Internet and choose a self imposed isolation. It affects the maintenance of relationships with their friends. 22 We need automatic method for hiding spoilers.
  23. 23. Related Researches Spoiler Alert targeted movie reviews and conducted an evaluation of machine learning approaches to find spoilers in social media posts. Spoiler detection in TV program tweets proposed a method of detecting spoilers in comments on Twitter about television programs. [Boyd-Graber et al. 2013] [Jeon et al. 2015] 23
  24. 24. Related Researches Spoiler Alert targeted movie reviews and conducted an evaluation of machine learning approaches to find spoilers in social media posts. Spoiler detection in TV program tweets proposed a method of detecting spoilers in comments on Twitter about television programs. [Boyd-Graber et al. 2013] [Jeon et al. 2015] 24 How to detect sports spoilers on SNSs with high accuracy was insufficient
  25. 25. [Our goal in this work] Examining methods for detecting football spoilers with high accuracy on Twitter. 25
  26. 26. Contents 1. Generating spoiler dataset 2. Experiment of spoiler detection 26
  27. 27. Generating Spoiler Dataset 27 We collect tweets of 9 games by the Japan national football team. Match Score Day 2015 Women’s World Cup “Japan vs. England” JPN 2 – 1 ENG 07/01/15 2015 Women’s World Cup “Japan vs. United States” JPN 2 – 5 USA 07/05/15 2015 EAFF East Asian Cup “Japan vs. South Korea” JPN 1 – 1 KOR 08/05/15 2015 Women’s EAFF East Asian Cup “Japan vs. China” JPN 2 – 0 CHN 08/08/15 2015 EAFF East Asian Cup “Japan vs. China” JPN 1 – 1 CHN 08/09/15 World Cup Qualifiers “Japan vs. Cambodia” JPN 3 – 0 KHM 09/03/15 World Cup Qualifiers “Japan vs. Afghanistan” JPN 6 – 0 AFG 09/08/15 Friendlies “Japan vs. Iran” JPN 1 – 1 IRI 10/13/15 World Cup Qualifiers “Japan vs. Singapore” JPN 3 – 0 SIN 11/12/15
  28. 28. Generating Spoiler Dataset 28 5 labelers helped us label the tweets. Evaluation system
  29. 29. Generating Spoiler Dataset 29 5 labelers helped us label the tweets. Evaluation system
  30. 30. Generating Spoiler Dataset 30 5 labelers helped us label the tweets. 1000 tweets × 9 games × 5 labelers = 45000 data Evaluation system
  31. 31. Some Examples of The Dataset We defined spoiler’s as tweets labeled more than half number. We got 1651 spoiler tweets. 31 Tweet Elapsed time Label “Ooooh! Kagawa scores a goal!!!” 20 spoiler “Already allowed two goals (´Д` )” 0 spoiler “Now kick off” 0 non-spoiler “Hmm. A missed pass is no good” 20 non-spoiler “I saw a sweeping victory for the first time in a very long time” 120 spoiler
  32. 32. Some Examples of The Dataset We defined spoiler’s as tweets labeled more than half number. We got 1651 spoiler tweets. 32 Tweet Elapsed time Label “Ooooh! Kagawa scores a goal!!!” 20 spoiler “Already allowed two goals (´Д` )” 0 spoiler “Now kick off” 0 non-spoiler “Hmm. A missed pass is no good” 20 non-spoiler “I saw a sweeping victory for the first time in a very long time” 120 spoiler Words themselves are like spoiler...
  33. 33. Terms Used Frequently in Spoiler 33 Winning Losing Tying Terms TF-IDF Terms TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 Goal 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 ※[Player]: player names, [Team]: team names, [Num]: number
  34. 34. Terms Used Frequently in Spoiler 34 Winning Losing Tying Terms TF-IDF Terms TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 Goal 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Some terms have the characteristics of spoiler directly. ※[Player]: player names, [Team]: team names, [Num]: number
  35. 35. Terms Used Frequently in Spoiler 35 Winning Losing Tying Terms TF-IDF Terms TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 Goal 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 ※[Player]: player names, [Team]: team names, [Num]: number
  36. 36. Terms Used Frequently in Spoiler 36 Winning Losing Tying Terms TF-IDF Terms TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 Goal 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 ※[Player]: player names, [Team]: team names, [Num]: number
  37. 37. Terms Used Frequently in Spoiler 37 Winning Losing Tying Terms TF-IDF Terms TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 Goal 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 The terms of spoilers differed depending on the game status.
  38. 38. Experiment of Spoiler Detection We examined methods for detecting spoilers with high accuracy. Compared methods 3 word based methods Evaluation points Precision, Recall, F-measure 38
  39. 39. Game 1: Game 2: Game 3: Our Proposed method(SVM with Status of Match) 0-1 0-2Kick-off 1-0 1-11-0 Full time 39 The method uses different learning models of SVM depending on game status.
  40. 40. Game 1: Game 2: Game 3: Our Proposed method(SVM with Status of Match) 0-1 0-2Kick-off 1-0 1-11-0 Full time Winning model Winning model 40 The method uses different learning models of SVM depending on game status.
  41. 41. Game 1: Game 2: Game 3: Our Proposed method(SVM with Status of Match) 0-1 0-2Kick-off 1-0 1-11-0 Full time Losing model Losing model Winning model Winning model 41 The method uses different learning models of SVM depending on game status.
  42. 42. Game 1: Game 2: Game 3: Our Proposed method(SVM with Status of Match) 0-1 0-2Kick-off 1-0 1-11-0 Full time Tying model Tying model Tying model Tying model Losing model Losing model Winning model Winning model 42 The method uses different learning models of SVM depending on game status.
  43. 43. Game 1: Game 2: Game 3: Our Proposed method(SVM with Status of Match) 0-1 0-2Kick-off 1-0 1-11-0 Full time 1-0 Tying model Tying model Tying model Tying model Losing model Losing model Winning model Winning model Tying model Winning model Detect Detect 43 Target Game: The method uses different learning models of SVM depending on game status. (A system monitors game status)
  44. 44. Methods Used in Experiment 3 word-based methods (player names, team names, number are normalized) Pattern matching (from Jeon’s research) Tweets that containing spoiler keywords were judged as spoilers Spoiler keywords were terms with 0.100 or higher of a TF-IDF value SVM (from Jeon’s research) The SVM model detected spoilers Features for SVM were generated using a Bag of Words of each tweet SVM with Status of Match (Our proposed method) The SVM model detected spoilers Features for SVM were generated using the Bag of Words of each tweet 44 [Jeon et al. 2015] [Jeon et al. 2015]
  45. 45. Bag of Words We conducted word division After it, we selected noun, verb, adjective, adverb, interjection and generated the Bag of Words’ features 45 Tweet Japan win fight Japan is winning 1 1 0 Fight! Japan!! 1 0 1 For example Features for SVM of tweets about “Japan is winning” and “Fight! Japan!!”
  46. 46. How to calculate the result value 46 Game 9Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Game 8
  47. 47. How to calculate the result value 47 Game 9Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Game 8 spoiler keywords or SVM model Generate
  48. 48. How to calculate the result value 48 Game 9Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Game 8 Detect spoiler keywords or SVM model
  49. 49. How to calculate the result value 49 Game 9Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Game 8 Detect Value 1 spoiler keywords or SVM model
  50. 50. How to calculate the result value 50 Game 9Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Game 8 Generate Value 1 spoiler keywords or SVM model
  51. 51. How to calculate the result value 51 Game 9Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Game 8 Detect Value 1 spoiler keywords or SVM model
  52. 52. How to calculate the result value 52 Game 9Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Game 8 Detect Value 2Value 1 spoiler keywords or SVM model
  53. 53. How to calculate the result value 53 Game 9Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Game 8 Value 3 Value 4 Value 5 Value 6 Value 7 Value 8 Value 9Value 2Value 1
  54. 54. How to calculate the result value 54 Game 9Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Game 8 9 = The result value Value 3 Value 4 Value 5 Value 6 Value 7 Value 8 Value 9Value 2Value 1
  55. 55. Result 55 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611
  56. 56. The evaluation points Precision = 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒔𝒑𝒐𝒊𝒍𝒆𝒓′ 𝒔 𝒅𝒆𝒕𝒆𝒄𝒕𝒆𝒅 𝒄𝒐𝒓𝒓𝒆𝒄𝒕𝒍𝒚 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒅𝒆𝒕𝒆𝒄𝒕𝒆𝒅 𝒕𝒘𝒆𝒆𝒕𝒔 Recall = 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒔𝒑𝒐𝒊𝒍𝒆𝒓′ 𝒔 𝒅𝒆𝒕𝒆𝒄𝒕𝒆𝒅 𝒄𝒐𝒓𝒓𝒆𝒄𝒕𝒍𝒚 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒔𝒑𝒐𝒊𝒍𝒆𝒓 𝒕𝒘𝒆𝒆𝒕𝒔 F-measure = 2・(𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧)・(𝐫𝐞𝐜𝐚𝐥𝐥) (𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧)+(𝐫𝐞𝐜𝐚𝐥𝐥) 56
  57. 57. Result 57 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611
  58. 58. Result 58 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611
  59. 59. Result 59 Pattern matching often detect a spoiler only from the player name. SVM related detect a spoiler not only from the player name. e.g.: “Kagawa’s missed pass is scary because of the heavy turf” Pattern matching: Spoiler SVM related: Non-spoiler Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611
  60. 60. Result 60 Pattern matching often detect a spoiler only from the player name. SVM related detect a spoiler not only from the player name. e.g.: “Kagawa’s missed pass is scary because of the heavy turf” Pattern matching: Spoiler SVM related: Non-spoiler Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611
  61. 61. Result 61 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611
  62. 62. Result 62 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611
  63. 63. Result 63 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Mistakenly learned tweets by SVM were no longer learned for every time zone by SVM with Status of Match. e.g.: “I saw a national team match for the first time in a very long time!!” SVM: Spoiler SVM with Status of Match: Non-spoiler
  64. 64. Result 64 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Mistakenly learned tweets by SVM were no longer learned for every time zone by SVM with Status of Match. e.g.: “I saw a national team match for the first time in a very long time!!” Cause: Mistakenly learned tweets such as “I saw a sweeping victory for the first time in a very long time”
  65. 65. Result 65 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Mistakenly learned tweets by SVM were no longer learned for every time zone by SVM with Status of Match. Ex) “I saw a national team match for the first time in a very long time!!” Cause: Mistakenly learned of “I saw a sweeping victory for the first time in a very long time” The detection accuracy can be improved by considering the statuses of matches.
  66. 66. Result 66 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611
  67. 67. Result 67 We don’t set up the standard for labeling. e.g.: “Nagatomo got a cramp!” was judged to be spoilers in the dataset Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611
  68. 68. Generating new spoiler dataset 68 A labeler predicts the game‘s result. (We defined more confident tweet is more crucial spoiler) Choose spoilers ↓ Predict the results and it’s confidence Not sureLoseDrawWinPrediction: Confidence: Not sureLoseDrawWinPrediction: Confidence: I wonder if Haaril has an idea. Japan is netted the rebound. Loooooooooong pass [120 minutes elapsed] [40 minutes elapsed] [20 minutes elapsed]
  69. 69. Result(new dataset) 69 Method Precision Recall F-measure SVM with Status of Match (old dataset) 0.698 0.565 0.611 SVM with Status of Match (new dataset) 0.831 0.880 0.852 We regarded tweets about 50 or more average confidence as spoiler and less than 50 average confidence as non-spoiler.
  70. 70. Result(new dataset) 70 Method Precision Recall F-measure SVM with Status of Match (old dataset) 0.698 0.565 0.611 SVM with Status of Match (new dataset) 0.831 0.880 0.852 We regarded tweets about 50 or more average confidence as spoiler and less than 50 average confidence as non-spoiler.
  71. 71. Result(new dataset) 71 Method Precision Recall F-measure SVM with Status of Match (old dataset) 0.698 0.565 0.611 SVM with Status of Match (new dataset) 0.831 0.880 0.852 We regarded tweets about 50 or more average confidence as spoiler and less than 50 average confidence as non-spoiler. If judging only more crucial spoiler, it can be judged with more 0.8 accuracy.
  72. 72. Application to real-world contents 72 Non-spoiler digestNormal digest
  73. 73. Application to real-world contents 73 Number of spoilers? time eliminate eliminate Non-spoiler digest
  74. 74. Current Problem Precision was about 0.5 if keeping high recall. 74 Precision-Recall curve
  75. 75. Current Problem Precision was about 0.5 if keeping high recall. 75 Future task: We need to think other methods to design a high-recall model first and then create models realizing higher precision. Precision-Recall curve
  76. 76. Findings 1. Terms themselves have the characteristics of spoiler and the terms of spoilers differed depending on the game status. 2. SVM with Status of Match method is the best about the detection accuracy. 3. If judging only more crucial spoiler, it can be judged with more 0.8 accuracy. 76
  77. 77. Summarize Examining methods for detecting spoilers with high accuracy on Twitter.  The detection accuracy can be improved by considering the statuses of matches.  If judging only more crucial spoiler, it can be judged with more 0.8 accuracy. [Future works]  Improvement of detection method.  Implementation of Twitter client. 77

×