Workshop Sentiment, Emotion, Demography, and Bot Detection
•
11 likes•2,409 views
This document summarizes a workshop on sentiment, emotion, demography, and bot detection using Drone Emprit Academic, a big data system that analyzes conversations on social media like Twitter. It discusses how Drone Emprit collects Twitter data using keywords and filters, then performs analysis including sentiment analysis using multiple models, emotion analysis using a lexicon approach, demography analysis of user locations, and bot analysis using the Botometer API. Visualizations of the analysis results are also presented.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Workshop Sentiment, Emotion, Demography, and Bot Detection
Similar to Workshop Sentiment, Emotion, Demography, and Bot Detection (20)
More from Ismail Fahmi
More from Ismail Fahmi (20)
Recently uploaded
Recently uploaded (20)
Workshop Sentiment, Emotion, Demography, and Bot Detection
- 1. WORKSHOP: SENTIMENT, EMOTION, DEMOGRAPHY, AND BOT DETECTION Ismail Fahmi, Ph.D. Director PT. Media Kernels Indonesia (a.k.a Drone Emprit) Ismail.fahmi@gmail.com SEMINAR & WORKSHOP DEA UII YOGYAKARTA 19 November 2019 UNIVERSITAS ISLAM INDONESIA ACADEMIC
- 3. ACADEMIC TENTANG DRONE EMPRIT ACADEMIC Drone Emprit Academic adalah sebuah sistem big data yang menangkap dan menganalisis percakapan di media sosial khususnya Twitter, yang dikembangkan oleh PT Media Kernels Indonesia, dan dipasang di data center Badan Sistem Informasi (BSI) Universitas Islam Indonesia. Drone Emprit menggunakan layanan API (Applications Programming Interface) dari Twitter untuk menangkap percakapan secara semi realtime melalui metode streaming. Ismail Fahmi, 2017. Drone Emprit: Konsep dan Teknologi, IT Camp on Big data and Data Mining, Jakarta 3
- 5. ACADEMIC DATA WORKFLOW 5 Data Population (backtrack) Analysis Visualization Sentiment analysis, opinion analysis, bot analysis, demography analysis, etc Keyword dan filter
- 6. ACADEMIC DATA POPULATION 6 Sensus, sp2020, .. Sensus penduduk Keyword à Twitter Sumber daya yang sangat terbatas Filter à Data Lake Membatasi hasil pencarian
- 7. ACADEMIC BERHEMAT DENGAN KEYWORD (DIAGRAM VENN) 7 Sensus, sp2020, .. Sensus penduduk
- 9. ACADEMIC KEYWORDS: SUMBER DAYA YANG SANGAT TERBATAS 9 TWITTER Max 400 keywords Server IP Addr 1 Server IP Addr 2 Server IP Addr n Max 400 keywords Max 400 keywords DRONE EMPRIT ACADEMIC DATA LAKE
- 10. ACADEMICDrone Emprit Big Data Architecture 10 News Crawler Twitter Crawler Twitter Streaming FB Page Crawler Data Pipeline Data SOLR Indexer 1 SOLR Indexer 2 SOLR Indexer 3 SOLR Indexer 4 Hadoop Framework Physical Hardware Insight DataIngest Management&Queue RealtimeJob Processing Google Custom Search Database Framework ScheduledJob Processing Map Reduce Sentiment Analysis Other Processings Data&Workflow Management Access Visualization Other sources Analytics UI
- 12. ACADEMIC Contoh: Free Twitter Search 12 History: 7 days Start search 100% results
- 13. ACADEMIC API: Realtime (Sample) 13 Random SampleAll Statuses Kurang dari 10%
- 15. ACADEMIC API: Realtime (Filter) 15 Filtered StatusesAll Statuses ~ 100% POST statuses/filter Filter max 400 keywords Filter: Max 400 keywords
- 16. ACADEMIC ANALYSIS AND VISUALIZATION 16 Trends DASHBOARD Comparison Topic Map NEWS PORTAL Latest News Media ANALYTICS News Sites Page Ranks Sentiment Analysis PF-Chart Engagement Exposure Retweets TOPICS Replies Most Shared URLs Most Shared Videos Topic Map Word Cloud Impact INFLUENCERS Engagement Reach Most Engaged Followers Influencer Network SNA Topic Network PR-Values Reach Hashtags Posts Bubble Map Twitter User Map DEMOGRAPHY User Locations Edit Sentiments MENTIONS Training & Learning Backtracking Compare SNA COMPARE Compare Projects Popularity vs Favorability Background Jobs Upload Report REPORTING Download Report User Management ADMIN Project Management Client Management Source Management Label and Training OPINION ANALYSIS Opinion Chart Insight Explorer
- 21. ACADEMIC Sentiment Analysis 21 Netral ? MENTIONS Untuk Hakim Cepi Iskandar
- 23. ACADEMIC Evaluasi 23 http://www.sciencedirect.com/science/article/pii/S2090447914000550 ”one model for all” tidak bisa memberi label yang tepat untuk setiap subyek. Lexicon base tergantung dari keberadaan kata dalam kamus sentimen, tidak bisa memberi label yang tepat untuk subyek yang berbeda.
- 24. ACADEMIC Sentiment Analysis Tools 24 https://breakthroughanalysis.com/2012/01/08/what-are- the-most-powerful-open-source-sentiment-analysis-tools/ Text Mining Module
- 25. ACADEMIC Sentiment Analysis: Drone Emprit 25 Adaptive Multiple Models
- 27. ACADEMIC EDIT SENTIMENT LABEL (DEA) 27
- 31. ACADEMIC Plutchik’s Wheel of Emotions 31 https://www.6seconds.org/2017/04/27/plu tchiks-model-of-emotions/
- 32. ACADEMIC EMOTION ANALYSIS DI DEA 32
- 33. ACADEMIC LEXICON BASED ANALYSIS • JOY • senang, bahagia, suka, seneng, hepi, menyenangkan, happy, menggembirakan, gembira, fun, sukacita, riang, ceria, … • TRUST • FEAR • SURPRISE • SADNESS • DISGUST • ANGER • ANTICIPATION 33
- 35. ACADEMIC 35
- 36. ACADEMIC 36
- 38. ACADEMIC DEMOGRAPHY ANALYSIS: DEA • Fitur ini sudah 80% dikembangkan, dan dalam waktu dekat akan ditambahkan ke delam dashboard Drone Emprit Academic. 38
- 40. ACADEMIC BOT ANALYSIS DI DEA 40
- 41. ACADEMIC METODE 41 Botometer Profile + 200 twits Bot scores https://rapidapi.com/OSoMe/api/botometer API
- 42. ACADEMIC HOW IT WORKS • Botometer is a machine learning algorithm trained to classify an account as bot or human based on tens of thousands of labeled examples. • When you check an account, you fetches its public profile and hundreds of its public tweets and mentions using the Twitter API. • This data is passed to the Botometer API, which extracts about 1,200 features to characterize the account's profile, friends, social network structure, temporal activity patterns, language, and sentiment. • Finally, the features are used by various machine learning models to compute the bot scores. 42
- 43. ACADEMIC VISUALISASI HASIL BOT ANALYSIS 43 HUMAN CYBORG ROBOT
- 45. ACADEMIC THANK YOU Ismail Fahmi, PhD.