Download as PDF, PPTX
![FB API (v2): Object
39
https://graph.facebook.com/$object_id/$type?
fields=id,
parent_id,
from,
to,
type,
status_type,
story,
message,
link,
likes.summary(true),
shares,
comments.order(reverse_chronological).summary(true),
created_time,
updated_time
&order=reverse_chronological
&access_token=$access_token&limit=$limit&until=$last_timestamp
$object_id = FB Page ID, etc
$type = [feed, comment, ...]](https://image.slidesharecdn.com/itcamp-drone-emprit-konsep-okt2017-171001135646/85/Drone-Emprit-Konsep-dan-Teknologi-39-320.jpg)
Uploaded byIsmail Fahmi
Drone Emprit: Konsep dan Teknologi
Dr. Ismail Fahmi gave a presentation about Drone Emprit, a natural language processing company he founded. Drone Emprit develops a big data system to monitor and analyze online and social media using technologies like machine learning and sentiment analysis. The system crawls data from sources like news sites, Twitter, Facebook and indexes the data. It then performs analytics like sentiment analysis, opinion analysis, social network analysis and visualizes the results. Examples of case studies analyzed include elections in West Java and analysis of pro-contra PKI issues.
Drone Emprit: Konsep dan Teknologi
- 1. Drone Emprit Konsep danTeknologi Ismail Fahmi, PhD. Drone Emprit Media Kernels Indonesia Ismail.fahmi@gmail.com IT CAMP – BIG DATA & DATA MINING Onno Center, Situ Gintung - Jakarta 1 Oktober 2017
- 2. 2 1992 – 1997S1, Teknik Elektro, ITB 2003 – 2004 S2, Computational Linguistics, Universitas Groningen, Belanda 2004 – 2009 S3, Computational Linguistics, Universitas Groningen, Belanda 2000 – 2003 Inisiator IndonesiaDLN (Digital Library Network pertama di Indonesia) Mengembangkan Ganesha Digital Library (GDL) Mendirikan Knowledge Management Research Group (KMRG) ITB Membangun Digital Library ITB 2009 – Sekarang Engineer di Weborama, Perusahaan berbasis big data (Paris/Amsterdam) 2012 – Sekarang Co-Founder Awesometrics, Media Monitoring & Analytics Company 2014 – Sekarang Founder PT. Media Kernels Indonesia, a Natural Language Processing Company 2015 – Sekarang Konsultan Perpustakaan Nasional, Inisiator Indonesia OneSearch 2017 – Sekarang Dosen Tetap Magister Teknik Informatika Universitas Islam Indonesia Ismail Fahmi, PhD. Ismail.fahmi@gmail.com
- 3. Agenda SESI 1 • Konsep •Tentang Drone Emprit • Data, tambang emas baru • Arsitektur & Fitur • Teknologi • Crawler • Twitter • Facebook • Online News • Indexing • Sharding • Replication • Analytics • Sentiment Analysis • Opinion Analysis • Term Extraction • Clustering • Social Network Analysis • Visualisasi SESI 2 • Studi Kasus • Analisis Pilkada Jawa Barat • Analisis Pro-Kontra PKI • Membaca Agenda Setting Media • Demo • Membuat topik monitoring baru • Membaca hasil analisis • Edit sentimen • Social Network Analysis 3
- 4.
- 5. Media Kernels a.k.aDrone Emprit • Sebuah sistem untuk memonitor dan menganalisa media online dan sosial berbasis teknologi big data. • Dikembangkan sejak tahun 2009 di Amsterdam, Belanda, oleh anak bangsa, melalui Media Kernels Netherlands B.V. • Mulai tahun 2012 digunakan di Indonesia. • Berbasis teknologi Artificial Intelligent (Machine Learning) dan Natural Language Processing (NLP). • Dikenal sebagai ‘Drone Emprit’ dalam berbagai pemberitaan di TV dan Media Nasional. 5 Drone Emprit
- 6. 6 2-8 Januari 2017 TEMPO Topik:Peternakan hoax di media sosial Media Kernels: • Diberitakan dengan name ‘Drone Emprit’. • Menyajikan peta Social Network Analysis (SNA) tentang bagaimana sebuah hoax berasal, menyebar, siapa influencers utama, dan siapa groupnya. • Beberapa isu yang dianalisis: 10 Juta Tenaga Kerja China, dan Aleppo (ISIS). LAPORAN UTAMA TEMPO, 2-8 Januari 2017
- 7. confidential 7 12 Januari 2017 KANTORSTAF PRESIDEN Kasus: Isu hoax menyerang pemerintah tentang 10 Juta Tenaga Kerja China Illegal. Media Kernels: • Menyajikan dua studi kasus: 10 Juta tenaga kerja china illegal, dan sentimen negatif terhadap gerakan anti hoax. • Menunjukkan timeline resonansi isu, dan peta percakapan dengan fitur SNA. • Menunjukkan kurang efektifnya komunikasi pemerintah, dan apa yang bisa dilakukan untuk perbaikan. FGD KEHUMASAN SELURUH KEMENTERIAN DAN LEMBAGA DI KANTOR STAF PRESIDEN (KSP)
- 8. confidential 8 22 Maret 2017 MATANAJWA Kasus: Virus Dusta (alias Hoax) Nara Sumber: • Stanley (Dewan Pers) • Johan Budi (Stafsus Presiden) • Boy Rafli (Humas Polri) • Ismail Fahmi (MK) • Septiaji & Khairul Anshar (Masy. Anti Hoax) Media Kernels: • Menyajikan analisis ttg 10 Juta Tenaga Kerja China Illegal. • Hoax Panglima TNI vs PKI. MATA NAJWA LIVE ‘VIRUS DUSTA’
- 9. Data is NewGold 9
- 10.
- 11. Data Collection: Gold= Expensive 11
- 12.
- 13. Twitter Analysis: WorldEco. Forum 2016 13 https://medium.com/@swainjo/wef16-davos-twitter-sna-analysis-4c38cf4bc46d
- 14.
- 15.
- 16. MK Big DataArchitecture confidential 16 News Crawler Twitter Crawler Twitter Streaming FB Page Crawler Data Pipeline Data SOLR Indexer 1 SOLR Indexer 2 SOLR Indexer 3 SOLR Indexer 4 Hadoop Framework Physical Hardware Insight DataIngest Management&Queue RealtimeJob Processing Google Custom Search Database Framework ScheduledJob Processing Map Reduce Sentiment Analysis Other Processings Data&Workflow Management Access Visualization Other sources Analytics UI
- 17. 17 Social Media Twitter Facebook Search+JSON Detik (ID) Reuters(EN) Etc.. RSS+HTML Gatra (ID) Bloomberg (EN) Etc.. HTML Kaskus Detik Forum Etc.. HTML Online News Forums Twitter StreamJSON Kompas TEXT Warta Ekonomi Etc.. Print PUSHJSON Subscriber Projects Storage Search + Account Crawler RSS + HTML Crawler HTML Crawler HTML Crawler SOLR Nodes Shard 1 SOLR Nodes Shard N Index Servers Redis Queue Cache Manager Mentions Storage Keywords + Accounts Filters deletes Sentiment Analysis Sentiment Models Backtrack Filters Sentiment Analysis Analyses Control Room Screens Smart phones, tablets Desktops Client(s) Converter System Architecture
- 18. Fitur-fitur Media Kernels confidential 18 Trends DASHBOARD Comparison TopicMap NEWS PORTAL Latest News Media ANALYTICS News Sites Page Ranks Sentiment Analysis PF-Chart Engagement Exposure Retweets TOPICS Replies Most Shared URLs Most Shared Videos Topic Map Word Cloud Impact INFLUENCERS Engagement Reach Most Engaged Followers Influencer Network SNA Topic Network PR-Values Reach Hashtags Posts Bubble Map Twitter User Map DEMOGRAPHY User Locations Edit Sentiments MENTIONS Training & Learning Backtracking Compare SNA COMPARE Compare Projects Popularity vs Favorability Background Jobs Upload Report REPORTING Download Report User Management ADMIN Project Management Client Management Source Management Label and Training OPINION ANALYSIS Opinion Chart Insight Explorer
- 19.
- 20. Online News 20 Dan RatusanMedia Non-mainstream
- 21. Crawling Online News 21 CrawlerIndeks Server
- 22.
- 23. Web Crawler Tools(2) 23 http://bigdata-madesimple.com/top-50-open-source- web-crawlers-for-data-mining/
- 24.
- 25. Web Crawler DroneEmprit 25 Bikin sendiri, powered by:
- 26. Anatomi: Metadata danFullteks 26 Ambil: Tanggal, judul, isi berita, penulis, url gambar Buang: Iklan, daftar headline, komentar.
- 27.
- 28.
- 29. Contoh: Free TwitterSearch 29 History: 7 days Start search 100% results
- 30. API: Realtime (Sample) 30 RandomSampleAll Statuses Kurang dari 10%
- 31.
- 32. API: Realtime (Filter) 32 FilteredStatusesAll Statuses ~ 100% POST statuses/filter Filter max 400 keywords Filter: Max 400 keywords
- 33. API: > 400keywords? 33 All Statuses Max 400 keywords Server IP Addr 1 Server IP Addr 2 Server IP Addr n Max 400 keywords Max 400 keywords
- 34.
- 35. Twitter API: DroneEmprit 35 Net::Twitter AnyEvent::Twitter::Stream
- 36.
- 37. FB API (v1):Public Search 37 April 2014 à distop Facebook
- 38. FB API (v2):Searching 38
- 39. FB API (v2):Object 39 https://graph.facebook.com/$object_id/$type? fields=id, parent_id, from, to, type, status_type, story, message, link, likes.summary(true), shares, comments.order(reverse_chronological).summary(true), created_time, updated_time &order=reverse_chronological &access_token=$access_token&limit=$limit&until=$last_timestamp $object_id = FB Page ID, etc $type = [feed, comment, ...]
- 40.
- 41. FB API: DroneEmprit 41 WWW::Curl Bikin sendiri, powered by:
- 42. Question: Perl orPython? 42 Of course!
- 43. Why Perl? 43 Perl yangmenolong manusia setelah jatuh di bumi, dan tentu lebih ‘nyunah’ Python yang bikin Adam-Hawa tergoda, lalu turun dari surga
- 44.
- 45. Full Text Indexing 45 DataSources Search Engine
- 46. Full Text SearchEngines 46
- 47. Search Engine: DroneEmprit 47 Simple - Powerful - Robust - Scalable
- 48.
- 49.
- 50.
- 51.
- 52. Analytics: Server Configuration 52 SlaveAnalysis Results Analysis Processes
- 53. Analytics Engine 53 Search by Keywords News,Twits, Statuses, etc Sentiment Analysis Opinion Analysis Term Extraction Segmentation Quote Extraction Named Entity Recognition Search Results
- 54.
- 55.
- 56.
- 57.
- 58.
- 59. Sentiment Analysis 59 Netral ? MENTIONS Untuk HakimCepi Iskandar
- 60.
- 61. Evaluasi 61 http://www.sciencedirect.com/science/article/pii/S2090447914000550 ”one model forall” tidak bisa memberi label yang tepat untuk setiap subyek. Lexicon base tergantung dari keberadaan kata dalam kamus sentimen, tidak bisa memberi label yang tepat untuk subyek yang berbeda.
- 62.
- 63. Sentiment Analysis: DroneEmprit 63 Adaptive Multiple Models
- 64.
- 65.
- 66.
- 67. Bersama DivHumas Polridi Kompas Petang 67
- 68.
- 69.
- 70. Membaca Voice, bukanNoise 70
- 71. Analisis Terpengaruh Noise 71 Sayang,analisis berbasis ‘noise’ ini yang menjadi viral.
- 72. Opinion Analysis Techniques 72 DroneEmprit Regular Expression Opinion Analysis
- 73.
- 74.
- 75. Quote Extraction: DroneEmprit 75 Pattern Matching dengan Regular Expression
- 76.
- 77. Named Entity Recognition 77 LOCATIONPERSON ORGANIZATION
- 78.
- 79.
- 80.
- 81.
- 82.
- 83.
- 84.
- 85. Topic Map: DocumentClustering 85
- 86.
- 87. SNA: Social NetworkAnalysis • SNA adalah pemetaan terhadap relasi antar orang, organisasi, topik, lokasi, dan entitas informasi lainnya. • Node atau titik di dalam jaringan menggambarkan orang, organisasi, lokasi, atau entitas informasi. • Garis sambungan antar titik menggambarkan relasi antar titik. 87
- 88. Betweenness Centrality 88 Betweenness Centrality: ameasure of centrality. Highest betweenness centrality (8 connections) Lowest betweenness centrality (4 connections)
- 89.
- 90.
- 91.
- 92. Link Functions: Retweet/ Mention 92
- 93.
- 94.
- 95.
- 96. 96 Information arbitrage: translate informationacross groups
- 97.
- 98.
- 99.
- 100.
- 101. Drone Emprit isHiring 101 System Administrator & Programmer
- 102. Terimakasih 102 Ismail Fahmi, PhD DroneEmprit PT Media Kernels Indonesia Email: ismail.fahmi@gmail.com Hp: 0812 8908 3894