Getting Insight from Big Data
•Download as PPTX, PDF•
1 like•94 views
The document provides an overview of extracting insights from big data. It discusses sources of big data, the process of extracting information from big data including data acquisition, pre-processing, exploration and modeling, visualization and reporting. It covers characteristics of big and small data, challenges of big data, and methods used in machine learning including clustering, classification, forecasting, and their applications to Twitter data including analysis of people, tweets, and geolocation information. Examples of research using big data from social media are also mentioned.
Recommended
Recommended
More Related Content
Similar to Getting Insight from Big Data
Similar to Getting Insight from Big Data (20)
Recently uploaded
Recently uploaded (20)
Getting Insight from Big Data
- 1. Getting Insight from Big Data Canggih Puspo Wibowo Ujang Fahmi Forum Penelitian Bulanan organized by MKP, FISIPOL, UGM
- 2. Purposes 1. Mengetahui dan megenal sumber-sumber big data 2. Mengetahui proses ekstraksi informasi dari big data 3. Mengetahui metode-metode yang sering digunakan dalam pengelolaan Big Data 4. Melakukan percobaan untuk memahami data, proses, dan hasil pengolahan data melalui dashboard
- 3. What, Why...? Large volumes of data that are produced routinely by organizations and are too complex for standard software packages to process (Mayer-Schonberger & Cukier, 2013). In the United Kingdom, local governments have been identified as one segment of the public sector which can mostly benefit from the systematic exploitation of Big Data, with some commentators suggesting it can help them save up to £25.4 billion over five years (Policy Exchange, 2015).
- 4. Characteristic Small data Big data Data sources Traditional enterprise data, user studies, personal data Social media, sensor data, log data, device data, video, images Volume Up to Gigabytes Terabytes or more Velocity Slow: batch to real time, not always a fast response is needed Fast: often real time, immediate response needed Variety Structured, semi-structured, unstructured Unstructured, structured, multi-structured Veracity Easier Harder Value Business intelligence, analysis and reporting Complex, advanced, predictive business analysis and insights Scope Partial: small populations, samples Exhaustive: continuous streams, entire populations Privacy Usually private Private but also anonymous Density Coarse to dense Dense Identity Weak to Precise Precise Relations Weak to strong Strong Flexibility Low to Middle High Small vs BIG data
- 6. Information extraction STRUCTURED/ UNSTRUCTURED DATA OFFLINE SOURCES PROBLEM/ CURIOSITY ONLINE SOURCES ARTIFICIAL INTELLIGENCE
- 7. AI vs ML vs DL vs DS Artificial Intelligence Machine Learning Deep Learning Data Science
- 8. Symbolists • Source: Logic, Philosophy • Best: Knowledge Composition • Ex: Decision Tree, Inverse Deduction, etc Connectionists • Source: Neuroscience • Best: Credit Assignment • Ex: Neural Network, Deep Learning, etc Bayesians • Source: Statistics • Best: Uncertainty • Ex: Hidden Markov Model, Graphical Model, etc Evolutionaries • Source: Evolutionary Biology • Best: Structure Discovery • Ex: Genetic Algorithm, Evolutionary Programming, etc Analogizers • Source: Psychology • Best: Similarity • Ex: k-Nearest Neighbors, Support Vector Machine, etc 5 Tribes of Machine Learning Techniques
- 9. Machine Learning Application Clustering Classification Forecasting
- 10. Clustering http://mropengate.blogspot.com/2015/06/ai-ch16-5-k-introduction-to-clustering.html Ex: - kNN - K-Means - DBSCAN - Agglomerative Clustering - Affinity Propagation - etc
- 11. Classification - Training ++ + + + + + + + --- - - - - -- - - ++ + + + + + + + --- - - - - -- - - classifier
- 12. Classification - Predicting ? + Ex: - Decision Tree - Naïve Bayes - SVM - ANN - Logistic Regression - Etc. many more
- 16. People Network Social Network Analysis Network Centrality Community Structure
- 17. People Network Network Centrality Centrality Degree Eigenvector Betweenness Closeness
- 18. People Network Centrality Degree Centrality “who has the most friends”
- 19. People Network Centrality Eigenvector Centrality “who has the most influence”
- 20. People Network Centrality Betweenness Centrality “who is the bridge”
- 21. People Network Centrality Closeness Centrality “who is the hub”
- 26. People Time-Series - Daily tweet count - Daily positive sentiment count - etc Trend Discovery Methods - Regression (linear, polynomial, multiple)
- 27. People Time-Series - Predict negative tweet count next week - etc Forecasting Methods - ARIMA
- 30. Tweet Clustering Sample Tweets Mas @ujang, ayo main PES Dulu pas masih murah gak mau, sekarang? Duet CR-Dybala kerenn bangett #forzajuve Gaji tenaga pendidik memang harus naik.. Cuma aku kah yang gak nonton avenger? CBOW lebih mudah konvergen dibanding Skipgram? Skipgram dengan 100 vektor data sudah cukup kok, kebanyakan nanti bikin lambat Cluster A Cluster B Cluster C
- 31. Tweet Classification Sample Tweets Mas @ujang, ayo main PES Dulu pas masih murah gak mau, sekarang? Duet CR-Dybala kerenn bangett #forzajuve Gaji tenaga pendidik memang harus naik.. Cuma aku kah yang gak nonton avenger? CBOW lebih mudah konvergen dibanding Skipgram? Skipgram dengan 100 vektor data sudah cukup kok, kebanyakan nanti bikin lambat Bola Lain-lain Komputer
- 32. Tweet Classification Sample Tweets Mas @ujang, ayo main PES Dulu pas masih murah gak mau, sekarang? Duet CR-Dybala kerenn bangett #forzajuve Gaji tenaga pendidik memang harus naik.. Cuma aku kah yang gak nonton avenger? CBOW lebih mudah konvergen dibanding Skipgram? Skipgram dengan 100 vektor data sudah cukup kok, kebanyakan nanti bikin lambat Positive Negative Sentiment Neutral
- 33. Tweet Topic Model Sample Tweets Mas @ujang, ayo main PES Dulu pas masih murah gak mau, sekarang? Duet CR-Dybala kerenn bangett #forzajuve Gaji tenaga pendidik memang harus naik.. Cuma aku kah yang gak nonton avenger? CBOW lebih mudah konvergen dibanding Skipgram? Skipgram dengan 100 vektor data sudah cukup kok, kebanyakan nanti bikin lambat Topic A: bola, game, main Topic B: film, main, uang Topic C: vektor, skipgram, cbow Topic D: … ALL Tweets Approach
- 34. Tweet Topic Model Sample Tweets Mas @ujang, ayo main PES Dulu pas masih murah gak mau, sekarang? Duet CR-Dybala kerenn bangett #forzajuve Gaji tenaga pendidik memang harus naik.. Cuma aku kah yang gak nonton avenger? CBOW lebih mudah konvergen dibanding Skipgram? Skipgram dengan 100 vektor data sudah cukup kok, kebanyakan nanti bikin lambat Topic each tweet Topic A, Topic E, Topic B, .. Topic C, Topic A, Topic F, .. Topic A, Topic B, Topic C, … … … … … EACH Tweet Approach
- 37. Geolocation Longitude, Latitude Tweet Lat Long ….. -7.538873 110.785338 ….. …..
- 38. Geolocation Address Tweet Location ….. Yogyakarta ….. Aceh ….. Jawa Tengah
- 39. Research Health Modeling Spread of Disease from Social Interactions Sadilek, A., Kautz, H. A., & Silenzio, V. (2012, June). Modeling Spread of Disease from Social Interactions. In ICWSM (pp. 322- 329).
- 40. Research Linguistic Diffusion of Lexical Change in Social Media Eisenstein, J., O'Connor, B., Smith, N. A., & Xing, E. P. (2014). Diffusion of lexical change in social media. PloS one, 9(11), e113114.
- 41. Research Psychology Modeling Public Mood and Emotion: Twitter Sentiment and Socio- Economic Phenomena Bollen, J., Mao, H., & Pepe, A. (2011). Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. Icwsm, 11, 450-453.
- 42. Bollen, J., Mao, H., & Zeng, X. (2011). Twitter mood predicts the stock market. Journal of computational science, 2(1), 1-8. Research Economy Twitter mood predicts the stock market