slide ini membahas dari penggunaan Regular Expression, Classification, Topic Modeling, Text Similarity.

1. Penerapan Text Mining
menggunakan Python
Andreas Chandra
linkedin.com/in/chandraandreas

2. Content
● Pengenalan Text Mining
● Bekerja dengan Regular Expression
● Dasar - dasar Natural Language Processing
● Text Classification
● Topic Modeling

3. Pengenalan Text Mining

4. Background

5. Hal yang bisa dilakukan dengan teks
● Menguraikan teks
● Ekstrak informasi dari teks
● Mengklasifikasi dokumen teks
● Mencari teks dokumen yang relevan
● Analisis sentimen

6. Hirarchy
Dokumen
Kalimat
Kata
Karakter

7. Python - string functions
● len(text)
● word.istitle()
● word.startwith(<letter>)
● word.endwith(<letter>)
● set(text_list)

8. Python - String Operations
● string.lower(); string.upper(); string.titlecase()
● string.split(<spliter>)
● string.join()
● string.strip()
● string.find(<word>)

9. Bekerja Dengan RegEx

10. Regular Expression
Excercise
String1 = “#indonesia hebat”
String 2=”@DKIJakarta macet banget nih”
String4 =””

11. Regular Expression
. : match any char
^: start of a string
$: end of string
[]: matches one of the set of char within []
[a-z]: matches one of the range of chars a,b,c,d, …, z

12. Dasar - dasar Natural Language
Processing

13. Definisi
● Kemampuan untuk memahami bahasa manusia
● Memahami bahasa manusia untuk mendapatkan informasi tentang kata-kata dan bagaimana
memahami struktur bahasa manusia

14. NLP Goals
● Menghitung kata
● Menemukan batas kalimat
● POS Tagging
● Menguraikan struktur kalimat (S + P + O + K)
● Mengidentifikasi sematic roles
● Mengidentifikasi entitas dalam kalimat
● Menemukan kata kepunyaan mana yang dimaksudkan ke entitas

15. NLTK
● Toolkit
● Open source
● Wrapper of scikit learn for NLP
● Terdapat beberapa korpus populer

16. Python
● Import nltk
● ...

17. NLP in Action
● nltk.word_tokenize()
● nltk.FreqDist()

18. Lemma VS Stem
What is Lemma?
What is Stem?

19. POS Tagging
In nutshell: mengidentifikasi apakah kata tersebut merupakan kata kerja, kata benda, kata keterangan,
dll

20. Python
text = “You should go now”
text2 = nltk.word_tokenize(text)
nltk.pos_tag(text2)

21. Issues in POS Tagging
---> Go to blackboard

22. Text Classification

23. Kategori mana yang tepat untuk teks dibawah ini?
http://ekonomi.kompas.com/read/2017/10/25/102555326/apbn-2018-diharapkan-bisa-menjadi-sentimen-positif
● Pembangunan
● Keuangan
● Politik

24. Penggunaan Text Classification
● Analisis Sentimen: apakah review film ini negatif atau positif
● Deteksi Spam: apakah email ini spam atau bukan?
● Identifikasi Topik: apakah berita ini topik teknologi, olahraga atau kesehatan?
● Spelling correction: bener atau benar?

25. Supervised Learning
● Machine learning task
● Labeled training data
● Learning from labeled data

26. Hal mendasar dalam klasifikasi
● Binary Classification
● Multi-class Classification
● Multi-label Classification

27. Text Features
● Kata
○ Most common words
○ Stop words
○ Normalization
○ Stemming / Lemmatization
● Kalimat
○ Pos Tagging
○ Struktur grammar
○ Kata yang similar

28. Algoritma
1. Naive Bayes Classifier
2. Support Vector Machine

29. Naive Bayes
https://www.analyticsvidhya.com/blog/2017/09/naive-bayes-explained/

30. Naive Bayes
P(Yes | Sunny) = P( Sunny | Yes) * P(Yes) / P (Sunny)
P (Sunny |Yes) = 3/9 = 0.33, P(Sunny) = 5/14 = 0.36, P( Yes)=
9/14 = 0.64
Now, P (Yes | Sunny) = 0.33 * 0.64 / 0.36 = 0.60, which has
higher probability.
https://www.analyticsvidhya.com/blog/2017/09/naive-bayes-explained/

31. Support Vector MAchine

32. Python
Jupyter Notebook
Install Scikit-learn
from sklearn import <algorithm>

33. Python -Naive Bayes
From sklearn import naive_bayes
NB = naive_bayes.MultinomialNB()
Model = NB.fit(train_data, train_labels)
Predict = NB.predict(test_data)
metrics.f1_score(test_labels, predict, average = ‘micro’)

34. Python - SVM
From sklearn import svm
svm = svm.SVC(kernel = ‘linear’, C=0.1)
model = svm.fit(train_data, train_labels)
predict = model.predict(test_data)
metrics.f1_score(test_labels, predict, average = ‘micro’)

35. Supervised algorithms in NLTK
● NaiveBayesClassifier
● DecisionTreeClassifier
● ConditionalExponentialClassifier
● MaxentClassifier
● WekaClassifier
● SklearnClassifier

36. Topic Modelling

37. Topic Modeling
● Semantic text similarity
● Topic Modeling

38. Penggunaan Semantic Similarity
● Semantic similarity is the practical, widely used approach to address
the natural language understanding issue in many core NLP tasks such
as paraphrase identification, Question Answering, Natural Language
Generation, and Intelligent Tutoring Systems

39. Semantic Similarity
Wordnet

40. https://www.coursera.org/learn/python-text-mining

41. Pengukuran similaritas
● Path Similarity
● Lowest Common subsumer
● Lin Similarity

42. Topic Modeling
● Discovering hidden topical patterns that are present across the
collection
● Annotating documents according to these topics
● Using these annotations to organize, search and summarize texts

43. Implementasi
● Latent Dirichlet Allocation
● TextRank
https://www.kdnuggets.com/2016/07/text-mining-101-topic-modeling.html