Korean Text mining

Kyunghoon Kim
UNIST
Department of Mathematical Sciences
December 12, 2017
kyunghoon@unist.ac.kr
A Mathematical Measurement for Korean Text mining
and its applications

Difﬁculty of Korean Language
Kyunghoon Kim (UNIST) A Mathematical Measurement for Korean Text mining and its applications Dec 12, 2017 2 / 83
• New Concepts 
- Korean Alphabet ( , , , , …) 
- End of a word ( ) ( , , , …) 
- Postposition ( ) ( , , , , …) 
- Word order ( ) (SOV, …) 
- …
Language Destruction

Outline
1. Text Summarization
Korean Text Mining
2. Text Clustering
3. Learning of
Text Relationship
Korean
Language 
Feature V2
Syllable
Vector
Heterogeneous Word2Vec
( Law2Vec )
Fuzzy System Term-Frequency Matrix
( LSI, NMF )
Artiﬁcial Neural Network
( Word2Vec )
2013’ 2015’ 2017’

1. Text Summarization | Motivation
< Raw News Article > < Summarized News >
March, 2013
How about Korean?
News Article
Summarized
Sentences

1. Text Summarization | Process
Document Preprocessing Feature Selection Scoring by Model Reﬁnement & Sorting by score
NNP,*,T, ,*,*,*,*
JKB,*,F, ,*,*,*,*

NNG,*,T, ,*,*,*,*
NNG,*,T, ,*,*,*,*
JC,*,F, ,*,*,*,*

NNG,*,T, ,*,*,*,*
XSN,*,T, ,*,*,*,*

NNG,*,T, ,*,*,*,*
NNG,*,T, ,*,*,*,*
XSN,*,T, ,*,*,*,*

NNG,*,F, ,*,*,*,*
JC,*,F, ,*,*,*,*

NNG,*,T, ,*,*,*,*
NNG,*,F,
,Compound,*,*, /NNG/*+ /
NNG/*
JKS,*,F, ,*,*,*,*

MAG, / ,F,
,*,*,*,*

VV,*,F, ,*,*,*,*

EC,*,F, ,*,*,*,*

VX,*,T, ,*,*,*,*

EF,*,F, ,*,*,*,*

. SF,*,*,*,*,*,*,*
• Content word(Keyword) feature
• Title word feature
• Sentence location feature
• Sentence Length feature
• Proper Noun feature
• Upper-case word feature
• Cue-Phrase feature
• Biased Word feature
• Font based feature
• Pronouns
• Sentence-to-Sentence Cohesion
• Sentence-to-Centroid Cohesion
• Occurrence of non-essential information
• Discourse analysis
Only for
English features

1. Text Summarization | Feature measurements
Feature based on English
Feature based on Korean
, , , , , ... , , , , ...

1. Text Summarization | Fuzzy Set

1. Text Summarization | Calculating the score of sentences

1. Text Summarization | Korean Text Summarization
http://summ-dev.ap-northeast-2.elasticbeanstalk.com/

1. Text Summarization | Patent, 2013
https://goo.gl/blkjwf
Korean News Summarization System And Method

2. Text Clustering
Text Clustering

2. Text Clustering
MatrixDocuments
1. Select Matrix 
 
 
 
2. Calculating similarity 
between each column of matrix
3. Clustering by the degree of similarity
A =
0
B
B
B
@
a11 a12 ··· a1n
a21 a22 ··· a2n
...
...
...
...
am1 am2 ··· amn
1
C
C
C
A
Convert
A. Basic (using raw matrix)
B. LSI (Latent Semantic Indexing)
C. NMF (Non-negative Matrix Factorization)

2. Text Clustering | Term-Frequency Matrix
= { apple, banana, kiwi }
= { apple, banana, store }
= { store }
d1
d2
d3
A =
2
4 d1 · · · dn
3
5
Term-Frequency Matrix
Frequency
Document vector

2. Text Clustering | Singular Value Decomposition (SVD)
d1 d2 d3
w1 1 0 0
w2 0 1 0
w3 1 1 1
w4 1 1 0
w5 0 0 1
-0.27 0.21 0.70 -0.53 0.30
-0.27 0.21 -0.70 -0.53 0.30
-0.71 -0.33 0 -0.10 -0.60
-0.55 0.43 0 0.64 0.29
-0.15 -0.77 0 0.10 0.60
2.35 0 0
0 1.19 0
0 0 1.00
0 0 0
0 0 0
-0.65 0.26 0.70
-0.65 0.26 -0.70
-0.36 -0.92 0
=

2. Text Clustering | Latent Semantic Indexing (LSI)

2. Text Clustering | Non-negative Matrix Factorization (NMF)

2. Text Clustering | Non-negative Matrix Factorization (NMF)
Doc1
Doc2
Doc3
Feature
1
Feature
2
Feature 1
Feature 2
Term
1
Term
2
Term
3
Term
4
Term
5

2. Text Clustering | Term-Frequency Matrix
Large Dimension Matrix
for large-scale set
Proposed method
Syllable Vector

2. Text Clustering | Syllable-n Vector
about 1,200
dimension

2. Text Clustering | Dimension reduction using Syllable-n vector
Dimension Reduction
by Syllable Vector
Syllable-1 Syllable-2 Syllable-3

2. Text Clustering | Syllable-n-All Vector
Syllable-1-All Syllable-2-All
, , , , , , , ,
✓
lj
n
◆
length of word wj
Take all combination of syllable-n

2. Text Clustering | Benchmark Dataset HKIB-20000
Dimension reduction
How about information loss?

2. Text Clustering | Similarity
✓
a
b
sim(d1, d2) =
v
u
u
t2 1
2/9
p
3/9
p
3/9
!
= 0.8164
sim(d2, d3) = 0.919
sim(d1, d3) = 1.414

2. Text Clustering | Similarity
Source :
Doc Number 5222
Target :
Other all documents

2. Text Clustering | Correlation
Basic
LSI
NMF

2. Text Clustering | Evaluation of Text Clustering

2. Text Clustering | Precision
Real
Answer
TP
FP
Precision =
5
7
= 0.71

2. Text Clustering | Evaluation Set
Doc 1
Doc 2
Doc 3
Doc 4
Doc 5
Doc 6
…

2. Text Clustering | Standard for Evaluation
1
2
3
4
5
Nearest
neighbors
Limited
Radius

2. Text Clustering | Evaluation of text clustering
Radius Threshold
Syl-2
Syl-3
Word

Count Threshold

Precision Speed
n = 5 , count threshold
LSI LSI

Syl-2 for LSI
is BEST!

2. Text Clustering | Patent
https://goo.gl/fskHxTKorean Text Clustering System and Method

2. Text Clustering | Limitation of word-based method
These words are NOT important
to understand the given text!
Limitation of word-based method

3. Learning of Text Relationship
Word-based
Citation Relation
Find similar documents
using citation information

3. Learning of Text Relationship | Natural Language Processing (NLP)
https://www.upwork.com/hiring/for-clients/artiﬁcial-intelligence-and-natural-language-processing-in-big-data/

3. Learning of Text Relationship | Word2Vec
2013, Hot Model in NLP
“Word2Vec” (Google)
http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
(맥도날드가, 햄버거는)
(맥도날드가, 맛있다.)
(맛있다., 맥도날드가)
(맛있다., 감자튀김도)
(감자튀김도, 맛있다.)
(감자튀김도, 맛있었는데..)
(맘스터치도, 햄버거는)
(맘스터치도, 맛있다.)
(맛있다., 맘스터치도)
(맛있다., 패티가)
Source Text
Red : Target keyword, Blue : Context Keyword
Training Set

(맥도날드가, 햄버거는)
(맥도날드가, 맛있다.)
Input, Output

Shortage of Word2Vec
• Only Word-based Method 
=> Meaningless words are also counted. 
• Only Same vocabulary set for input, output 
=> Dimensions of input, output are ﬁxed. 
• Only use a context information of target word 
=> depends entirely on context with windows size N.

3. Learning of Text Relationship | Heterogeneous Word2Vec
Heterogeneous Word2Vec
Input Output

1
2
3
4
5
1
2
3
4
5
0
6

1
0
0
0
0
0
0
0
0
0
1
0
0
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
@
1
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
A
0
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
@
1
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
A

1
0
0
0
0
0
1
0
0
0
0
0
0
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
@
1
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
A
0
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
@
1
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
A

Ch 4. Learning for number relationship
1
2
3
4
5
1
2
3
4
5
0
6

1
2
3
4
5
1
2
3
4
5
Similarity
( 0 is best )
Matrix (Vectors)

Input Output

3. Learning of Text Relationship | Law2Vec
Legal information comprises mainly of legislation and case.
• CL ( Case - Legislation )
• CC ( Case - Case )
• CLC ( Case - Legislation, Case )

3. Learning of Text Relationship | Law2Vec CL Model, CC Model
Cited legislations Cited cases
Case Case

3. Learning of Text Relationship | Law2Vec CLC Model
Case

3. Learning of Text Relationship | Evaluation of Law2Vec

3. Learning of Text Relationship | Evaluation of CL Model
Cited legislations
Case

3. Learning of Text Relationship | Evaluation of Law2Vec : W_1

3. Learning of Text Relationship | Evaluation of Law2Vec : W_2

3. Learning of Text Relationship | Evaluation of CC Model
Cited cases
Case

3. Learning of Text Relationship | Evaluation of CC Model

3. Learning of Text Relationship | Evaluation of CLC Model
Case

3. Learning of Text Relationship | Evaluation of CLC Model

3. Learning of Text Relationship | Result of Law2Vec

3. Learning of Text Relationship | Expansion of Data set
< Lawyer Oh’s Answer Sheet >

3. Learning of Text Relationship | Law2vec for Sample Data
CL Model
Iteration
10000
CL Model
Iteration
60000
CC Model
Iteration
10000
CC Model
Iteration
60000
CLC Model
Iteration
10000
CLC Model
Iteration
60000

3. Learning of Text Relationship | Link Prediction

Conclusion
1. Main Contribution
Korean Language Feature V2 (JKB, JX)
Syllable Vector
Heterogeneous Word2Vec ( Law2Vec )
2. Advantage
Chapter 2.
Text Summarization
Chapter 3.
Text Clustering
Chapter 4.
Text Relational Learning To summarize by linguistic feature for Korean
To get the dimension reduction with a small amount of information
loss using Syllable vector and to make eﬃcient computing for large-
scale document set.
To learn of heterogeneous data by using the relationship
between them without text(word) data

Conclusion
3. Interest to readerChapter 2.
Text Summarization
Chapter 3.
Text Clustering
Chapter 4.
Text Relational Learning
To apply Fuzzy Concept to text mining considering Language
features
=> Define your idea and apply it to system easily
Korean language has more efficient for large-scale document set
=> Korean language is adequate to compress text data
Design the NN system to fit the structure of your data
=> Meta-data is a good enough material to learn the relationship
between them.

Korean Text mining

More Related Content

What's hot

Similar to Korean Text mining

More from Kyunghoon Kim

Recently uploaded

Korean Text mining