1. The document discusses n-grams and their use in natural language processing tasks. 2. It provides examples of n-grams of different sizes and describes how n-grams are used to calculate weights and similarity measures between words or phrases. 3. The document also mentions using n-grams with the Markov Clustering Algorithm to cluster similar words or phrases together.

1. n-gram
1

2. NLP
2

3. ✓ n-gram
- n-gram
- n-gram
✓ n-gram
-
-
✓ Ameba n-gram
- n-gram
-
-
3

4. n-gram
4

5. N
n-1 n
....
5

6. N
n-1 n
....
_
_
_ _
6

7. n
2-gram
_
_ _
_ _ _CD
_ _ _ _
7

8. n
2-gram
_
_ _
_ _ _CD
_ _ _ _
8

9. n
2-gram
_
_ _
_ _ _CD
_ _ _ _
9

10. n
2-gram
_
_ _
_ _ _CD
_ _ _ _
10

11. 11

12. n-gram
12

13. 13

14. GSK
※
n
MeCab
14

15. 15

16. ※
n
MeCab ipadic
16

17. 17

18. 18

19. 19

20. ✓ n-gram
- n-gram
- n-gram
✓ n-gram
-
-
✓ Ameba n-gram
- n-gram
-
-
20

21. Baidu
21

22. 22

23. ※Google
23

24. 24

25. •
•
Baidu
25

26. − −
NG
26

27. − −
27

28. _ _ _ _ _ _ _
_ _ _ _ _ _ _
_ _ _ _ _ _
_ _2038_ _ _ _ _ _ _ _
_ _ _ _
_ _ _ _ _ _
28

29. _ _ _ _ _ _ _
_ _ _ _ _ _ _
_ _ _ _ _ _
_ _2038_ _ _ _ _ _ _ _
_ _ _ _
_ _ _ _ _ _
29

30. 1. w v
※ n-gram
2. w v
Weight
3. wi wj
Measure
30

31. n-gram
n-gram
n > 1) 1-gram
31

32. Weight
freq(w) w
freq(v) v
freq(w,v) w v
32

33. Measure
[2010 ] Simpson Jaccard
Simpson-Jaccard
※Vi Vj wi,wj
※Simpson Jaccard
33

34. [2010 ]http://nlp.nagaokaut.ac.jp/arc/10/10NLP-asakura.pdf
34

35. 35

36. 10
...
36

37. 1.
2. Simpson-Jaccard [ 2010]
3. 2+ [ 2010]
•
•
37

38. 38

39. 39

40. • →
→
• 5 9
•
40

41. 10 ×
[ ]
→
→ .....
100
41

42. 42

43. •
•
•
• BootStrap
43

44. 44

45. - FX 0.0701096506659567
- 0.06259698742992559
-
0.06042366216085525
-
0.058799844151470174
- 0.05735695738694635
-
0.055435815549429605
-
0.05232985446935419
-
0.04980454574094188
-
0.04927509063333216
-
0.04824717092086649
45

46. -
0.13373671096183018
- 0.0918404883164821
- 0.08504703320048444
- 0.07961197269662577
-
0.07711954365644495
-
0.07702996997666198
-
0.07662113919947415
-
0.0725147419575286
-
0.06882302518118967
-
0.06849300043195472
46

47. -
0.46469658771106315
- 0.11975034678985662
- 0.1097484574232766
- 0.09879163577196441
- 0.09813513570488815
- <EMOJI_546> 0.09719796981762122
- 0.09570307958644202
- 0.09472409933583924
- □ 0.09466494353310652
- 0.0939216442782036
47

48. - 0.1315343475643398
- 0.09995353106829284
- <EMOJI_195> 0.07684385370515374
- 0.0668208753259571
- 0.064557866506281
- 0.06399098585323154
- 0.0631873841230999
-
0.059873448412962174
- 0.058499392875879214
- 0.05843526443912576
48

49. -
0.05180321648370331
-
0.04568992635712712
- 0.03513069449040805
- 0.027690371652517847
-
0.024491134248894916
- 0.024070717542493413
- 0.013918762159274874
- <EMOJI_1BE> 0.013795160758545503
- 0.011283944598435737
- 0.009334867761007871
49

50. •
•
50

51. 51

52. Baidu n-gram
UI
1.
2.
3. 2. Trie Double Array Trie Tree
4. Trie Common Prefix Search
52

53. 53

54. Amazon EC2
High Performance Computing(HPC)
2010/07/13
1.6$ / hour ( 150
54

55. Amazon EC2 HPC
4GB
55

56. ✓ n-gram
- n-gram
- n-gram
✓ n-gram
-
-
✓ Ameba n-gram
- n-gram
-
-
56

57. Ameba
n-gram
57

58. Ameba
n-gram
n 7
MeCab ipadic
1 122,500 / 2 1,195,628
3 1,586,205 / 4 1,021,938
5 499,788 / 6 231,064
7 117,515
58

59. n
59

60. •
•
•
• (<EMOJI_xxx>)
•
7,500
• Ameba
60

61. • MeCab ipadic
• Hadoop
• Map/Reduce 5
• -gram
•
• →
• →
• Unicode NFKC
•
•
61

62. Unicode
※
•
•
•
•
62

63. •
•
63

64. 64

65. <EMOJI_031> - <EMOJI_123> 0.25245663283888076
<EMOJI_031> - 0.23865945647496203
<EMOJI_031> - 0.2233787922434289
<EMOJI_031> - <EMOJI_139> 0.21505030767963476
<EMOJI_031> - <EMOJI_320> 0.20905272608162664
<EMOJI_031> - <EMOJI_176> 0.20767832299973635
<EMOJI_031> - ww 0.20396530274266259
<EMOJI_031> - <EMOJI_035> 0.1952646415960877
<EMOJI_031> - <EMOJI_029> 0.19118380193173534
<EMOJI_031> - 0.18532801238518545
65

66. <EMOJI_022> - 5.05433098481788
<EMOJI_022> - <EMOJI_106> 0.21255982715221317
<EMOJI_022> - <EMOJI_178> 0.20371824309787556
<EMOJI_022> - <EMOJI_105> 0.19694261277461542
<EMOJI_022> - <EMOJI_324> 0.1960739883132334
<EMOJI_022> - <EMOJI_028> 0.19591157145392102
<EMOJI_022> - <EMOJI_143> 0.19367459051404434
<EMOJI_022> - <EMOJI_089> 0.1894972913601391
<EMOJI_022> - <EMOJI_319> 0.1853140205685605
<EMOJI_022> - <EMOJI_187> 0.18316119166397835
66

67. <EMOJI_021> - 0.1980766824375306
<EMOJI_021> - 0.160789228797033
<EMOJI_021> - <EMOJI_143> 0.15832172025846883
<EMOJI_021> - <EMOJI_325> 0.15752899334689507
<EMOJI_021> - 0.15600103135195192
<EMOJI_021> - <EMOJI_089> 0.15588927293536636
<EMOJI_021> - <EMOJI_144> 0.15288842785042273
<EMOJI_021> - <EMOJI_141> 0.15264128047608577
<EMOJI_021> - <EMOJI_022> 0.1524110403297163
<EMOJI_021> - 0.1520281112500744
67

68. <EMOJI_077> - 0.20100120017122702
<EMOJI_077> - <EMOJI_075> 0.16814782000472359
<EMOJI_077> - <EMOJI_188> 0.16407662484190644
<EMOJI_077> - <EMOJI_134> 0.1598560562097877
<EMOJI_077> - 0.15753861586037332
<EMOJI_077> - <EMOJI_007> 0.1520971768068388
<EMOJI_077> - <EMOJI_014> 0.15101611087607572
<EMOJI_077> - <EMOJI_162> 0.15081721574211884
<EMOJI_077> - <EMOJI_044> 0.14580103348348722
<EMOJI_077> - 0.13778630432134603
68

69. <EMOJI_181> - <EMOJI_150> 0.0998223599336455
<EMOJI_181> - <EMOJI_169> 0.09619906986114168
<EMOJI_181> - 0.09407484828035773
<EMOJI_181> - 0.0751037382171799
<EMOJI_181> - <EMOJI_008> 0.06918233066750942
<EMOJI_181> - 0.05804747259025906
<EMOJI_181> - 0.05277048261006679
<EMOJI_181> - 0.05059336124231738
<EMOJI_181> - 0.049448742078198986
<EMOJI_181> - <EMOJI_113> 0.04768875635910332
69

70. 70

71. Markov Clustering Algorithm
( = ≒ )
※MCL:http://www.micans.org/mcl/
71

72. ( <>
< > < > < > < > < > < > < > < > < >
< > < > < > < > < > < > <
<end> < > < > < > < >
>
< > < > < > < > < > < > < >
< > < > < > < > < > < > < > < >
< > < > < > < > < > < >
< > < > < > < > < > < > <CD> < >
< > < > < > < > < > < > < > < >
< > < > < > < > < > < > < > < > <
< > < > <CD> < >
> < > < > < >
< > < > < > < > < > < > < >
< > < > < > <
< > < > < >
>
< > < > <CD> < > < > < > < > < > <
< > < > < >
> < > < >
< > < > < > <NEW> < > < > <NEW> < >
< > < > < > < > < >
72

73. < > < > < > <
> < >
< > < > < > < >
< > < > <
> < >
< > < > < > <NEW>
< > < > < > < >
< > < > < > < > < >
< > < > < > < > < > <
> < >
73

74. •
•
74

75. 75

76. (` ω )
(` ) - 0.1768694301100037
(` ) - ⊃ 0.09913242362438676
(` ) - ({( 0.09405018119300235
(` ) - 0.031062102211042445
(` ) - 0.028857016807085342
(` ) - 0.02608930784914413
(` ) - σ 0.022498088286870093
(` ) - 0.016193426515270032
(` ) - ω 0.016122858488317616
(` ) - 0.01345138088760404
76

77. (^∀^)
(^∀^) -
0.06916344408426699
(^∀^) - 0.06890434881805171
(^∀^) - 0.06795858079282872
(^∀^) - 0.05823929001282186
(^∀^) -
0.05591209864361552
(^∀^) - 0.055199102468491
(^∀^) - 0.0529766713691993
(^∀^) -
0.04651788654030653
(^∀^) - 0.04506786291842584
77

78. (*́д`)
(*́д`) - 0.11168727359492218
(*́д`) - 0.10079146007557346
(*́д`) - 0.09144542975809382
(*́д`) - 0.08668152231221557
(*́д`) - 0.0828216114116598
(*́д`) - 0.07886784003012988
(*́д`) - 0.07516063088790395
(*́д`) - д 0.04393643491834669
(*́д`) - 0.04391789518745296
(*́д`) - 0.041770270015884854
78

79. (/ω)
(/ω) - 0.16208491557582996
(/ω) - /// 0.12872038557250623
(/ω) - 0.1280711164456965
(/ω) - 0.11152828905321341
(/ω) - 0.09964963425187295
(/ω) - 0.09765195595553885
(/ω) - 0.09551633971030135
(/ω) - 0.08766541396665446
(/ω) -
0.0857860000269694
(/ω) - 0.07871911676808412
79

80. (^0^)/
(^0^)/ - 0.09445225976273072
(^0^)/ - 0.036873637365873665
(^0^)/ - 0.026846223878548046
(^0^)/ - 0 0.024911172513660745
(^0^)/ - 0.019634091253330398
(^0^)/ - 0.01680880907554066
(^0^)/ - ^ 0.016789481994256183
(^0^)/ - 0.016144113065194438
(^0^)/ - 0.015391342223916664
(^0^)/ - .01494851710964888
0
80

81. Google Docs 200
http://docs.google.com/leaf?
id=0B5kuQ0wVMAkMNmRhMzM3YzYtYjNlOS00YWI0LWFhYjktZDM2MDFiNWQwY2Zh&hl=ja&a
uthkey=CLbOpdEE
81

82. •
82

83. 83

84. •
• n-gram
84

85. • 100%
•
85

86. 86