Successfully reported this slideshow.
Your SlideShare is downloading. ×

自然言語処理における構文解析器の構造と解釈

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 101 Ad

More Related Content

Recently uploaded (20)

Advertisement

自然言語処理における構文解析器の構造と解釈

  1. 1. ⮬↛ゝㄒฎ⌮䛻䛚䛡䜛 ᵓᩥゎᯒჾ䛾ᵓ㐀䛸ゎ㔘 Structure and Implementation of the Parsers for Natural Language Processing JOIss2014 @zakuro9715
  2. 2. ⮬ᕫ⤂௓
  3. 3. YuZakuro @zakuro9715 ᫂▼㧗ᑓ ༟⌫(㏆␥3఩) HN twitter Ꮫᰯ ㊃࿡
  4. 4. ᡂᯝ≀
  5. 5. https://github.com/zakuro9715/shino
  6. 6. 䛸䛣䜝䛷
  7. 7. 䝇䝷䜲䝗䛾ᯛᩘ䛜
  8. 8. 101ᯛ
  9. 9. 䛒䜛䛾䛷ᛴ䛞┠䛷䜔䜚䜎䛩
  10. 10. What is Parser? ᵓᩥゎᯒჾ 䛳䛶䠛
  11. 11. ᵓᩥゎᯒჾ ● ୚䛘䜙䜜䛯䝖䞊䜽䞁ิ䜢ゎᯒ䛧䚸 ᩥ䛾ᵓ㐀䜢⾲䛩䝕䞊䝍ᵓ㐀(୰㛫⾲⌧)䛻ኚ᥮ 䛩䜛䚹
  12. 12. ᵓᩥゎᯒჾ ● ୚䛘䜙䜜䛯䝖䞊䜽䞁ิ䜢ゎᯒ䛧䚸 ᩥ䛾ᵓ㐀䜢⾲䛩䝕䞊䝍ᵓ㐀(୰㛫⾲⌧)䛻ኚ᥮ 䛩䜛䚹 構文木・AST・S式など
  13. 13. せ䛩䜛䛻…
  14. 14. I am You
  15. 15. ᩥ ྡモྃ ே⛠௦ྡモ I ືモྃ ືモ am ྡモྃ ᅛ᭷ྡモ You
  16. 16. Tips 1. ゎᯒᑐ㇟䛿䝖䞊䜽䞁ิ 2. 䛔䜝䛔䜝䛺୰㛫⾲⌧
  17. 17. Tips 1. ゎᯒᑐ㇟䛿䝖䞊䜽䞁ิ 2. 䛔䜝䛔䜝䛺୰㛫⾲⌧
  18. 18. ゎᯒᑐ㇟䛿䝖䞊䜽䞁ิ ● ᵓᩥゎᯒ䛻䛛䛡䜛䛯䜑䛻䛿䚸ᩥ䜢䝖䞊䜽䞁 䛻ศゎ䛩䜛ᚲせ䛜䛒䜛 ● 䝖䞊䜽䞁䜈䛾ศゎ䛿Ꮠྃゎᯒჾ䛜⾜䛖
  19. 19. You are PRO ᩥᏐิ
  20. 20. [You, are, PRO] 䝖䞊䜽䞁ิ
  21. 21. 䛒䛺䛯䛿䝥䝻䛷䛩 ᩥᏐิ
  22. 22. [䛒䛺䛯, 䛿, 䝥䝻, 䛷䛩] 䝖䞊䜽䞁ิ
  23. 23. 䝥䝻 ᩥᏐิ
  24. 24. [䝥䝻] 䝖䞊䜽䞁ิ
  25. 25. Tips 1. ゎᯒᑐ㇟䛿䝖䞊䜽䞁ิ 2. 䛔䜝䛔䜝䛺୰㛫⾲⌧
  26. 26. S-Expression (S (NP (PeopleN (‘I’))) (VP (V (‘am’)) (NP (ProperN ('You')))))
  27. 27. Syntax Tree S NP PeopleN I VP V am NP ProperN You
  28. 28. 䛭䛾௚䛔䜝䛔䜝 ● Abstract Syntax Tree ● Reverse Polish Notation ● Polish Notation
  29. 29. Context Free Language ᩥ⬦⮬⏤ᩥἲ
  30. 30. ᩥ⬦⮬⏤ᩥἲ ● ᩥἲ䜢ᐃ⩏䛩䜛䛯䜑䛻౑⏝䛥䜜䜛ᩥἲ ● ṇつ⾲⌧䛷䛿ᐃ⩏䛷䛝䛺䛔ᩥἲ䜢ᐃ⩏䛷䛝䜛
  31. 31. ౛ S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes” a cat likes the dog 䛾ᵝ䛺ᩥἲ䜢⏕ᡂ䛩䜛CFG
  32. 32. Implements of typical parsers ௦⾲ⓗ䛺ᵓᩥゎᯒჾ䛸䛭䛾ᐇ⿦
  33. 33. ௦⾲ⓗ䛺ᵓᩥゎᯒჾ ● Recursive Descent Parser ● Shift-Reduce Parser
  34. 34. ௦⾲ⓗ䛺ᵓᩥゎᯒჾ ● Recursive Descent Parser ● Shift-Reduce Parser
  35. 35. ≉ᚩ ● ẚ㍑ⓗᐇ⿦䛜ᐜ᫆䛷䛒䜛 ● 䜶䝷䞊䝯䝑䝉䞊䝆䜢⾲♧䛩䜛䛾䛜ᐜ᫆
  36. 36. ཎ⌮ ● ධຊ䛻ᑐ䛧䛶ᩥἲ䜢๓䛛䜙㡰䛻㐺⏝䛥䛫䜛
  37. 37. a cat likes a dog S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes”
  38. 38. S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes” a cat likes a dog
  39. 39. S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes” a cat likes a dog
  40. 40. S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes” a cat likes a dog
  41. 41. S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes” a cat likes a dog
  42. 42. S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes” a cat likes a dog
  43. 43. S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes” a cat likes a dog
  44. 44. a cat likes a dog S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes”
  45. 45. a cat likes a dog S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes”
  46. 46. a cat likes a dog S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes”
  47. 47. a cat likes a dog S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes”
  48. 48. a cat likes a dog S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes”
  49. 49. a cat likes a dog S NP N Det VP V -> -> -> -> -> -> NP VP Det N “cat” | “dog” “the” | “a” V NP “likes”
  50. 50. ௨ୗ␎
  51. 51. ᙅⅬ ● ᕥ෌ᖐ ● Ỵᐃ୙⬟ᛶ
  52. 52. ᕥ෌ᖐ ● ྑ䛾䜘䛖䛺CFG䜢⪃䛘䜛 䛣䛾CFG䛿௵ពಶ䛾”a”䜢୪䜉䛯ᩥᏐ ิ䜢⾲䛩䛜䚸䝟䞊䝇䛩䜛䛸↓㝈䝹䞊䝥 䛻㝗䜛 S -> S A | A A -> “a”
  53. 53. a a a a a S A -> -> S A | A “a”
  54. 54. a a a a a S A -> -> S A | A “a”
  55. 55. a a a a a S A -> -> S A | A “a”
  56. 56. a a a a a S A -> -> S A | A “a”
  57. 57. a a a a a S A -> -> S A | A “a”
  58. 58. a a a a a S A -> -> S A | A “a”
  59. 59. a a a a a S A -> -> S A | A “a”
  60. 60. a a a a a S A -> -> S A | A “a”
  61. 61. ゎỴ᪉ἲ ● ᩥἲ䜢௨ୗ䛾䜘䛖䛻ኚᙧ䛩䜛 S S_ A -> -> -> A S_ “” | A S_ “a”
  62. 62. Ỵᐃ୙⬟ᛶ ● ௒ぢ䛶䛔䜛䝖䞊䜽䞁䛷ᵓᩥ䜢Ỵᐃ䛩䜛䛾䛷䚸ఝ䛯 ᵓᩥ䛜䛒䜛ሙྜ䛿Ỵᐃ䛷䛝䛺䛔䛣䛜䛒䜛 ● 䛯䛸䛘䜀ୗ䛾䜘䛖䛺ᵓᩥ䛿A䜢ぢ䛯᫬Ⅼ䛷䛿Ỵ ᐃ䛷䛝䛺䛔 S A B C -> -> -> -> A B | A C “a” “b” “c”
  63. 63. ゎỴ᪉ἲ ● A䜢ぢ䛯᫬Ⅼ䛷䛿Ỵᐃ䛷䛝䛺䛔䛾䛷䚸䜒䛖䜂䛸䛴 ඛ䜢ぢ䛶ุ᩿䛩䜛䚹 S A B C -> -> -> -> A B | A C “a” “b” “c”
  64. 64. ゎỴ᪉ἲ ● ඛㄞ䜏䜢ቑ䜔䛩䜋䛹ィ⟬㔞䛜ቑ䛘䛶䛔䛟䛯䜑⌧ ᐇⓗ䛷䛿䛺䛔
  65. 65. ௦⾲ⓗ䛺ᵓᩥゎᯒჾ ● Recursive Descent Parser ● Shift-Reduce Parser
  66. 66. ≉ᚩ ● LLἲ䛜ⱞᡭ䛸䛩䜛ᩥἲ䜢ゎᯒ䛷䛝䜛 ● 䜶䝷䞊䝯䝑䝉䞊䝆䜢⾲♧䛩䜛䛾䛜㞴䛧䛔
  67. 67. ཎ⌮ ● 䝖䞊䜽䞁䜢䝇䝍䝑䜽䛻✚䜏䛴䛴䚸䝇䝍䝑䜽䛾ୖ䛛䜙 䝹䞊䝹䜢㐺⏝䛥䛫䜛 ● 䝇䝍䝑䜽䛻✚䜐ືస䜢Shift, 䝹䞊䝹䜢㐺⏝䛩䜛 ືస䜢Reduce䛸࿧䜆
  68. 68. a cat likes a dog
  69. 69. cat likes a dog a Shift
  70. 70. cat likes a dog Reduce Det
  71. 71. likes a dog Det Shift cat
  72. 72. likes a dog Reduce N Det
  73. 73. likes a dog Reduce NP
  74. 74. a dog NP Shift likes
  75. 75. a dog Reduce V NP
  76. 76. dog a NP Shift V
  77. 77. dog Reduce Det V NP
  78. 78. dog Det NP Shift V
  79. 79. Reduce N Det V NP
  80. 80. Reduce NP V NP
  81. 81. Reduce S
  82. 82. Recursive Descent Parser 䛾ᙅⅬ ● ᕥ෌ᖐ ● Ỵᐃ୙⬟ᛶ
  83. 83. ᕥ෌ᖐ ● ⮬᫂䛻Ⓨ⏕䛧䛺䛔
  84. 84. ᕥ෌ᖐ ● ᕥ෌ᖐ䛷↓㝈䝹䞊䝥䛻㝗䜛ཎᅉ䛿䚸ᩥἲ䛻ᑐ 䛧䛶䝖䞊䜽䞁䜢ᙜ䛶䛶䜖䛟䛛䜙 ● 䝖䞊䜽䞁䛻ᑐ䛧䛶ᩥἲ䜢ᙜ䛶䜛᪉ἲ䛷䛿Ⓨ⏕䛧 䛺䛔
  85. 85. Ỵᐃ୙⬟ᛶ ● Recursive Descent Parser䛸ྠᵝ䛾ၥ㢟䛜Ⓨ⏕ 䛧䛺䛔䛣䛸䛿᫂䜙䛛
  86. 86. Ỵᐃ୙⬟ᛶ ● Recursive Descent Parser 䛷䛿᭱ึ䛾䝖䞊䜽䞁 䜢Ⓨぢ䛧䛯᫬Ⅼ䛷ᩥἲ䜢Ỵᐃ䛧䛯 ● 䝖䞊䜽䞁䜢ぢ䛶䚸㐺⏝䛷䛝䜛ᩥἲ䜢᥈䛩᪉ἲ䛷䛿 ྠᵝ䛾ၥ㢟䛿Ⓨ⏕䛧䛺䛔
  87. 87. LLἲ䛾ᙅⅬ䜢ඞ᭹
  88. 88. 䜔䛳䛯䛽
  89. 89. 䜔䛳䛯䛽 䛷䜒…
  90. 90. Shift-Reduce Parser 䛾ᙅⅬ ● Shift-Reduce⾪✺ ● Reduce-Reduce⾪✺
  91. 91. Shift-Reduce Parser 䛾ᙅⅬ ● Shift-Reduce⾪✺ ● Reduce-Reduce⾪✺
  92. 92. Shift-Reduce ⾪✺ ● ୗ䛾䜘䛖䛺ᩥἲ䛷䚸”a”䛜䝇䝍䝑䜽䛻䛒䜛䛸䛝 Reduce䛩䜛䛛Shift䛩䜛䛛䛜Ỵᐃ䛷䛝䛺䛔 S A B -> -> -> A | A B “a” “b”
  93. 93. ゎỴ᪉ἲ ● ඛㄞ䜏
  94. 94. Shift-Reduce Parser 䛾ᙅⅬ ● Shift-Reduce⾪✺ ● Reduce-Reduce⾪✺
  95. 95. Reduce-Reduce ⾪✺ ● ୗ䛾䜘䛖䛺ᩥἲ䛷䚸”a”䛜䝇䝍䝑䜽䛻䛒䜛䛸䛝䚸A1 䜢㐺⏝䛩䜛䛛A2䜢㐺⏝䛩䜛䛛Ỵᐃ䛷䛝䛺䛔 S A1 A2 -> -> -> A1 ‘x’ | A2 ‘y’ “a” “a”
  96. 96. ゎỴ᪉ἲ ● ඛㄞ䜏
  97. 97. Summary 䜎䛸䜑
  98. 98. ● Recursive Descent Parser 䛿ᐇ⿦䛜⡆༢ ○ 䛯䛰䛧ᙅⅬ䜒ከ䛔 ● Shift-Reduce Parser 䛿ᗄ䛴䛛䛾ᙅⅬ䜢ゎᾘ ○ ᪂䛧䛔ᙅⅬ䛜…
  99. 99. ⤖ㄽ
  100. 100. ඛㄞ䜏
  101. 101. 䛤Ύ⫈䛒䜚䛜䛸䛖䛤䛦䛔䜎䛧䛯

×