Japanese Linguistics in Lucene and Solr

1. Japanese linguisticsin Apache Lucene™ and Apache Solr™ May 9th, 2012 Christian Moen christian@atilika.com

2. About me• MSc. in computer science, University of Oslo, Norway• Worked with search at FAST (now Microsoft) for 10 years • 5 years in R&D building FAST Enterprise Search Platform in Oslo, Norway • 5 years in Services doing solution delivery, sales, etc. in Tokyo, Japan• Founded アティリカ株式会社 in 2009 • We help companies innovate using search technologies and good ideas • We know information retrieval, natural language processing and big data • We are based in Tokyo, but we have clients everywhere• Newbie Lucene & Solr Committer • Mostly been working on Japanese language support (Kuromoji) so far• Please write me on christian@atilika.com or cm@apache.org

3. Today’s topics

4. Today’s topics• Japanese 101 - ordering beer and toasting• Japanese language processing• Japanese features in Lucene/Solr

7. Japanese 101

8. ビールください bi-ru kudasai

9. ビールください bi-ru kudasaiA beer, please

10. ありがとうございます！ arigatō gozaimasu!

11. ありがとうございます！ arigatō gozaimasu!Thank you very much!

12. 乾杯！kanpai!

13. 乾杯！kanpai!Cheers!

14. ＪＲ新宿駅の近くにビールを飲みに行こうか？JR Shinjuku eki no chikaku ni bi-ru ō nomi ni ikō ka?

15. ＪＲ新宿駅の近くにビールを飲みに行こうか？JR Shinjuku eki no chikaku ni bi-ru ō nomi ni ikō ka? Shall we go for a beer near JR Shinjuku station?

16. ＪＲ新宿駅の近くにビールを飲みに行こうか？

17. Romaji - ローマ字・Latin characters (26+)・Used for proper nouns, etc. ＪＲ新宿駅の近くにビールを飲みに行こうか？

18. Katakana - カタカナ・Phonetic script (~50) ・Typically used for loan wordsＪＲ新宿駅の近くにビールを飲みに行こうか？

19. ＪＲ新宿駅の近くにビールを飲みに行こうか？Kanji - 漢字・Chinese characters (50,000+)・Used for stems & proper nouns

20. ＪＲ新宿駅の近くにビールを飲みに行こうか？ Hiragana - ひらがな・Phonetic script (~50) ・Used for inﬂections & particles

21. Romaji - ローマ字 Katakana - カタカナ・Latin characters (26+) ・Phonetic script (~50)・Used for proper nouns, etc. ・Typically used for loan words ＪＲ新宿駅の近くにビールを飲みに行こうか？Kanji - 漢字 Hiragana - ひらがな・Chinese characters (50,000+) ・Phonetic script (~50)・Used for stems & proper nouns ・Used for inﬂections & particles

22. ＪＲ新宿駅の近くにビールを飲みに行こうか？

23. ＪＲ新宿駅の近くにビールを飲みに行こうか？? What are the words in this sentence?

24. ＪＲ新宿駅の近くにビールを飲みに行こうか？? What are the words in this sentence?! Words are implicit in Japanese - there is no white space that separates them

25. ＪＲ新宿駅の近くにビールを飲みに行こうか？? How do we index this for search, then?

26. ＪＲ新宿駅の近くにビールを飲みに行こうか？? How do we index this for search, then?! We need to segment text into tokens ﬁrst

27. ! Two major approaches for segmentation 1. n-gramming 2. morphological analysis (statistical approach)

28. n-gramming (n=2)ＪＲ新宿駅の近くにビールを飲みに行こうか？ Shall we go for a beer near JR Shinjuku station?

29. n-gramming (n=2)ＪＲ新宿駅の近くにビールを飲みに行こうか？ＪＲ Shall we go for a beer near JR Shinjuku station? n=2ＪＲ

30. n-gramming (n=2)ＪＲ新宿駅の近くにビールを飲みに行こうか？ＪＲ Shall we go for a beer near JR Shinjuku station? n=2 Ｒ新ＪＲＲ新

31. n-gramming (n=2)ＪＲ新宿駅の近くにビールを飲みに行こうか？ＪＲ Shall we go for a beer near JR Shinjuku station? n=2 Ｒ新新宿ＪＲＲ新新宿

32. n-gramming (n=2)ＪＲ新宿駅の近くにビールを飲みに行こうか？ＪＲ Shall we go for a beer near JR Shinjuku station? n=2 Ｒ新新宿宿駅ＪＲＲ新新宿宿駅

33. n-gramming (n=2)ＪＲ新宿駅の近くにビールを飲みに行こうか？ＪＲ Shall we go for a beer near JR Shinjuku station? n=2 Ｒ新新宿宿駅駅のＪＲＲ新新宿宿駅駅の

34. n-gramming (n=2)ＪＲ新宿駅の近くにビールを飲みに行こうか？ＪＲ Shall we go for a beer near JR Shinjuku station? n=2 Ｒ新新宿宿駅駅のの近ＪＲＲ新新宿宿駅駅のの近

35. n-gramming (n=2)ＪＲ新宿駅の近くにビールを飲みに行こうか？ＪＲ Shall we go for a beer near JR Shinjuku station? n=2 Ｒ新新宿宿駅駅のの近近くＪＲＲ新新宿宿駅駅のの近近く

36. Problems with n-gramming

37. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ...

38. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ... ●

39. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ... ● ×

40. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ... ● × ●

41. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ... ● × ● × change of semantics! means ‘post town’, ‘relay station’ or ‘stage’

42. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ... ● × ● × × change of semantics! means ‘post town’, ‘relay station’ or ‘stage’

43. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ... ● × ● × × × change of semantics! means ‘post town’, ‘relay station’ or ‘stage’

44. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ... ● × ● × × × ● change of semantics! means ‘post town’, ‘relay station’ or ‘stage’

45. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ... ● × ● × × × ● change of semantics! means ‘post town’, ‘relay station’ or ‘stage’• Does not preserve meaning well and often changes semantics • Impacts on ranking - search precision (many false positives)Generates many terms per document or queryImpacts on index size and search performanceSometimes appropriate for certain search applicationsCompliance, e-commerce with non product names, ...

46. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ... ● × ● × × × ● change of semantics! means ‘post town’, ‘relay station’ or ‘stage’• Does not preserve meaning well and often changes semantics • Impacts on ranking - search precision (many false positives)• Also generates many terms per document or query • Impacts on index size and performanceSometimes appropriate for certain search applicationsCompliance, e-commerce with non product names, ...

47. Problems with n-gramming ＪＲＲ新新宿宿駅駅のの近近く ... ● × ● × × × ● change of semantics! means ‘post town’, ‘relay station’ or ‘stage’• Does not preserve meaning well and often changes semantics • Impacts on ranking - search precision (many false positives)• Also generates many terms per document or query • Impacts on index size and performance• Still sometimes appropriate for certain search applications • Compliance, e-commerce with special product names, ...

48. Morphological analysisＪＲ新宿駅の近くにビールを飲みに行こうか？ Shall we go for a beer near JR Shinjuku station?

49. Morphological analysisＪＲ新宿駅の近くにビールを飲みに行こうか？ Shall we go for a beer near JR Shinjuku station?ＪＲ新宿駅の近くにビールを飲みに行こうか？

50. Morphological analysisＪＲ新宿駅の近くにビールを飲みに行こうか？ Shall we go for a beer near JR Shinjuku station?ＪＲ新宿駅の近くにビールを飲みに行こうか？ ● ● ● ● ● ● ● ● ● ● ● ● ● ●

51. Morphological analysisＪＲ新宿駅の近くにビールを飲みに行こうか？ Shall we go for a beer near JR Shinjuku station?ＪＲ新宿駅の近くにビールを飲みに行こうか？ ● ● ● ● ● ● ● ● ● ● ● ● ● ● • Tokens reﬂect what a Japanese speaker consider as words • Machine-learned statistical approach • CRFs decoded using Viterbi • Also does part-of-speech tagging, readings for kanji, etc. • Several statistical models available with high accuracy (F > 0.97) • Models/dictionaries are available as IPADIC, UniDic, ...

52. Morphological analysisＪＲ新宿駅の近くにビールを飲みに行こうか？ Shall we go for a beer near JR Shinjuku station?ＪＲ新宿駅の近くにビールを飲みに行こうか？ ● ● ● ● ● ● ● ● ● ● ● ● ● ● • Tokens reﬂect what a Japanese speaker consider as words • Machine-learned statistical approach • Conditional Random Fields (CRFs) decoded using Viterbi • Also does part-of-speech tagging, extract readings for kanji, etc. • Several statistical models available with high accuracy (F > 0.97) • Models/dictionaries are available as IPADIC, UniDic, ...

53. Morphological analysisＪＲ新宿駅の近くにビールを飲みに行こうか？ Shall we go for a beer near JR Shinjuku station?ＪＲ新宿駅の近くにビールを飲みに行こうか？ ● ● ● ● ● ● ● ● ● ● ● ● ● ● • Tokens reﬂect what a Japanese speaker consider as words • Machine-learned statistical approach • Conditional Random Fields (CRFs) decoded using Viterbi • Also does part-of-speech tagging, readings for kanji, etc. • Several statistical models available with high accuracy (F > 0.97) • Models/dictionaries are available as IPADIC, UniDic, ...

54. How does this actually work?

55. Demo

56. Japanese support in Lucene and Solr

57. Japanese in Lucene/Solr

58. Japanese in Lucene/Solr! New feature in Lucene/Solr 3.6

59. Japanese in Lucene/Solr! New feature in Lucene/Solr 3.6! Available out-of-the-box

60. Japanese in Lucene/Solr! New feature in Lucene/Solr 3.6! Available out-of-the-box! Easy to use with reasonable defaults

61. Japanese in Lucene/Solr! New feature in Lucene/Solr 3.6! Available out-of-the-box! Easy to use with reasonable defaults! Provides sophisticated Japanese linguistics

62. Japanese in Lucene/Solr! New feature in Lucene/Solr 3.6! Available out-of-the-box! Easy to use with reasonable defaults! Provides sophisticated Japanese linguistics! Customisable

63. How do we use it?

64. How do we use it? ! Use JapaneseAnalyzer

65. How do we use it? ! Use JapaneseAnalyzer ! Use ﬁeld type “text_ja” in example schema.xml

66. Demo

67. Feature summary / text_ja analyzer chain Segments Japanese text into tokens with very high accuracy JapaneseTokenizer • Token attributes for part-of-speech, base form, readings, etc. • Compound segmentation with compound synonyms • Segmentation is customisable using user dictionaries

68. Feature summary / text_ja analyzer chain Segments Japanese text into tokens with very high accuracy JapaneseTokenizer • Token attributes for part-of-speech, base form, readings, etc. • Compound segmentation with compound synonyms • Segmentation is customisable using user dictionariesJapaneseBaseFormFilter Adjective and verb lemmatisation (by reduction)

69. Feature summary / text_ja analyzer chain Segments Japanese text into tokens with very high accuracy JapaneseTokenizer • Token attributes for part-of-speech, base form, readings, etc. • Compound segmentation with compound synonyms • Segmentation is customisable using user dictionaries JapaneseBaseFormFilter Adjective and verb lemmatisation (by reduction) Stop-words removal based on part-of-speech tagsJapanesePartOfSpeechStopFilter See example/solr/conf/lang/stoptags_ja.txt

70. Feature summary / text_ja analyzer chain Segments Japanese text into tokens with very high accuracy JapaneseTokenizer • Token attributes for part-of-speech, base form, readings, etc. • Compound segmentation with compound synonyms • Segmentation is customisable using user dictionaries JapaneseBaseFormFilter Adjective and verb lemmatisation (by reduction) Stop-words removal based on part-of-speech tagsJapanesePartOfSpeechStopFilter See example/solr/conf/lang/stoptags_ja.txt CJKWidthFilter Character width normalisation (fast Unicode NFKC subset)

71. Feature summary / text_ja analyzer chain Segments Japanese text into tokens with very high accuracy JapaneseTokenizer • Token attributes for part-of-speech, base form, readings, etc. • Compound segmentation with compound synonyms • Segmentation is customisable using user dictionaries JapaneseBaseFormFilter Adjective and verb lemmatisation (by reduction) Stop-words removal based on part-of-speech tagsJapanesePartOfSpeechStopFilter See example/solr/conf/lang/stoptags_ja.txt CJKWidthFilter Character width normalisation (fast Unicode NFKC subset) Stop-words removal StopFilter See example/solr/conf/lang/stopwords_ja.txt

72. Feature summary / text_ja analyzer chain Segments Japanese text into tokens with very high accuracy JapaneseTokenizer • Token attributes for part-of-speech, base form, readings, etc. • Compound segmentation with compound synonyms • Segmentation is customisable using user dictionaries JapaneseBaseFormFilter Adjective and verb lemmatisation (by reduction) Stop-words removal based on part-of-speech tagsJapanesePartOfSpeechStopFilter See example/solr/conf/lang/stoptags_ja.txt CJKWidthFilter Character width normalisation (fast Unicode NFKC subset) Stop-words removal StopFilter See example/solr/conf/lang/stopwords_ja.txt JapaneseKatakanaStemFilter Normalises common katakana spelling variations

73. Feature summary / text_ja analyzer chain Segments Japanese text into tokens with very high accuracy JapaneseTokenizer • Token attributes for part-of-speech, base form, readings, etc. • Compound segmentation with compound synonyms • Segmentation is customisable using user dictionaries JapaneseBaseFormFilter Adjective and verb lemmatisation (by reduction) Stop-words removal based on part-of-speech tagsJapanesePartOfSpeechStopFilter See example/solr/conf/lang/stoptags_ja.txt CJKWidthFilter Character width normalisation (fast Unicode NFKC subset) Stop-words removal StopFilter See example/solr/conf/lang/stopwords_ja.txt JapaneseKatakanaStemFilter Normalises common katakana spelling variations LowerCaseFilter Lowercases

74. Feature details

75. Compound nouns? How do we deal with compound nouns?

76. Compound nouns? How do we deal with compound nouns? Japanese English 関西国際空港 Kansai International Airportシニアソフトウェアエンジニア Senior Software Engineer

77. Compound nouns? How do we deal with compound nouns? Japanese English 関西国際空港 Kansai International Airportシニアソフトウェアエンジニア Senior Software Engineer! These are one word in Japanese, so searching for 空港 (airport) doesn’t match

78. Compound nouns? How do we deal with compound nouns? Japanese English 関西国際空港 Kansai International Airportシニアソフトウェアエンジニア Senior Software Engineer! These are one word in Japanese, so searching for 空港 (airport) doesn’t match! We need to segment the compounds, too

79. Compound segmentation 関西国際空港Kansai International Airportシニアソフトウェアエンジニナ Senior Software Engineer ! We are using a heuristic to implement this

80. Compound segmentation 関西国際空港関西Kansai International Airport Kansaiシニアソフトウェアエンジニナシニア Senior Software Engineer Senior ! We are using a heuristic to implement this

81. Compound segmentation 関西国際空港関西国際Kansai International Airport Kansai Internationalシニアソフトウェアエンジニナシニアソフトウェア Senior Software Engineer Senior Software ! We are using a heuristic to implement this

82. Compound segmentation 関西国際空港関西国際空港Kansai International Airport Kansai International Airportシニアソフトウェアエンジニナシニアソフトウェアエンジニナ Senior Software Engineer Senior Software Engineer ! We are using a heuristic to implement this

83. Compound synonym tokens Position 1 Position 2 Position 3 関西国際空港関西国際空港• Segment the compounds into its part • Good for recall - we can also search and match 空港 (airport)• We keep the compound itself as a synonym • Good for precision with an exact hit because of IDF• Approach beneﬁts both precision and recall for overall good ranking • JapaneseTokenizer actually returns a graph of tokens

84. Compound synonym tokens Position 1 Position 2 Position 3 関西国際空港関西国際空港• Segment the compounds into its parts • Good for recall - we can also search and match 空港 (airport)• We keep the compound itself as a synonym • Good for precision with an exact hit because of IDF• Approach beneﬁts both precision and recall for overall good ranking • JapaneseTokenizer actually returns a graph of tokens

87. Character width normalisation? How do we deal with character widths? Half-width・半角 Full-width・全角 Lucene Ｌｕｃｅｎｅｶﾀｶﾅカタカナ 123 １２３

88. Character width normalisation? How do we deal with character widths? Half-width・半角 Full-width・全角 Lucene Ｌｕｃｅｎｅｶﾀｶﾅカタカナ 123 １２３! Use CJKWidthFilter to normalise them (Unicode NFKC subset) Input text Ｌｕｃｅｎｅｶﾀｶﾅ１２３ CJKWidthFilter Lucene カタカナ 123 half-width full-width half-width

89. Katakana end-vowel stemming? A common spelling variation in katakana is a end long-vowel sound English Japanese spelling variations manager マネージャーマネージャマネジャー

90. Katakana end-vowel stemming ? A common spelling variation in katakana is a end long-vowel sound English Japanese spelling variations manager マネージャーマネージャマネジャー ! We JapaneseKatakanaStemFilter to normalise/stem end-vowel for long terms Input text コピーマネージャーマネージャマネジャーJapaneseKatakanaStemFilter コピーマネージャマネージャマネジャ copy manager manager “manager”

91. Lemmatisation? Japanese adjectives and verbs are highly inﬂected, how do we deal with that?

92. Lemmatisation? Japanese adjectives and verbs are highly inﬂected, how do we deal with that? Dictionary form 買う kau to buy

93. Lemmatisation? Japanese adjectives and verbs are highly inﬂected, how do we deal with that? Dictionary form Inﬂected forms (not exhaustive) 買いなさい買いませんでしたら買える買わせられる買う買いなさるな買いましたら買いませんでしたり買いませんなら買おう買った買わせる買わない買いましたり買うだろう買ったら買わないだろう kau 買いまして買いましょう買うでしょう買うな買ったり買って買わないで買わないでしょう買わせない to buy 買います買うまい買わなかった買いますまい買え買わせます買わなかったら買いませば買えない買わせません買わなかったり買いません買えば買わせられない買わなければ買いませんで買えます買わせられます買われない買いませんでした買えません買わせられません買われます

94. Lemmatisation? Japanese adjectives and verbs are highly inflected, how do we deal with that? Dictionary form Inflected forms (not exhaustive) 買いなさい買いませんでしたら買える買わせられる買う買いなさるな買いましたら買いませんでしたり買いませんなら買おう買った買わせる買わない買いましたり買うだろう買ったら買わないだろう kau 買いまして買いましょう買うでしょう買うな買ったり買って買わないで買わないでしょう買わせない to buy 買います買うまい買わなかった買いますまい買え買わせます買わなかったら買いませば買えない買わせません買わなかったり買いません買えば買わせられない買わなければ買いませんで買えます買わせられます買われない買いませんでした買えません買わせられません買われます ! Use JapaneseBaseformFilter to normalise inflected adjectives and verbs to dictionary form (lemmatisation by reduction)

95. User dictionaries• Own dictionaries can be used for ad hoc segmentation, i.e. to override default model• File format is simple and there’s no need to assign weights, etc. before using them• Example custom dictionary:# Custom segmentation and POS entry for long entries関西国際空港,関西国際空港,カンサイコクサイクウコウ,カスタム名詞# Custom reading and POS former sumo wrestler Asashoryu朝青龍,朝青龍,アサショウリュウ,カスタム人名

96. Japanese focus in 4.0• Improvements in JapaneseTokenizer • Improved search mode for katakana compounds • Improved unknown word segmentation • Some performance improvements• CharFilters for various character normalisations • Dates and numbers • Repetition marks (odoriji)• Japanese spell-checker • Robert and Koji almost got this into 3.6, but it got postponed because of API changes being necessary

97. AcknowledgementsRobert MuirThanks for the heavy lifting integrating Kuromoji into Luceneand always reviewing my patches quickly and friendly helpMichael McCandlessThanks for streaming Viterbi and synonym compounds!Uwe SchindlerThanks for performance improvements + being the policemanSimon WillnauerThanks for doing the Kuromoji code donation process so wellGaute Lambertsen & Gerry HocksThanks for presentation feedback and being great colleagues

98. Q&A

99. ありがとうございました！ arigatō gozaimashita!Thank you very much!