Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

非構造データをAsakusaで扱うための考え方と実装

540 views

Published on

  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

非構造データをAsakusaで扱うための考え方と実装

  1. 1. 㠀ᵓ㐀䝕䞊䝍䜢Asakusa䛷ᢅ 䛖䛯䜑䛾⪃䛘᪉䛸ᐇ⿦ @pochi_black
  2. 2. @pochi_black.inspect ● Ruby䛜኱ዲ䛝 ● ᭱㏆䛿Elixir䜔䛳䛶䜎䛩 ● 䝡䝑䜾䝕䞊䝍䜔䛳䛶䜎䛩 ● AsakusaṔ2ᖺ䠛 ● ึLT
  3. 3. Asakusa䝞䝑䝏㛤Ⓨ ● 4✀㢮䛾䝻䜾䝣䜯䜲䝹䜢⤖ྜ䛧䛶ᑟ⥺䜢సᡂ ● 3✀㢮䛜AP䝃䞊䝞䛾㠀ᵓ㐀䝻䜾 ● 㠀ᵓ㐀䝻䜾䛿ᚲせ䛺᝟ሗ䜢䜰䜴䝖䝥䝑䝖䛩䜛タィ䛿䛥䜜䛶䛔 䜛䛜䚸ศᯒ䛥䜜䜛๓ᥦ䛾タィ䛿䛥䜜䛶䛔䛺䛔 ログA ログB 中間導線1 中間導線2 導線 ログD ログF
  4. 4. ௒᪥䛾䛚ヰ DMDL䛷ᆺ䜢䜒䛳䛯Asakusa䛻ᑐ䛧䛶 䝻䜾䛾䜘䛖䛺ศᯒ䜢ព㆑䛧䛶䛔䛺䛔䝣䜯䜲䝹䜢 䛹䛖ᢅ䛳䛯䛛䛸䛔䛖䛣䛸䛻䛴䛔䛶䛚ヰ䛧䛧䜎䛩
  5. 5. 㠀ᵓ㐀䝕䞊䝍䜈䛾䜰䝥䝻䞊䝏 ● TSVFormat䛸CSVFormat䛷ᑐᛂ䛩䜛 ● Asakusa䛻⊂⮬䝣䜷䞊䝬䝑䝖䜢㏣ຍ䛩䜛 ● Asakusa䛻ㄞ䜏㎸䜎䛫䜛ᡭ๓䛻⣲䛾 MapReduce䛷஦๓ฎ⌮䜢⾜䛖
  6. 6. 㠀ᵓ㐀䝻䜾䛾ྲྀ䜚ᢅ䛔 ● Asakusaⓗ䜰䝥䝻䞊䝏䛷ྲྀ䜚⤌䜐 ○ DirectIO/CSV䝣䜯䜲䝹㐃ᦠ䜢౑䛖 ○ DirectIO/TSV䝣䜯䜲䝹㐃ᦠ䜢౑䛖 ● 䝻䜾䛻䛿ព࿡䛾ษ䜜┠䛷䛿䛺䛔䛸䛣䜝䛻”,”䛛䝍 䝤䜻䞊䛜ධ䛳䛶䛔䜛䛸䜶䝷䞊䛻䛺䜛
  7. 7. 㠀ᵓ㐀䝻䜾䛾ྲྀ䜚ᢅ䛔 ● OSSⓗ䜰䝥䝻䞊䝏䛷ྲྀ䜚⤌䜐 ○ TSVParser䛾䜘䛖䛺ឤ䛨䛷ఱ䜒䝟䞊䝇䛧䛺䛔Parser䜢స䜜 䜀䛔䛔䜣䛨䜓䛺䛔䛛 ● ᩘ䚻䛾୙Ᏻ ○ Javaຊప䛔䞉䞉䞉 ○ RecordParser࿘䜚䛻኱䛝䛺ኚ᭦㉳䛝䛯䜙䛹䛖䛧䜘䛖 ○ 䝗䜻䝳䝯䞁䝖ᮍᩚഛ䠚䠘
  8. 8. 㠀ᵓ㐀䝻䜾䛾ྲྀ䜚ᢅ䛔 ● Asakusa䛻䜲䞁䝥䝑䝖䛩䜛஦๓ฎ⌮䛷ᑐᛂ ○ ༢⣧䛺MapReduce䛷䝻䜾䛜Asakusa䛻ㄞ䜏㎸䜑䛶䛛䛴 ព࿡䛜䛺䛟䛺䜙䛺䛔䜘䛖䛻ຍᕤ ● ᭱ప㝈䛾ฎ⌮䛰䛡ᐇ᪋ ○ TSV䛻䠍⾜䛸䛧䛶ㄞ䜎䛫䜛䛯䜑䛻䝍䝤䜢๐㝖 ○ ୙Ᏻ䜢ᾘ䛩䛯䜑䛻኱㔞䛾༢య䝔䝇䝖䜢᭩䛝䜎䛧䛯
  9. 9. ᭱⤊⣔ 非構造ログ 事前処理 MapReduce Asakusa 導線
  10. 10. 䜎䛸䜑 ● Asakusa䛷䝞䝑䝏䜢᭩䛟䛸Ᏻᚰឤ䛜䛒䜛 ● 䝍䝤䜔CSV䜢ព㆑䛷䛝䛶䛔䛺䛔䝻䜾䝣䜯䜲䝹䛾ᢅ 䛔䛿ᮍ䛰䛻ṇ䛧䛔䜔䜚᪉䛜ศ䛛䜙䛺䛔 ● DMDL䝥䝷䜾䜲䞁䝗䜻䝳䝯䞁䝖䛜䛒䜜䜀䛜䜣䜀䜜䛯 䛛䜒䞉䞉䞉
  11. 11. 䛒䜚䛜䛸䛖䛤䛦䛔䜎䛧䛯

×