Your SlideShare is downloading. ×
0
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Norikra in action
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Norikra in action

8,128

Published on

Published in: Technology
2 Comments
10 Likes
Statistics
Notes
No Downloads
Views
Total Views
8,128
On Slideshare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
14
Comments
2
Likes
10
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Norikra in action Data/Stream Processing Meetup (2013/06/28) TAGOMORI Satoshi (@tagomoris) 13年6月29日土曜日
  • 2. TAGOMORI Satoshi (@tagomoris) LINE corp. Ruby, Perl, Node.js, Hadoop, ... 13年6月29日土曜日
  • 3. 13年6月29日土曜日
  • 4. System Overview Web Servers Fluentd Cluster Archive Storage (scribed) Fluentd Watchers Graph Tools Notifications (IRC) Hadoop Cluster (HDFS, YARN) webhdfs Huahin Manager hive server STREAM Shib ShibUI BATCH SCHEDULED BATCH Norikra 13年6月29日土曜日
  • 5. Stream query Custom fluentd plugin: not so casual enough xQL: declarative language streams processing for optional data fields no more schema management connectivity with Fluentd 13年6月29日土曜日
  • 6. Stream query: vs stored data query No more query wait time Immediate result for time batch No more storages No more query execution management Once register query, runs forever 13年6月29日土曜日
  • 7. Esper "Esper and Event Processing Language (EPL) provide a highly scalable, memory-efficient, in- memory computing, SQL-standard, minimal latency, real-time streaming Big Data processing engine for medium to high-velocity and high- variety data." http://esper.codehaus.org/ 13年6月29日土曜日
  • 8. Norikra: is not for only Fluentd. 13年6月29日土曜日
  • 9. Norikra query: vs Fluentd custom plugin SQL!!! No more restart for new queries register queries whenever we want No more private plugins No more fat Fluentd configurations 13年6月29日土曜日
  • 10. Norikra Full feature of Esper over JRuby Simple RPC: msgpack-rpc-over-http Simple RPC Server: mizuno (jetty + rack) Simple Client Library: norikra-client Just same code for cruby/jruby 13年6月29日土曜日
  • 11. Norikra Norikra Server (on JVM) Esper Instance (Query Engine) Type Definition Manager Output Event Pool Norikra Engine RPC Server mizuno (Jetty + Rack) Rack RPC Handler Norikra Client Norikra Client JRUBY CRUBY msgpack-rpc-over-http 13年6月29日土曜日
  • 12. Norikra Query: target "sales" goods_id:5 price:49.8 num:1 shop:"LINE" goods_id:2 price:12.5 num:3 shop:"Cookpad" goods_id:4 price:36.6 num:10 shop:"Cookpad" SELECT shop, sum(price*num) AS amount FROM sales.win:time_batch(10 minutes) GROUP BY shop goods_id:5 price:49.8 num:1 shop:"LINE" goods_id:2 price:12.5 num:3 shop:"Cookpad" affiliate:"BiS" SELECT affiliate, count(*) AS cnt FROM sales.win:time_batch(1 hour) GROUP BY affiliate 13年6月29日土曜日
  • 13. Esper and Norikra Esper: queries for streams stream: a set of field-type pairs of events users need to know for field set variations (or manage 'map subtypes' on your own) Norikra: queries for targets target: virtual name of union of field set variation users don't need to know for detail of target 13年6月29日土曜日
  • 14. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx minimal fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' 13年6月29日土曜日
  • 15. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx event data fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' product_code: 'string' charge: 'integer' shop_code: 'long'e_xxxxxxxx1 13年6月29日土曜日
  • 16. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx e_xxxxxxxx1 e_xxxxxxxx2 event data fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' product_code: 'string' charge: 'integer' shop_code: 'long' affiliate: 'string' 13年6月29日土曜日
  • 17. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx e_xxxxxxxx1 e_xxxxxxxx2 new query: SELECT count(*) FROM target.win:time_batch(1min) WHERE affiliate.length() > 0 13年6月29日土曜日
  • 18. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx e_xxxxxxxx1 e_xxxxxxxx2' event data fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' affiliate: 'string' q_xxxxxxxx0 new query: SELECT count(*) FROM target.win:time_batch(1min) WHERE affiliate.length() > 0 13年6月29日土曜日
  • 19. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx e_xxxxxxxx1 e_xxxxxxxx2' q_xxxxxxxx0 Registered EPL: SELECT count(*) FROM q_xxxxxxxx0.win:time_batch(1min) WHERE affiliate.length() > 0 13年6月29日土曜日
  • 20. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx e_xxxxxxxx1' e_xxxxxxxx2' q_xxxxxxxx0 e_xxxxxxxx3' q_xxxxxxxx1 13年6月29日土曜日
  • 21. Output data pooling Output event data: pushed Event pushing brings many problems Pooling + fetch typical usecase: aggregation -> not so many outputs 13年6月29日土曜日
  • 22. fluent-plugin-norikra Fluentd plugin to use Norikra Norikra server autostart Automatically defined target Pre-defined queries for each targets 13年6月29日土曜日
  • 23. fluent-plugin-norikra installation `gem install fluent-plugin-norikra` configuration see DEMO 13年6月29日土曜日
  • 24. Demo: bootstrap rbenv shell jruby-1.7.4 gem install norikra which norikra rbenv shell 2.0.0-pxxx gem install fluent-plugin-norikra vi demo.conf fluentd -c demo.conf 13年6月29日土曜日
  • 25. Demo: query streams some messages over fluent-cat register queries with norikra-client more messages over fluent-cat & norikra-client 13年6月29日土曜日
  • 26. roadmap of norikra Norikra is still UNDER DEVELOPMENT Norikra feature updates (JOINs, etc) Web GUI query & target list management save & restore metadata Distributed & orchestrated nodes 13年6月29日土曜日
  • 27. See also: http://fluentd.org/ http://fluentd.org/plugin/ https://github.com/tagomoris/norikra https://github.com/tagomoris/norikra-client https://github.com/tagomoris/fluent-plugin-norikra http://esper.codehaus.org/ "Fluentd: The ruby based middleware across the world" http://www.slideshare.net/tagomoris/fluentd-in-tkrk10 "Log analysis system with Hadoop in livedoor 2013 Winter" http://www.slideshare.net/tagomoris/log-analysis-with-hadoop-in-livedoor-2013 13年6月29日土曜日

×