Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Norikra in action
Data/Stream Processing Meetup (2013/06/28)
TAGOMORI Satoshi (@tagomoris)
13年6月29日土曜日
TAGOMORI Satoshi (@tagomoris)
LINE corp.
Ruby, Perl, Node.js, Hadoop, ...
13年6月29日土曜日
13年6月29日土曜日
System Overview
Web
Servers Fluentd
Cluster
Archive
Storage
(scribed)
Fluentd
Watchers
Graph
Tools
Notifications
(IRC)
Hado...
Stream query
Custom fluentd plugin: not so casual
enough
xQL: declarative language
streams processing
for optional data fiel...
Stream query:
vs stored data query
No more query wait time
Immediate result for time batch
No more storages
No more query ...
Esper
"Esper and Event Processing Language (EPL)
provide a highly scalable, memory-efficient, in-
memory computing, SQL-sta...
Norikra:
is not for only Fluentd.
13年6月29日土曜日
Norikra query:
vs Fluentd custom plugin
SQL!!!
No more restart for new queries
register queries whenever we want
No more p...
Norikra
Full feature of Esper over JRuby
Simple RPC: msgpack-rpc-over-http
Simple RPC Server: mizuno (jetty + rack)
Simple...
Norikra
Norikra Server (on JVM)
Esper Instance (Query Engine)
Type Definition
Manager
Output Event
Pool
Norikra Engine
RPC ...
Norikra Query: target "sales"
goods_id:5 price:49.8 num:1 shop:"LINE"
goods_id:2 price:12.5 num:3 shop:"Cookpad"
goods_id:...
Esper and Norikra
Esper:
queries for streams
stream: a set of field-type pairs of events
users need to know for field set va...
automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
minimal fieldset defini...
automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
event data fieldset
de...
automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
e_xxxxxxxx1 e_xxxxxxx...
automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
e_xxxxxxxx1 e_xxxxxxx...
automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
e_xxxxxxxx1 e_xxxxxxx...
automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
e_xxxxxxxx1 e_xxxxxxx...
automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
e_xxxxxxxx1' e_xxxxxx...
Output data pooling
Output event data: pushed
Event pushing brings many problems
Pooling + fetch
typical usecase: aggregat...
fluent-plugin-norikra
Fluentd plugin to use Norikra
Norikra server autostart
Automatically defined target
Pre-defined queries...
fluent-plugin-norikra
installation
`gem install fluent-plugin-norikra`
configuration
see DEMO
13年6月29日土曜日
Demo: bootstrap
rbenv shell jruby-1.7.4
gem install norikra
which norikra
rbenv shell 2.0.0-pxxx
gem install fluent-plugin-...
Demo: query streams
some messages over fluent-cat
register queries with norikra-client
more messages over fluent-cat & norik...
roadmap of norikra
Norikra is still UNDER DEVELOPMENT
Norikra feature updates (JOINs, etc)
Web GUI
query & target list man...
See also:
http://fluentd.org/
http://fluentd.org/plugin/
https://github.com/tagomoris/norikra
https://github.com/tagomoris/n...
Upcoming SlideShare
Loading in …5
×

Norikra in action

11,393 views

Published on

Published in: Technology

Norikra in action

  1. 1. Norikra in action Data/Stream Processing Meetup (2013/06/28) TAGOMORI Satoshi (@tagomoris) 13年6月29日土曜日
  2. 2. TAGOMORI Satoshi (@tagomoris) LINE corp. Ruby, Perl, Node.js, Hadoop, ... 13年6月29日土曜日
  3. 3. 13年6月29日土曜日
  4. 4. System Overview Web Servers Fluentd Cluster Archive Storage (scribed) Fluentd Watchers Graph Tools Notifications (IRC) Hadoop Cluster (HDFS, YARN) webhdfs Huahin Manager hive server STREAM Shib ShibUI BATCH SCHEDULED BATCH Norikra 13年6月29日土曜日
  5. 5. Stream query Custom fluentd plugin: not so casual enough xQL: declarative language streams processing for optional data fields no more schema management connectivity with Fluentd 13年6月29日土曜日
  6. 6. Stream query: vs stored data query No more query wait time Immediate result for time batch No more storages No more query execution management Once register query, runs forever 13年6月29日土曜日
  7. 7. Esper "Esper and Event Processing Language (EPL) provide a highly scalable, memory-efficient, in- memory computing, SQL-standard, minimal latency, real-time streaming Big Data processing engine for medium to high-velocity and high- variety data." http://esper.codehaus.org/ 13年6月29日土曜日
  8. 8. Norikra: is not for only Fluentd. 13年6月29日土曜日
  9. 9. Norikra query: vs Fluentd custom plugin SQL!!! No more restart for new queries register queries whenever we want No more private plugins No more fat Fluentd configurations 13年6月29日土曜日
  10. 10. Norikra Full feature of Esper over JRuby Simple RPC: msgpack-rpc-over-http Simple RPC Server: mizuno (jetty + rack) Simple Client Library: norikra-client Just same code for cruby/jruby 13年6月29日土曜日
  11. 11. Norikra Norikra Server (on JVM) Esper Instance (Query Engine) Type Definition Manager Output Event Pool Norikra Engine RPC Server mizuno (Jetty + Rack) Rack RPC Handler Norikra Client Norikra Client JRUBY CRUBY msgpack-rpc-over-http 13年6月29日土曜日
  12. 12. Norikra Query: target "sales" goods_id:5 price:49.8 num:1 shop:"LINE" goods_id:2 price:12.5 num:3 shop:"Cookpad" goods_id:4 price:36.6 num:10 shop:"Cookpad" SELECT shop, sum(price*num) AS amount FROM sales.win:time_batch(10 minutes) GROUP BY shop goods_id:5 price:49.8 num:1 shop:"LINE" goods_id:2 price:12.5 num:3 shop:"Cookpad" affiliate:"BiS" SELECT affiliate, count(*) AS cnt FROM sales.win:time_batch(1 hour) GROUP BY affiliate 13年6月29日土曜日
  13. 13. Esper and Norikra Esper: queries for streams stream: a set of field-type pairs of events users need to know for field set variations (or manage 'map subtypes' on your own) Norikra: queries for targets target: virtual name of union of field set variation users don't need to know for detail of target 13年6月29日土曜日
  14. 14. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx minimal fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' 13年6月29日土曜日
  15. 15. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx event data fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' product_code: 'string' charge: 'integer' shop_code: 'long'e_xxxxxxxx1 13年6月29日土曜日
  16. 16. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx e_xxxxxxxx1 e_xxxxxxxx2 event data fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' product_code: 'string' charge: 'integer' shop_code: 'long' affiliate: 'string' 13年6月29日土曜日
  17. 17. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx e_xxxxxxxx1 e_xxxxxxxx2 new query: SELECT count(*) FROM target.win:time_batch(1min) WHERE affiliate.length() > 0 13年6月29日土曜日
  18. 18. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx e_xxxxxxxx1 e_xxxxxxxx2' event data fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' affiliate: 'string' q_xxxxxxxx0 new query: SELECT count(*) FROM target.win:time_batch(1min) WHERE affiliate.length() > 0 13年6月29日土曜日
  19. 19. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx e_xxxxxxxx1 e_xxxxxxxx2' q_xxxxxxxx0 Registered EPL: SELECT count(*) FROM q_xxxxxxxx0.win:time_batch(1min) WHERE affiliate.length() > 0 13年6月29日土曜日
  20. 20. automated stream inheritance of norikra's target Base typedef Query typedef Data typedef b_xxxxxxxxx e_xxxxxxxx1' e_xxxxxxxx2' q_xxxxxxxx0 e_xxxxxxxx3' q_xxxxxxxx1 13年6月29日土曜日
  21. 21. Output data pooling Output event data: pushed Event pushing brings many problems Pooling + fetch typical usecase: aggregation -> not so many outputs 13年6月29日土曜日
  22. 22. fluent-plugin-norikra Fluentd plugin to use Norikra Norikra server autostart Automatically defined target Pre-defined queries for each targets 13年6月29日土曜日
  23. 23. fluent-plugin-norikra installation `gem install fluent-plugin-norikra` configuration see DEMO 13年6月29日土曜日
  24. 24. Demo: bootstrap rbenv shell jruby-1.7.4 gem install norikra which norikra rbenv shell 2.0.0-pxxx gem install fluent-plugin-norikra vi demo.conf fluentd -c demo.conf 13年6月29日土曜日
  25. 25. Demo: query streams some messages over fluent-cat register queries with norikra-client more messages over fluent-cat & norikra-client 13年6月29日土曜日
  26. 26. roadmap of norikra Norikra is still UNDER DEVELOPMENT Norikra feature updates (JOINs, etc) Web GUI query & target list management save & restore metadata Distributed & orchestrated nodes 13年6月29日土曜日
  27. 27. See also: http://fluentd.org/ http://fluentd.org/plugin/ https://github.com/tagomoris/norikra https://github.com/tagomoris/norikra-client https://github.com/tagomoris/fluent-plugin-norikra http://esper.codehaus.org/ "Fluentd: The ruby based middleware across the world" http://www.slideshare.net/tagomoris/fluentd-in-tkrk10 "Log analysis system with Hadoop in livedoor 2013 Winter" http://www.slideshare.net/tagomoris/log-analysis-with-hadoop-in-livedoor-2013 13年6月29日土曜日

×