Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Norikra in Action
TAGOMORI Satoshi (@tagomoris)
LINE Corp.
2014/01/31 (Fri) at University of Tsukuba
the 2nd half

14年1月31...
Norikra:
Schema-less Stream Processing with SQL
Open source software (GPLv2)
http://norikra.github.io/
https://github.com/...
Norikra:
Schema-less event stream:
Add/Remove data fields whenever you want
SQL:
No more restarts to add/remove queries
w/ ...
Norikra Queries: (1)

SELECT name, age
FROM events

target

14年1月31日金曜日
Norikra Queries: (1)
{“name”:”tagomoris”,
“age”:34, “address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}

SELECT name, a...
Norikra Queries: (1)
{“name”:”tagomoris”,
“address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}

SELECT name, age
FROM ev...
Norikra Queries: (2)
{“name”:”tagomoris”,
“age”:34, “address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}

SELECT name, a...
Norikra Queries: (2)
{“name”:”kawashima”,
“age”:99, “address”:”Tsukuba”,
“corp”:”Univ”, “current”:”Dream”}

SELECT name, a...
Norikra Queries: (3)

SELECT age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
GROUP BY age

14年1月31日金曜日
Norikra Queries: (3)
{“name”:”tagomoris”,
“age”:34, “address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}

SELECT age, CO...
Norikra Queries: (4)
{“name”:”tagomoris”,
“age”:34, “address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}

SELECT age, CO...
Norikra Queries: (5)
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Tsukuba”,
“speake...
Norikra Queries: (5)
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Tsukuba”,
“speake...
Norikra Queries: (5)
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Kyoto”,
“speaker”...
Norikra and Esper
Esper:
CEP engine library, Java, GPLv2
EPL: Event Processing Language (SQL + window)
Streams: schema-ful...
Norikra query execution
1.accept query
2.parse -> find target / field set
3.(if target is not opened) wait for first event
4....
Target mapping

Ignore unused fields
Field set matching between streams and queries
Generate field set inheritance tree

14年...
automated stream inheritance
of norikra's target
Base fieldset

Query fieldset

Data fieldset

14年1月31日金曜日

b_xxxxxxxxx

mini...
automated stream inheritance
of norikra's target
Base fieldset

b_xxxxxxxxx

Query fieldset

Data fieldset

14年1月31日金曜日

e_xx...
automated stream inheritance
of norikra's target
Base fieldset
event data fieldset definition:
name: 'string'
id: 'long'
vali...
automated stream inheritance
of norikra's target
Base fieldset

b_xxxxxxxxx
new query:
SELECT count(*)
FROM target.win:time...
new query:
SELECT count(*)
FROM target.win:time_batch(1min)
WHERE affiliate.length() > 0

automated stream inheritance
of n...
Registered
automated stream inheritance EPL:
SELECT count(*)
of norikra's targetFROM q_xxxxxxxx0.win:time_batch(1min)
WHER...
automated stream inheritance
of norikra's target
Base fieldset

b_xxxxxxxxx

Query fieldset

q_xxxxxxxx0

q_xxxxxxxx1

Data ...
Query rewriting
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Kyoto”,
“speaker”:true...
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Kyoto”,
“speaker”:true,
“attend”:[true...
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Kyoto”,
“speaker”:true,
“attend”:[true...
{“name”:”tagomoris”,
“user.age”:34,
“user.corp”:”LINE”,
“user.address”:”Tokyo”,
“current”:”Kyoto”,
“speaker”:true,
“attend...
{“user$age”:34,
“current”:”Kyoto”,
“attend$$0”:true,
“attend$$1”:true,
}

SELECT user.age, COUNT(*) as cnt
FROM events.win...
{“user$age”:34,
“current”:”Kyoto”,
“attend$$0”:true,
“attend$$1”:true,
}
SELECT

user.age, COUNT(*) as cnt
FROM events.win...
Norikra internal:
Jump
from schema-full world
to schema-less world.

14年1月31日金曜日
Upcoming SlideShare
Loading in …5
×

Norikra in Action (ver. 2014 spring)

6,144 views

Published on

筑波大学 集中講義資料
2014/01/31

Published in: Technology

Norikra in Action (ver. 2014 spring)

  1. 1. Norikra in Action TAGOMORI Satoshi (@tagomoris) LINE Corp. 2014/01/31 (Fri) at University of Tsukuba the 2nd half 14年1月31日金曜日
  2. 2. Norikra: Schema-less Stream Processing with SQL Open source software (GPLv2) http://norikra.github.io/ https://github.com/norikra/norikra 14年1月31日金曜日
  3. 3. Norikra: Schema-less event stream: Add/Remove data fields whenever you want SQL: No more restarts to add/remove queries w/ JOINs, w/ SubQueries w/ UDF Truly Complex events: Nested Hash/Array, accessible directly from SQL 14年1月31日金曜日
  4. 4. Norikra Queries: (1) SELECT name, age FROM events target 14年1月31日金曜日
  5. 5. Norikra Queries: (1) {“name”:”tagomoris”, “age”:34, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Tsukuba”} SELECT name, age FROM events {“name”:”tagomoris”,”age”:34} 14年1月31日金曜日
  6. 6. Norikra Queries: (1) {“name”:”tagomoris”, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Tsukuba”} SELECT name, age FROM events nothing 14年1月31日金曜日
  7. 7. Norikra Queries: (2) {“name”:”tagomoris”, “age”:34, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Tsukuba”} SELECT name, age FROM events WHERE current=”Tsukuba” {“name”:”tagomoris”,”age”:34} 14年1月31日金曜日
  8. 8. Norikra Queries: (2) {“name”:”kawashima”, “age”:99, “address”:”Tsukuba”, “corp”:”Univ”, “current”:”Dream”} SELECT name, age FROM events WHERE current=”Tsukuba” nothing 14年1月31日金曜日
  9. 9. Norikra Queries: (3) SELECT age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) GROUP BY age 14年1月31日金曜日
  10. 10. Norikra Queries: (3) {“name”:”tagomoris”, “age”:34, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Tsukuba”} SELECT age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) GROUP BY age every 5 mins {”age”:34,”cnt”:3}, {“age”:33,”cnt”:1}, ... 14年1月31日金曜日
  11. 11. Norikra Queries: (4) {“name”:”tagomoris”, “age”:34, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Tsukuba”} SELECT age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) GROUP BY age SELECT max(age) as max FROM events.win:time_batch(5 mins) {”age”:34,”cnt”:3}, {“age”:33,”cnt”:1}, ... {“max”:51} 14年1月31日金曜日 every 5 mins
  12. 12. Norikra Queries: (5) {“name”:”tagomoris”, “user:{“age”:34, “corp”:”LINE”, “address”:”Tokyo”}, “current”:”Tsukuba”, “speaker”:true, “attend”:[true,true,false, ...] } SELECT age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) GROUP BY age 14年1月31日金曜日
  13. 13. Norikra Queries: (5) {“name”:”tagomoris”, “user:{“age”:34, “corp”:”LINE”, “address”:”Tokyo”}, “current”:”Tsukuba”, “speaker”:true, “attend”:[true,true,false, ...] } SELECT user.age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) GROUP BY user.age 14年1月31日金曜日
  14. 14. Norikra Queries: (5) {“name”:”tagomoris”, “user:{“age”:34, “corp”:”LINE”, “address”:”Tokyo”}, “current”:”Kyoto”, “speaker”:true, “attend”:[true,true,false, ...] } SELECT user.age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) WHERE current=”Tsukuba” AND attend.$0 AND attend.$1 GROUP BY user.age 14年1月31日金曜日
  15. 15. Norikra and Esper Esper: CEP engine library, Java, GPLv2 EPL: Event Processing Language (SQL + window) Streams: schema-full flat field set Norikra: Using Esper internally Schema-less stream -> schema-full stream conversion Rewriting compiled queries 14年1月31日金曜日
  16. 16. Norikra query execution 1.accept query 2.parse -> find target / field set 3.(if target is not opened) wait for first event 4.compile query 5.rewrite target name into stream name 6.rewrite field names 7.register query 8.input events 14年1月31日金曜日
  17. 17. Target mapping Ignore unused fields Field set matching between streams and queries Generate field set inheritance tree 14年1月31日金曜日
  18. 18. automated stream inheritance of norikra's target Base fieldset Query fieldset Data fieldset 14年1月31日金曜日 b_xxxxxxxxx minimal fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string'
  19. 19. automated stream inheritance of norikra's target Base fieldset b_xxxxxxxxx Query fieldset Data fieldset 14年1月31日金曜日 e_xxxxxxxx1 event data fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' product_code: 'string' charge: 'integer' shop_code: 'long'
  20. 20. automated stream inheritance of norikra's target Base fieldset event data fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' product_code: 'string' Query fieldset charge: 'integer' shop_code: 'long' affiliate: 'string' Data fieldset 14年1月31日金曜日 e_xxxxxxxx1 b_xxxxxxxxx e_xxxxxxxx2
  21. 21. automated stream inheritance of norikra's target Base fieldset b_xxxxxxxxx new query: SELECT count(*) FROM target.win:time_batch(1min) WHERE affiliate.length() > 0 Query fieldset Data fieldset 14年1月31日金曜日 e_xxxxxxxx1 e_xxxxxxxx2
  22. 22. new query: SELECT count(*) FROM target.win:time_batch(1min) WHERE affiliate.length() > 0 automated stream inheritance of norikra's target Base fieldset b_xxxxxxxxx Query fieldset Data fieldset 14年1月31日金曜日 event data fieldset definition: name: 'string' id: 'long' valid: 'boolean' action_type: 'string' affiliate: 'string' q_xxxxxxxx0 e_xxxxxxxx1 e_xxxxxxxx2'
  23. 23. Registered automated stream inheritance EPL: SELECT count(*) of norikra's targetFROM q_xxxxxxxx0.win:time_batch(1min) WHERE affiliate.length() > 0 Base fieldset b_xxxxxxxxx Query fieldset Data fieldset 14年1月31日金曜日 q_xxxxxxxx0 e_xxxxxxxx1 e_xxxxxxxx2'
  24. 24. automated stream inheritance of norikra's target Base fieldset b_xxxxxxxxx Query fieldset q_xxxxxxxx0 q_xxxxxxxx1 Data fieldset e_xxxxxxxx1' e_xxxxxxxx2' e_xxxxxxxx3' 14年1月31日金曜日
  25. 25. Query rewriting {“name”:”tagomoris”, “user:{“age”:34, “corp”:”LINE”, “address”:”Tokyo”}, “current”:”Kyoto”, “speaker”:true, “attend”:[true,true,false, ...] } SELECT user.age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) WHERE current=”Tsukuba” AND attend.$0 AND attend.$1 GROUP BY user.age 14年1月31日金曜日
  26. 26. {“name”:”tagomoris”, “user:{“age”:34, “corp”:”LINE”, “address”:”Tokyo”}, “current”:”Kyoto”, “speaker”:true, “attend”:[true,true,false, ...] } SELECT user.age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) WHERE current=”Tsukuba” AND attend.$0 AND attend.$1 GROUP BY user.age 14年1月31日金曜日
  27. 27. {“name”:”tagomoris”, “user:{“age”:34, “corp”:”LINE”, “address”:”Tokyo”}, “current”:”Kyoto”, “speaker”:true, “attend”:[true,true,false, ...] } {“name”:”tagomoris”, “user.age”:34, “user.corp”:”LINE”, “user.address”:”Tokyo”, “current”:”Kyoto”, “speaker”:true, “attend.$0”:true, “attend.$1”:true, “attend.$2”:false, ... } 14年1月31日金曜日
  28. 28. {“name”:”tagomoris”, “user.age”:34, “user.corp”:”LINE”, “user.address”:”Tokyo”, “current”:”Kyoto”, “speaker”:true, “attend.$0”:true, “attend.$1”:true, “attend.$2”:false, ... } {“user$age”:34, “current”:”Kyoto”, “attend$$0”:true, “attend$$1”:true, } 14年1月31日金曜日
  29. 29. {“user$age”:34, “current”:”Kyoto”, “attend$$0”:true, “attend$$1”:true, } SELECT user.age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) WHERE current=”Tsukuba” AND attend.$0 AND attend.$1 GROUP BY user.age 14年1月31日金曜日
  30. 30. {“user$age”:34, “current”:”Kyoto”, “attend$$0”:true, “attend$$1”:true, } SELECT user.age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) WHERE current=”Tsukuba” AND attend.$0 AND attend.$1 GROUP BY user.age Conversions for compiled query object in fact. SELECT user$age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) WHERE current=”Tsukuba” AND attend$$0 AND attend$$1 GROUP BY user$age 14年1月31日金曜日
  31. 31. Norikra internal: Jump from schema-full world to schema-less world. 14年1月31日金曜日

×