This document summarizes recent updates to Norikra, an open source stream processing server. Key updates include: 1) The addition of suspended queries, which allow queries to be temporarily stopped and resumed later, and NULLABLE fields, which handle missing fields as null values. 2) New listener plugins that allow processing query outputs in customizable ways, such as pushing to users, enqueueing to Kafka, or filtering records. 3) Dynamic plugin reloading that loads newly installed plugins without requiring a restart, improving uptime.

1. Norikra Recent Updates
Norikra meetup #2, 2015/06/03
TAGOMORI Satoshi(@tagomoris)

2. @tagomoris
Treasure Data, Inc.
Fluentd, Norikra, Hadoop, ...

3. Norikra:
Open source stream processing server with SQL
• v1.0.0: 2014/05/20
• "Stream Processing and Norikra"
• http://www.slideshare.net/tagomoris/stream-processing-and-norikra
• "Landscape of Norikra Features"
• http://www.slideshare.net/tagomoris/norikra-meetup-features
• (... 16 releases)
• v1.3.1: 2015/05/07

4. Changes of Norikra
• "Changes" section in README
• only for major updates
• "Changes.md"
• for details
• https://github.com/norikra/norikra/blob/master/Changes.md
• Commit logs

5. Features newly added
• Suspended queries
• NULLABLE fields
• Listener plugins
• Dynamic plugin reloading

6. Suspended queries
• temporarily stop queries / resume these later
• good for tests, try&errors
• [caution]
• suspended queries doesn't saved in stats file

7. NULLABLE fields
• Queries process events just with all fields
• NULLABLE(field)
• queries handle field as NULL if missing
• works well for multi data source situation

8. NULLABLE fields(2)
SELECT a, b,
COUNT(*) AS count
FROM s.win:time(1 min)
GROUP BY a, b
{"a":1, "b":2} {"a":2}
{"a":1, "b":2, "count":1}
{"a":2, "b":2, "count":1}
{"a":2, "b":2}
ignored!

9. NULLABLE fields(3)
SELECT a, NULLABLE(b),
COUNT(*) AS count
FROM s.win:time(1 min)
GROUP BY a, b
{"a":1, "b":2} {"a":2}
{"a":1, "b":2, "count":1}
{"a":2, "b":2, "count":1}
{"a":2, "b":null, "count":1}
{"a":2, "b":2}

10. Listener
• Listener
• process output data of queries
• default listener: memory pool
• Built-in listeners: specified by group
• LOOPBACK(target)
• STDOUT()

11. Listener plugin: do anything
• 3rd party listener plugin
• syntax: NAME(arg)
• JRuby + rubygem
• We can do whatever we want :D
• by writing very simple code
• https://github.com/norikra/norikra-lilstener-mock

12. Listener plugin: patterns
query
push to users
enqueue to
kafka
memory
pool
filter records
other target
sample records

13. Listener plugin: sync vs async
Sync listener:
> called per every query's output
> roughly specified by views (.win:...)
> select this if you wanna deliver events ASAP
Async listener:
> called per specified interval [0.1sec]
> process buffered output data
> very easy to write

14. Dynamic plugin reloading
• SIGHUP
• Norikra loads plugins newly installed in
JRuby runtime
• doesn't reload plugins already installed
• UDF plugins, Listener plugins
• Less restarts!

15. Other updates:
• /engine_statistics, --ui-context-path
• GC stats in engine statistics
• SO_REUSEADDR
• Shutoff mode under high memory pressure
• --log4j-properties-path, -javaagent
• Esper 5.2, msgpack.gem, ...
• and many other bug fixes!

16. Norikra improvement
never lasting!
Next:
• Custom views? Tables?
• Query distribution?
Thanks!