Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Norikra Recent Updates

6,457 views

Published on

#norikra meetup 2

Published in: Technology
  • Be the first to comment

Norikra Recent Updates

  1. 1. Norikra Recent Updates Norikra meetup #2, 2015/06/03 TAGOMORI Satoshi(@tagomoris)
  2. 2. @tagomoris Treasure Data, Inc. Fluentd, Norikra, Hadoop, ...
  3. 3. Norikra: Open source stream processing server with SQL • v1.0.0: 2014/05/20 • "Stream Processing and Norikra" • http://www.slideshare.net/tagomoris/stream-processing-and-norikra • "Landscape of Norikra Features" • http://www.slideshare.net/tagomoris/norikra-meetup-features • (... 16 releases) • v1.3.1: 2015/05/07
  4. 4. Changes of Norikra • "Changes" section in README • only for major updates • "Changes.md" • for details • https://github.com/norikra/norikra/blob/master/Changes.md • Commit logs
  5. 5. Features newly added • Suspended queries • NULLABLE fields • Listener plugins • Dynamic plugin reloading
  6. 6. Suspended queries • temporarily stop queries / resume these later • good for tests, try&errors • [caution] • suspended queries doesn't saved in stats file
  7. 7. NULLABLE fields • Queries process events just with all fields • NULLABLE(field) • queries handle field as NULL if missing • works well for multi data source situation
  8. 8. NULLABLE fields(2) SELECT a, b, COUNT(*) AS count FROM s.win:time(1 min) GROUP BY a, b {"a":1, "b":2} {"a":2} {"a":1, "b":2, "count":1} {"a":2, "b":2, "count":1} {"a":2, "b":2} ignored!
  9. 9. NULLABLE fields(3) SELECT a, NULLABLE(b), COUNT(*) AS count FROM s.win:time(1 min) GROUP BY a, b {"a":1, "b":2} {"a":2} {"a":1, "b":2, "count":1} {"a":2, "b":2, "count":1} {"a":2, "b":null, "count":1} {"a":2, "b":2}
  10. 10. Listener • Listener • process output data of queries • default listener: memory pool • Built-in listeners: specified by group • LOOPBACK(target) • STDOUT()
  11. 11. Listener plugin: do anything • 3rd party listener plugin • syntax: NAME(arg) • JRuby + rubygem • We can do whatever we want :D • by writing very simple code • https://github.com/norikra/norikra-lilstener-mock
  12. 12. Listener plugin: patterns query push to users enqueue to kafka memory pool filter records other target sample records
  13. 13. Listener plugin: sync vs async Sync listener: > called per every query's output > roughly specified by views (.win:...) > select this if you wanna deliver events ASAP Async listener: > called per specified interval [0.1sec] > process buffered output data > very easy to write
  14. 14. Dynamic plugin reloading • SIGHUP • Norikra loads plugins newly installed in JRuby runtime • doesn't reload plugins already installed • UDF plugins, Listener plugins • Less restarts!
  15. 15. Other updates: • /engine_statistics, --ui-context-path • GC stats in engine statistics • SO_REUSEADDR • Shutoff mode under high memory pressure • --log4j-properties-path, -javaagent • Esper 5.2, msgpack.gem, ... • and many other bug fixes!
  16. 16. Norikra improvement never lasting! Next: • Custom views? Tables? • Query distribution? Thanks!

×