Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
JRuby with Java Code
in Data Processing World
JRubyConf.EU at 31 Jul 2015
Satoshi Tagomori (@tagomoris)
Satoshi "Moris" Tagomori
(@tagomoris)
Fluentd, Norikra, MessagePack-Ruby,...
Docker logging driver for Fluentd (docker v1....
https://jobs.lever.co/treasure-data
We're hiring!
OSS team (developer / community manager)
Distributed system engineer (Ha...
Data Processing World
Data Processing World
Java
Data Processing World
Data Processing World
Hadoop, Spark, Tez, Flink, Storm, Kafka, ...
Hive, Pig, Drill, Impala, Presto, ....
Java + Scala, Clojure + C++, ....
Data Processing World
on JVM
Data Processing World
Many CPU cores, Large memory, High rate Disk I/O, ...
High throughput data processing
Hadoop YARN/Ma...
Two OSS using Java&JRuby
Norikra:
Stream Processing with SQL for everybody
Server software, written in JRuby, runs on JVM
Open source software (GPL...
What Norikra does:
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path O...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
{"pa...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
{"pa...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
{"pa...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
{"pa...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
{"pa...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
{"pa...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
{"pa...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
{"pa...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
{"pa...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
{"pa...
SELECT path, SUM(bytes) AS s
FROM www_access_logs.win:length_batch(10)
WHERE status=200
GROUP BY path ORDER BY s DESC
10
{...
Norikra and Java
Norikra is written in JRuby, and using Esper
Key factor: productivity (33days until first release)
Esper:J...
Plugins
as rubygems
Norikra Server (on JVM)
Esper (Query Engine)
Type Definition

Manager
Output Event
Pool
Norikra Engine
...
Embulk
"Embulk is a open-source bulk data loader
that helps data transfer between various
databases, storages, file formats...
Embulk:
makes painful data integration work relaxed
Plugin-based parallel bulk data loader
Open source software (Apache Li...
HDFS
MySQL
Amazon S3
Embulk
CSV Files
SequenceFile
Salesforce.com
Elasticsearch
Cassandra
Hive
Redis
✓ Parallel execution
...
#ccc_cd4 / #embulk
InputPlugin OutputPlugin
Executor plugin
Filter plugin
Filter plugin
Filter plugins
records
Threads,
Ma...
#ccc_cd4 / #embulk
InputPlugin
FileInput plugin
OutputPlugin
FileOutput plugin
Encoder plugin
Formatter plugin
Decoder plu...
Embulk and Java
Embulk core is written in Java
mainly for performance
Embulk plugins:
are loaded over API based on JRuby
a...
InputPlugin
module Embulk
class InputExample < InputPlugin
Plugin.register_input('example', self)
def self.transaction(con...
OutputPlugin
module Embulk
class OutputExample < OutputPlugin
Plugin.register_output('example', self)
def self.transaction...
Plugin management: Norikra
Esper instance
Engine
Plugin management
UDF Listener
plugins as gems
plugin loader written in
J...
Plugin management: Embulk
Embulk core
Plugin management
input/output/filter
parser/formatter
Java JRuby
decoder/encoder
file...
Pluggable software
on JVM & Java API
Java? Scala? Clojure? JRuby?: JRuby
Plugin packaging: jar? gem?: gem
rubygem.org >>> ...
JRuby in Japan
Not so many users :(
CRuby is super major software in Japan
Java -> Ruby -> Scala? Golang?
Make your software pluggable.
Make eco-system&community.
with JRuby!
Thanks!
Upcoming SlideShare
Loading in …5
×

JRuby with Java Code in Data Processing World

5,735 views

Published on

presentation in JRubyConf.EU 2015

Published in: Technology
  • Be the first to comment

JRuby with Java Code in Data Processing World

  1. 1. JRuby with Java Code in Data Processing World JRubyConf.EU at 31 Jul 2015 Satoshi Tagomori (@tagomoris)
  2. 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, Norikra, MessagePack-Ruby,... Docker logging driver for Fluentd (docker v1.8) Treasure Data, Inc.
  3. 3. https://jobs.lever.co/treasure-data We're hiring! OSS team (developer / community manager) Distributed system engineer (Hadoop, queue/workers) Front-end engineer (RoR)
  4. 4. Data Processing World
  5. 5. Data Processing World
  6. 6. Java Data Processing World
  7. 7. Data Processing World Hadoop, Spark, Tez, Flink, Storm, Kafka, ... Hive, Pig, Drill, Impala, Presto, ....
  8. 8. Java + Scala, Clojure + C++, .... Data Processing World on JVM
  9. 9. Data Processing World Many CPU cores, Large memory, High rate Disk I/O, ... High throughput data processing Hadoop YARN/MapReduce/HDFS API compatibility
  10. 10. Two OSS using Java&JRuby
  11. 11. Norikra: Stream Processing with SQL for everybody Server software, written in JRuby, runs on JVM Open source software (GPLv2) http://norikra.github.io/ https://github.com/norikra/norikra Distributed on rubygems.org "gem i norikra"
  12. 12. What Norikra does: SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC
  13. 13. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":200, "bytes":300, "duration":0.03, "referer":"...", "user-agent":"...." path:"/", s:301 1
  14. 14. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/download/a", "status":200, "bytes":10240, "duration":0.53, "referer":"...", "user-agent":"...." path:"/", s:301 path:"/download/a", s:10240 2
  15. 15. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":404, "bytes":0, "duration":0.08, "referer":"...", "user-agent":"...." path:"/", s:301 path:"/download/a", s:10240 3
  16. 16. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":200, "bytes":301, "duration":0.01, "referer":"...", "user-agent":"...." path:"/", s:602 path:"/download/a", s:10240 4
  17. 17. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/download/b", "status":200, "bytes":678, "duration":0.11, "referer":"...", "user-agent":"...." path:"/", s:602 path:"/download/a", s:10240 path:"/download/b", s:678 5
  18. 18. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/download/b", "status":200, "bytes":678, "duration":0.13, "referer":"...", "user-agent":"...." path:"/", s:602 path:"/download/a", s:10240 path:"/download/b", s:1356 6
  19. 19. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":200, "bytes":301, "duration":0.02, "referer":"...", "user-agent":"...." path:"/", s:903 path:"/download/a", s:10240 path:"/download/b", s:1356 7
  20. 20. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":200, "bytes":301, "duration":0.09, "referer":"...", "user-agent":"...." path:"/", s:1204 path:"/download/a", s:10240 path:"/download/b", s:1356 8
  21. 21. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/download/a", "status":200, "bytes":10240, "duration":1.1, "referer":"...", "user-agent":"...." path:"/", s:1204 path:"/download/a", s:20480 path:"/download/b", s:1356 9
  22. 22. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":200, "bytes":301, "duration":0.05, "referer":"...", "user-agent":"...." path:"/", s:1505 path:"/download/a", s:20480 path:"/download/b", s:1356 10
  23. 23. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC 10 {"path":"/download/a", "s":20480} {"path":"/", "s":1505} {"path":"/download/b", "s":1356}
  24. 24. Norikra and Java Norikra is written in JRuby, and using Esper Key factor: productivity (33days until first release) Esper:Java library, provides Complex Event Processing SQL parser, executor Many features and good performance Licensed under GPLv2
  25. 25. Plugins as rubygems Norikra Server (on JVM) Esper (Query Engine) Type Definition Manager Output Event Pool Norikra Engine RPC Server mizuno (Jetty + Rack) Rack RPC Handler Listener UDF UDF User-Defined Functions "gem i norikra-udf-xxx" written in Java, or JRuby (compiled to Java) works in Esper instance: must be a Java class Listener handler for output data of queries, written in JRuby "gem i norikra-listener-xxx"
  26. 26. Embulk "Embulk is a open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services." http://www.embulk.org/docs/
  27. 27. Embulk: makes painful data integration work relaxed Plugin-based parallel bulk data loader Open source software (Apache License v2.0) http://www.embulk.org/ https://github.com/embulk/embulk Distributed as .jar or on rubygems.org Plugins are on rubygems.org http://www.slideshare.net/frsyuki/embuk-making-data-integration-works-relaxed http://www.slideshare.net/HiroshiNakamura/embulk-20150411
  28. 28. HDFS MySQL Amazon S3 Embulk CSV Files SequenceFile Salesforce.com Elasticsearch Cassandra Hive Redis ✓ Parallel execution ✓ Data validation ✓ Error recovery ✓ Deterministic behavior ✓ Idempotet retrying Plugins Plugins bulk load
  29. 29. #ccc_cd4 / #embulk InputPlugin OutputPlugin Executor plugin Filter plugin Filter plugin Filter plugins records Threads, MapReduce records convert, … input, … output. 29 records config
  30. 30. #ccc_cd4 / #embulk InputPlugin FileInput plugin OutputPlugin FileOutput plugin Encoder plugin Formatter plugin Decoder plugin Parser plugin HDFS, S3,
 Riak CS, … gzip, bzip2,
 aes, … CSV, JSON,
 pcap, … buffer buffer buffer buffer Filter plugin Filter plugin Filter plugins recordsrecords Executor plugin 30 records config
  31. 31. Embulk and Java Embulk core is written in Java mainly for performance Embulk plugins: are loaded over API based on JRuby are written in JRuby or Java JRuby for early release Java for performance
  32. 32. InputPlugin module Embulk class InputExample < InputPlugin Plugin.register_input('example', self) def self.transaction(config, &control) # read config task = { 'message' => config.param('message', :string, default: nil) } threads = config.param('threads', :int, default: 2) columns = [ Column.new(0, 'col0', :long), Column.new(1, 'col1', :double), Column.new(2, 'col2', :string), ] # BEGIN here commit_reports = yield(task, columns, threads) # COMMIT here puts "Example input finished" return {} end def run(task, schema, index, page_builder) puts "Example input thread #{@index}…" 10.times do |i| @page_builder.add([i, 10.0, "example"]) end @page_builder.finish commit_report = { } return commit_report end end end
  33. 33. OutputPlugin module Embulk class OutputExample < OutputPlugin Plugin.register_output('example', self) def self.transaction( config, schema, processor_count, &control) # read config task = { 'message' => config.param('message', :string, default: "record") } puts "Example output started." commit_reports = yield(task) puts "Example output finished. Commit reports = #{commit_reports.to_json}" return {} end def initialize(task, schema, index) puts "Example output thread #{index}..." super @message = task.prop('message', :string) @records = 0 end def add(page) page.each do |record| hash = Hash[schema.names.zip(record)] puts "#{@message}: #{hash.to_json}" @records += 1 end end def finish end def abort end def commit commit_report = { "records" => @records } return commit_report end end end
  34. 34. Plugin management: Norikra Esper instance Engine Plugin management UDF Listener plugins as gems plugin loader written in JRuby Java JRuby
  35. 35. Plugin management: Embulk Embulk core Plugin management input/output/filter parser/formatter Java JRuby decoder/encoder file-input/output executor plugins as gems plugin loader written in JRuby
  36. 36. Pluggable software on JVM & Java API Java? Scala? Clojure? JRuby?: JRuby Plugin packaging: jar? gem?: gem rubygem.org >>> maven central (or others) especially for plugin authors Plugin loader: Class Loader? "require"?: require
  37. 37. JRuby in Japan Not so many users :( CRuby is super major software in Japan Java -> Ruby -> Scala? Golang?
  38. 38. Make your software pluggable. Make eco-system&community. with JRuby! Thanks!

×