Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Three Ruby usages 
Kouhei Sutou 
ClearCode Inc. 
RubyKaigi 2014 
2014/09/20 
Three Ruby usages Powered by Rabbit 2.1.4
Silver sponsor 
Three Ruby usages Powered by Rabbit 2.1.4
Goal 
You know three Ruby usages 
✓High-level interface 
✓Glue 
✓Embed 
✓ 
✓You can remember them later 
Three Ruby usages...
Targets 
High-level interface 
✓Pure Rubyists 
✓ 
Glue 
✓Rubyists who can write C/C++ 
✓ 
Embed 
✓Rubyists who also write ...
Case study 
Implement distributed 
full-text search engine in 
Ruby 
Abbreviation: DFTSE = Distributed Full-Text Search En...
DFTSE? 
Distrubuted 
Full- 
Text 
Search 
Engine 
Full- 
Text 
Search 
Engine 
1: Full-text search 
2: Distribute 
sub req...
Why do we use DFTSE? 
I'm developing 
Droonga 
(A DFTSE implementation in Ruby) 
 
Three Ruby usages Powered by Rabbit 2....
High-level interface 
Three Ruby usages 
High-level interface 
Target: ✓ Pure Rubyists 
✓ 
✓Glue 
✓Embed 
Three Ruby usage...
High-level interface 
Provides 
lower layer feature to 
higher layer 
✓ 
With simpler/✓ convenience API 
Three Ruby usages...
High-level interface 
Higher layer users 
High-level 
interface 
Feature 
by Yukihiro Matsumoto 
Application/Library 
Thre...
Example 
Vagrant Active Record 
Developers 
Vagrantfile 
Build 
development 
environment 
Vagrant 
b y Y u k i h i r o M a...
Droonga: High-level IF 
DFTSE components 
Full-✓ text search engine 
✓Messaging system 
✓Cluster management 
✓Process mana...
Messaging system 
1: Full-text search DTFSE 
FTSE 
2: Distribute 
sub requests 
3: Merge 
responses 
Worker 
process 
Mess...
Messaging system 
Provides 
distributed search feature 
✓Plan how to search 
✓Distribute requests 
✓Merge responses 
✓ 
✓U...
Characteristic 
Plan how to search 
May speed up/✓ down over 100 times 
✓ 
Distribute requests 
✓Network bound operation 
...
Point 
Algorithm is important 
Need to find new/existing better 
algorithm 
✓ 
"Rapid prototype and measure" 
feedback loo...
Glue 
Three Ruby usages 
✓High-level interface 
Glue 
Target: 
Rubyists who can write C/C++ 
✓ 
✓ 
✓Embed 
Three Ruby usag...
Glue 
Export 
a feature 
Feature 
Combine 
features 
Glue 
Ruby 
Other 
Language 
Three Ruby usages Powered by Rabbit 2.1....
Example 
mysql2 gem 
Vagrant 
Access to MySQL VM Provision 
libmysqlclient.so 
(VirtualBox) (Chef) 
Feature 
Active Record...
Why do we glue? 
Reuse ✓ existing features 
Three Ruby usages Powered by Rabbit 2.1.4
How to glue 
Use external library 
Implement ✓ bindings (mysql2 gem) 
✓ 
Use external command 
✓Spawn command (Vagrant) 
✓...
Glue in Droonga 
Rroonga: Groonga bindings 
Groonga: ✓ FTSE C library (and server) 
✓ 
Cool.io: libev bindings 
libev: Eve...
Rroonga in Droonga 
1: Full-text search DTFSE 
FTSE 
2: Distribute 
sub requests 
3: Merge 
responses 
Worker 
process 
Me...
FTSE in Droonga 
✓Must be fast! 
✓CPU bound processing 
Three Ruby usages Powered by Rabbit 2.1.4
For fast Rroonga 
Do heavy processing in C 
Nice to ✓ have Ruby-ish API 
✓ 
Less memory allocation 
✓Cache internal buffer...
Search 
Groonga::Database.open(ARGV[0]) 
entries = Groonga["Entries"] 
entries.select do |record| 
record.description =~ "...
Search - Pure Ruby (ref) 
Groonga::Database.open(ARGV[0]) 
entries = Groonga["Entries"] 
entries.find_all do |record| 
# T...
Search impl. 
# (2) Evaluate expression in C 
entries.select do |record| 
# (1) Build expression in Ruby 
# This block is ...
Search impl. - Fig. 
Ruby 
by Groonga project 
C 
entries.select do |record| 
record.description =~ "Ruby" 
end 
Build Sea...
Search - Benchmark 
✓Ruby (It's already showed) 
✓C 
Three Ruby usages Powered by Rabbit 2.1.4
Search - C 
grn_obj *expr; 
grn_obj *variable; 
const gchar *filter = "description @ "Ruby""; 
grn_obj *result; 
GRN_EXPR_...
Search - Benchmark 
Ruby impl. is fast enough  
Impl. Elapsed time 
C 0.6ms 
Ruby 0.8ms 
(Full-text search with "Ruby" ag...
Embed 
Three Ruby usages 
✓High-level interface 
✓Glue 
Embed 
Target: 
Rubyists who also write C/C++ 
✓ 
✓ 
Three Ruby us...
Embed 
Internal engine Interface 
C/C++ application 
C/C++ library 
Implement 
some features 
in Ruby 
Plugin API 
Conifug...
Examples 
Internal engine Interface 
Implement 
query optimizer 
in Ruby 
vim-ruby by Yukihiro Matsumoto 
by Yukihiro Mats...
Embed in Droonga 
by Yukihiro Matsumoto 
by Yukihiro Matsumoto 
... 
by The Groonga Project 
by The Groonga Project 
by Th...
CRuby vs. mruby 
CRuby 
✓Full featured! 
✓Signal handler isn't needed  
✓ 
mruby 
✓Multi-interpreters in a process! 
✓You...
mruby in Groonga 
✓Query optimizer 
Command interface (plan) 
✓Interface and also high-level interface! 
✓ 
Plugin API (pl...
Query optimizer 
Query 
Optimize Optimized 
Query 
Optimizer 
query 
Full-text search 
Evaluator 
by Yukihiro Matsumoto 
R...
Query optimizer 
Plan how to search 
✓It's a bother  
✓ 
✓Light operation than FTS 
Depends on data 
(Choose effective in...
Example 
rank < 200 && rank > 100 
Three Ruby usages Powered by Rabbit 2.1.4
Simple impl. 
rank 1 2 ... 100 ... 200 ... ... 
10000 
rank > 100 
rank < 200 
101 199 
&& 
101 ... 199 
rank < 200 && ran...
Simple impl. 
Slow against 
many out of range data 
✓ 
Three Ruby usages Powered by Rabbit 2.1.4
Optimized impl. 
rank 1 2 ... 100 101 ... 199 
200 ... ... 
10000 
100 < rank < 200 
101 ... 199 
rank < 200 && rank > 100...
Is embedding reasonable? 
Measure 
Three Ruby usages Powered by Rabbit 2.1.4
Measure 
✓mruby overhead 
✓Speed-up by optimization 
Three Ruby usages Powered by Rabbit 2.1.4
Overhead 
Small overhead: Reasonable 
# conds mruby Elapsed 
1 ○ 0.24ms 
1 × 0.16ms 
4 ○ 0.45ms 
4 × 0.19ms 
Three Ruby u...
Speed-up 
Fast for many data:Reasonable 
# records mruby no mruby 
1000 0.29ms 0.31ms 
10000 0.31ms 2.3ms 
100000 0.26ms ...
Note 
Embedding needs many works 
Write bindings, import mruby your 
build system and ... 
✓ 
✓ 
How to test your mruby pa...
Conclusion
Conclusion 1 
Describe three Ruby usages 
✓High-level interface 
✓Glue 
✓Embed 
✓ 
Three Ruby usages Powered by Rabbit 2.1...
Conclusion 2 
High-level interface 
✓ Target: Pure Rubyists 
✓ 
Provides lower layer feature to 
higher layer w/ usable in...
Conclusion 3 
Glue 
Target: 
Rubyists who can write C/C++ 
✓ 
Why: Reuse ✓ existing feature 
✓To be fast, do the process i...
Conclusion 4 
Embed 
Target: 
Rubyists who also write C/C++ 
✓ 
Why: 
Avoid bother programming by Ruby 
✓ 
✓ 
Three Ruby u...
Conclusion 5 
Embed 
Is it reasonable ✓ for your case? 
✓You need many works 
✓ 
Very powerful 
if your case is reasonable...
Announcement 
ClearCode Inc. 
✓A silver sponsor 
✓Is recruiting 
✓Will do readable code workshop 
✓ 
The next Groonga conf...
Upcoming SlideShare
Loading in …5
×

Three Ruby usages

914 views

Published on

This talk describes three Ruby usages:

* Implementing high-level interface by Ruby.
* Using Ruby as a glue language.
* Embedding Ruby into a C program for flexibility.

All of them have pros and cons. They are trade-off. If a case can
ignore cons, we will have pros for the case.

Most of Rubyists implement their Ruby applications only by Ruby. This
talk shows another options to use Ruby as your convenient tool. If you
know another options and trade-offs of them, you will implement your
Ruby applications more effectively.

This talk uses Droonga, a distributed full-text search engine, as a
sample application to describe these Ruby usages. Droonga uses these
three Ruby usages.

Published in: Technology
  • Be the first to comment

Three Ruby usages

  1. 1. Three Ruby usages Kouhei Sutou ClearCode Inc. RubyKaigi 2014 2014/09/20 Three Ruby usages Powered by Rabbit 2.1.4
  2. 2. Silver sponsor Three Ruby usages Powered by Rabbit 2.1.4
  3. 3. Goal You know three Ruby usages ✓High-level interface ✓Glue ✓Embed ✓ ✓You can remember them later Three Ruby usages Powered by Rabbit 2.1.4
  4. 4. Targets High-level interface ✓Pure Rubyists ✓ Glue ✓Rubyists who can write C/C++ ✓ Embed ✓Rubyists who also write C/C++ ✓ Three Ruby usages Powered by Rabbit 2.1.4
  5. 5. Case study Implement distributed full-text search engine in Ruby Abbreviation: DFTSE = Distributed Full-Text Search Engine Three Ruby usages Powered by Rabbit 2.1.4
  6. 6. DFTSE? Distrubuted Full- Text Search Engine Full- Text Search Engine 1: Full-text search 2: Distribute sub requests 3: Merge responses Three Ruby usages Powered by Rabbit 2.1.4
  7. 7. Why do we use DFTSE? I'm developing Droonga (A DFTSE implementation in Ruby)  Three Ruby usages Powered by Rabbit 2.1.4
  8. 8. High-level interface Three Ruby usages High-level interface Target: ✓ Pure Rubyists ✓ ✓Glue ✓Embed Three Ruby usages Powered by Rabbit 2.1.4
  9. 9. High-level interface Provides lower layer feature to higher layer ✓ With simpler/✓ convenience API Three Ruby usages Powered by Rabbit 2.1.4
  10. 10. High-level interface Higher layer users High-level interface Feature by Yukihiro Matsumoto Application/Library Three Ruby usages Powered by Rabbit 2.1.4
  11. 11. Example Vagrant Active Record Developers Vagrantfile Build development environment Vagrant b y Y u k i h i r o M a t s u m o t o Developers Object based API b y Y u k i h i r o M a t s u m o t o Access data in RDBMS Active Record Three Ruby usages Powered by Rabbit 2.1.4
  12. 12. Droonga: High-level IF DFTSE components Full-✓ text search engine ✓Messaging system ✓Cluster management ✓Process management Three Ruby usages Powered by Rabbit 2.1.4
  13. 13. Messaging system 1: Full-text search DTFSE FTSE 2: Distribute sub requests 3: Merge responses Worker process Messaging system Three Ruby usages Powered by Rabbit 2.1.4
  14. 14. Messaging system Provides distributed search feature ✓Plan how to search ✓Distribute requests ✓Merge responses ✓ ✓Users don't know details Three Ruby usages Powered by Rabbit 2.1.4
  15. 15. Characteristic Plan how to search May speed up/✓ down over 100 times ✓ Distribute requests ✓Network bound operation ✓ Merge responses ✓CPU and network bound operation ✓ Three Ruby usages Powered by Rabbit 2.1.4
  16. 16. Point Algorithm is important Need to find new/existing better algorithm ✓ "Rapid prototype and measure" feedback loop is helpful ✓ Ruby is ✓ good at rapid dev. ✓ Three Ruby usages Powered by Rabbit 2.1.4
  17. 17. Glue Three Ruby usages ✓High-level interface Glue Target: Rubyists who can write C/C++ ✓ ✓ ✓Embed Three Ruby usages Powered by Rabbit 2.1.4
  18. 18. Glue Export a feature Feature Combine features Glue Ruby Other Language Three Ruby usages Powered by Rabbit 2.1.4
  19. 19. Example mysql2 gem Vagrant Access to MySQL VM Provision libmysqlclient.so (VirtualBox) (Chef) Feature Active Record Glue Three Ruby usages Powered by Rabbit 2.1.4
  20. 20. Why do we glue? Reuse ✓ existing features Three Ruby usages Powered by Rabbit 2.1.4
  21. 21. How to glue Use external library Implement ✓ bindings (mysql2 gem) ✓ Use external command ✓Spawn command (Vagrant) ✓ Use external service ✓Implement client ✓ Three Ruby usages Powered by Rabbit 2.1.4
  22. 22. Glue in Droonga Rroonga: Groonga bindings Groonga: ✓ FTSE C library (and server) ✓ Cool.io: libev bindings libev: Event loop C library (Based on I/O multiplexing and non-blocking I/O) ✓ ✓ ✓Serf: Clustering tool (in Droonga) Three Ruby usages Powered by Rabbit 2.1.4
  23. 23. Rroonga in Droonga 1: Full-text search DTFSE FTSE 2: Distribute sub requests 3: Merge responses Worker process Messaging system Three Ruby usages Powered by Rabbit 2.1.4
  24. 24. FTSE in Droonga ✓Must be fast! ✓CPU bound processing Three Ruby usages Powered by Rabbit 2.1.4
  25. 25. For fast Rroonga Do heavy processing in C Nice to ✓ have Ruby-ish API ✓ Less memory allocation ✓Cache internal buffer ✓ Multiprocessing ✓Groonga supports multiprocessing ✓ Three Ruby usages Powered by Rabbit 2.1.4
  26. 26. Search Groonga::Database.open(ARGV[0]) entries = Groonga["Entries"] entries.select do |record| record.description =~ "Ruby" end Three Ruby usages Powered by Rabbit 2.1.4
  27. 27. Search - Pure Ruby (ref) Groonga::Database.open(ARGV[0]) entries = Groonga["Entries"] entries.find_all do |record| # This block is evaluated for each record /Ruby/ =~ record.description end Three Ruby usages Powered by Rabbit 2.1.4
  28. 28. Search impl. # (2) Evaluate expression in C entries.select do |record| # (1) Build expression in Ruby # This block is evaluated only once record.description =~ "Ruby" end Three Ruby usages Powered by Rabbit 2.1.4
  29. 29. Search impl. - Fig. Ruby by Groonga project C entries.select do |record| record.description =~ "Ruby" end Build Search request Expression Result set Evaluate expression by Groonga project Three Ruby usages Powered by Rabbit 2.1.4
  30. 30. Search - Benchmark ✓Ruby (It's already showed) ✓C Three Ruby usages Powered by Rabbit 2.1.4
  31. 31. Search - C grn_obj *expr; grn_obj *variable; const gchar *filter = "description @ "Ruby""; grn_obj *result; GRN_EXPR_CREATE_FOR_QUERY(&ctx, table, expr, variable); grn_expr_parse(&ctx, expr, filter, strlen(filter), NULL, GRN_OP_MATCH, GRN_OP_AND, GRN_EXPR_SYNTAX_SCRIPT); result = grn_table_select(&ctx, table, expr, NULL, GRN_OP_OR); grn_obj_unlink(&ctx, expr); grn_obj_unlink(&ctx, result); Three Ruby usages Powered by Rabbit 2.1.4
  32. 32. Search - Benchmark Ruby impl. is fast enough  Impl. Elapsed time C 0.6ms Ruby 0.8ms (Full-text search with "Ruby" against 72632 records) Three Ruby usages Powered by Rabbit 2.1.4
  33. 33. Embed Three Ruby usages ✓High-level interface ✓Glue Embed Target: Rubyists who also write C/C++ ✓ ✓ Three Ruby usages Powered by Rabbit 2.1.4
  34. 34. Embed Internal engine Interface C/C++ application C/C++ library Implement some features in Ruby Plugin API Conifugration by Yukihiro Matsumoto by Yukihiro Matsumoto C/C++ application C/C++ library Three Ruby usages Powered by Rabbit 2.1.4
  35. 35. Examples Internal engine Interface Implement query optimizer in Ruby vim-ruby by Yukihiro Matsumoto by Yukihiro Matsumoto VIM Three Ruby usages Powered by Rabbit 2.1.4
  36. 36. Embed in Droonga by Yukihiro Matsumoto by Yukihiro Matsumoto ... by The Groonga Project by The Groonga Project by The Groonga Project mruby by Yukihiro Matsumoto Three Ruby usages Powered by Rabbit 2.1.4
  37. 37. CRuby vs. mruby CRuby ✓Full featured! ✓Signal handler isn't needed  ✓ mruby ✓Multi-interpreters in a process! ✓You may miss some features  ✓ Three Ruby usages Powered by Rabbit 2.1.4
  38. 38. mruby in Groonga ✓Query optimizer Command interface (plan) ✓Interface and also high-level interface! ✓ Plugin API (plan) ✓Interface! ✓ Three Ruby usages Powered by Rabbit 2.1.4
  39. 39. Query optimizer Query Optimize Optimized Query Optimizer query Full-text search Evaluator by Yukihiro Matsumoto Result set Three Ruby usages Powered by Rabbit 2.1.4
  40. 40. Query optimizer Plan how to search ✓It's a bother  ✓ ✓Light operation than FTS Depends on data (Choose effective index, use table scan and so on) ✓ Three Ruby usages Powered by Rabbit 2.1.4
  41. 41. Example rank < 200 && rank > 100 Three Ruby usages Powered by Rabbit 2.1.4
  42. 42. Simple impl. rank 1 2 ... 100 ... 200 ... ... 10000 rank > 100 rank < 200 101 199 && 101 ... 199 rank < 200 && rank > 100 Three Ruby usages Powered by Rabbit 2.1.4
  43. 43. Simple impl. Slow against many out of range data ✓ Three Ruby usages Powered by Rabbit 2.1.4
  44. 44. Optimized impl. rank 1 2 ... 100 101 ... 199 200 ... ... 10000 100 < rank < 200 101 ... 199 rank < 200 && rank > 100 Three Ruby usages Powered by Rabbit 2.1.4
  45. 45. Is embedding reasonable? Measure Three Ruby usages Powered by Rabbit 2.1.4
  46. 46. Measure ✓mruby overhead ✓Speed-up by optimization Three Ruby usages Powered by Rabbit 2.1.4
  47. 47. Overhead Small overhead: Reasonable # conds mruby Elapsed 1 ○ 0.24ms 1 × 0.16ms 4 ○ 0.45ms 4 × 0.19ms Three Ruby usages Powered by Rabbit 2.1.4
  48. 48. Speed-up Fast for many data:Reasonable # records mruby no mruby 1000 0.29ms 0.31ms 10000 0.31ms 2.3ms 100000 0.26ms 21.1ms 1000000 0.26ms 210.2ms Three Ruby usages Powered by Rabbit 2.1.4
  49. 49. Note Embedding needs many works Write bindings, import mruby your build system and ... ✓ ✓ How to test your mruby part? ✓And how to debug? ✓ Three Ruby usages Powered by Rabbit 2.1.4
  50. 50. Conclusion
  51. 51. Conclusion 1 Describe three Ruby usages ✓High-level interface ✓Glue ✓Embed ✓ Three Ruby usages Powered by Rabbit 2.1.4
  52. 52. Conclusion 2 High-level interface ✓ Target: Pure Rubyists ✓ Provides lower layer feature to higher layer w/ usable interface ✓Ruby's flexibility is useful ✓ Three Ruby usages Powered by Rabbit 2.1.4
  53. 53. Conclusion 3 Glue Target: Rubyists who can write C/C++ ✓ Why: Reuse ✓ existing feature ✓To be fast, do the process in C ✓ Three Ruby usages Powered by Rabbit 2.1.4
  54. 54. Conclusion 4 Embed Target: Rubyists who also write C/C++ ✓ Why: Avoid bother programming by Ruby ✓ ✓ Three Ruby usages Powered by Rabbit 2.1.4
  55. 55. Conclusion 5 Embed Is it reasonable ✓ for your case? ✓You need many works ✓ Very powerful if your case is reasonable ✓ Three Ruby usages Powered by Rabbit 2.1.4
  56. 56. Announcement ClearCode Inc. ✓A silver sponsor ✓Is recruiting ✓Will do readable code workshop ✓ The next Groonga conference ✓It's held at 11/29 ✓ Three Ruby usages Powered by Rabbit 2.1.4

×