Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

GoFFIng around with Ruby #RubyConfPH

516 views

Published on

This talk is about how to use Foreign Function Interface (FFI) with Ruby and Go programs.

Published in: Software
  • Be the first to comment

GoFFIng around with Ruby #RubyConfPH

  1. 1. GoFFIng around with Ruby @gautamrege
  2. 2. @joshsoftware
  3. 3. FFI
  4. 4. Foreign Function Interface *In stdlib since Ruby 1.9.3
  5. 5. Why do we need FFI? • Resque / Sidekiq / delayed_job • 3rd party API requests • Sending newsletters • Heavy computation • Fast searches, aggregation.
  6. 6. Why Go? • Power of multi-core. • Goroutines to the rescue • Language is restricted focussed. • Programming Ethics • Compiled If it compiles, it works
  7. 7. What is FFI anyway? • Language interoperability! • Calling convention. • libffi.so • ruby-ffi • Fiddle (in Stdlib since 1.9.3)
  8. 8. FFI or Good ol’ C-ext? • FFI Easier to write and maintain. • Gems are now easier to install - don’t need build dependencies. • Portable code • No more “Building native extensions…” while installing gems.
  9. 9. Simple Ruby Code def race(legs, benchmark) tids = [] legs.times do |i| tids << Thread.new(i) do |leg| array = (1..benchmark).map { |i| i } p "Done leg #{leg} benchmark #{benchmark}" end end tids.each { |t| t.join } end race(4, 50_000_000)
  10. 10. Benchmark - Ruby $ time ruby run1.rb Done leg 1 benchmark 50000000 Done leg 0 benchmark 50000000 Done leg 3 benchmark 50000000 Done leg 2 benchmark 50000000 real 0m18.242s user 0m16.626s sys 0m1.292s
  11. 11. Simple Go Code package main import "fmt" func race(leg int, benchmark int) { arr := []int{} for i := 0; i < benchmark; i++ { arr = append(arr, i) } fmt.Println("Done leg:", leg) } func main() { for i := 0; i < 4; i++ { race(i, 50000000) } }
  12. 12. Simple Go Code package main import "fmt" func race(leg int, benchmark int) { arr := []int{} for i := 0; i < benchmark; i++ { arr = append(arr, i) } fmt.Println("Done leg:", leg) } func main() { for i := 0; i < 4; i++ { race(i, 50000000) } }
  13. 13. Simple Go Code package main import "fmt" func race(leg int, benchmark int) { arr := []int{} for i := 0; i < benchmark; i++ { arr = append(arr, i) } fmt.Println("Done leg:", leg) } func main() { for i := 0; i < 4; i++ { race(i, 50000000) } }
  14. 14. Simple Go Code package main import "fmt" func race(leg int, benchmark int) { arr := []int{} for i := 0; i < benchmark; i++ { arr = append(arr, i) } fmt.Println("Done leg:", leg) } func main() { for i := 0; i < 4; i++ { race(i, 50000000) } }
  15. 15. Benchmark - Go $ time go run run1.go Done leg: 0 Done leg: 1 Done leg: 2 Done leg: 3 real 0m2.774s user 0m2.564s sys 0m0.929s
  16. 16. Calling Go from Ruby • Build a C library using Go code. • Build a FFI extension in Ruby • Call it !
  17. 17. c-shared Go library package main import ( "C" "fmt" ) func race(leg int, benchmark int) { arr := []int{} for i := 0; i < benchmark; i++ { arr = append(arr, i) } fmt.Println("Done leg:", leg) } //export churn func churn(leg C.int, benchmark C.int) { for i := 0; i < int(leg); i++ { race(i, int(benchmark)) } } func main() { }
  18. 18. c-shared Go library package main import ( "C" "fmt" ) func race(leg int, benchmark int) { arr := []int{} for i := 0; i < benchmark; i++ { arr = append(arr, i) } fmt.Println("Done leg:", leg) } //export churn func churn(leg C.int, benchmark C.int) { for i := 0; i < int(leg); i++ { race(i, int(benchmark)) } } func main() { }
  19. 19. c-shared Go library package main import ( "C" "fmt" ) func race(leg int, benchmark int) { arr := []int{} for i := 0; i < benchmark; i++ { arr = append(arr, i) } fmt.Println("Done leg:", leg) } //export churn func churn(leg C.int, benchmark C.int) { for i := 0; i < int(leg); i++ { race(i, int(benchmark)) } } func main() { } $ go build -o librun.so -buildmode=c-shared run2.go
  20. 20. Ruby FFI extension require 'fiddle' module Fun def self.race(leg, benchmark) librun = Fiddle.dlopen(‘./librun.so') race = Fiddle::Function.new( librun['churn'], [Fiddle::TYPE_INT, Fiddle::TYPE_INT], Fiddle::TYPE_VOID ) race.call(leg, benchmark) end end
  21. 21. Call it! require './fun' Fun.race(4, 50_000_000) $ time ruby run2.rb Done leg: 0 Benchmark 50000000 Done leg: 1 Benchmark 50000000 Done leg: 2 Benchmark 50000000 Done leg: 3 Benchmark 50000000 real 0m2.776s user 0m2.719s sys 0m0.969s
  22. 22. Practical Example
  23. 23. Practical Example Make Model Variant
  24. 24. Practical Example Make Model Variant
  25. 25. Practical Example
  26. 26. Practical Example Insurance Vendor
  27. 27. Practical Example Insurance Vendor 68 columns
  28. 28. Doing the Math! 18 STATES 428 CARS 68 VENDORS = 523,872 cells
  29. 29. If that was not enough … For each of the 523,872 cells Calculate Depreciation Index in ElasticSearch
  30. 30. for cmake, v := range segments { for model, j := range v { for submodel, k := range j { for insurance, segment := range k { wg.Add(1) go func(cmake string, model string, submodel string, insurance string, segment string) { defer wg.Done() // data := es_data{...} // prepare_es_data(...) es_add_depreciation(&data) }(cmake, model, submodel, insurance, segment) } } } } Practical Example About 350,000 go-routines
  31. 31. for cmake, v := range segments { for model, j := range v { for submodel, k := range j { for insurance, segment := range k { wg.Add(1) go func(cmake string, model string, submodel string, insurance string, segment string) { defer wg.Done() // data := es_data{...} // prepare_es_data(...) es_add_depreciation(&data) }(cmake, model, submodel, insurance, segment) } } } } Practical Example About 350,000 go-routines
  32. 32. Performance Benefits
  33. 33. Performance Benefits • Sidekiq: 2 GB memory & ~18 minutes
  34. 34. Performance Benefits • Sidekiq: 2 GB memory & ~18 minutes • Go FFI: 135 MB memory & 1 minute 13 seconds.
  35. 35. Performance Benefits • Sidekiq: 2 GB memory & ~18 minutes • Go FFI: 135 MB memory & 1 minute 13 seconds. ElasticSearch indexing took 48 seconds
  36. 36. Complexities of FFI • Memory Management (alloc / free) • Pointer management • Fiddle::Pointer • Fiddle::Closure Don’t use them unless you HAVE to!
  37. 37. Resources • Fiddle http://ruby-doc.org/stdlib-2.0.0/libdoc/fiddle/rdoc/ Fiddle.html • FFI Core Concepts https://github.com/ffi/ffi/wiki/Core- Concepts • Github: ffi/ffi • Effective Go ( https://golang.org/doc/effective_go.html ) • A Tour of Go ( https://tour.golang.org/welcome/1 )
  38. 38. @joshsoftware @gautamrege http://www.codecuriosity.org

×