0
ElasticsearchAnd RubyKarel Minařík
http://karmi.cz                  Elasticsearch and Ruby
{elasticsearch in a nutshell}Built on top of Apache LuceneSearching and analyzing big dataScalabilityREST API, JSON DSLGre...
{ }      Elasticsearch and Ruby
{ }  It all started in this gist… (< 200 LOC)                  Elasticsearch and Ruby
{ }      Elasticsearch and Ruby
Example class Results   include Enumerable   attr_reader :query, :curl, :time, :total, :results, :facets  def initialize(s...
elasticsearch’s Query DSLcurl  -­‐X  POST  "http://localhost:9200/articles/_search?pretty=true"  -­‐d  {    "query"  :  { ...
Example Tire.search(articles) do   query do     boolean do       must { terms :tags, [ruby, python] }       must { string ...
Example tags_query = lambda do |boolean|   boolean.must { terms :tags, [ruby, python] } end published_on_query = lambda do...
Example search = Tire.search articles do   query do     string title:T*   end   filter :terms, tags: [ruby]   facet tags, ...
TEH PROBLEM     Designing the Tire library as domain-specific language, from the higher level, and     consequently doing ...
”Blocks with arguments”      (alternative DSL syntax)     Tire.search do       query do         text :name, params[:q]    ...
The Git(Hub) (r)evolution‣ Lots of contributions... but less feedback‣ Many contributions focus on specific use case‣ Many...
Tire’s Ruby on Rails integration$  rails  new  myapp        -­‐m  "https://raw.github.com/karmi/tire/master/examples/rails...
Rails integration baked in‣ No proper separation of concerns / layers‣ People expect everything to be as easy as that‣ Tir...
…Persistence extensionRails extensionsActiveRecord extensionsActiveModel integrationThe Ruby DSLBase library (HTTP, JSON, ...
https://rubygems.orghttps://github.com/rubygems/rubygems.org/pull/455
“Search”class Rubygem < ActiveRecord::Base  # ...  def self.search(query)    conditions = <<-SQL      versions.indexed and...
123456    Adding search to an existing application
https://github.com/karmi/rubygems.org/compare/search-steps
“Hello Cloud” with Chef Serverhttp://git.io/chef-hello-cloud‣   Deploy Rubygems.org on EC2 (or locally with Vagrant) from ...
Thanks!  d
Upcoming SlideShare
Loading in...5
×

Elasticsearch And Ruby [RuPy2012]

3,043

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,043
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
39
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Transcript of "Elasticsearch And Ruby [RuPy2012]"

  1. 1. ElasticsearchAnd RubyKarel Minařík
  2. 2. http://karmi.cz Elasticsearch and Ruby
  3. 3. {elasticsearch in a nutshell}Built on top of Apache LuceneSearching and analyzing big dataScalabilityREST API, JSON DSLGreat fit for dynamic languages and web-oriented workflows / architectureshttp://www.elasticsearch.org Elasticsearch and Ruby
  4. 4. { } Elasticsearch and Ruby
  5. 5. { } It all started in this gist… (< 200 LOC) Elasticsearch and Ruby
  6. 6. { } Elasticsearch and Ruby
  7. 7. Example class Results include Enumerable attr_reader :query, :curl, :time, :total, :results, :facets def initialize(search) response = JSON.parse( Slingshot.http.post("http://localhost:9200/#{search.indices}/_search", search.to_json) ) @query = search.to_json @curl = %Q|curl -X POST "http://localhost:9200/#{search.indices}/_search?pretty" -d #{@query}| @time = response[took] @total = response[hits][total] @results = response[hits][hits] @facets = response[facets] end def each(&block) @results.each(&block) end end Elasticsearch plays nicely with Ruby… Elasticsearch and Ruby
  8. 8. elasticsearch’s Query DSLcurl  -­‐X  POST  "http://localhost:9200/articles/_search?pretty=true"  -­‐d  {    "query"  :  {        "filtered"  :  {            "filter"  :  {                "range"  :  {                    "date"  :  {                        "from"  :  "2012-­‐01-­‐01",                        "to"      :  "2012-­‐12-­‐31"                    }                }            },            "query"  :  {                "bool"  :  {                    "must"  :  {                        "terms"  :  {                            "tags"  :  [  "ruby",  "python"  ]                        }                    },                    "must"  :  {                        "match"  :  {                            "title"  :  {                                "query"  :  "conference",                                "boost"  :  10.0                            }                        }                    }                }            }        }    }}
  9. 9. Example Tire.search(articles) do query do boolean do must { terms :tags, [ruby, python] } must { string published_on:[2011-01-01 TO 2011-01-02] } end end end Elasticsearch and Ruby
  10. 10. Example tags_query = lambda do |boolean| boolean.must { terms :tags, [ruby, python] } end published_on_query = lambda do |boolean| boolean.must { string published_on:[2011-01-01 TO 2011-01-02] } end Tire.search(articles) do query { boolean &tags_query } end Tire.search(articles) do query do boolean &tags_query boolean &published_on_query end end Elasticsearch and Ruby
  11. 11. Example search = Tire.search articles do query do string title:T* end filter :terms, tags: [ruby] facet tags, terms: tags sort { by :title, desc } end search = Tire::Search::Search.new(articles) search.query { string(title:T*) } search.filter :terms, :tags => [ruby] search.facet(tags) { terms :tags } search.sort { by :title, desc } Elasticsearch and Ruby
  12. 12. TEH PROBLEM Designing the Tire library as domain-specific language, from the higher level, and consequently doing a lot of mistakes in the lower levels. ‣ Class level settings (Tire.configure); cannot connect to two elasticsearch clusters in one codebase ‣ Inconsistent access (methods vs Hashes) ‣ Not enough abstraction and separation of concerns Elasticsearch and Ruby
  13. 13. ”Blocks with arguments” (alternative DSL syntax) Tire.search do query do text :name, params[:q] end endTire.search do |search| search.query do |query| query.text :name, params[:q] endend Elasticsearch and Ruby
  14. 14. The Git(Hub) (r)evolution‣ Lots of contributions... but less feedback‣ Many contributions focus on specific use case‣ Many contributions don’t take the bigger picture and codebase conventions into account‣ Almost every patch needs to be processed, polished, amended‣ Maintainer: lots of curation, less development — even on this small scale (2K LOC, 7K LOT)‣ Contributors very eager to code, but a bit afraid to talk
  15. 15. Tire’s Ruby on Rails integration$  rails  new  myapp        -­‐m  "https://raw.github.com/karmi/tire/master/examples/rails-­‐application-­‐template.rb"‣ Generate a fully working Rails application with a single command‣ Downloads elasticsearch if not running, creates the application, commits every step, seeds the example data, launches the application on a free port, …‣ Tire::Results::Item fully compatible with Rails view / URL helpers‣ Any ActiveModel compatible OxM supported‣ Rake task for importing data (using pagination libraries) Elasticsearch and Ruby
  16. 16. Rails integration baked in‣ No proper separation of concerns / layers‣ People expect everything to be as easy as that‣ Tire::Results::Item baked in, not opt-in, masquerades as models‣ People consider ActiveRecord the only OxM in the world Elasticsearch and Ruby
  17. 17. …Persistence extensionRails extensionsActiveRecord extensionsActiveModel integrationThe Ruby DSLBase library (HTTP, JSON, API)
  18. 18. https://rubygems.orghttps://github.com/rubygems/rubygems.org/pull/455
  19. 19. “Search”class Rubygem < ActiveRecord::Base # ... def self.search(query) conditions = <<-SQL versions.indexed and (upper(name) like upper(:query) or upper(translate(name, #{SPECIAL_CHARACTERS}, #{ *SPECIAL_CHARACTERS.length})) like upper(:query)) SQL where(conditions, {:query => "%#{query.strip}%"}). includes(:versions). by_downloads endendhttps://github.com/rubygems/rubygems.org/blob/master/app/models/rubygem.rb Elasticsearch and Ruby
  20. 20. 123456 Adding search to an existing application
  21. 21. https://github.com/karmi/rubygems.org/compare/search-steps
  22. 22. “Hello Cloud” with Chef Serverhttp://git.io/chef-hello-cloud‣ Deploy Rubygems.org on EC2 (or locally with Vagrant) from a “zero state”‣ 1 load balancer (HAproxy), 3 application servers (Thin+Nginx)‣ 1 database node (PostgreSQL, Redis)‣ 2 elasticsearch nodes‣ Install Ruby 1.9.3 via RVM‣ Clone the application from GitHub repository‣ init.d scripts and full configuration for every component‣ Restore data from backup (database dump) and import into search index‣ Monitor every part of the stack Elasticsearch and Ruby
  23. 23. Thanks! d
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×