Elasticsearch and Ruby document summarized in 3 sentences: Elasticsearch is an open source search and analytics engine built on Apache Lucene that provides scalable searching and analyzing of big data. It is a great fit for dynamic languages like Ruby and web-oriented workflows due to its REST API and JSON DSL. The document provides examples of using the Ruby library Tire to interface with Elasticsearch to perform searches and facets from Ruby applications and Rails frameworks.

1. Elasticsearch
And Ruby
Karel Minařík

2. http://karmi.cz
Elasticsearch and Ruby

3. {elasticsearch in a nutshell}
Built on top of Apache Lucene
Searching and analyzing big data
Scalability
REST API, JSON DSL
Great fit for dynamic languages and web-oriented workflows / architectures
http://www.elasticsearch.org Elasticsearch and Ruby

4. { }
Elasticsearch and Ruby

5. { }
It all started in this gist… (< 200 LOC)
Elasticsearch and Ruby

6. { }
Elasticsearch and Ruby

7. Example
class Results
include Enumerable
attr_reader :query, :curl, :time, :total, :results, :facets
def initialize(search)
response = JSON.parse( Slingshot.http.post("http://localhost:9200/#{search.indices}/_search", search.to_json) )
@query = search.to_json
@curl = %Q|curl -X POST "http://localhost:9200/#{search.indices}/_search?pretty" -d '#{@query}'|
@time = response['took']
@total = response['hits']['total']
@results = response['hits']['hits']
@facets = response['facets']
end
def each(&block)
@results.each(&block)
end
end
Elasticsearch plays nicely with Ruby…
Elasticsearch and Ruby

8. elasticsearch’s Query DSL
curl -­‐X POST "http://localhost:9200/articles/_search?pretty=true" -­‐d '
{
"query" : {
"filtered" : {
"filter" : {
"range" : {
"date" : {
"from" : "2012-­‐01-­‐01",
"to" : "2012-­‐12-­‐31"
}
}
},
"query" : {
"bool" : {
"must" : {
"terms" : {
"tags" : [ "ruby", "python" ]
}
},
"must" : {
"match" : {
"title" : {
"query" : "conference",
"boost" : 10.0
}
}
}
}
}
}
}
}

9. Example
Tire.search('articles') do
query do
boolean do
must { terms :tags, ['ruby', 'python'] }
must { string 'published_on:[2011-01-01 TO 2011-01-02]' }
end
end
end
Elasticsearch and Ruby

10. Example
tags_query = lambda do |boolean|
boolean.must { terms :tags, ['ruby', 'python'] }
end
published_on_query = lambda do |boolean|
boolean.must { string 'published_on:[2011-01-01 TO 2011-01-02]' }
end
Tire.search('articles') do
query { boolean &tags_query }
end
Tire.search('articles') do
query do
boolean &tags_query
boolean &published_on_query
end
end
Elasticsearch and Ruby

11. Example
search = Tire.search 'articles' do
query do
string 'title:T*'
end
filter :terms, tags: ['ruby']
facet 'tags', terms: tags
sort { by :title, 'desc' }
end
search = Tire::Search::Search.new('articles')
search.query { string('title:T*') }
search.filter :terms, :tags => ['ruby']
search.facet('tags') { terms :tags }
search.sort { by :title, 'desc' }
Elasticsearch and Ruby

12. TEH PROBLEM
Designing the Tire library as domain-specific language, from the higher level, and
consequently doing a lot of mistakes in the lower levels.
‣ Class level settings (Tire.configure); cannot connect to two elasticsearch clusters in one codebase
‣ Inconsistent access (methods vs Hashes)
‣ Not enough abstraction and separation of concerns
Elasticsearch and Ruby

13. ”Blocks with arguments”
(alternative DSL syntax)
Tire.search do
query do
text :name, params[:q]
end
end
Tire.search do |search|
search.query do |query|
query.text :name, params[:q]
end
end Elasticsearch and Ruby

14. The Git(Hub) (r)evolution
‣ Lots of contributions... but less feedback
‣ Many contributions focus on specific use case
‣ Many contributions don’t take the bigger picture and codebase conventions into account
‣ Almost every patch needs to be processed, polished, amended
‣ Maintainer: lots of curation, less development — even on this small scale (2K LOC, 7K LOT)
‣ Contributors very eager to code, but a bit afraid to talk

15. Tire’s Ruby on Rails integration
$ rails new myapp
-­‐m "https://raw.github.com/karmi/tire/master/examples/rails-­‐application-­‐template.rb"
‣ Generate a fully working Rails application with a single command
‣ Downloads elasticsearch if not running, creates the application, commits
every step, seeds the example data, launches the application on a free port, …
‣ Tire::Results::Item fully compatible with Rails view / URL helpers
‣ Any ActiveModel compatible OxM supported
‣ Rake task for importing data (using pagination libraries)
Elasticsearch and Ruby

16. Rails integration baked in
‣ No proper separation of concerns / layers
‣ People expect everything to be as easy as that
‣ Tire::Results::Item baked in, not opt-in, masquerades as models
‣ People consider ActiveRecord the only OxM in the world
Elasticsearch and Ruby

17. …
Persistence extension
Rails extensions
ActiveRecord extensions
ActiveModel integration
The Ruby DSL
Base library (HTTP, JSON, API)

18. https://rubygems.org
https://github.com/rubygems/rubygems.org/pull/455

19. “Search”
class Rubygem < ActiveRecord::Base
# ...
def self.search(query)
conditions = <<-SQL
versions.indexed and
(upper(name) like upper(:query) or
upper(translate(name, '#{SPECIAL_CHARACTERS}', '#{' ' *
SPECIAL_CHARACTERS.length}')) like upper(:query))
SQL
where(conditions, {:query => "%#{query.strip}%"}).
includes(:versions).
by_downloads
end
end
https://github.com/rubygems/rubygems.org/blob/master/app/models/rubygem.rb Elasticsearch and Ruby

20. 1
2
3
4
5
6
Adding search to an existing application

21. https://github.com/karmi/rubygems.org/compare/search-steps

22. “Hello Cloud” with Chef Server
http://git.io/chef-hello-cloud
‣ Deploy Rubygems.org on EC2 (or locally with Vagrant) from a “zero state”
‣ 1 load balancer (HAproxy), 3 application servers (Thin+Nginx)
‣ 1 database node (PostgreSQL, Redis)
‣ 2 elasticsearch nodes
‣ Install Ruby 1.9.3 via RVM
‣ Clone the application from GitHub repository
‣ init.d scripts and full configuration for every component
‣ Restore data from backup (database dump) and import into search index
‣ Monitor every part of the stack
Elasticsearch and Ruby

23. Thanks!
d