Solving the Riddle of Search: Using Sphinx with Rails

Solving the
Riddle of Search
Using Sphinx with Rails

http://freelancing-gods.com

http://twitter.com/pat

Similar to Ferret,
Solr, Xapian, etc

... and Microsoft
SQL Server and
Oracle in 0.9.9

No prizes for
guessing this
either.

Does not talk to
your database.

No, not Microsoft
Word docs

A document is a
single database
record

Documents have
fields and
attributes

Attributes, on
the other hand...

Integer values
indicating
alphabetical
order of strings.

Search queries
don’t look at
attribute values.

But what’s
Thinking
Sphinx?

Configure
Sphinx with
Ruby and YAML

What’s in it for
me though?

• Find all people
• Where any word matches
• Part of any of four fields

# given a query of ‘Pat Allan’
# assuming MySQL

SELECT * FROM people
WHERE first_name LIKE '%Pat%'
OR first_name LIKE '%Allan%'
OR last_name LIKE '%Pat%'
OR last_name LIKE '%Allan%'
OR location LIKE '%Pat%'
OR location LIKE '%Allan%'
OR profile LIKE '%Pat%'
OR profile LIKE '%Allan%';

# given a query of ‘Pat Allan’

Person.search quot;Pat Allanquot;,
:match_mode => :any

What textual
data do we want
indexed?

class Person < ActiveRecord::Base
# ...

define_index do
indexes first_name, last_name,
location, profile
end

# ...
end

# Combine columns into one field
indexes [first_name, last_name],
:as => :name

# Avoid core Ruby class methods
indexes :name

# Use association columns
indexes photos.captions,
:as => :photos

What do we want
to sort and filter
by?

Our friend the
has
method

class Person < ActiveRecord::Base
# ...

define_index do
# ...
has created_at, admin, email, :id
end

# ...
end

# Multi-Value Attributes (MVAs)
has tags.id, :as => :tag_ids

What if I want to
sort by a field?

indexes first_name, last_name,
:sortable => true

Defines a SQL
query for Sphinx

Complexity
leads to slower
indexing

So don’t forget
to add database
indexes as well!

Warning for fans
of fixtures...

# In your define_index block:
set_property(
:sql_range_step => 10_000_000
)

Can only use
model columns,
not model methods

Maybe that’s not
good enough?

has quot;YEAR(created_at)quot;,
:as => :year, :type => :integer

So, we’ve set up
our index.
Now what?

(in /Users/pat/Code/ruby/ts_test_2.2.0)
Generating Configuration to /Users/pat/Code/ruby/ts_test_2.2.0/
config/development.sphinx.conf
indexer --config /Users/pat/Code/ruby/ts_test_2.2.0/config/
development.sphinx.conf --all
Sphinx 0.9.8-release (r1371)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file '/Users/pat/Code/ruby/ts_test_2.2.0/config/
development.sphinx.conf'...
indexing index 'person_core'...
collected 1000 docs, 0.0 MB
collected 0 attr values
sorted 0.0 Mvalues, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 1000 docs, 14436 bytes
total 0.191 sec, 75523.42 bytes/sec, 5231.60 docs/sec
indexing index 'person_delta'...
collected 0 docs, 0.0 MB
collected 0 attr values
sorted 0.0 Mvalues, nan% done
total 0 docs, 0 bytes
total 0.010 sec, 0.00 bytes/sec, 0.00 docs/sec
distributed index 'person' can not be directly indexed; skipping.

searchd --pidfile --config /Users/pat/Code/ruby/ts_test_2.2.0/
config/development.sphinx.conf

Started successfully (pid 6220).


stop: succesfully sent SIGTERM to pid 6220
Stopped search daemon (pid 6220).

rake ts:in
rake ts:start
rake ts:stop

Person.search quot;Pat Allanquot;

Focus that
search on a
specific field

Person.search(quot;Rubyquot;,
:conditions => {:name => quot;Pat Allanquot;}
)

Conditions are
not exactly like
ActiveRecord

And conditions
should always
be a Hash.

:with => {:admin => true}
)

:with => {
:created_at => 1.week.ago..Time.now
}
)

:with => {:tag_ids => [1, 2, 3]}
)

:with_all => {:tag_ids => [1, 2, 3]}
)

:without => {:admin => true}
)

All words must
exist in a
document

Person.search(quot;Pat Allanquot;,
:match_mode => :any
)

One of the words
must exist in a
document

The exact query
must exist in the
document

Use boolean
logic in search
queries

Can match on
specific fields,
and apply
boolean logic

Person.search(quot;Pat Allanquot;,
:field_weights => {
quot;namequot; => 100,
quot;profilequot; => 50,
quot;locationquot; => 30
}
)

define_index do
# ...

set_property :field_weights => {
quot;namequot; => 100,
quot;profilequot; => 50,
quot;locationquot; => 30
}
end

Or Fields flagged
as :sortable

:order => :created_at

:order => :created_at,
:sort_mode => :desc

:order => quot;last_name ASC,
first_name ASCquot;

:sort_mode => :expr,
:sort_by => quot;@weight * rankingquot;

:sort_mode => :time_segments,
:sort_by => quot;created_atquot;

• Last Hour
• Last Day
• Last Week
• Last Month
• Last 3 Months
• Everything Else

Sorted by
Segment, then
Relevance

ThinkingSphinx::Search.search(
quot;Pat Allanquot;
)

:page => params[:page]

You can request
really big pages
though.

:per_page => 1000,
:page => params[:page]

:page => params[:page],
:per_page => 1_000_000,
:max_matches => 1_000_000

# config/sphinx.yml
development:
max_matches: 1000000
test:
production:

Don’t forget to
restart Sphinx

script/plugin install
git://github.com/freelancing-god/
thinking-sphinx.git

git clone git://github.com/
freelancing-god/thinking-sphinx.git

sudo gem install
freelancing-god-thinking-sphinx
--source http://gems.github.com

# inside config/environment.rb
config.gem(
quot;freelancing-god-thinking-sphinxquot;,
:version => quot;1.1.6quot;,
:lib => quot;thinking_sphinxquot;
)

# at the end of your Rakefile:
require 'thinking_sphinx/tasks'

apt, yum, etc.
Should be
painless.

Well... it can
be a little
complicated

./configure
make
sudo make install

./configure --with-pgsql=/usr/local/
include/postgresql
make
sudo make install

... if you’re using
it for MySQL or
PostgreSQL
as well

git://github.com/freelancing-
god/sphinx-tute.git

Can only process
a full index.

Delta Index
holds the
changes

We can track
changes as
they happen

... at the cost of a
little extra
overhead.

1. Add a boolean
column ‘delta’ to
your model.

2. Tell your index
to use a delta.

define_index do
# ...

set_property :delta => true
end

rake thinking_sphinx:stop
rake thinking_sphinx:index
rake thinking_sphinx:start

... that’s the
default
approach

define_index do
# ...

set_property :delta => :datetime,
:threshold => 1.day
end

Use existing
updated_at
column

rake thinking_sphinx:index:delta

Please Note: Not
entirely reliable

create_table :delayed_jobs do |t|
# ...
end

define_index do
# ...

set_property :delta => :delayed
end

rake thinking_sphinx:delayed_delta

Removes
overheard from
your web stack

define_index do
# ...
indexes location, country,
:facet => true

has admin, :facet => true
end

{
:country => {
quot;Australiaquot; => 15,
quot;USAquot; => 11,
quot;Germanyquot; => 8,
quot;Canadaquot; => 12
},
:location => {
quot;Melbournequot; => 3,
# ...
}

@facets.for(:country => quot;Australiaquot;)

Common
Example:
Partial Matching

development:
enable_star: 1
min_infix_len: 1
test:
enable_star: 1
min_infix_len: 1
production:
enable_star: 1
min_infix_len: 1

# config/deploy.rb
load 'vendor/plugins/thinking-sphinx/lib/
thinking_sphinx/deploy/capistrano'

cap thinking_sphinx:configure
cap thinking_sphinx:index
cap thinking_sphinx:install:sphinx
cap thinking_sphinx:install:ts
cap thinking_sphinx:rebuild
cap thinking_sphinx:restart
cap thinking_sphinx:shared_sphinx_folder
cap thinking_sphinx:start
cap thinking_sphinx:stop

Set Sphinx index
location

Sphinx does not
like symlinks

# config/sphinx.yml
production:
searchd_file_path: quot;/var/www/
sphinx-tute/shared/db/sphinx/
production/quot;

# config/deploy.rb
after quot;deploy:setupquot;,
quot;thinking_sphinx:shared_sphinx_folderquot;

Need latitude
and longitude
attributes

define_index do
# ...

has latitude, longitude
end

Bar.search(
quot;Las Vegasquot;,
:geo => [@lat, @lng],
:sort => quot;@geodist ASCquot;
)

@bars.each_with_geodist do |bar,
distance|
# distance is in metres
end

define_index do
# ...

set_property(
:latitude_attr => :updown,
:longitude_attr => :leftright
)
end

Geo-searching
across multiple
models?

ThinkingSphinx::Search.search(
quot;pancakesquot;,
:geo => [@lat, @lng]
:latitude_attr => :lat,
:longitude_attr => :lng
)

http://ts.freelancing-gods.com

http://groups.google.com/
group/thinking-sphinx

http://peepcode.com/products/
thinking-sphinx-pdf

Solving the Riddle of Search: Using Sphinx with Rails

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Solving the Riddle of Search: Using Sphinx with Rails

Similar to Solving the Riddle of Search: Using Sphinx with Rails (20)

More from freelancing_god

More from freelancing_god (6)

Recently uploaded

Recently uploaded (20)

Solving the Riddle of Search: Using Sphinx with Rails