This document provides an overview of full text search capabilities including: - Tokenization, stemming, stop words, and word boundaries to preprocess text for searching - Language translation examples showing Japanese translations of an English query - Accuracy considerations for language translation in search - Performance factors like hosted databases, search indexing, and Rails integration - Code examples configuring ActiveRecord models to work with a full text search engine

1. Adventures
in
Full Text Search
Sarah Allen @ultrasaurus

2. class Article < ActiveRecord::Base
acts_as_solr
end

3. 3

8. Tokyo Dystopia

9. Language
Relevance
Accuracy
Speed

10. Text as Language

11. stemming
synonyms
stop words
word boundaries

12. SELECT text FROM phrases WHERE text like '%run%';
Can you run this to the post office for me?
I'm going for a run, want to come along?
Cross country running
I'm too drunk to drive.
I am running out of battery power.
Work is not like wolf - it won't run away.

13. SELECT text FROM phrases WHERE
vectors @@ 'run'::tsquery;
Can you run this to the post office for me?
Sorry I am running really late.
I'm going for a run, want to come along?
Cross country running
I am running out of battery power.
Work is not like wolf - it won't run away.

15. Tokenization and Stemming
Google App Engine /JRuby / Lucene
http://full-text-search.appspot.com
http://
github.com/
ultrasaurus/
full-text-search-appengine

16. hAp://full‐text‐search.appspot.com/
16

17. hAp://full‐text‐search.appspot.com/
17

18. hAp://full‐text‐search.appspot.com/
18

19. hAp://localhost:8080/_ah/admin/datastore?kind=Notes
19

20. ./script/generate scaffold note
content:string index:List -f --skip-migration
./script/generate dd_model note content:string index:List -f

21. class Note
include DataMapper::Resource
property :id, Serial
property :content, String, :required => true, :length => 500
property :index, List, :required => true
timestamps :at
end

22. java_import org.apache.lucene.analysis.snowball.SnowballAnalyzer
java_import java.io.StringReader

23. before :valid?, :update_index
def update_index
analyzer = SnowballAnalyzer.new("English")
s = StringReader.new(content)
token_stream = analyzer.tokenStream(nil, s)
terms = []
while (token = token_stream.next) do
terms << token.term
end
self.index = terms
end

24. before :valid?, :update_index
def update_index
analyzer = SnowballAnalyzer.new("English")
s = StringReader.new(content)
token_stream = analyzer.tokenStream(nil, s)
terms = []
while (token = token_stream.next) do
terms << token.term
end
self.index = terms
end

25. hAp://full‐text‐search.appspot.com/
25

26. a about above after again against all am an and any are
aren't as at be because been before being below between
both but by can't cannot could couldn't did didn't do does
doesn't doing don't down during each few for from further had
hadn't has hasn't have haven't having he he'd he'll he's her
here here's hers herself him himself his how how's i i'd i'll i'm
i've if in into is isn't it it's its itself let's me more most mustn't
my myself no nor not of off on once only or other ought our
ours ourselves out over own same shan't she she'd she'll
she's should shouldn't so some such than that that's the their
theirs them themselves then there there's these they they'd
they'll they're they've this those through to too under until up
very was wasn't we we'd we'll we're we've were weren't what
what's when when's where where's which while who who's
whom why why's with won't would wouldn't you you'd you'll
you're you've your yours yourself yourselves
http://www.ranks.nl/resources/stopwords.html

27. Word Boundaries

32.
I love horses

33.
I love horses

34.
I love horses
Horses are beauSful

35.
I love horses
Horses are beauSful

36.
I love horses
Horses are beauSful
deer in the forest

37.
I love horses
Horses are beauSful
deer in the forest

38.
I love horses
Horses are beauSful
deer in the forest
deer live in the woods

39.
I love horses
Horses are beauSful
deer in the forest
deer live in the woods

40.
I love horses
Horses are beauSful
deer in the forest
deer live in the woods

41.
I love horses
Horses are beauSful
deer in the forest
deer live in the woods
You are an idiot.

42. Relevance

47. Accuracy

48. Speed

51. Write
Hosted
Database
Search
Rails

52. Read
Hosted
Database
Search
Rails

54. Target Target Source
Text Language Language
We’re running out of daylight en ja
Could you run this? en ja
Cross‐country running en ja
I’m going for a run, want to come along? en ja

55. I’m going for a run, want to come along? en ja

56. I’m going for a run, want to come along? en ja

57. I’m going for a run, want to come along? en ja
ha shi ri ni iku ke do iAtsho ni ki ma su ka?

58. I’m going for a run, want to come along? en ja
ha shi ri ni iku ke do iAtsho ni ki ma su ka?
Ikuko Kobayashi

59. I’m going for a run, want to come along? en ja
ha shi ri ni iku ke do iAtsho ni ki ma su ka?
Ikuko Kobayashi
2009‐11‐29 20:36:47 UTC

60. I’m going for a run, want to come along? en ja
ha shi ri ni iku ke do iAtsho ni ki ma su ka?
Ikuko Kobayashi
2009‐11‐29 20:36:47 UTC
hAp://….16ec695a‐8fce‐4277‐bdd4.flv

61. I’m going for a run, want to come along? en ja
ha shi ri ni iku ke do iAtsho ni ki ma su ka?
Ikuko Kobayashi
2009‐11‐29 20:36:47 UTC
hAp://….16ec695a‐8fce‐4277‐bdd4.flv
hAp://….Japanese_ikuko_kobayashi.jpg

62. 62

63. class Page < ActiveRecord::Base
acts_as_tsearch :fields => [ ... ]
end

64. Page.send :acts_as_tsearch, :fields => [:title]
PagePart.send :acts_as_tsearch, :fields =>
[:content]
ProgramPropertyList.send :acts_as_tsearch,
:fields
=>[:instructor, :program_desc,
:program_detail, :resource]

65. @pages = Page.find_by_tsearch(@query)

66. 66

69. 69

70. 70

71. 71

72. class Phrase < ActiveRecord::Base
acts_as_tsearch :fields => [:text]
end

73. Phrase.find_by_tsearch(term,
:conditions => {:language_id =>
target_language.id})

75. When you think about
search...

77. Questions?