Search and information discovery is a huge part of almost any modern site. Solr is an incredibly powerful search tool that allows us to quickly add advanced search capabilities such as full-text search, faceting, autocomplete and spelling suggestions to our projects without much effort. We will be using 'django-haystack' to communicate between Django and Solr.

1. ADVANCED SEARCH
WITH
SOLR + DJANGO-HAYSTACK
MARCEL CHASTAIN
LA DJANGO – 2014-09-30

2. WHAT WE’LL COVER
1. THE PITCH:
The Problem With Search
The Solution(s)
Overall Architecture of System with Django/Solr/Haystack
2. THE GOOD STUFF:
Indexing Data for Search
Querying the Search Index
Advanced Search Methods
Resources

3. THE PITCH
OR, “WHY ANY OF THIS MATTERS”

4. THE PROBLEM
1. Sites with stored information are
ONLY as useful as they are at
retrieving and displaying that
information

5. THE PROBLEM
2. Users have high expectations of
search (thanks, Google)

6. THE PROBLEM
2. Users have high expectations of
search
• Spelling Suggestions:

7. THE PROBLEM
2. Users have high expectations of
search
• Hit Highlighting:

8. THE PROBLEM
2. Users have high expectations of
search
• “Related Searches”
• Distance/GeoSpatial Search

9. THE PROBLEM
2. Users have high expectations of
search
• Faceting:

10. THE PROBLEM
3. Good search involves lots of
challenges

11. THE PROBLEM
3. Good search involves lots of
challenges
• Stemming:
User Searches For Word “Stem”
“argue”
“argues”
“argued”
“argu”
“argument”
“arguments”
“argument”

12. THE PROBLEM
3. Good search involves lots of
challenges
And more..!
• Synonyms
• Acronyms
• Non-ASCII characters
• Stop words (“and”, “to”, “a”)
• Calculating relevance
• Performance with millions/billions(!) of documents

13. THE SOLUTION
“Information Retrieval Systems”
a.k.a Search Engines

14. THE SOLUTION
“Information Retrieval Systems”
a.k.a Search Engines

15. SOLR
THE BACKEND

16. WHAT IS SOLR?
Open-source enterprise search
Java-based
Created in 2004
Built on Apache Lucene
Most popular enterprise search engine
Apache 2.0 License
Built for millions or billions of documents

17. WHAT DOES IT DO?
• Full-text search
• Hit highlighting
• Faceted search
• Clustering/replication/sharding
• Database integration
• Rich document (word, pdf, etc) handling
• Geospatial search
• Spelling corrections/suggestions
• … loads and loads more

18. WHO USES SOLR?

19. HOW CAN WE USE IT
WITH DJANGO?
Haystack
From the homepage:
(http://haystacksearch.org/)

20. LOOK FAMILIAR?
Query style
Declarative search index definitions

21. THE GOOD
STUFF
INSTALLING, CONFIGURING & USING
SOLR/HAYSTACK

22. WHO DOES WHAT
Solr:
• Provides API for submitting to & querying from index
• Stores actual index data
• Manages fields/data types in xml config (‘schema.xml’)
Haystack:
• Manages connection(s) to solr
• Provides familiar API for querying
• Uses templates and declarative search index definitions
• Helps generate solr xml config
• Management commands to index content
• Generic views/forms for common search use-cases
• Hooks into signals to keep data up-to-date

23. PART 1:
LET’S MAKE AN INDEX

24. 0. GITHUB REPO
git clone https://github.com/marcelchastain/haystackdemo

25. 1. SETUP SOLR
(from github repo root)
./solr_download.sh
(or, manually)
wget http://apache.mirrors.pair.com/lucene/solr/4.10.1/solr-4.10.1.tgz
tar –xzvf solr-4.10.1.tgz
ln –s ./solr-4.10.1 ./solr
The one file to care about:
• solr/example/solr/collection1/conf/schema.xml
Stores field definitions and data types. Frequently updated during
development

26. 2. RUN SOLR
(from github repo root)
./solr_start.sh
(or, manually)
cd solr/example && java –jar start.jar
Requires java 1.7+. To install on debian/ubuntu:
sudo apt-get install openjdk-7-jre-headless

27. 3. INSTALL HAYSTACK
(CWD haystackdemo/)
apt-get install python-pip python-virtualenv
virtualenv env && source env/bin/activate
(from github repo root)
pip install –r requirements.txt
(or, manually)
pip install Django==1.6.7 django-haystack

28. 4. HAYSTACK SETTINGS
INSTALLED_APPS = [
# ‘django.contrib.admin’, etc
‘haystack’,
# then your usual apps
‘myapp’,
]
HAYSTACK_CONNECTIONS = {
‘default’: {
‘ENGINE’: ‘haystack.backends.solr_backend.SolrEngine’,
‘URL’: ‘http://127.0.0.1:8983/solr’
},
}
HAYSTACK_SIGNAL_PROCESSOR = ‘haystack.signals.RealtimeSignalProcessor’

29. 5. THE MODEL(S)

30. 6. SYNCDB & INITIAL DATA
(CWD haystackdemo/demo/)
./manage.py syncdb
./manage.py loaddata restaurants

31. 7. DEFINE SEARCH INDEX
myapp/search_indexes.py

32. 7.5 BOOSTING FIELD
RELEVANCE
Some fields are simply more relevant!
(Note: changes to field boosts require reindex)

33. 8. CREATE A TEMPLATE
FOR INDEXED TEXT
templates/search/indexes/myapp/note_text.txt

34. 9. UPDATE SOLR SCHEMA
(CWD: haystackdemo/demo/)
./manage.py build_solr_schema >
../solr/example/solr/collection1/conf/schema.xml
Which adds:
*Restart solr for changes to go into effect

35. 10. REBUILD INDEX
(CWD hackstackdemo/demo/)
$ ./manage.py update_index
Indexing 6 notes

36. 10. REBUILD INDEX
(CWD hackstackdemo/demo/)
$ ./manage.py update_index
Indexing 6 notes

37. PART 2:
LET’S GET TO QUERYIN’

38. SIMPLE
SEARCHQUERYSETS

39. GREAT, WHAT ABOUT
FROM A BROWSER?

40. EASY MODE
Full-document search
urls.py
templates/search/search.html

41. HAYSTACK COMPONENTS TO
EXTEND
• haystack.forms.SearchForm
django form with extendable .search() method. Define additional
fields on the form, then incorporate them in the .search()
method’s logic
• haystack.views.SearchView
Class-based view made to be flexible for common search cases

42. PART 3: FEATURES

43. HIT HIGHLIGHTING
Instead of referring to a context variable directly, use the {% highlight %} tag

44. SPELLING
SUGGESTIONS
Update connection’s settings dictionary + reindex
Use spelling_suggestion() method

45. AUTOCOMPLETE
Create another search index field using EdgeNgramField + reindex
Use the .autocomplete() method on a SearchQuerySet

46. FACETING
Add faceting to search index definition
Regenerate schema.xml and reindex content
./manage.py build_solr_schema >
../solr/example/solr/collection1/conf/schema.xml
./manage.py update_index

47. FACETING
From a shell:

48. RESOURCES
LET’S SAVE YOU A GOOGLE TRIP

49. RESOURCES
Solr in Action ($45)
Apr 2014
Haystack Documentation
http://django-haystack.readthedocs.org/
IRC (freenode):
#django
#haystack
#solr