1. tl;dr: Solr

2. Dumbledore: "I use the Pensieve. One simply siphons the excess thoughts from one's mind,
pours them into the basin, and examines them at one's leisure. It becomes
easier to spot patterns and links, you understand, when they are in this form."
Harry: "You mean... that stuff's your thoughts?"
Dumbledore: "Certainly."

3. Dumbledore: "I use the Pensieve. One simply siphons the excess thoughts from one's mind,
pours them into the basin, and examines them at one's leisure. It becomes
easier to spot patterns and links, you understand, when they are in this form."
Harry: "You mean... that stuff's your thoughts?"
Dumbledore: "Certainly."

4. Solr is Lucene­based

Lucene = text search engine library written in Java

All kinds of crazy goodies:

Ranked search

Multiple indexing

Simultaneous read & write

Date­range search

...the list goes on

Platform­independent (thanks, Java!)

Fast & efficient

Index size ~= 20­30% size of indexed data

Very high throughput indexing (95GB/hour)

5. Solr is NoSQL

NoSQL == Non­relational database

RDBMS metaphor:

One database

One table

Denormalized data

Query parameters instead of SQL

“Documents” instead of rows

Bottom line: it's a persistent datastore, and we use it to store data
persistently.

6. Vocabulary

Master

Slave

Replication

Document

API

7. Master

There can be only one

Read & write operations

Must be secure

Younger, stronger brother of production DB

Home base for Solr slaves

8. Slave

There are many copies

They have a plan: replication

Read­only

Gets copy of index from the Solr master every k
minutes

Responds to queries

9. Replication

Slaves –­HTTP GET­­> Master

Replication is differential

Configuration is set in solrconfig.xml

http://tinyurl.com/DESolrRepl

10. Document

RDBMS = row; Solr = document

Denormalized relational data
my friend,

Flatten a bunch of related RDBMS rows into a
single Solr document

11. API

Application programming interface

Primary means of communicating with Solr is an
HTTP API

12. The Good Stuff:
Unix & Diagnostics
“This is the Unix philosophy: Write programs that
do one thing and do it well. Write programs to
work together. Write programs to handle text
streams, because that is a universal interface.”
­ Doug McIlroy

Examples of things beyond the scope of this talk:

Cat

Awk

Grep

Sed

Cut

Wc

Sort

Tail

Head

Great read: http://matt.might.net/articles/sql­in­the­shell/

13. The Good Stuff:
Unix & Diagnostics

You cannot effectively troubleshoot without parsing logs

You cannot effectively parse logs without good text­parsing tools:

Cat

Awk

Grep

Sed

Cut

Wc

Sort

Tail

Head

No *nix OS? PowerShell!

14. The Good Stuff:
Unix & Diagnostics

Example commands:

tail -f /var/log/celery/project.log

Output the Celery log to stdout, in real time

cat /ebs2/log/celery/project.log|grep -oE 'BUID:([0-9]
{0,5})'|grep -oE '[0-9]{0,5}'|sort --unique

Parse the Celery log, printing a list of unique BUIDs

cat /ebs2/log/celery/project.log|grep -B 15
"DocumentInvalid"|grep -E 'Download complete for BUID ([0-9]
{1,5})'|awk '{sub(/[/, "");print $1 " " $2 " " $7 ":" $8}'

Parse the Celery log, outputting a list of BUID the feed
file for which failed for some reason:

15. Conclusion

RTFreakingM

http://wiki.apache.org/solr/SolrQuerySyntax

http://wiki.apache.org/solr/SolrCaching

http://wiki.apache.org/solr/SchemaXml

http://django­haystack.readthedocs.org/en/latest/

Experiment & tinker & reinvent the wheel

Get comfortable with the command line – you can't effectively administer Solr
(or any sufficiently complex system) with a web GUI

Read the logs

Connect Solr behavior to application operations