Open source indexing and
search engine

Web scale

Lucene
Inverted
index

Lucene
Inverted
index
Results

Lucene
Inverted
index
Servlet container
J2EE application server

WARNING
Java approaching!

Java is strongly
object orientated

my @gene_names = ();
push(@gene_names, $gene);
print @gene_names;
Perl
Java
Array gene_names = new Array();
gene_names.add(gene);
System.out.println(gene_names.toString)

my $gene = Gene->new(‘ENS12345’);
$gene->set_name(‘BRCA2’);
Perl
Java
Gene gene = new Gene(‘ENS12345’);
gene.set_name(‘BRCA2’);

Java is strongly typed

my $number = “100”;
$number = $number + 400
print $number;
Perl
Java
Integer number = new Integer(100);
number = number + 400;
System.out.println(number + 400);

Java is good at error
handling

eval ($gene->transform);
warn $@ if $@;
Perl
Java
try {
gene->transform
} catch (IOException e) {
e.printStackTrace;
}

Java is surprisingly
easy to learn

Conditionals and loops
Variables have scope
Extras from CPAN
Performance is important
Perl
Java
Conditionals and loops
Variables have scope
Extras available as JAR files
Performance is important

Recipe 1:
Indexing a collection of
documents

org.ensembl.lucene.Writer

public static void main(String[] args) {
HashMap<String, String> arguments = new HashMap<String, String>();
String key = null;
for (String s: args) {
if (key == null) {
key = s;
} else {
arguments.put(key, s);
key = null;
}
}
Writer writer = new Writer();
writer.setIndexLocation(arguments.get(\"-index\"));
writer.setInputLocation(arguments.get(\"-input\"));
if (arguments.get(\"-mergefactor\") != null) {
writer.setMergeFactor(Integer.valueOf(arguments.get(\"-mergefactor\")));
}
if (arguments.get(\"-maxmergedocs\") != null) {
writer.setMaxMergeDocs(Integer.valueOf(arguments.get(\"-maxmergedocs\")));
}
try {
writer.index();
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(\"Indexing complete\");
}

public static void main(String[] args) {
HashMap<String, String> arguments = new HashMap<String, String>();
String key = null;
for (String s: args) {
if (key == null) {
key = s;
} else {
arguments.put(key, s);
key = null;
}
}
Writer writer = new Writer();
writer.setIndexLocation(arguments.get(\"-index\"));
writer.setInputLocation(arguments.get(\"-input\"));
if (arguments.get(\"-mergefactor\") != null) {
writer.setMergeFactor(Integer.valueOf(arguments.get(\"-mergefactor\")));
}
if (arguments.get(\"-maxmergedocs\") != null) {
writer.setMaxMergeDocs(Integer.valueOf(arguments.get(\"-maxmergedocs\")));
}
try {
writer.index();
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(\"Indexing complete\");
}

public static void main(String[] args) {
HashMap<String, String> arguments = new HashMap<String, String>();
String key = null;
for (String s: args) {
if (key == null) {
key = s;
} else {
arguments.put(key, s);
key = null;
}
}
Writer writer = new Writer();
writer.setIndexLocation(arguments.get(\"-index\"));
writer.setInputLocation(arguments.get(\"-input\"));
if (arguments.get(\"-mergefactor\") != null) {
writer.setMergeFactor(Integer.valueOf(arguments.get(\"-mergefactor\")));
}
if (arguments.get(\"-maxmergedocs\") != null) {
writer.setMaxMergeDocs(Integer.valueOf(arguments.get(\"-maxmergedocs\")));
}
try {
writer.index();
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(\"Indexing complete\");
}

Max-merge-docs
how many documents are added to a
segment

Merge-factor
how often Lucene merges index segments
when adding documents

public static void main(String[] args) {
HashMap<String, String> arguments = new HashMap<String, String>();
String key = null;
for (String s: args) {
if (key == null) {
key = s;
} else {
arguments.put(key, s);
key = null;
}
}
Writer writer = new Writer();
writer.setIndexLocation(arguments.get(\"-index\"));
writer.setInputLocation(arguments.get(\"-input\"));
if (arguments.get(\"-mergefactor\") != null) {
writer.setMergeFactor(Integer.valueOf(arguments.get(\"-mergefactor\")));
}
if (arguments.get(\"-maxmergedocs\") != null) {
writer.setMaxMergeDocs(Integer.valueOf(arguments.get(\"-maxmergedocs\")));
}
try {
writer.index();
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(\"Indexing complete\");
}

public void index() throws IOException {
File index = new File(getIndexLocation());
File location = new File(getInputLocation());
IndexWriter writer = new IndexWriter(index, new StandardAnalyzer(), true);
writer.setMergeFactor(getMergeFactor());
writer.setMaxMergeDocs(getMaxMergeDocs());
indexDocuments(writer, location);
writer.optimize();
writer.close();
}
private static void indexDocuments(IndexWriter writer, Filelocation) throws IOException {
if (location.canRead()) {
if (location.isDirectory()) {
String[] files = location.list();
if (files != null) {
for (int i = 0; i < files.length; i++) {
indexDocuments(writer, new File(location, files[i]));
}
}
} else {
System.out.println(\"Indexing \" + location);
try {
GeneFileDocument.index(writer, location);
}
catch (FileNotFoundException e) {
System.out.println(\"Caught exception: \" + e);
}
}
}
}

public void index() throws IOException {
File index = new File(getIndexLocation());
File location = new File(getInputLocation());
IndexWriter writer = new IndexWriter(index, new StandardAnalyzer(), true);
writer.setMergeFactor(getMergeFactor());
writer.setMaxMergeDocs(getMaxMergeDocs());
indexDocuments(writer, location);
writer.optimize();
writer.close();
}
private static void indexDocuments(IndexWriter writer, Filelocation) throws IOException {
if (location.canRead()) {
if (location.isDirectory()) {
String[] files = location.list();
if (files != null) {
for (int i = 0; i < files.length; i++) {
indexDocuments(writer, new File(location, files[i]));
}
}
} else {
System.out.println(\"Indexing \" + location);
try {
GeneFileDocument.index(writer, location);
}
catch (FileNotFoundException e) {
System.out.println(\"Caught exception: \" + e);
}
}
}
}

public void index() throws IOException {
File index = new File(getIndexLocation());
File location = new File(getInputLocation());
IndexWriter writer = new IndexWriter(index, new StandardAnalyzer(), true);
writer.setMergeFactor(getMergeFactor());
writer.setMaxMergeDocs(getMaxMergeDocs());
indexDocuments(writer, location);
writer.optimize();
writer.close();
}
private static void indexDocuments(IndexWriter writer, Filelocation) throws IOException {
if (location.canRead()) {
if (location.isDirectory()) {
String[] files = location.list();
if (files != null) {
for (int i = 0; i < files.length; i++) {
indexDocuments(writer, new File(location, files[i]));
}
}
} else {
System.out.println(\"Indexing \" + location);
try {
GeneFileDocument.index(writer, location);
}
catch (FileNotFoundException e) {
System.out.println(\"Caught exception: \" + e);
}
}
}
}

public void index() throws IOException {
File index = new File(getIndexLocation());
File location = new File(getInputLocation());
IndexWriter writer = new IndexWriter(index, new StandardAnalyzer(), true);
writer.setMergeFactor(getMergeFactor());
writer.setMaxMergeDocs(getMaxMergeDocs());
indexDocuments(writer, location);
writer.optimize();
writer.close();
}
private static void indexDocuments(IndexWriter writer, Filelocation) throws IOException {
if (location.canRead()) {
if (location.isDirectory()) {
String[] files = location.list();
if (files != null) {
for (int i = 0; i < files.length; i++) {
indexDocuments(writer, new File(location, files[i]));
}
}
} else {
System.out.println(\"Indexing \" + location);
try {
GeneFileDocument.index(writer, location);
}
catch (FileNotFoundException e) {
System.out.println(\"Caught exception: \" + e);
}
}
}
}

org.ensembl.lucene.
GeneFileDocument

public static void index(IndexWriter writer, File f) throws IOException {
String fields[] = {\"subtype\", \"id\", \"url\", \"keywords\", \"description\"};
FileReader input = new FileReader(f);
BufferedReader bufRead = new BufferedReader(input);
String line;
line = bufRead.readLine();
while (line != null){
Document doc = new Document();
int count = 0;
String terms[] = line.split(\"\\t\");
while (count < terms.length) {
String field = fields[count];
String item = terms[count];
doc.add(new Field(field, item, Field.Store.YES, Field.Index.TOKENIZED));
count++;
}
writer.addDocument(doc);
line = bufRead.readLine();
}
}

public static void index(IndexWriter writer, File f) throws IOException {
String fields[] = {\"subtype\", \"id\", \"url\", \"keywords\", \"description\"};
FileReader input = new FileReader(f);
BufferedReader bufRead = new BufferedReader(input);
String line;
line = bufRead.readLine();
while (line != null){
Document doc = new Document();
int count = 0;
String terms[] = line.split(\"\\t\");
while (count < terms.length) {
String field = fields[count];
String item = terms[count];
doc.add(new Field(field, item, Field.Store.YES, Field.Index.TOKENIZED));
count++;
}
writer.addDocument(doc);
line = bufRead.readLine();
}
}

public static void index(IndexWriter writer, File f) throws IOException {
String fields[] = {\"subtype\", \"id\", \"url\", \"keywords\", \"description\"};
FileReader input = new FileReader(f);
BufferedReader bufRead = new BufferedReader(input);
String line;
line = bufRead.readLine();
while (line != null){
Document doc = new Document();
int count = 0;
String terms[] = line.split(\"\\t\");
while (count < terms.length) {
String field = fields[count];
String item = terms[count];
doc.add(new Field(field, item, Field.Store.YES, Field.Index.TOKENIZED));
count++;
}
writer.addDocument(doc);
line = bufRead.readLine();
}
}

Quite a lot of memory
~1.5Gb

Creates index

Merge indices to form
master search index

Recipe 2:
Finding documents
containing a search
term

Easy

org.ensembl.lucene.Search

public static void main(String args[]) {
Timer timer = new Timer();
String index = \"index\";
try {
timer.start();
Searcher searcher = new IndexSearcher(index);
timer.stop();
System.out.println(\"Loaded \" + searcher.maxDoc() + \" documents in \" + timer.elapsed() + \"ms\");
search(searcher, \"subtype\", \"Vega_havana processed_pseudogene Gene\");
search(searcher, \"id\", \"OTTHUMG00000000423\");
searcher.close();
} catch (Exception e) {
e.printStackTrace();
}
}

public static void main(String args[]) {
Timer timer = new Timer();
String index = \"index\";
try {
timer.start();
Searcher searcher = new IndexSearcher(index);
timer.stop();
System.out.println(\"Loaded \" + searcher.maxDoc() + \" documents in \" + timer.elapsed() + \"ms\");
search(searcher, \"subtype\", \"Vega_havana processed_pseudogene Gene\");
search(searcher, \"id\", \"OTTHUMG00000000423\");
searcher.close();
} catch (Exception e) {
e.printStackTrace();
}
}

public static void main(String args[]) {
Timer timer = new Timer();
String index = \"index\";
try {
timer.start();
Searcher searcher = new IndexSearcher(index);
timer.stop();
System.out.println(\"Loaded \" + searcher.maxDoc() + \" documents in \" + timer.elapsed() + \"ms\");
search(searcher, \"subtype\", \"Vega_havana processed_pseudogene Gene\");
search(searcher, \"id\", \"OTTHUMG00000000423\");
searcher.close();
} catch (Exception e) {
e.printStackTrace();
}
}

public static void main(String args[]) {
Timer timer = new Timer();
String index = \"index\";
try {
timer.start();
Searcher searcher = new IndexSearcher(index);
timer.stop();
System.out.println(\"Loaded \" + searcher.maxDoc() + \" documents in \" + timer.elapsed() + \"ms\");
search(searcher, \"subtype\", \"Vega_havana processed_pseudogene Gene\");
search(searcher, \"id\", \"OTTHUMG00000000423\");
searcher.close();
} catch (Exception e) {
e.printStackTrace();
}
}

private static void search(Searcher searcher, String field, String queryString) throws
ParseException, IOException {
Timer timer = new Timer();
timer.start();
System.out.println(\"Search (\" + field + \"): \" + queryString);
QueryParser parser = new QueryParser(field, new StandardAnalyzer());
Query query = parser.parse(queryString);
Hits hits = searcher.search(query);
Integer count = 1;
Iterator<Hit> hiterator = hits.iterator();
while (hiterator.hasNext()) {
Hit hit = hiterator.next();
Document document = hit.getDocument();
System.out.println(count + \": ID: \" + document.get(\"id\"));
System.out.println(count + \": Subtype: \" + document.get(\"subtype\"));
count++;
}
int hitCount = hits.length();
timer.stop();
System.out.println(\"Hits: \" + hitCount);
System.out.println(\"Completed in \" + timer.elapsed() + \"ms\");

private static void search(Searcher searcher, String field, String queryString) throws
ParseException, IOException {
Timer timer = new Timer();
timer.start();
System.out.println(\"Search (\" + field + \"): \" + queryString);
QueryParser parser = new QueryParser(field, new StandardAnalyzer());
Query query = parser.parse(queryString);
Hits hits = searcher.search(query);
Integer count = 1;
Iterator<Hit> hiterator = hits.iterator();
while (hiterator.hasNext()) {
Hit hit = hiterator.next();
Document document = hit.getDocument();
System.out.println(count + \": ID: \" + document.get(\"id\"));
System.out.println(count + \": Subtype: \" + document.get(\"subtype\"));
count++;
}
int hitCount = hits.length();
timer.stop();
System.out.println(\"Hits: \" + hitCount);
System.out.println(\"Completed in \" + timer.elapsed() + \"ms\");

private static void search(Searcher searcher, String field, String queryString) throws
ParseException, IOException {
Timer timer = new Timer();
timer.start();
System.out.println(\"Search (\" + field + \"): \" + queryString);
QueryParser parser = new QueryParser(field, new StandardAnalyzer());
Query query = parser.parse(queryString);
Hits hits = searcher.search(query);
Integer count = 1;
Iterator<Hit> hiterator = hits.iterator();
while (hiterator.hasNext()) {
Hit hit = hiterator.next();
Document document = hit.getDocument();
System.out.println(count + \": ID: \" + document.get(\"id\"));
System.out.println(count + \": Subtype: \" + document.get(\"subtype\"));
count++;
}
int hitCount = hits.length();
timer.stop();
System.out.println(\"Hits: \" + hitCount);
System.out.println(\"Completed in \" + timer.elapsed() + \"ms\");

private static void search(Searcher searcher, String field, String queryString) throws
ParseException, IOException {
Timer timer = new Timer();
timer.start();
System.out.println(\"Search (\" + field + \"): \" + queryString);
QueryParser parser = new QueryParser(field, new StandardAnalyzer());
Query query = parser.parse(queryString);
Hits hits = searcher.search(query);
Integer count = 1;
Iterator<Hit> hiterator = hits.iterator();
while (hiterator.hasNext()) {
Hit hit = hiterator.next();
Document document = hit.getDocument();
System.out.println(count + \": ID: \" + document.get(\"id\"));
System.out.println(count + \": Subtype: \" + document.get(\"subtype\"));
count++;
}
int hitCount = hits.length();
timer.stop();
System.out.println(\"Hits: \" + hitCount);
System.out.println(\"Completed in \" + timer.elapsed() + \"ms\");

private static void search(Searcher searcher, String field, String queryString) throws
ParseException, IOException {
Timer timer = new Timer();
timer.start();
System.out.println(\"Search (\" + field + \"): \" + queryString);
QueryParser parser = new QueryParser(field, new StandardAnalyzer());
Query query = parser.parse(queryString);
Hits hits = searcher.search(query);
Integer count = 1;
Iterator<Hit> hiterator = hits.iterator();
while (hiterator.hasNext()) {
Hit hit = hiterator.next();
Document document = hit.getDocument();
System.out.println(count + \": ID: \" + document.get(\"id\"));
System.out.println(count + \": Subtype: \" + document.get(\"subtype\"));
count++;
}
int hitCount = hits.length();
timer.stop();
System.out.println(\"Hits: \" + hitCount);
System.out.println(\"Completed in \" + timer.elapsed() + \"ms\");

private static void search(Searcher searcher, String field, String queryString) throws
ParseException, IOException {
Timer timer = new Timer();
timer.start();
System.out.println(\"Search (\" + field + \"): \" + queryString);
QueryParser parser = new QueryParser(field, new StandardAnalyzer());
Query query = parser.parse(queryString);
Hits hits = searcher.search(query);
Integer count = 1;
Iterator<Hit> hiterator = hits.iterator();
while (hiterator.hasNext()) {
Hit hit = hiterator.next();
Document document = hit.getDocument();
System.out.println(count + \": ID: \" + document.get(\"id\"));
System.out.println(count + \": Subtype: \" + document.get(\"subtype\"));
count++;
}
int hitCount = hits.length();
timer.stop();
System.out.println(\"Hits: \" + hitCount);
System.out.println(\"Completed in \" + timer.elapsed() + \"ms\");

Recipe 3:
Querying a remote
document index

Wrap everything into a
single file

Copy that file to an
application server

Restart the application
server

Voilà!

(almost never that easy)

You will need...

Bonus recipe!
Automate tasks with Ant

XML based configuration

Automated compiles

Automated test runner

Automated deployment

Platform independent

Flexible
(but complex)

ant deploy

clean code
clean index
compile
build index
build jar
build war
deploy

Could this work for
Ensembl?

lucene.apache.org

Java IDEs rock:
get stuck in

Thank you