BioSolr, funded by the BBSRC, is a collaboration between open source search experts Flax and the European Bioinformatics Institute (EBI), aiming to significantly advance the state of the art with regard to indexing and querying biomedical data with freely available open source software
recruitment government e-commerce news & media bioinformatics consulting law
PDBe is the European resource for the collection, organisation and dissemination of data on biological macromolecular structures… collate, maintain and provide access to the global repositories of macromolecular structure data SPOT team – 3 subteams Mouse informatics, Functional Genomics Production, Gene ontology editorial office. The FGPT subteam develops ontologies such as the Experimental Factor Ontology (EFO) , the Cell Line Ontology, delivers ontology tooling, provides curation tools for Gene Expression Atlas, BioSamples databases
Grant met heads of SPOT and PDBE teams, did Solr presentation BBSRC = Biotechnology and Biological Sciences Research Council
BBSRC = Biotechnology and Biological Sciences Research Council
PDBe is the European resource for the collection, organisation and dissemination of data on biological macromolecular structures… collate, maintain and provide access to the global repositories of macromolecular structure data actively involved in an effort to integrate data from major biomedical resources at EMBL-EBI and across the world. This &quot;Structure Integration with Function, Taxonomy and Sequence&quot; (SIFTS) initiative integrates data from a number of resources and is used by major global sequence, structure and protein-family resources.
Sources: FASTA (fast- ay) – search tool for protein databases PHMMER – search protein sequences against a protein sequence database
Distributed search, but very distributed EMBL has member states across Europe, plus associate member countries – Australia, Argentina
Ontologies generally hierarchical – root node, child nodes, etc. Additional relationships between nodes at any level though, so not strictly a tree structure.
SPARQL is an RDF query language for searching across triple stores. Triple stores are databases where each item represents a subject, a predicate and an object – eg. “the heart is part of the human body”
Annotations vary between ontologies as to which data is a synonym, definintion, and so on. Need Solr config to tell it which annotation applies for the current ontology.
SWAT4LS – Semantic web applications and tools for life sciences EBI workshop – hands on, interactive, understanding different search technologies for biomedical data
BioSolr - Searching the stuff of life - Lucene/Solr Revolution 2015
OCTOBER 13-16, 2016 • AUSTIN, TX
Searching the Stuff of Life: BioSolr
Matt Pearce & Alan Woodward
Senior Developers, Flax – www.flax.co.uk
•Building open source search applications since 2001
•Independent, honest advice and analysis
•Expert design & development, Apache Solr committers
•UK Authorized Partner of
•Test-driven relevancy and performance tuning
•Custom training & mentoring
•The European Bioinformatics Institute
•Part of the European Molecular Biology Laboratory
•Based on the Wellcome Genome Campus in Hinxton, Cambridge, UK.
•Maintains the world’s most comprehensive range of freely available and up-to-date
molecular databases, serving millions of researchers – indexing over 1 billion items.
•BioSolr project involves two teams from EMBL-EBI:
•Protein Data Bank in Europe (PDBe)
•Samples, Phenotypes and Ontologies (SPOt)
The genesis of BioSolr
•Grant Ingersoll visits the Wellcome Campus in July ’13
•Around 90 people attend
•Show of hands indicates 75% using Lucene/Solr
•Sameer Velankar of EMBL-EBI identifies grant funding
•Flax and EMBL-EBI apply successfully to the BBSRC
•One year BBSRC funded project from September 2014
•“to significantly advance the state of the art with regard to indexing and querying biomedical data
with freely available open source software”
•Papers & presentations
•Software (Open Source, of course!)
•Inputs: from the PDBe & SPOt teams
•Working on site with Sameer Velankar & the PDBe team
•Facet.contains, Xjoin, Federated Search
•Working on site with Tony Burdett & the SPOt team
•The problem – you have data in an external data store which is not suitable
for indexing in Solr
•The data may be from a live source, for example.
•You need to match data from your search results against data from one or
more of these external sources for display or analysis.
•Xjoin is implemented as a Solr search component.
•There should be one configured instance per external source.
•XJoinResultsFactory interface defines the search behaviour:
•Communicates with the external source to carry out the query
•Configured in solrconfig.xml, along with presets
•Returns results as an XjoinResults object
•Results are keyed by a string ID, defined in the configuration
•User is required to provide the implementation of this interface for each
external source being used
•Configure the XJoinSearchComponent in solrconfig.xml with details of your
•Add the search component to the search request handler
•Needs to be in both first-components and last-components sections.
Xjoin results handling
•Uses a query parser based on TermsQParserPlugin
•Allows the same methods as TermsQParserPlugin
•Enables the use of multiple external sources, joined using Boolean
•Results are returned in a separate block
•Similar to how highlights are returned.
Federated search - introduction
•Problem: we need to search data sets split across multiple locations (even
•Records may contain different fields.
•Similar to pre-SolrCloud distributed search
Federated search challenges: result counts
•The same document may appear in more than one shard, so they need to be
•Also applies to facet counts.
•Shards return all document IDs rather than the number found.
•Aggregator builds a set of unique documents from the returned results.
•Simple for small result sets but inefficient for large sets.
•Estimate the result count, using statistical methods:
•If two shards always return similar counts, overlap likely to be high;
•If they don’t, overlap will be small so dataset can be treated as independent and
number found added to total.
Federated search challenges: merging document sets
•Problem: documents are not unique across data sets.
•Default behaviour is to use the first instance of a document, ignoring others.
•Datasets may contain different fields – we need all versions.
•Use a custom MergeStrategy to build the ID list.
•Cannot use a Grouper – prevents result grouping in the query.
•Scoring is also a challenge!
Federated search challenges: merging document data
•Problem: document data may not be the same across data sets.
•Merge documents together into a single, composite document.
•Potentially use an aggregation schema describing merge process.
•Merge strategy must be capable of merging disparate field types.
BioSolr & SPOt – Indexing ontologies
Washington, N. & Lewis, S. (2008) Ontologies: Scientific
Data Sharing Made Easy. Nature Education 1(3):5
Indexing ontologies – the problem
•You have a collection of documents annotated with ontology references.
•You want to search both the documents and the associated ontology data.
•This may include associated nodes – “has location”, “is part of”, etc.
•Faceting the ontology references would be nice!
•(especially if the facets can be presented in a tree)
Keep the data separate
Approach 1 - steps
•Index the documents with the node annotations but no further
•Index the ontology in its own core.
•Search the documents, then cross-match against the ontology.
•BUT requires multiple calls, doesn’t allow searching both
cores at the same time.
Add some ontology data to your documents
Approach 2 – step 1
•Index the documents with the node annotations.
•While indexing, look up the node references, with their labels
•Easier to include the ontology references in your search.
•Can boost fields as required.
•Faster to search
Approach 2 – step 2
•Expand the ontology data being stored.
•Include single (or multi) level parent and child nodes, with their
•Use dynamic fields to store additional relationships.
•Dynamic fields allow searches across specific relationship
types (“is part of”, “has location”, etc.)
•BUT requires additional Solr look-ups to be fully dynamic
•(using /admin/luke to look through the current schema for dynamic fields).
Approach 2 – search screen
•General search box
•Options to include child and parent labels (one step
•Dynamically-generated additional relationship search
Approach 2 – final result
•We have developed an UpdateProcessor to do this as part of the update
•The user defines the ontology location (file, URL) and the field to
•Aims for convention over configuration for remaining properties.
•Field names and ontology annotation details all customisable.
•A similar plugin for ElasticSearch has also been developed.
Aside: facet trees
•Additional search component to return facets from ontology references in
•Takes initial facets from results, searches hierarchical references to build
the tree during facet generation.
•Avoids multiple calls to Solr from client-side to build the tree.
•A second collection can be used to search the ontologies...
•… BUT it must be part of the same Solr instance
Aside: facet trees - challenges
•Nodes with multiple parents may appear more
•… not yet found a solution for this!
•Default behaviour is to return entire tree.
•Not always useful – may need to drill through
several layers to get to useful entries.
•Solution: prune the tree!
Aside: facet trees - pruning
•Multiple pruning options available:
•Simple pruning: remove layers with no useful
•Datapoint pruning: return the highest counts at
top-level, remainder in “Other” section.
•Search the ontology, and cross-match with the documents.
•Allow SPARQL queries over the ontology index.
•Enables complex searches over ontology relationships.
•(SPARQL is a semantic query language)
Adding Apache Jena
•We use Apache Jena to provide TDB-querying with SPARQL.
•Jena uses Solr to search specified text fields – set via configuration files.
•Uses its own Triple Store for other fields and relationship queries.
•Return the reference URI in the returned fields to cross-match with
•Use a filter query to choose the matched documents.
•Not that different from Xjoin!
Generic ontology indexer
•Stand-alone application to index different ontologies
•Allows separate configuration for each ontology
•Plugins may be used:
•At item level, to external data some/all nodes;
•At ontology level, to push the ontology into MongoDB, etc.
•Cross-pollination with the EBI's Ontology Lookup Service site (currently in
Conclusions – what have we achieved?
•Searching across multiple external datasets (xjoin)
•See SOLR-7341 (and please up-vote!)
•Searching across multiple Solr nodes across different campuses.
•Indexing ontologies – both with and without document data.
•Enriching documents with ontology data.
•(…and facet trees)
•Check out the github page: https://github.com/flaxsearch/BioSolr
•Vote for Xjoin: https://issues.apache.org/jira/browse/SOLR-7341
•7th December: SWAT4LS tutorial session in Cambridge, UK
•Will cover ontology indexing and search
•3rd/4th February: Workshop at the EBI campus, Hinxton, UK
Thank you for listening!
+44 (0) 8700 118334