More Related Content

Slideshows for you(20)


Similar to BioSolr - Searching the stuff of life - Lucene/Solr Revolution 2015(20)

Recently uploaded(20)


BioSolr - Searching the stuff of life - Lucene/Solr Revolution 2015

  1. OCTOBER 13-16, 2016 • AUSTIN, TX
  2. Searching the Stuff of Life: BioSolr Matt Pearce & Alan Woodward Senior Developers, Flax –
  3. •Building open source search applications since 2001 •Independent, honest advice and analysis •Expert design & development, Apache Solr committers •UK Authorized Partner of •Test-driven relevancy and performance tuning •Custom training & mentoring 02
  4. 02
  5. 01 •The European Bioinformatics Institute •Part of the European Molecular Biology Laboratory •Based on the Wellcome Genome Campus in Hinxton, Cambridge, UK. •Maintains the world’s most comprehensive range of freely available and up-to-date molecular databases, serving millions of researchers – indexing over 1 billion items. •BioSolr project involves two teams from EMBL-EBI: •Protein Data Bank in Europe (PDBe) •Samples, Phenotypes and Ontologies (SPOt)
  6. 02 The genesis of BioSolr •Grant Ingersoll visits the Wellcome Campus in July ’13 •Around 90 people attend •Show of hands indicates 75% using Lucene/Solr •Sameer Velankar of EMBL-EBI identifies grant funding •Flax and EMBL-EBI apply successfully to the BBSRC
  7. 03 BioSolr •One year BBSRC funded project from September 2014 •“to significantly advance the state of the art with regard to indexing and querying biomedical data with freely available open source software” •Outputs: •Workshops •Papers & presentations •Software (Open Source, of course!) •Documentation •Inputs: from the PDBe & SPOt teams
  8. 01 BioSolr •Tom Winch •Working on site with Sameer Velankar & the PDBe team •Facet.contains, Xjoin, Federated Search •Matt Pearce •Working on site with Tony Burdett & the SPOt team •Indexing ontologies
  9. 01 BioSolr & PDBe - Introduction •Protein Data Bank (PDBe) •facet.contains (autosuggest) • •In Solr 5.1 •Xjoin (searching external sources) • •Federated search
  10. 01 Xjoin concepts •The problem – you have data in an external data store which is not suitable for indexing in Solr •The data may be from a live source, for example. •You need to match data from your search results against data from one or more of these external sources for display or analysis.
  11. 01 Xjoin implementation •Xjoin is implemented as a Solr search component. •There should be one configured instance per external source. •XJoinResultsFactory interface defines the search behaviour: •Communicates with the external source to carry out the query •Configured in solrconfig.xml, along with presets •Returns results as an XjoinResults object •Results are keyed by a string ID, defined in the configuration •User is required to provide the implementation of this interface for each external source being used
  12. 01 Xjoin configuration •Configure the XJoinSearchComponent in solrconfig.xml with details of your XJoinResultsFactory implementations •Add the search component to the search request handler •Needs to be in both first-components and last-components sections.
  13. 01 Xjoin results handling •Uses a query parser based on TermsQParserPlugin •Allows the same methods as TermsQParserPlugin •Enables the use of multiple external sources, joined using Boolean operators. •Results are returned in a separate block •Similar to how highlights are returned.
  14. 01 Federated search - introduction •Problem: we need to search data sets split across multiple locations (even different countries) •Records may contain different fields. •Similar to pre-SolrCloud distributed search
  15. 01 Federated search challenges: result counts •The same document may appear in more than one shard, so they need to be aggregated. •Also applies to facet counts. One solution: •Shards return all document IDs rather than the number found. •Aggregator builds a set of unique documents from the returned results. •Simple for small result sets but inefficient for large sets. •Estimate the result count, using statistical methods: •If two shards always return similar counts, overlap likely to be high; •If they don’t, overlap will be small so dataset can be treated as independent and number found added to total.
  16. 01 Federated search challenges: merging document sets •Problem: documents are not unique across data sets. •Default behaviour is to use the first instance of a document, ignoring others. •Datasets may contain different fields – we need all versions. One solution: •Use a custom MergeStrategy to build the ID list. •Cannot use a Grouper – prevents result grouping in the query. •Scoring is also a challenge!
  17. 01 Federated search challenges: merging document data •Problem: document data may not be the same across data sets. A solution: •Merge documents together into a single, composite document. •Potentially use an aggregation schema describing merge process. •Merge strategy must be capable of merging disparate field types.
  18. 01 BioSolr & SPOt – Indexing ontologies Washington, N. & Lewis, S. (2008) Ontologies: Scientific Data Sharing Made Easy. Nature Education 1(3):5
  19. 01 Indexing ontologies – the problem •You have a collection of documents annotated with ontology references. •You want to search both the documents and the associated ontology data. •This may include associated nodes – “has location”, “is part of”, etc. •Faceting the ontology references would be nice! •(especially if the facets can be presented in a tree)
  20. 01 Approach 1 Keep the data separate documents Documents Indexer Documents Indexer ontology Ontology Indexer
  21. 01 Approach 1 - steps •Index the documents with the node annotations but no further detail. •Index the ontology in its own core. •Search the documents, then cross-match against the ontology. •BUT requires multiple calls, doesn’t allow searching both cores at the same time.
  22. 01 Approach 2 Add some ontology data to your documents Documents Indexer Ontology documents
  23. 01 Approach 2 – step 1 •Index the documents with the node annotations. •While indexing, look up the node references, with their labels and synonyms. •Easier to include the ontology references in your search. •Can boost fields as required. •Faster to search
  24. 01 Approach 2 – step 2 •Expand the ontology data being stored. •Include single (or multi) level parent and child nodes, with their labels. •Use dynamic fields to store additional relationships. •Dynamic fields allow searches across specific relationship types (“is part of”, “has location”, etc.) •BUT requires additional Solr look-ups to be fully dynamic •(using /admin/luke to look through the current schema for dynamic fields).
  25. 01 Approach 2 – search screen •General search box •Options to include child and parent labels (one step removed) •Dynamically-generated additional relationship search options
  26. 01 Approach 2 – search results
  27. 01 Approach 2 – final result •We have developed an UpdateProcessor to do this as part of the update chain. •The user defines the ontology location (file, URL) and the field to reference. •Aims for convention over configuration for remaining properties. •Field names and ontology annotation details all customisable. •A similar plugin for ElasticSearch has also been developed.
  28. 01 Aside: facet trees •Additional search component to return facets from ontology references in tree form. •Extends FacetComponent •Takes initial facets from results, searches hierarchical references to build the tree during facet generation. •Avoids multiple calls to Solr from client-side to build the tree. •A second collection can be used to search the ontologies... •… BUT it must be part of the same Solr instance
  29. 01 Aside: facet trees - challenges •Nodes with multiple parents may appear more than once… •… not yet found a solution for this! •Default behaviour is to return entire tree. •Not always useful – may need to drill through several layers to get to useful entries. •Solution: prune the tree!
  30. 01 Aside: facet trees - pruning •Multiple pruning options available: •Simple pruning: remove layers with no useful information. •Datapoint pruning: return the highest counts at top-level, remainder in “Other” section.
  31. 01 Approach 3 •Search the ontology, and cross-match with the documents. •Allow SPARQL queries over the ontology index. •Enables complex searches over ontology relationships. •(SPARQL is a semantic query language)
  32. 01 Approach 3
  33. 01 Adding Apache Jena •We use Apache Jena to provide TDB-querying with SPARQL. •Jena uses Solr to search specified text fields – set via configuration files. •Uses its own Triple Store for other fields and relationship queries. •Return the reference URI in the returned fields to cross-match with documents. •Use a filter query to choose the matched documents. •Not that different from Xjoin!
  34. 01 Generic ontology indexer •Stand-alone application to index different ontologies •Allows separate configuration for each ontology •Plugins may be used: •At item level, to external data some/all nodes; •At ontology level, to push the ontology into MongoDB, etc. •Cross-pollination with the EBI's Ontology Lookup Service site (currently in beta)
  35. 01 Conclusions – what have we achieved? •Searching across multiple external datasets (xjoin) •See SOLR-7341 (and please up-vote!) •Searching across multiple Solr nodes across different campuses. •Indexing ontologies – both with and without document data. •Enriching documents with ontology data. •(…and facet trees)
  36. 01 Get Involved! •Check out the github page: •Vote for Xjoin:
  37. 01 Upcoming events •7th December: SWAT4LS tutorial session in Cambridge, UK •Will cover ontology indexing and search • •3rd/4th February: Workshop at the EBI campus, Hinxton, UK •
  38. 01 Thank you for listening! Matt Pearce +44 (0) 8700 118334 Twitter: @flaxsearch

Editor's Notes

  1. Test-drive relevancy done via Quepid
  2. recruitment government e-commerce news & media bioinformatics consulting law
  3. PDBe is the European resource for the collection, organisation and dissemination of data on biological macromolecular structures… collate, maintain and provide access to the global repositories of macromolecular structure data SPOT team – 3 subteams Mouse informatics, Functional Genomics Production, Gene ontology editorial office. The FGPT subteam develops ontologies such as the Experimental Factor Ontology (EFO) , the Cell Line Ontology, delivers ontology tooling, provides curation tools for Gene Expression Atlas, BioSamples databases
  4. Grant met heads of SPOT and PDBE teams, did Solr presentation BBSRC = Biotechnology and Biological Sciences Research Council
  5. BBSRC = Biotechnology and Biological Sciences Research Council
  6. PDBe is the European resource for the collection, organisation and dissemination of data on biological macromolecular structures… collate, maintain and provide access to the global repositories of macromolecular structure data actively involved in an effort to integrate data from major biomedical resources at EMBL-EBI and across the world. This "Structure Integration with Function, Taxonomy and Sequence" (SIFTS) initiative integrates data from a number of resources and is used by major global sequence, structure and protein-family resources.
  7. Sources: FASTA (fast- ay) – search tool for protein databases PHMMER – search protein sequences against a protein sequence database
  8. Distributed search, but very distributed EMBL has member states across Europe, plus associate member countries – Australia, Argentina
  9. Ontologies generally hierarchical – root node, child nodes, etc. Additional relationships between nodes at any level though, so not strictly a tree structure.
  10. SPARQL is an RDF query language for searching across triple stores. Triple stores are databases where each item represents a subject, a predicate and an object – eg. “the heart is part of the human body”
  11. Annotations vary between ontologies as to which data is a synonym, definintion, and so on. Need Solr config to tell it which annotation applies for the current ontology.
  12. SWAT4LS – Semantic web applications and tools for life sciences EBI workshop – hands on, interactive, understanding different search technologies for biomedical data