Bio solr building a better search for bioinformatics
•Download as PPT, PDF•
3 likes•2,416 views
Presentation given to the London Lucene/Solr Meetup on BioSolr, a project to improve search for bioinformaticians using Apache Solr
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (6)
Similar to Bio solr building a better search for bioinformatics
Similar to Bio solr building a better search for bioinformatics (20)
More from Charlie Hull
More from Charlie Hull (11)
Recently uploaded
Recently uploaded (20)
Bio solr building a better search for bioinformatics
- 1. Tom Winch & Matt Pearce 21st April 2015 charlie@flax.co.uk www.flax.co.uk/blog +44 (0) 8700 118334 Twitter: @FlaxSearch BioSolr building a better search for bioinformatics
- 2. The European Bioinformatics Institute Part of the European Molecular Biology Laboratory Based on the Wellcome Genome Campus in Hinxton, Cambridge Maintains the world’s most comprehensive range of freely available and up-to-date molecular databases, serving millions of researchers – indexing over 1 billion items BioSolr project involves two teams from EMBL-EBI: Protein Data Bank in Europe (PDBe) Samples, Phenotypes and Ontologies (SPOt)
- 3. The genesis of BioSolr Grant Ingersoll visits the Wellcome Campus in July '13 Around 90 people attend Show of hands indicates 75% using Lucene/Solr Sameer Velankar of EMBL-EBI identifies grant funding Flax and EMBL-EBI apply successfully to the BBSRC
- 4. BioSolr One year BBSRC funded project from September 2014 “to significantly advance the state of the art with regard to indexing and querying biomedical data with freely available open source software” Outputs: – Workshops – Papers & presentations – Software (Open source of course!) – Documentation Inputs: from the PDBe & SPOt teams
- 5. BioSolr Tom Winch – Working on site with Sameer Velankar & the PDBe team – Facet.contains & Xjoin Matt Pearce – Working on site with Tony Burdett & the SPOt team – Indexing ontologies
- 6. BioSolr & PDBe - Introduction Protein Data Bank (PDBe) facet.contains – autosuggest https://issues.apache.org/jira/browse/SOLR-1387 In Solr 5.1 DNA sequence similarity
- 7. BioSolr & PDBe – Xjoin concepts The problem - sequences come from a live source Joining with data from an external source Custom SOLR code
- 8. BioSolr & PDBe – Solr classes XJoinResultsFactory, XJoinResults XJoinSearchComponent XJoinQParserPlugin XJoinValueSourceParser
- 9. BioSolr & PDBe – What next? SOLR contrib – SOLR-7341 https://issues.apache.org/jira/browse/SOLR-7341 Joining from multiple external sources Federated search
- 10. Washington, N. & Lewis, S. (2008) Ontologies: Scientific Data Sharing Made Easy. Nature Education 1(3):5 BioSolr & SPOt – Indexing Ontologies
- 11. Indexing Ontologies - the problem You have a collection of documents annotated with ontology references. You want to search both the documents and the associated ontology data. This may include associated nodes – “has location”, “is part of”, etc. Faceting by ontology reference would be nice!
- 12. Approach 1 – Keep the data separate documents Documents Indexer Documents Indexer ontology Ontology Indexer
- 13. Approach 1 - steps Index the documents, with the node annotations, but no further detail. Index the ontology in its own core. Search the documents, then cross-match against the ontology. BUT - Requires multiple calls, doesn't allow searching both cores at the same time.
- 14. Approach 2 • Add some ontology data to your documents. Documents Indexer Ontology documents
- 15. Approach 2 – step 1 Index node references, plus their labels and synonyms. Easier to include the ontology references in your search. Can boost fields over others.
- 16. Approach 2 – step 2 Expand the ontology data being stored. Include single (or multi)-level parent and child nodes, with labels. Use dynamic fields to store additional relationships. Dynamic fields allow searches across specific relation types. BUT Requires some additional Solr look-ups to be fully dynamic.
- 17. Approach 2 – search screen
- 18. Approach 2 – search results
- 19. Approach 3 Search the ontology, and cross-match with documents. Allow SPARQL queries over the ontology index. SPARQL is a semantic query language
- 20. Approach 3
- 21. Adding Apache Jena To allow SPARQL queries, we use Apache Jena to provide TDB-querying. Jena uses Solr to search label fields. Uses its own Triple Store for other fields. Need to include reference URI in returned fields.
- 22. Integrating Jena results Returned Jena data needs to be cross-matched against document index. Use a filter query to choose the matching documents.
- 24. Summary so far We can search documents and ontology data with a single call to Solr. We can dynamically search over additional related ontology nodes. We can use SPARQL to search. Can facet on individual ontology annotations...but we still can't present the facets in a tree. https://github.com/flaxsearch/BioSolr/tree/master/spot
- 25. The ultimate goal A generic ontology indexer using Solr. Multiple ontologies stored in the same index. Unique integer keys for each node, allowing cross- matching from document indexes. Optional customisation, allowing for additional lookups or data manipulation.
- 26. BioSolr conclusions Final workshop at EMBL-EBI in September https://github.com/flaxsearch/BioSolr Investigating funding to continue the project – We have some ideas around federated Solr search...
- 27. Thankyou! Any questions? charlie@flax.co.uk www.flax.co.uk/blog +44 (0) 8700 118334 Twitter: @FlaxSearch