Bio4j: A pioneer graph based database for the integration of biological Big Data


Published on

1. Bio4j
2. What’s Bio4j?: Data included
3. What’s Bio4j?: A completely new and powerful framework for protein
4. What’s Bio4j?: Neo4j --> very scalable
5. What's Bio4j?: Everything in Bio4j is open source released under AGPLv3
6. Bioinformatics DBs and Graphs: Highly interconnected overlapping knowledge spread throughout different databases
7. Bioinformatics DBs and Graphs: Data is in most cases modeled in relational databases, (sometimes even just as plain CSV files)
8. Bioinformatics DBs and Graphs: Problems of a relational paradigm
9. Bioinformatics DBs and Graphs: Life + Biology like a graph
10. Bioinformatics DBs and Graphs: NoSQL
11. Bioinformatics DBs and Graphs: NoSQLdata models
12. Bioinformatics DBs and Graphs: The Graph DB model: representation
13. Bioinformatics DBs and Graphs: Neo4j
14. Initial motivation: Why starting all this?
15. Initial motivation: Processes had to be automated for BG7 (

Published in: Education, Technology
1 Comment
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Bio4j: A pioneer graph based database for the integration of biological Big Data

  1. 1. Bio4j: A pioneer graph based database for the integration of biological Big
  2. 2. What’s Bio4j? Bio4j is a bioinformatics graph based DB including most data available in : Uniprot (SwissProt + Trembl) Gene Ontology (GO) UniRef (50,90,100) NCBI Taxonomy RefSeq Enzyme
  3. 3. What’s Bio4j? It provides a completely new and powerful framework for protein related information querying and management. Since it relies on a high-performance graph engine, data is stored in a way that semantically represents its own
  4. 4. What’s Bio4j? Bio4j uses Neo4j technology, a "high-performance graph engine with all the features of a mature and robust database". Thanks to both being based on Neo4j DB and the API provided, Bio4j is also very scalable, allowing anyone to easily incorporate his own data making the best out of
  5. 5. What’s Bio4j? Everything in Bio4j is open source ! released under
  6. 6. Bioinformatics Highly interconnected overlapping knowledgeDBs and Graphs spread throughout different DBsInitial motivationBio4j structureSome samplesWhy Bio4j?Bio4j and theCloudUpcoming
  7. 7. Bioinformatics However all this data is in most cases modeled in relational databases.DBs and Graphs Sometimes even just as plain CSV filesInitial motivation As the amount and diversity of data grows, domain models become crazily complicated!Bio4j structureSome samplesWhy Bio4j?Bio4j and theCloudUpcoming
  8. 8. Bioinformatics With a relational paradigm, the double implicationDBs and Graphs Entity  TableInitial motivation does not go both ways.Bio4j structure You get „auxiliary‟ tables that have no relationship with the small piece of reality you are modeling.Some samples You need ‘artificial’ IDs only for connecting entities, (and these are mixedWhy Bio4j? with IDs that somehow live in reality)Bio4j and the Entity-relationship models are cool but in the end you always have toCloud deal with ‘raw’ tables plus SQL. Integrating/incorporating new knowledge into already existingUpcoming features databases is hard and sometimes even not possible without changing the domain
  9. 9. Bioinformatics Life in general and biology in particular are probably not 100% like a graph…DBs and GraphsInitial motivationBio4j structureSome samplesWhy Bio4j?Bio4j and theCloudUpcoming features but one thing’s sure, they are not a set of tables!
  10. 10. BioinformaticsDBs and Graphs NoSQL (not only SQL)Initial motivation NoSQ… what !??Bio4j structureSome samples Let’s see what Wikipedia says…Why Bio4j? “NoSQL is a broad class of database management systems that differ from the classic model of the relational databaseBio4j and theCloud management system (RDBMS) in some significant ways. These data stores may not require fixed table schemas,Upcoming features usually avoid join operations and typically scale horizontally.”
  11. 11. Bioinformatics NoSQL data modelsDBs and GraphsInitial motivationBio4j structureSome samplesWhy Bio4j?Bio4j and theCloudUpcoming
  12. 12. BioinformaticsDBs and GraphsInitial motivation Cassandra is a highly scalable, eventually consistent,Bio4j structure distributed, structured key-value store. Cassandra brings together the distributed systems technologies from Dynamo and the data model from Googles BigTable.Some samplesWhy Bio4j?Bio4j and theCloud MongoDB (from "humongous") is an open source document- oriented NoSQL database system written in the C++ programmingUpcoming features
  13. 13. BioinformaticsDBs and GraphsInitial motivation Neo4j is a high-performance, NOSQL graph database with allBio4j structure the features of a mature and robust database.Some samples The programmer works with an object-oriented, flexible network structure rather than with strict and static tablesWhy Bio4j?Bio4j and the All the benefits of a fully transactional, enterprise-strengthCloud database.Upcoming features For many applications, Neo4j offers performance improvements on the order of 1000x or more compared to relational
  14. 14. Bioinformatics DBsand Graphs Ok, but why starting all this? Were you so bored…?!Initialmotivation It all started somehow around our need for massive access to protein GO (Gene Ontology) annotations.Bio4j structure At that point I had to develop my own MySQL DB based on the official GO SQL database, and problems started from the beginning:Some samples I got crazy ‘deciphering’ how to extract Uniprot protein annotationsWhy Bio4j? from GO official tables schemaBio4j and the Uniprot and GO official protein annotations were not always consistentCloud Populating my own DB took really long due to all the joins and subqueries needed in order to get and store the protein annotations.Upcoming features Soon enough we also had the need of having massive access to basic protein
  15. 15. Bioinformatics DBs These processes had to be automated for our (specificallyand Graphs designed for NGS data) bacterial genome annotation systemInitial BG7motivation Uniprot web services available were too limited:Bio4j structure - SlowSome samples - Number of queries limitationWhy Bio4j? - Too little information availableBio4j and theCloud So I downloaded the whole Uniprot DB in XML formatUpcoming features (Swiss-Prot + Trembl) and started to have some fun with it !
  16. 16. Bioinformatics DBs We got used to having massive direct access to all this proteinand Graphs related information…Initialmotivation So why not adding other resources we needed quite often in most projects and which now were becoming a sort of bottleneck compared to all those already included in Bio4j ?Bio4j structure Then came:Some samples - Isoform sequencesWhy Bio4j? - Protein interactions and features - Uniref 50, 90, and 100Bio4j and theCloud - RefSeq - NCBI TaxonomyUpcoming features - Enzyme Expasy
  17. 17. Bioinformatics DBs Let’s dig a bit about Bio4j structure:and GraphsInitial motivation Data sources and their relationships:Bio4j structureSome samplesWhy Bio4j?Bio4j and theCloudUpcoming
  18. 18. Bioinformatics DBsand Graphs The Graph DB model: representationInitial motivation Core abstractions:Bio4j structure Nodes Relationships between nodesSome samples Properties on bothWhy Bio4j?Bio4j and theCloudUpcoming
  19. 19. Bioinformatics DBs Let’s dig a bit about Bio4j structure:and GraphsInitial motivation How are things modeled?Bio4j structure Couldn’t be simpler!Some samplesWhy Bio4j? Entities Associations / RelationshipsBio4j and theCloudUpcoming features Nodes
  20. 20. Bioinformatics DBs Some examples of nodes would be:and GraphsInitial motivation GO term ProteinBio4j structure Genome ElementSome samplesWhy Bio4j? and relationships:Bio4j and theCloud Protein PROTEIN_GO_ANNOTATIONUpcoming features GO
  21. 21. Bioinformatics DBs We have developed a tool aimed to be used both as a reference manual andand Graphs initial contact for Bio4j domain model: Bio4jExplorer Bio4jExplorer allows you to:Initial motivation • Navigate through all nodes and relationshipsBio4j structure • Access the javadocs of any node or relationshipSome samples • Graphically explore the neighborhood of a node/relationshipWhy Bio4j? • Look up for the indexes that may serve as an entry point for a nodeBio4j and theCloud • Check incoming/outgoing relationships of a specific nodeUpcoming features • Check start/end nodes of a specific
  22. 22. Bioinformatics DBs Entry points and indexingand Graphs There are two kinds of entry points for the graph:Initial motivationBio4j structure Auxiliary relationships going from the reference node, e.g. - CELLULAR_COMPONENT: leads to the root of GO cellular componentSome samples sub-ontology - MAIN_DATASET: leads to both main datasets: Swiss-Prot and TremblWhy Bio4j? Node indexingBio4j and theCloud There are two types of node indexes: - Exact: Only exact values are considered hitsUpcoming features - Fulltext: Regular expressions can be
  23. 23. Bioinformatics DBs Retrieving protein info (Bio4jModel Java API)and Graphs //--creating manager and node retriever---- Bio4jManager manager = new Bio4jManager(“/mybio4jdb”);Initial motivation NodeRetriever nR= new NodeRetriever(manager); ProteinNode protein = nR.getProteinNodeByAccession(“P12345”);Bio4j structure Getting more related info...Some samples List<InterproNode> interpros = protein.getInterpro(); OrganismNode organism = protein.getOrganism(); List<GoTermNode> goAnnotations = protein.getGOAnnotations();Why Bio4j? List<ArticleNode> articles = protein.getArticleCitations();Bio4j and the for (ArticleNode article : articles) {Cloud System.out.println(article.getPubmedId()); }Upcoming features //And don’t forget to close the Bio4jManager manager.shutDown();
  24. 24. Bioinformatics DBs Proteins with Interpro motif ‘IPR000847’ (Bio4jModel Java API)and Graphs //--creating manager and node retriever----Initial motivation Bio4jManager manager = new Bio4jManager(“/mybio4jdb”); NodeRetriever nR= new NodeRetriever(manager);Bio4j structure InterproNode interpro = nR.getInterproById(“IPR000847”); ProteinInterproRel rel = ProteinInterproRel(null);Some samples Iterator<Relationship> iterator = interpro.getNode().getRelationships(rel, Direction.INCOMING);Why Bio4j? while(relIterator.hasNext()){ ProteinNode p = new ProteinNode(; System.out.println(p.getAccession());Bio4j and the }Cloud //And don’t forget to close the Bio4jManager manager.shutDown();Upcoming
  25. 25. Bioinformatics DBs Querying Bio4j with Cypherand GraphsInitial motivation Getting a keyword by its IDBio4j structure START k=node:keyword_id_index(keyword_id_index = "KW-0181") return, k.idSome samples Finding circuits/simple cycles of length 3 where at least one protein is from Swiss-Prot dataset:Why Bio4j? START d=node:dataset_name_index(dataset_name_index = "Swiss-Prot") MATCH d <-[r:PROTEIN_DATASET]- p,Bio4j and the circuit = (p) -[:PROTEIN_PROTEIN_INTERACTION]-> (p2) -Cloud [:PROTEIN_PROTEIN_INTERACTION]-> (p3) - [:PROTEIN_PROTEIN_INTERACTION]-> (p) return p.accession, p2.accession, p3.accessionUpcoming features Check this blog post for more info and our Bio4j Cypher
  26. 26. Bioinformatics DBsand GraphsInitial motivation Get protein by its accession number and return its full nameBio4j structure gremlin> g.idx(protein_accession_index)[[protein_accession_index:P12345]].full_nameSome samples ==> Aspartate aminotransferase, mitochondrial Get proteins (accessions) associated to an interpro motif (limited to 4 results)Why Bio4j? gremlin> g.idx(interpro_id_index)[[interpro_id_index:IPR023306]].inE(PROTEIN_INTERPRO).outVBio4j and the .accession[0..3]Cloud ==> E2GK26 ==> G3PMS4 ==> G3Q865Upcoming features ==> G3PIL8 Check our Bio4j Gremlin
  27. 27. Bioinformatics DBsand Graphs REST ServerInitial motivation You can also query/navigate through Bio4j with the REST API !Bio4j structure The default representation is json, both for responses and or data sent with POST/PUT requestsSome samples Get protein by its accession number: (Q9UR66)Why Bio4j? http://server_url:7474/db/data/index/node/protein_acc ession_index/protein_accession_index/Q9UR66Bio4j and theCloud Get outgoing relationships for protein Q9UR66Upcoming features http://server_url:7474/db/data/node/Q9UR66_node_id/re lationships/
  28. 28. Bioinformatics DBs Visualizations (1)  REST Server Data Browserand Graphs Navigate through Bio4j data in real time !Initial motivationBio4j structureSome samplesWhy Bio4j?Bio4j and theCloudUpcoming
  29. 29. Bioinformatics DBs Visualizations (2)  Bio4j + Gephiand Graphs Get really cool graph visualizations using Bio4j and Gephi visualization andInitial motivation exploration platformBio4j structureSome samplesWhy Bio4j?Bio4j and theCloudUpcoming
  30. 30. Bioinformatics DBs Visualizations (3)  Bio4j GO Toolsand GraphsInitial motivationBio4j structureSome samplesWhy Bio4j?Bio4j and theCloudUpcoming
  31. 31. Bioinformatics DBs Why would I use Bio4j ?and Graphs Massive access to protein/genome/taxonomy… relatedInitial motivation informationBio4j structure Integration of your own DBs/resources around common informationSome samples Development of services tailored to your needs built aroundWhy Bio4j? Bio4jBio4j and the Networks analysisCloud VisualizationsUpcoming features Besides many others I cannot think of myself… If you have something in mind for which Bio4j might be useful, please let us know so we can all see how it could help you meet your needs! ;)
  32. 32. Bioinformatics DBs Bio4j + Cloud (1)and Graphs We use AWS (Amazon Web Services) everywhere we can aroundInitial motivation Bio4j, giving us the following benefits:Bio4j structure Interoperability and data distributionSome samples Releases are available as public EBS Snapshots, giving AWS users the opportunity of creating and attaching to their instances Bio4j DB 100% ready volumes in just a few seconds.Why Bio4j?Bio4j and the CloudFormation templates:Cloud - Basic Bio4j DB InstanceUpcoming features - Bio4j REST Server
  33. 33. Bioinformatics DBs Bio4j + Cloud (2)and GraphsInitial motivation Backup and Storage using S3 (Simple Storage Service) We use S3 both for backup (indirectly through the EBS snapshots) andBio4j structure storage (directly storing RefSeq sequences as independent S3 files) What kind of benefits do we get from this?Some samples • Easy to useWhy Bio4j? • Flexible • Cost-EffectiveBio4j and theCloud • Reliable • Scalable and high-performanceUpcoming features •
  34. 34. Bioinformatics DBs Bio4j + Cloud (3)and GraphsInitial motivation Web servers and service providers in the cloud Deploying your own web server in AWS using Bio4j as back-end is reallyBio4j structure simple. A good example of this would be Bio4jTestServer, a continuouslySome samples developed server showcasing Web Services based on Bio4j.Why Bio4j?Bio4j and theCloudUpcoming
  35. 35. Bioinformatics DBsand Graphs Upcoming features - Relationship indexing for relationships going and coming from supernodesInitial motivation No one’s perfect, and Bio4j is not the exception. Relationship fetching can become a bottleneck whenever you have to dealBio4j structure with supernodes (unless you index these relationships). Fortunately this is something that Neo4j is going to fix in the next version(s).Some samples - More resources available (Reactome…)Why Bio4j? - Improvements in the importing process - A more complete version of Bio4jModelBio4j and theCloud Allowing users to perform almost all sorts of queries without having to worry about Neo4j core API.Upcoming - New tools, services and visualizations built around
  36. 36. Bioinformatics DBs Communityand Graphs Bio4j has a fast growing internet presence:Initial motivationBio4j structure - Twitter: check @bio4j for updates - Blog: go to http://blog.bio4j.comSome samples - Mail-list: ask any question you may have in our list.Why Bio4j? - LinkedIn: check the Bio4j groupBio4j and theCloud - Github issues: don’t be shy! open a new issue if you think something’s going wrong.Upcoming
  37. 37. Bioinformatics DBs and... Who’s behind all this?and Graphs Bio4j is being developed by Oh no sequences! Team andInitial motivation Era7 Bioinformatics members:Bio4j structure - Pablo Pareja Tobes: Main developer (that’s me!)Some samples - Eduardo Pareja Tobes: Technology and architecture main advisorWhy Bio4j? - Raquel Tobes: Bioinformatics main advisorBio4j and theCloud - Marina Manrique: Bioinformatics supportUpcoming features - Eduardo Pareja: Scientific
  38. 38. Bioinformatics DBsand GraphsInitial motivationBio4j structure That’s it !Some samplesWhy Bio4j? Thanks for your time ;)Bio4j and theCloudUpcoming