The need for a transparent data supply chainPaul Groth
Illustrating data supply chains and motivating the need for a more transparent data supply chain in the context of responsible data science. Presented at the 2018 KNAW-Royal Society bilateral meeting on responsible data science.
Drug Repurposing using Deep Learning on Knowledge GraphsDatabricks
Discovering new drugs is a lengthy and expensive process. This means that finding new uses for existing drugs can help create new treatments in less time and with less time. The difficulty is in finding these potential new uses.
How do we find these undiscovered uses for existing drugs?
We can unify the available structured and unstructured data sets into a knowledge graph. This is done by fusing the structured data sets, and performing named entity extraction on the unstructured data sets. Once this is done, we can use deep learning techniques to predict latent relationships.
In this talk we will cover:
Building the knowledge graph
Predicting latent relationships
Using the latent relationships to repurpose existing drugs
Next-Generation Search Engines for Information RetrievalWaqas Tariq
In the recent years, there have been significant advancements in the areas of scientific data management and retrieval techniques, particularly in terms of standards and protocols for archiving data and metadata. Scientific data is generally rich, not easy to understand, and spread across different places. In order to integrate these pieces together, a data archive and associated metadata should be generated. This data should be stored in a format that can be locatable, retrievable and understandable, more importantly it should be in a form that will continue to be accessible as technology changes, such as XML. New search technologies are being implemented around these protocols, which makes searching easy, fast and yet robust. One such system, Mercury, a metadata harvesting, data discovery, and access system, built for researchers to search to, share and obtain spatiotemporal data used across a range of climate and ecological sciences.
ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...Carole Goble
Keynote given by Carole Goble on 23rd July 2013 at ISMB/ECCB 2013
http://www.iscb.org/ismbeccb2013
How could we evaluate research and researchers? Reproducibility underpins the scientific method: at least in principle if not practice. The willing exchange of results and the transparent conduct of research can only be expected up to a point in a competitive environment. Contributions to science are acknowledged, but not if the credit is for data curation or software. From a bioinformatics view point, how far could our results be reproducible before the pain is just too high? Is open science a dangerous, utopian vision or a legitimate, feasible expectation? How do we move bioinformatics from one where results are post-hoc "made reproducible", to pre-hoc "born reproducible"? And why, in our computational information age, do we communicate results through fragmented, fixed documents rather than cohesive, versioned releases? I will explore these questions drawing on 20 years of experience in both the development of technical infrastructure for Life Science and the social infrastructure in which Life Science operates.
API-Centric Data Integration for Human Genomics Reference Databases: Achieve...Genomika Diagnósticos
API-Centric Data Integration for Human Genomics Reference Databases: Achievements, Lessons Learned and Challenges
X-Meeting 2015
Authors: Jamisson Freitas, Marcel Caraciolo, Victor Diniz, Rodrigo Alexandre and João Bosco Oliveira
The need for a transparent data supply chainPaul Groth
Illustrating data supply chains and motivating the need for a more transparent data supply chain in the context of responsible data science. Presented at the 2018 KNAW-Royal Society bilateral meeting on responsible data science.
Drug Repurposing using Deep Learning on Knowledge GraphsDatabricks
Discovering new drugs is a lengthy and expensive process. This means that finding new uses for existing drugs can help create new treatments in less time and with less time. The difficulty is in finding these potential new uses.
How do we find these undiscovered uses for existing drugs?
We can unify the available structured and unstructured data sets into a knowledge graph. This is done by fusing the structured data sets, and performing named entity extraction on the unstructured data sets. Once this is done, we can use deep learning techniques to predict latent relationships.
In this talk we will cover:
Building the knowledge graph
Predicting latent relationships
Using the latent relationships to repurpose existing drugs
Next-Generation Search Engines for Information RetrievalWaqas Tariq
In the recent years, there have been significant advancements in the areas of scientific data management and retrieval techniques, particularly in terms of standards and protocols for archiving data and metadata. Scientific data is generally rich, not easy to understand, and spread across different places. In order to integrate these pieces together, a data archive and associated metadata should be generated. This data should be stored in a format that can be locatable, retrievable and understandable, more importantly it should be in a form that will continue to be accessible as technology changes, such as XML. New search technologies are being implemented around these protocols, which makes searching easy, fast and yet robust. One such system, Mercury, a metadata harvesting, data discovery, and access system, built for researchers to search to, share and obtain spatiotemporal data used across a range of climate and ecological sciences.
ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...Carole Goble
Keynote given by Carole Goble on 23rd July 2013 at ISMB/ECCB 2013
http://www.iscb.org/ismbeccb2013
How could we evaluate research and researchers? Reproducibility underpins the scientific method: at least in principle if not practice. The willing exchange of results and the transparent conduct of research can only be expected up to a point in a competitive environment. Contributions to science are acknowledged, but not if the credit is for data curation or software. From a bioinformatics view point, how far could our results be reproducible before the pain is just too high? Is open science a dangerous, utopian vision or a legitimate, feasible expectation? How do we move bioinformatics from one where results are post-hoc "made reproducible", to pre-hoc "born reproducible"? And why, in our computational information age, do we communicate results through fragmented, fixed documents rather than cohesive, versioned releases? I will explore these questions drawing on 20 years of experience in both the development of technical infrastructure for Life Science and the social infrastructure in which Life Science operates.
API-Centric Data Integration for Human Genomics Reference Databases: Achieve...Genomika Diagnósticos
API-Centric Data Integration for Human Genomics Reference Databases: Achievements, Lessons Learned and Challenges
X-Meeting 2015
Authors: Jamisson Freitas, Marcel Caraciolo, Victor Diniz, Rodrigo Alexandre and João Bosco Oliveira
These slides were presented at AGU 2018 by Tanu Malik from DePaul University, in a session convened by Dr. Ian Foster, director of the Data Science and Learning division at Argonne National Laboratory.
Sources of Change in Modern Knowledge Organization SystemsPaul Groth
Talk covering how knowledge graphs are making us rethink how change occurs in Knowledge Organization Systems. Based on https://arxiv.org/abs/1611.00217
Keynote for Theory and Practice of Digital Libraries 2017
The theory and practice of digital libraries provides a long history of thought around how to manage knowledge ranging from collection development, to cataloging and resource description. These tools were all designed to make knowledge findable and accessible to people. Even technical progress in information retrieval and question answering are all targeted to helping answer a human’s information need.
However, increasingly demand is for data. Data that is needed not for people’s consumption but to drive machines. As an example of this demand, there has been explosive growth in job openings for Data Engineers – professionals who prepare data for machine consumption. In this talk, I overview the information needs of machine intelligence and ask the question: Are our knowledge management techniques applicable for serving this new consumer?
The Roots: Linked data and the foundations of successful Agriculture DataPaul Groth
Some thoughts on successful data for the agricultural domain. Keynote at Linked Open Data in Agriculture
MACS-G20 Workshop in Berlin, September 27th and 28th, 2017 https://www.ktbl.de/inhalte/themen/ueber-uns/projekte/macs-g20-loda/lod/
Combining Explicit and Latent Web Semantics for Maintaining Knowledge GraphsPaul Groth
A look at how the thinking about Web Data and the sources of semantics can help drive decisions on combining latent and explicit knowledge. Examples from Elsevier and lots of pointers to related work.
DataONE Education Module 01: Why Data Management?DataONE
Lesson 1 in a set of 10 created by DataONE on Best Practices fo Data Management. The full module can be downloaded from the DataONE.org website at: http://www.dataone.org/educaiton-modules. Released under a CC0 license, attribution and citation requested.
Research Data Sharing: A Basic FrameworkPaul Groth
Some thoughts on thinking about data sharing. Prepared for the 2016 LERU Doctoral Summer School - Data Stewardship for Scientific Discovery and Innovation.
http://www.dtls.nl/fair-data/fair-data-training/leru-summer-school/
Data management is a key skill in the age of large, complex data sets. Collaborative research makes the process of managing research data harder. This presentation will cover some key features of the Open Science Framework that facilitate collaborative research.
The literature contains a myriad of recommendations, advice, and strictures about what data providers should do to facilitate data reuse. It can be overwhelming. Based on recent empirical work (analyzing data reuse proxies at scale, understanding data sensemaking and looking at how researchers search for data), I talk about what practices are a good place to start for helping others to reuse your data.
With the explosion of interest in both enhanced knowledge management and open science, the past few years have seen considerable discussion about making scientific data “FAIR” — findable, accessible, interoperable, and reusable. The problem is that most scientific datasets are not FAIR. When left to their own devices, scientists do an absolutely terrible job creating the metadata that describe the experimental datasets that make their way in online repositories. The lack of standardization makes it extremely difficult for other investigators to locate relevant datasets, to re-analyse them, and to integrate those datasets with other data. The Center for Expanded Data Annotation and Retrieval (CEDAR) has the goal of enhancing the authoring of experimental metadata to make online datasets more useful to the scientific community. The CEDAR work bench for metadata management will be presented in this webinar. CEDAR illustrates the importance of semantic technology to driving open science. It also demonstrates a means for simplifying access to scientific data sets and enhancing the reuse of the data to drive new discoveries.
Tools and approaches for data deposition into nanomaterial databasesValery Tkachenko
Sustainable research progress in many scientific disciplines critically depends on the existence of robust specialized databases that integrate and structure all available experimental information in the respective fields. The need for such reference database is especially critical for nanoscience and nanomaterial research given the significant diversity of shapes, sizes, and properties of engineered nanomaterials and the difficulty of synthesizing engineered nanoparticles with controlled properties. The acquisition of data from public sources is inefficient, time consuming and limited in scope. Moreover, it is not clear where the resources come from to support this activity on a perpetual basis. The NIH has recently posted its intention to provide special funds toward data deposition by the experimental investigators through the ‘data sharing plan’ for each proposal. However, this points to a current weakness which is that all laboratories use different data collection approaches each of which requires interpretation by staff hosting the database. It would be far more efficient and useful if a template with key terms that could be modified to add new or important additional data or parameters for each investigator. We will discuss tools and approaches to facilitate collection and direct deposition of experimental data into Nanomaterial Registry (https://www.nanomaterialregistry.org/) - a versatile semantically enriched templates-based platform for registering diverse data pertaining to nanomaterials research.
Non techie journey in social internet age noiselessinnovationframeworks2go.com
a journey of a non techie in the social internet age. His realization of how product building has fundamentally changed with open source software, hardware, crowd sourcing, self learning etc. It pulls it all together and makes the case for a new way of innovations... Noiseless innovation
These slides were presented at AGU 2018 by Tanu Malik from DePaul University, in a session convened by Dr. Ian Foster, director of the Data Science and Learning division at Argonne National Laboratory.
Sources of Change in Modern Knowledge Organization SystemsPaul Groth
Talk covering how knowledge graphs are making us rethink how change occurs in Knowledge Organization Systems. Based on https://arxiv.org/abs/1611.00217
Keynote for Theory and Practice of Digital Libraries 2017
The theory and practice of digital libraries provides a long history of thought around how to manage knowledge ranging from collection development, to cataloging and resource description. These tools were all designed to make knowledge findable and accessible to people. Even technical progress in information retrieval and question answering are all targeted to helping answer a human’s information need.
However, increasingly demand is for data. Data that is needed not for people’s consumption but to drive machines. As an example of this demand, there has been explosive growth in job openings for Data Engineers – professionals who prepare data for machine consumption. In this talk, I overview the information needs of machine intelligence and ask the question: Are our knowledge management techniques applicable for serving this new consumer?
The Roots: Linked data and the foundations of successful Agriculture DataPaul Groth
Some thoughts on successful data for the agricultural domain. Keynote at Linked Open Data in Agriculture
MACS-G20 Workshop in Berlin, September 27th and 28th, 2017 https://www.ktbl.de/inhalte/themen/ueber-uns/projekte/macs-g20-loda/lod/
Combining Explicit and Latent Web Semantics for Maintaining Knowledge GraphsPaul Groth
A look at how the thinking about Web Data and the sources of semantics can help drive decisions on combining latent and explicit knowledge. Examples from Elsevier and lots of pointers to related work.
DataONE Education Module 01: Why Data Management?DataONE
Lesson 1 in a set of 10 created by DataONE on Best Practices fo Data Management. The full module can be downloaded from the DataONE.org website at: http://www.dataone.org/educaiton-modules. Released under a CC0 license, attribution and citation requested.
Research Data Sharing: A Basic FrameworkPaul Groth
Some thoughts on thinking about data sharing. Prepared for the 2016 LERU Doctoral Summer School - Data Stewardship for Scientific Discovery and Innovation.
http://www.dtls.nl/fair-data/fair-data-training/leru-summer-school/
Data management is a key skill in the age of large, complex data sets. Collaborative research makes the process of managing research data harder. This presentation will cover some key features of the Open Science Framework that facilitate collaborative research.
The literature contains a myriad of recommendations, advice, and strictures about what data providers should do to facilitate data reuse. It can be overwhelming. Based on recent empirical work (analyzing data reuse proxies at scale, understanding data sensemaking and looking at how researchers search for data), I talk about what practices are a good place to start for helping others to reuse your data.
With the explosion of interest in both enhanced knowledge management and open science, the past few years have seen considerable discussion about making scientific data “FAIR” — findable, accessible, interoperable, and reusable. The problem is that most scientific datasets are not FAIR. When left to their own devices, scientists do an absolutely terrible job creating the metadata that describe the experimental datasets that make their way in online repositories. The lack of standardization makes it extremely difficult for other investigators to locate relevant datasets, to re-analyse them, and to integrate those datasets with other data. The Center for Expanded Data Annotation and Retrieval (CEDAR) has the goal of enhancing the authoring of experimental metadata to make online datasets more useful to the scientific community. The CEDAR work bench for metadata management will be presented in this webinar. CEDAR illustrates the importance of semantic technology to driving open science. It also demonstrates a means for simplifying access to scientific data sets and enhancing the reuse of the data to drive new discoveries.
Tools and approaches for data deposition into nanomaterial databasesValery Tkachenko
Sustainable research progress in many scientific disciplines critically depends on the existence of robust specialized databases that integrate and structure all available experimental information in the respective fields. The need for such reference database is especially critical for nanoscience and nanomaterial research given the significant diversity of shapes, sizes, and properties of engineered nanomaterials and the difficulty of synthesizing engineered nanoparticles with controlled properties. The acquisition of data from public sources is inefficient, time consuming and limited in scope. Moreover, it is not clear where the resources come from to support this activity on a perpetual basis. The NIH has recently posted its intention to provide special funds toward data deposition by the experimental investigators through the ‘data sharing plan’ for each proposal. However, this points to a current weakness which is that all laboratories use different data collection approaches each of which requires interpretation by staff hosting the database. It would be far more efficient and useful if a template with key terms that could be modified to add new or important additional data or parameters for each investigator. We will discuss tools and approaches to facilitate collection and direct deposition of experimental data into Nanomaterial Registry (https://www.nanomaterialregistry.org/) - a versatile semantically enriched templates-based platform for registering diverse data pertaining to nanomaterials research.
Non techie journey in social internet age noiselessinnovationframeworks2go.com
a journey of a non techie in the social internet age. His realization of how product building has fundamentally changed with open source software, hardware, crowd sourcing, self learning etc. It pulls it all together and makes the case for a new way of innovations... Noiseless innovation
Microsoft Kinect & the Microsoft MIX11 Game PreviewDataLeader.io
Dave Drach, Managing Director for Emerging Businesses at Microsoft gave this presentation at my Microsoft Pre-MIX11 event ROCK!
1. Emerging Business Team Mission: Building Opportunity with VC & Startup Communities
2. Microsoft BizSpark: Global program designed to help accelerate the success of early stage startups
3. Building for the Kinect: the XBOX Dev Kit, integrates XBOX with PC Development environment
4. Kinect Sensor: a hybrid device with input devices, space control is done through a tilt monitor
5. XBOX Studio Overview
6. Human Depth Sensing: Object pattern similarity determines disparity
7. Kinect Depth Sensing: IR pattern similarity determines disparity
8. The Kinect Play Space
9. Player Framing
10. Tilting the Playing Field
11. Provided Data: Depth & Segmentation Map
12. Depth Map Format
13. Skeleton Tracking & Depth
14. Comparing Depth Map to Skeleton
15. Gaussian Filtering
16. Audio Overview
17. Kinect Audio Routing
18. Talking to Your Kinect
19. Biometric Data for Player Recognition
Data Harmonization for a Molecularly Driven Health SystemWarren Kibbe
Maximizing the value of data, computing, data science in an academic medical center, or 'towards a molecularly informed Learning Health System. Given in October at the University of Florida in Gainesville
dkNET Webinar: Creating and Sustaining a FAIR Biomedical Data Ecosystem 10/09...dkNET
Abstract
In this presentation, Susan Gregurick, Ph.D., Associate Director of Data Science and Director, Office of Data Science Strategy at the National Institutes of Health, will share the NIH’s vision for a modernized, integrated FAIR biomedical data ecosystem and the strategic roadmap that NIH is following to achieve this vision. Dr. Gregurick will highlight projects being implemented by team members across the NIH’s 27 institutes and centers and will ways that industry, academia, and other communities can help NIH enable a FAIR data ecosystem. Finally, she will weave in how this strategy is being leveraged to address the COVID-19 pandemic.
Presenter: Susan Gregurick, Ph.D., Associate Director of Data Science and Director, Office of Data Science Strategy at the National Institutes of Health
dkNET Webinar Information: https://dknet.org/about/webinar
Data Harmonization for a Molecularly Driven Health SystemWarren Kibbe
Seminar for Dr. Min Zhang's Purdue Bioinformatics Seminar Series. Touched on learning health systems, the Gen3 Data Commons, the NCI Genomic Data Commons, Data Harmonization, FAIR, and open science.
Building bioinformatics resources for the global communityExternalEvents
http://www.fao.org/about/meetings/wgs-on-food-safety-management/en/
Building bioinformatics resources for the global community. Presentation from the Technical Meeting on the impact of Whole Genome Sequencing (WGS) on food safety management and GMI-9, 23-25 May 2016, Rome, Italy.
Data-knowledge transition zones within the biomedical research ecosystemMaryann Martone
Overview of the Neuroscience Information Framework and how it brings together data, in the form of distributed databases, and knowledge, in the form of ontologies to show the mapping of the dataspace and places where there are mismatches between data and knowledge.
Data Communities - reusable data in and outside your organization.Paul Groth
Description
Data is a critical both to facilitate an organization and as a product. How can you make that data more usable for both internal and external stakeholders? There are a myriad of recommendations, advice, and strictures about what data providers should do to facilitate data (re)use. It can be overwhelming. Based on recent empirical work (analyzing data reuse proxies at scale, understanding data sensemaking and looking at how researchers search for data), I talk about what practices are a good place to start for helping others to reuse your data. I put this in the context of the notion data communities that organizations can use to help foster the use of data both within your organization and externally.
Knowledge Management in the AI Driven Scintific SystemSubhasis Dasgupta
In this dynamic talk, we'll explore the transformative role of AI in scientific knowledge management. We'll delve into how AI revolutionizes data organization, analysis, and hypothesis testing, enhancing efficiency and discovery. Highlighting the seamless integration with existing research processes, we'll address the training and ethical considerations of AI adoption. Through real-world examples, we'll demonstrate AI's impact on scientific breakthroughs, emphasizing the shift towards more collaborative and innovative research landscapes. This presentation aims to inspire the scientific community to embrace AI, leveraging its potential to redefine the boundaries of knowledge and innovation.
In this deck from the 2014 HPC User Forum in Seattle, Jack Collins from the National Cancer Institute presents: Genomes to Structures to Function: The Role of HPC.
Watch the video presentation: http://wp.me/p3RLHQ-d28
Semantic Web & Web 3.0 empowering real world outcomes in biomedical research ...Amit Sheth
Talk presented in Spain (WiMS 2013/UAM-Madrid, UMA-Malaga), June 2013.
Replaces earlier version at: http://www.slideshare.net/apsheth/semantic-technology-empowering-real-world-outcomes-in-biomedical-research-and-clinical-practices
Biomedical and translational research as well as clinical practice are increasingly data driven. Activities routinely involve large number of devices, data and people, resulting in the challenges associated with volume, velocity (change), variety (heterogeneity) and veracity (provenance, quality). Equally important is to realize the challenge of serving the needs of broader ecosystems of people and organizations, extending traditional stakeholders like drug makers, clinicians and policy makers, to increasingly technology savvy and information empowered patients. We believe that semantics is becoming centerpiece of informatics solutions that convert data into meaningful, contextually relevant information and insights that lead to optimal decisions for translational research and 360 degree health, fitness and well-being.
In this talk, I will provide a series of snapshots of efforts in which semantic approach and technology is the key enabler. I will emphasize real-world and in-use projects, technologies and systems, involving significant collaborations between my team and biomedical researchers or practicing clinicians. Examples include:
• Active Semantic Electronic Medical Record
• Semantics and Services enabled Problem Solving Environment for T.cruzi (SPSE)
• Data Mining of Cardiology data
• Semantic Search, Browsing and Literature Based Discovery
• PREscription Drug abuse Online Surveillance and Epidemiology (PREDOSE)
• kHealth: development of a knowledge-enhanced sensing and mobile computing applications (using low cost sensors and smartphone), along with ability to convert low level observations into clinically relevant abstractions
Further details are at http://knoesis.org/amit/hcls
The fourth paradigm: data intensive scientific discovery - Jisc Digifest 2016Jisc
There is broad recognition within the scientific community that the emerging data deluge will fundamentally alter disciplines in areas throughout academic research. A wide variety of researchers - from scientists and engineers to social scientists and humanities researchers - will require tools, technologies, and platforms that seamlessly integrate into standard scientific methodologies and processes.
'The fourth paradigm' refers to the data management techniques and the computational systems needed to manipulate, visualize, and manage large amounts of research data. This talk will illustrate the challenges researchers will face, the opportunities these changes will afford, and the resulting implications for data-intensive researchers.
In addition, the talk will review the global movement towards open access, research repositories and open science and the importance of curation of digital data. The talk concludes with some comments on the research requirements for campus e-infrastructure and the end-to-end performance of the network.
KnetMiner, with a silent "K" and standing for Knowledge Network Miner, is a suite of open-source software tools developed at Rothamsted Research for integrating and visualising large biological datasets in order to accelerate gene discovery. The software mines the myriad databases that describe an organism’s biology to present links between relevant pieces of information, such as genes, biological pathways, phenotypes or publications. The aim is to provide leads for scientists who are investigating the molecular basis for a particular trait or ways of improving the organism’s performance in some way
Building a Knowledge Graph with Spark and NLP: How We Recommend Novel Drugs t...Databricks
It is widely known that the discovery, development, and commercialization of new classes of drugs can take 10-15 years and greater than $5 billion in R&D investment only to see less than 5% of the drugs make it to market.
AstraZeneca is a global, innovation-driven biopharmaceutical business that focuses on the discovery, development, and commercialization of prescription medicines for some of the world’s most serious diseases. Our scientists have been able to improve our success rate over the past 5 years by moving to a data-driven approach (the “5R”) to help develop better drugs faster, choose the right treatment for a patient and run safer clinical trials.
However, our scientists are still unable to make these decisions with all of the available scientific information at their fingertips. Data is sparse across our company as well as external public databases, every new technology requires a different data processing pipeline and new data comes at an increasing pace. It is often repeated that a new scientific paper appears every 30 seconds, which makes it impossible for any individual expert to keep up-to-date with the pace of scientific discovery.
To help our scientists integrate all of this information and make targeted decisions, we have used Spark on Azure Databricks to build a knowledge graph of biological insights and facts. The graph powers a recommendation system which enables any AZ scientist to generate novel target hypotheses, for any disease, leveraging all of our data.
In this talk, I will describe the applications of our knowledge graph and focus on the Spark pipelines we built to quickly assemble and create projections of the graph from 100s of sources. I will also describe the NLP pipelines we have built – leveraging spacy, bioBERT or snorkel – to reliably extract meaningful relations between entities and add them to our knowledge graph.
Being FAIR: Enabling Reproducible Data ScienceCarole Goble
Talk presented at Early Detection of Cancer Conference, OHSU, Portland, Oregon USA, 2-4 Oct 2018, http://earlydetectionresearch.com/ in the Data Science session
Similar to 2015 GU-ICBI Poster (third printing) (20)
1. Background
The past decade has seen a significant increase in high-
throughput experimental studies that catalog variant
datasets using massively parallel sequencing experiments.
New insights of biological significance can be gained by this
information with multiple genomic locations based
annotations. However, efforts to obtain this information by
integrating and mining variant data have had limited success
so far and there has yet to be a method developed that can
be scalable, practical and applied to millions of variants and
their related annotations. We explored the use of graph data
structures as a proof of concept for scalable interpretation of
the impact of variant related data.
Methods and Materials
Results
Leveraging Graph Data Structures for Variant Data and Related Annotations
Chris Zawora1, Jesse Milzman1, Yatpang Cheung1, Akshay Bhushan1, Michael S. Atkins2, Hue Vuong3, F. Pascal Girard2, Uma Mudunuri3
1Georgetown University, Washington, DC, 2FedCentric Technologies, LLC, Fairfax, VA, 3Frederick National Laboratory for Cancer Research, Frederick, MD
Conclusion
References
The 1000 Genomes Project Consortium. (2010). A map of human
genome variation from population-scale sequencing. Nature, 467,
1061–1073.
Gregg, B. (2014). Systems performance: enterprise and the cloud.
Pearson Hall: Upper Saddle River, NJ.
Acknowledgments
FedCentric acknowledges Frank D’Ippolito, Shafigh
Mehraeen, Margreth Mpossi, Supriya Nittoor, and Tianwen
Chu for their invaluable assistance with this project.
This project has been funded in whole or in part with federal funds from the National
Cancer Institute, National Institutes of Health, under contract HHSN261200800001E. The
content of this publication does not necessarily reflect the views or policies of the
Department of Health and Human Services, nor does mention of trade names, commercial
products, or organizations imply endorsement by the U.S. Government.
Contract HHSN261200800001E - Funded by the National Cancer Institute
DEPARTMENT OF HEALTH AND HUMAN SERVICES • National Institutes of Health • National Cancer Institute
Frederick National Laboratory is a Federally Funded Research and Development Center
operated by Leidos Biomedical Research, Inc., for the National Cancer Institute
Introduction
Traditional approaches of data mining and integration in the
research field have relied on relational databases or
programming for deriving dynamic insights from research
related data. However, as more next generation sequencing
(NGS) becomes available, these approaches limit the
exploration of certain hypothesis. One such limitation is the
mining of variant data from publicly available databases
such as the 1000 genomes project and TCGA.
Although there are applications available for quickly finding
the public data with a certain set of variants or for finding
minor allele frequencies, there is no such application that
can be applied generically across all the projects allowing
researchers to globally mine and find patterns that would be
applicable to their specific research interests.
In this pilot project, we have investigated whether graph
database structures are applicable for mining variants from
individuals and populations in a scalable manner and
understanding their impact by integrating with known
annotations.
Phase I: As an initial evaluation of the graph structures we
ran several simple queries, also feasible through a
relational architecture, and measured performance
speeds.
Simple Query Examples
• Get all information for a single variant
• Find annotations within a range of genomic locations
• Find variants associated with specific clinical
phenotypes
Performance speeds
• Query times in milliseconds
• Better or equal to relational database query times
Queries
• Developed a new SQL-like query language called
SparkQL
• Eases writing queries for non-programmatic users
Ingestion Times
• Slower than expected
• Sparksee is a multi-thread single write database
• Writes one node/edge at a time
• Each write involves creating connections with existing
nodes
• Slows down as the graph size increases
Solution: Implement multi-threaded insertions in
combination with internal data structures to efficiently
find nodes and create edges
High Degree Vertices
• Nodes with millions of edges
• Stored in a non-distributed list like format
• Searches for a specific edge might be slow
Example: nodes representing individuals with millions
of variants
Solution: Explore other graph clustering approaches
that can essentially condense the information
presented
Our results indicate that a graph database, run on an in-
memory machine, can be a very powerful and useful tool
for cancer research. Performance using the graph database
for finding details on specific nodes or a list of nodes is
better or equal to a well-architected relational database. We
also see promising initial results for identifying correlations
between genetic changes and specific phenotype
conditions.
We conclude that an in-memory graph database
would allow researchers to run known queries while also
providing the opportunity to develop algorithms to explore
complex correlations. Graph models are useful for mining
complex data sets and could prove essential in the
development and implementation of tools aiding precision
medicine.
Results (cont.)
Hardware
• FedCentric Labs’ SGI UV300 system: x86/Linux system,
scales up to 1,152 cores & 64TB of memory
• Data in memory, very low latency, high performance
(Fig. 2)
Fig. 1: Graphs handle data complexity intuitively and interactively
Fig. 4: Results of spectral clustering of 1000 Genomes data
Methods and Materials (cont.)
Data
• SNPs from 1000 genomes project
• Phenotype conditions from ClinVar
• Gene mappings & mRNA transcripts from Entrez Gene
• Amino acid changes from UniProt
The Graph
• Variants and annotations mapped to reference genomic
locations (Fig. 3)
• Includes all chromosomes and genomic locations
• 180 million nodes and 12 billion edges.
Fig. 2: Latency matters
Fig. 3: The Graph Model
Graph Architecture
• Sparsity Technologies’ Sparsity Graph database
• API supports C, C++, C#, Python, and Java
• Implements graph and its attributes as maps and sparse
bitmap structures
• Allows it to scale with very limited memory requirements.
Phase II: We explored complex patterns and clusters inside
the graph and spectral clustering queries that were not
feasible through the relational architecture.
Complex Query Examples
• Compare variant profiles and find individuals that are
closely related
• Compare annotation profiles to find clusters of
populations
Phase II Results
• Eight populations with 25 individuals from each
population
• Strong eigenvalue support (near zero) for 3 main clusters
• Cluster pattern supported by population genetics (Fig. 4)
Performance speeds
• Spectral clustering took ca. 2 minutes