SlideShare a Scribd company logo

Google BigQuery for analysis of scientific datasets: Interactive exploration and analysis of the data using KNIME Analytics Platform

•
•
Greg Landrum
Greg Landrum

This document discusses using Google BigQuery for analyzing scientific datasets interactively with KNIME Analytics Platform. It acknowledges contributions from others and notes that BigQuery allows querying of giant tables with SQL. The author is enthusiastic about exploring data when questions are still being formulated rather than just searching. The workflow demonstrated will explore scientific data in BigQuery using KNIME, starting with initial database queries to pick a disease and examine compound classes.

1 of 16
Download to read offline
© 2019 KNIME AG. All Rights Reserved. Google BigQuery for analysis of scientific datasets: Interactive exploration and analysis of the data using KNIME Analytics Platform Greg Landrum Martyna Pawletta Jeanette Prinz greg.landrum@knime.com @dr_greg_landrum
© 2019 KNIME AG. All Rights Reserved. 2 Acknowledgements • Steve Boyer (Collabra) • Lutz Weber (OntoChem) • Ian Wetherbee (Google)
© 2019 KNIME AG. All Rights Reserved. 3 Google BigQuery? • A giant collection of tables that I can query with SQL • If the tables share common keys, I can do interesting things Might be an oversimplification. ☺
© 2019 KNIME AG. All Rights Reserved. 4 An aside: searching vs exploring a.k.a. why I’m enthusiastic about this project
© 2019 KNIME AG. All Rights Reserved. 5 An aside: searching vs exploring
© 2019 KNIME AG. All Rights Reserved. 6 An aside: searching vs exploring
© 2019 KNIME AG. All Rights Reserved. 7 An aside: searching vs exploring • There are definitely arguments for specialized interfaces that are tailored to make answering a particular question super efficient and easy • But! There are times when I’m still trying to figure out exactly what the question is • For this it’s nice to have a giant pile of data and a general purpose tool for exploring it
© 2019 KNIME AG. All Rights Reserved. 8 What we’re going to do here • Do some exploration of the scientific data that’s now in BigQuery… • … with KNIME
© 2019 KNIME AG. All Rights Reserved. 9 Workflow part 1
© 2019 KNIME AG. All Rights Reserved. 10 Workflow part 2
© 2019 KNIME AG. All Rights Reserved. 11 The first database queries
© 2019 KNIME AG. All Rights Reserved. 12 Picking the disease/condition
© 2019 KNIME AG. All Rights Reserved. 13 Results
© 2019 KNIME AG. All Rights Reserved. 14 Compound classes
© 2019 KNIME AG. All Rights Reserved. 15
© 2019 KNIME AG. All Rights Reserved. 16

Recommended

Building useful models for imbalanced datasets (without resampling)
Building useful models for imbalanced datasets (without resampling)Building useful models for imbalanced datasets (without resampling)
Building useful models for imbalanced datasets (without resampling)
Greg Landrum
 
Building useful models for imbalanced datasets (without resampling)
Building useful models for imbalanced datasets (without resampling)Building useful models for imbalanced datasets (without resampling)
Building useful models for imbalanced datasets (without resampling)
Greg Landrum
 
Moving from Artisanal to Industrial Machine Learning
Moving from Artisanal to Industrial Machine LearningMoving from Artisanal to Industrial Machine Learning
Moving from Artisanal to Industrial Machine Learning
Greg Landrum
 
Hadoop UK Strata Panel Discussion
Hadoop UK Strata Panel DiscussionHadoop UK Strata Panel Discussion
Hadoop UK Strata Panel Discussion
huguk
 
How Data Saves Time
How Data Saves TimeHow Data Saves Time
How Data Saves Time
NetApp
 
Introduction aux algorithmes génétiques
Introduction aux algorithmes génétiquesIntroduction aux algorithmes génétiques
Introduction aux algorithmes génétiques
JUG Lausanne
 
5 Crucial Considerations for Big data adoption
5 Crucial Considerations for Big data adoption5 Crucial Considerations for Big data adoption
5 Crucial Considerations for Big data adoption
Qubole
 
Scaling Deep Learning Algorithms on Extreme Scale Architectures
Scaling Deep Learning Algorithms on Extreme Scale ArchitecturesScaling Deep Learning Algorithms on Extreme Scale Architectures
Scaling Deep Learning Algorithms on Extreme Scale Architectures
inside-BigData.com
 

Recommended

Building useful models for imbalanced datasets (without resampling)
Building useful models for imbalanced datasets (without resampling)Building useful models for imbalanced datasets (without resampling)
Building useful models for imbalanced datasets (without resampling)
Greg Landrum
 
Building useful models for imbalanced datasets (without resampling)
Building useful models for imbalanced datasets (without resampling)Building useful models for imbalanced datasets (without resampling)
Building useful models for imbalanced datasets (without resampling)
Greg Landrum
 
Moving from Artisanal to Industrial Machine Learning
Moving from Artisanal to Industrial Machine LearningMoving from Artisanal to Industrial Machine Learning
Moving from Artisanal to Industrial Machine Learning
Greg Landrum
 
Hadoop UK Strata Panel Discussion
Hadoop UK Strata Panel DiscussionHadoop UK Strata Panel Discussion
Hadoop UK Strata Panel Discussion
huguk
 
How Data Saves Time
How Data Saves TimeHow Data Saves Time
How Data Saves Time
NetApp
 
Introduction aux algorithmes génétiques
Introduction aux algorithmes génétiquesIntroduction aux algorithmes génétiques
Introduction aux algorithmes génétiques
JUG Lausanne
 
5 Crucial Considerations for Big data adoption
5 Crucial Considerations for Big data adoption5 Crucial Considerations for Big data adoption
5 Crucial Considerations for Big data adoption
Qubole
 
Scaling Deep Learning Algorithms on Extreme Scale Architectures
Scaling Deep Learning Algorithms on Extreme Scale ArchitecturesScaling Deep Learning Algorithms on Extreme Scale Architectures
Scaling Deep Learning Algorithms on Extreme Scale Architectures
inside-BigData.com
 
SpaceCurve - Integrating with Hadoop
SpaceCurve - Integrating with HadoopSpaceCurve - Integrating with Hadoop
SpaceCurve - Integrating with Hadoop
Spacecurve
 
Visualising your Big Data: Eye Vegetables and Eye Candy
Visualising your Big Data: Eye Vegetables and Eye CandyVisualising your Big Data: Eye Vegetables and Eye Candy
Visualising your Big Data: Eye Vegetables and Eye Candy
DataWorks Summit
 
Scoring Metrics for Classification Models
Scoring Metrics for Classification ModelsScoring Metrics for Classification Models
Scoring Metrics for Classification Models
KNIMESlides
 
Analysis and interpretation of monitoring data
Analysis and interpretation of monitoring dataAnalysis and interpretation of monitoring data
Analysis and interpretation of monitoring data
corehard_by
 
Cluster vision at Amsterdam Tech Job Fair
Cluster vision at Amsterdam Tech Job FairCluster vision at Amsterdam Tech Job Fair
Cluster vision at Amsterdam Tech Job Fair
TechMeetups
 
Optalysys Optical Processing for HPC
Optalysys Optical Processing for HPCOptalysys Optical Processing for HPC
Optalysys Optical Processing for HPC
inside-BigData.com
 
DataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven OrganizationsDataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven Organizations
Ellen Friedman
 
kleemann8_12_16c
kleemann8_12_16ckleemann8_12_16c
kleemann8_12_16cGunnarKl
 
Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...
Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...
Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...
Jean-Claude Meteodyn
 
SGI Big Data Launch
SGI Big Data LaunchSGI Big Data Launch
SGI Big Data Launch
inside-BigData.com
 
Emerson Technology Group (ETG)
Emerson Technology Group (ETG)Emerson Technology Group (ETG)
Emerson Technology Group (ETG)
ebtsusa
 
Utilizing Human Data Validation For KPI Analysis And Machine Learning
Utilizing Human Data Validation For KPI Analysis And Machine LearningUtilizing Human Data Validation For KPI Analysis And Machine Learning
Utilizing Human Data Validation For KPI Analysis And Machine Learning
Jen Aman
 
ODSC data science to DataOps
ODSC data science to DataOpsODSC data science to DataOps
ODSC data science to DataOps
Christopher Bergh
 
Augmented OLAP for Big Data
Augmented OLAP for Big DataAugmented OLAP for Big Data
Augmented OLAP for Big Data
Luke Han
 
Augmented OLAP Analytics for Big Data
Augmented OLAP Analytics for Big DataAugmented OLAP Analytics for Big Data
Augmented OLAP Analytics for Big Data
Tyler Wishnoff
 
Open Source Story and what’s new in KNIME Software
Open Source Story and what’s new in KNIME SoftwareOpen Source Story and what’s new in KNIME Software
Open Source Story and what’s new in KNIME Software
KNIMESlides
 
Your Data Nerd Friends Need You!
Your Data Nerd Friends Need You! Your Data Nerd Friends Need You!
Your Data Nerd Friends Need You!
DataKitchen
 
Interactive and reproducible data analysis with the open-source KNIME Analyti...
Interactive and reproducible data analysis with the open-source KNIME Analyti...Interactive and reproducible data analysis with the open-source KNIME Analyti...
Interactive and reproducible data analysis with the open-source KNIME Analyti...
Greg Landrum
 
Advance Data Visualization and Storytelling Virtual Workshop
Advance Data Visualization and Storytelling Virtual WorkshopAdvance Data Visualization and Storytelling Virtual Workshop
Advance Data Visualization and Storytelling Virtual Workshop
CCG
 
Google vs bing
Google vs bingGoogle vs bing
Google vs bing
gitam university( anjali)
 
Prototype: Its methods, techniques, and key features.
Prototype: Its methods, techniques, and key features.Prototype: Its methods, techniques, and key features.
Prototype: Its methods, techniques, and key features.
ONE BCG
 
What is Prototype,Rapid prototyping and Methods.
What is Prototype,Rapid prototyping and Methods. What is Prototype,Rapid prototyping and Methods.
What is Prototype,Rapid prototyping and Methods.
Taniya K
 

More Related Content

What's hot

SpaceCurve - Integrating with Hadoop
SpaceCurve - Integrating with HadoopSpaceCurve - Integrating with Hadoop
SpaceCurve - Integrating with Hadoop
Spacecurve
 
Visualising your Big Data: Eye Vegetables and Eye Candy
Visualising your Big Data: Eye Vegetables and Eye CandyVisualising your Big Data: Eye Vegetables and Eye Candy
Visualising your Big Data: Eye Vegetables and Eye Candy
DataWorks Summit
 
Scoring Metrics for Classification Models
Scoring Metrics for Classification ModelsScoring Metrics for Classification Models
Scoring Metrics for Classification Models
KNIMESlides
 
Analysis and interpretation of monitoring data
Analysis and interpretation of monitoring dataAnalysis and interpretation of monitoring data
Analysis and interpretation of monitoring data
corehard_by
 
Cluster vision at Amsterdam Tech Job Fair
Cluster vision at Amsterdam Tech Job FairCluster vision at Amsterdam Tech Job Fair
Cluster vision at Amsterdam Tech Job Fair
TechMeetups
 
Optalysys Optical Processing for HPC
Optalysys Optical Processing for HPCOptalysys Optical Processing for HPC
Optalysys Optical Processing for HPC
inside-BigData.com
 
DataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven OrganizationsDataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven Organizations
Ellen Friedman
 
kleemann8_12_16c
kleemann8_12_16ckleemann8_12_16c
kleemann8_12_16cGunnarKl
 
Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...
Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...
Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...
Jean-Claude Meteodyn
 
SGI Big Data Launch
SGI Big Data LaunchSGI Big Data Launch
SGI Big Data Launch
inside-BigData.com
 
Emerson Technology Group (ETG)
Emerson Technology Group (ETG)Emerson Technology Group (ETG)
Emerson Technology Group (ETG)
ebtsusa
 
Utilizing Human Data Validation For KPI Analysis And Machine Learning
Utilizing Human Data Validation For KPI Analysis And Machine LearningUtilizing Human Data Validation For KPI Analysis And Machine Learning
Utilizing Human Data Validation For KPI Analysis And Machine Learning
Jen Aman
 
ODSC data science to DataOps
ODSC data science to DataOpsODSC data science to DataOps
ODSC data science to DataOps
Christopher Bergh
 

What's hot (13)

SpaceCurve - Integrating with Hadoop
SpaceCurve - Integrating with HadoopSpaceCurve - Integrating with Hadoop
SpaceCurve - Integrating with Hadoop
 
Visualising your Big Data: Eye Vegetables and Eye Candy
Visualising your Big Data: Eye Vegetables and Eye CandyVisualising your Big Data: Eye Vegetables and Eye Candy
Visualising your Big Data: Eye Vegetables and Eye Candy
 
Scoring Metrics for Classification Models
Scoring Metrics for Classification ModelsScoring Metrics for Classification Models
Scoring Metrics for Classification Models
 
Analysis and interpretation of monitoring data
Analysis and interpretation of monitoring dataAnalysis and interpretation of monitoring data
Analysis and interpretation of monitoring data
 
Cluster vision at Amsterdam Tech Job Fair
Cluster vision at Amsterdam Tech Job FairCluster vision at Amsterdam Tech Job Fair
Cluster vision at Amsterdam Tech Job Fair
 
Optalysys Optical Processing for HPC
Optalysys Optical Processing for HPCOptalysys Optical Processing for HPC
Optalysys Optical Processing for HPC
 
DataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven OrganizationsDataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven Organizations
 
kleemann8_12_16c
kleemann8_12_16ckleemann8_12_16c
kleemann8_12_16c
 
Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...
Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...
Wind meteodyn WT cfd micro scale modeling combined statistical learning for s...
 
SGI Big Data Launch
SGI Big Data LaunchSGI Big Data Launch
SGI Big Data Launch
 
Emerson Technology Group (ETG)
Emerson Technology Group (ETG)Emerson Technology Group (ETG)
Emerson Technology Group (ETG)
 
Utilizing Human Data Validation For KPI Analysis And Machine Learning
Utilizing Human Data Validation For KPI Analysis And Machine LearningUtilizing Human Data Validation For KPI Analysis And Machine Learning
Utilizing Human Data Validation For KPI Analysis And Machine Learning
 
ODSC data science to DataOps
ODSC data science to DataOpsODSC data science to DataOps
ODSC data science to DataOps
 

Similar to Google BigQuery for analysis of scientific datasets: Interactive exploration and analysis of the data using KNIME Analytics Platform

Augmented OLAP for Big Data
Augmented OLAP for Big DataAugmented OLAP for Big Data
Augmented OLAP for Big Data
Luke Han
 
Augmented OLAP Analytics for Big Data
Augmented OLAP Analytics for Big DataAugmented OLAP Analytics for Big Data
Augmented OLAP Analytics for Big Data
Tyler Wishnoff
 
Open Source Story and what’s new in KNIME Software
Open Source Story and what’s new in KNIME SoftwareOpen Source Story and what’s new in KNIME Software
Open Source Story and what’s new in KNIME Software
KNIMESlides
 
Your Data Nerd Friends Need You!
Your Data Nerd Friends Need You! Your Data Nerd Friends Need You!
Your Data Nerd Friends Need You!
DataKitchen
 
Interactive and reproducible data analysis with the open-source KNIME Analyti...
Interactive and reproducible data analysis with the open-source KNIME Analyti...Interactive and reproducible data analysis with the open-source KNIME Analyti...
Interactive and reproducible data analysis with the open-source KNIME Analyti...
Greg Landrum
 
Advance Data Visualization and Storytelling Virtual Workshop
Advance Data Visualization and Storytelling Virtual WorkshopAdvance Data Visualization and Storytelling Virtual Workshop
Advance Data Visualization and Storytelling Virtual Workshop
CCG
 
Google vs bing
Google vs bingGoogle vs bing
Google vs bing
gitam university( anjali)
 
Prototype: Its methods, techniques, and key features.
Prototype: Its methods, techniques, and key features.Prototype: Its methods, techniques, and key features.
Prototype: Its methods, techniques, and key features.
ONE BCG
 
What is Prototype,Rapid prototyping and Methods.
What is Prototype,Rapid prototyping and Methods. What is Prototype,Rapid prototyping and Methods.
What is Prototype,Rapid prototyping and Methods.
Taniya K
 
Slicing heuristics - Techniques for improving value generation, speed to mark...
Slicing heuristics - Techniques for improving value generation, speed to mark...Slicing heuristics - Techniques for improving value generation, speed to mark...
Slicing heuristics - Techniques for improving value generation, speed to mark...
Killick Agile Consulting Services
 
U4 l01 What is big data?
U4 l01 What is big data?U4 l01 What is big data?
U4 l01 What is big data?
Chapelgate Christian Academy
 
Why i love Apache Spark?
Why i love Apache Spark?Why i love Apache Spark?
Why i love Apache Spark?
Jean-Georges Perrin
 
10 reasons why you should choose big data hadoop as career in 2018
10 reasons why you should choose big data hadoop as career in 201810 reasons why you should choose big data hadoop as career in 2018
10 reasons why you should choose big data hadoop as career in 2018
JanBask Training
 
Deltaplan - SEO Search
Deltaplan - SEO SearchDeltaplan - SEO Search
Deltaplan - SEO Search
Roy Huiskes
 
Cross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics ShortcutsCross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics Shortcuts
Craig Sullivan
 
Webinar-Building a Strong Brand For Your Organization -2017-03-07
Webinar-Building a Strong Brand For Your Organization -2017-03-07Webinar-Building a Strong Brand For Your Organization -2017-03-07
Webinar-Building a Strong Brand For Your Organization -2017-03-07
TechSoup
 
Making Sense of Cyberspace, keynote for Software Engineering Institute Cyber ...
Making Sense of Cyberspace, keynote for Software Engineering Institute Cyber ...Making Sense of Cyberspace, keynote for Software Engineering Institute Cyber ...
Making Sense of Cyberspace, keynote for Software Engineering Institute Cyber ...Jason Hong
 
How to Use Big Data by Onehub
How to Use Big Data by OnehubHow to Use Big Data by Onehub
How to Use Big Data by Onehub
Charles Mount
 
Opportunities and Pitfalls of Prototyping with Artificial Intelligence berl...
Opportunities and Pitfalls of Prototyping with Artificial Intelligence berl...Opportunities and Pitfalls of Prototyping with Artificial Intelligence berl...
Opportunities and Pitfalls of Prototyping with Artificial Intelligence berl...
DAIN Studios
 
How to make sure your new website won't be a failure? - Digital Elite Camp 2016
How to make sure your new website won't be a failure? - Digital Elite Camp 2016How to make sure your new website won't be a failure? - Digital Elite Camp 2016
How to make sure your new website won't be a failure? - Digital Elite Camp 2016
AGConsult
 

Similar to Google BigQuery for analysis of scientific datasets: Interactive exploration and analysis of the data using KNIME Analytics Platform (20)

Augmented OLAP for Big Data
Augmented OLAP for Big DataAugmented OLAP for Big Data
Augmented OLAP for Big Data
 
Augmented OLAP Analytics for Big Data
Augmented OLAP Analytics for Big DataAugmented OLAP Analytics for Big Data
Augmented OLAP Analytics for Big Data
 
Open Source Story and what’s new in KNIME Software
Open Source Story and what’s new in KNIME SoftwareOpen Source Story and what’s new in KNIME Software
Open Source Story and what’s new in KNIME Software
 
Your Data Nerd Friends Need You!
Your Data Nerd Friends Need You! Your Data Nerd Friends Need You!
Your Data Nerd Friends Need You!
 
Interactive and reproducible data analysis with the open-source KNIME Analyti...
Interactive and reproducible data analysis with the open-source KNIME Analyti...Interactive and reproducible data analysis with the open-source KNIME Analyti...
Interactive and reproducible data analysis with the open-source KNIME Analyti...
 
Advance Data Visualization and Storytelling Virtual Workshop
Advance Data Visualization and Storytelling Virtual WorkshopAdvance Data Visualization and Storytelling Virtual Workshop
Advance Data Visualization and Storytelling Virtual Workshop
 
Google vs bing
Google vs bingGoogle vs bing
Google vs bing
 
Prototype: Its methods, techniques, and key features.
Prototype: Its methods, techniques, and key features.Prototype: Its methods, techniques, and key features.
Prototype: Its methods, techniques, and key features.
 
What is Prototype,Rapid prototyping and Methods.
What is Prototype,Rapid prototyping and Methods. What is Prototype,Rapid prototyping and Methods.
What is Prototype,Rapid prototyping and Methods.
 
Slicing heuristics - Techniques for improving value generation, speed to mark...
Slicing heuristics - Techniques for improving value generation, speed to mark...Slicing heuristics - Techniques for improving value generation, speed to mark...
Slicing heuristics - Techniques for improving value generation, speed to mark...
 
U4 l01 What is big data?
U4 l01 What is big data?U4 l01 What is big data?
U4 l01 What is big data?
 
Why i love Apache Spark?
Why i love Apache Spark?Why i love Apache Spark?
Why i love Apache Spark?
 
10 reasons why you should choose big data hadoop as career in 2018
10 reasons why you should choose big data hadoop as career in 201810 reasons why you should choose big data hadoop as career in 2018
10 reasons why you should choose big data hadoop as career in 2018
 
Deltaplan - SEO Search
Deltaplan - SEO SearchDeltaplan - SEO Search
Deltaplan - SEO Search
 
Cross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics ShortcutsCross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics Shortcuts
 
Webinar-Building a Strong Brand For Your Organization -2017-03-07
Webinar-Building a Strong Brand For Your Organization -2017-03-07Webinar-Building a Strong Brand For Your Organization -2017-03-07
Webinar-Building a Strong Brand For Your Organization -2017-03-07
 
Making Sense of Cyberspace, keynote for Software Engineering Institute Cyber ...
Making Sense of Cyberspace, keynote for Software Engineering Institute Cyber ...Making Sense of Cyberspace, keynote for Software Engineering Institute Cyber ...
Making Sense of Cyberspace, keynote for Software Engineering Institute Cyber ...
 
How to Use Big Data by Onehub
How to Use Big Data by OnehubHow to Use Big Data by Onehub
How to Use Big Data by Onehub
 
Opportunities and Pitfalls of Prototyping with Artificial Intelligence berl...
Opportunities and Pitfalls of Prototyping with Artificial Intelligence berl...Opportunities and Pitfalls of Prototyping with Artificial Intelligence berl...
Opportunities and Pitfalls of Prototyping with Artificial Intelligence berl...
 
How to make sure your new website won't be a failure? - Digital Elite Camp 2016
How to make sure your new website won't be a failure? - Digital Elite Camp 2016How to make sure your new website won't be a failure? - Digital Elite Camp 2016
How to make sure your new website won't be a failure? - Digital Elite Camp 2016
 

More from Greg Landrum

Chemical registration
Chemical registrationChemical registration
Chemical registration
Greg Landrum
 
Mike Lynch Award Lecture, ICCS 2022
Mike Lynch Award Lecture, ICCS 2022Mike Lynch Award Lecture, ICCS 2022
Mike Lynch Award Lecture, ICCS 2022
Greg Landrum
 
ACS San Diego - The RDKit: Open-source cheminformatics
ACS San Diego - The RDKit: Open-source cheminformaticsACS San Diego - The RDKit: Open-source cheminformatics
ACS San Diego - The RDKit: Open-source cheminformatics
Greg Landrum
 
Let’s talk about reproducible data analysis
Let’s talk about reproducible data analysisLet’s talk about reproducible data analysis
Let’s talk about reproducible data analysis
Greg Landrum
 
How Do You Build and Validate 1500 Models and What Can You Learn from Them?
How Do You Build and Validate 1500 Models and What Can You Learn from Them? How Do You Build and Validate 1500 Models and What Can You Learn from Them?
How Do You Build and Validate 1500 Models and What Can You Learn from Them?
Greg Landrum
 
Processing malaria HTS results using KNIME: a tutorial
Processing malaria HTS results using KNIME: a tutorialProcessing malaria HTS results using KNIME: a tutorial
Processing malaria HTS results using KNIME: a tutorial
Greg Landrum
 
Big (chemical) data? No Problem!
Big (chemical) data? No Problem!Big (chemical) data? No Problem!
Big (chemical) data? No Problem!
Greg Landrum
 
Is one enough? Data warehousing for biomedical research
Is one enough? Data warehousing for biomedical researchIs one enough? Data warehousing for biomedical research
Is one enough? Data warehousing for biomedical research
Greg Landrum
 
Some "challenges" on the open-source/open-data front
Some "challenges" on the open-source/open-data frontSome "challenges" on the open-source/open-data front
Some "challenges" on the open-source/open-data front
Greg Landrum
 
Large scale classification of chemical reactions from patent data
Large scale classification of chemical reactions from patent dataLarge scale classification of chemical reactions from patent data
Large scale classification of chemical reactions from patent data
Greg Landrum
 
Machine learning in the life sciences with knime
Machine learning in the life sciences with knimeMachine learning in the life sciences with knime
Machine learning in the life sciences with knime
Greg Landrum
 
Open-source from/in the enterprise: the RDKit
Open-source from/in the enterprise: the RDKitOpen-source from/in the enterprise: the RDKit
Open-source from/in the enterprise: the RDKit
Greg Landrum
 
Open-source tools for querying and organizing large reaction databases
Open-source tools for querying and organizing large reaction databasesOpen-source tools for querying and organizing large reaction databases
Open-source tools for querying and organizing large reaction databases
Greg Landrum
 
Is that a scientific report or just some cool pictures from the lab? Reproduc...
Is that a scientific report or just some cool pictures from the lab? Reproduc...Is that a scientific report or just some cool pictures from the lab? Reproduc...
Is that a scientific report or just some cool pictures from the lab? Reproduc...Greg Landrum
 
Reproducibility in cheminformatics and computational chemistry research: cert...
Reproducibility in cheminformatics and computational chemistry research: cert...Reproducibility in cheminformatics and computational chemistry research: cert...
Reproducibility in cheminformatics and computational chemistry research: cert...
Greg Landrum
 

More from Greg Landrum (15)

Chemical registration
Chemical registrationChemical registration
Chemical registration
 
Mike Lynch Award Lecture, ICCS 2022
Mike Lynch Award Lecture, ICCS 2022Mike Lynch Award Lecture, ICCS 2022
Mike Lynch Award Lecture, ICCS 2022
 
ACS San Diego - The RDKit: Open-source cheminformatics
ACS San Diego - The RDKit: Open-source cheminformaticsACS San Diego - The RDKit: Open-source cheminformatics
ACS San Diego - The RDKit: Open-source cheminformatics
 
Let’s talk about reproducible data analysis
Let’s talk about reproducible data analysisLet’s talk about reproducible data analysis
Let’s talk about reproducible data analysis
 
How Do You Build and Validate 1500 Models and What Can You Learn from Them?
How Do You Build and Validate 1500 Models and What Can You Learn from Them? How Do You Build and Validate 1500 Models and What Can You Learn from Them?
How Do You Build and Validate 1500 Models and What Can You Learn from Them?
 
Processing malaria HTS results using KNIME: a tutorial
Processing malaria HTS results using KNIME: a tutorialProcessing malaria HTS results using KNIME: a tutorial
Processing malaria HTS results using KNIME: a tutorial
 
Big (chemical) data? No Problem!
Big (chemical) data? No Problem!Big (chemical) data? No Problem!
Big (chemical) data? No Problem!
 
Is one enough? Data warehousing for biomedical research
Is one enough? Data warehousing for biomedical researchIs one enough? Data warehousing for biomedical research
Is one enough? Data warehousing for biomedical research
 
Some "challenges" on the open-source/open-data front
Some "challenges" on the open-source/open-data frontSome "challenges" on the open-source/open-data front
Some "challenges" on the open-source/open-data front
 
Large scale classification of chemical reactions from patent data
Large scale classification of chemical reactions from patent dataLarge scale classification of chemical reactions from patent data
Large scale classification of chemical reactions from patent data
 
Machine learning in the life sciences with knime
Machine learning in the life sciences with knimeMachine learning in the life sciences with knime
Machine learning in the life sciences with knime
 
Open-source from/in the enterprise: the RDKit
Open-source from/in the enterprise: the RDKitOpen-source from/in the enterprise: the RDKit
Open-source from/in the enterprise: the RDKit
 
Open-source tools for querying and organizing large reaction databases
Open-source tools for querying and organizing large reaction databasesOpen-source tools for querying and organizing large reaction databases
Open-source tools for querying and organizing large reaction databases
 
Is that a scientific report or just some cool pictures from the lab? Reproduc...
Is that a scientific report or just some cool pictures from the lab? Reproduc...Is that a scientific report or just some cool pictures from the lab? Reproduc...
Is that a scientific report or just some cool pictures from the lab? Reproduc...
 
Reproducibility in cheminformatics and computational chemistry research: cert...
Reproducibility in cheminformatics and computational chemistry research: cert...Reproducibility in cheminformatics and computational chemistry research: cert...
Reproducibility in cheminformatics and computational chemistry research: cert...
 

Recently uploaded

Hemoglobin metabolism_pathophysiology.pptx
Hemoglobin metabolism_pathophysiology.pptxHemoglobin metabolism_pathophysiology.pptx
Hemoglobin metabolism_pathophysiology.pptx
muralinath2
 
Mammalian Pineal Body Structure and Also Functions
Mammalian Pineal Body Structure and Also FunctionsMammalian Pineal Body Structure and Also Functions
Mammalian Pineal Body Structure and Also Functions
YOGESH DOGRA
 
In silico drugs analogue design: novobiocin analogues.pptx
In silico drugs analogue design: novobiocin analogues.pptxIn silico drugs analogue design: novobiocin analogues.pptx
In silico drugs analogue design: novobiocin analogues.pptx
AlaminAfendy1
 
NuGOweek 2024 Ghent - programme - final version
NuGOweek 2024 Ghent - programme - final versionNuGOweek 2024 Ghent - programme - final version
NuGOweek 2024 Ghent - programme - final version
pablovgd
 
Cancer cell metabolism: special Reference to Lactate Pathway
Cancer cell metabolism: special Reference to Lactate PathwayCancer cell metabolism: special Reference to Lactate Pathway
Cancer cell metabolism: special Reference to Lactate Pathway
AADYARAJPANDEY1
 
What is greenhouse gasses and how many gasses are there to affect the Earth.
What is greenhouse gasses and how many gasses are there to affect the Earth.What is greenhouse gasses and how many gasses are there to affect the Earth.
What is greenhouse gasses and how many gasses are there to affect the Earth.
moosaasad1975
 
Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...
Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...
Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...
Sérgio Sacani
 
general properties of oerganologametal.ppt
general properties of oerganologametal.pptgeneral properties of oerganologametal.ppt
general properties of oerganologametal.ppt
IqrimaNabilatulhusni
 
Nutraceutical market, scope and growth: Herbal drug technology
Nutraceutical market, scope and growth: Herbal drug technologyNutraceutical market, scope and growth: Herbal drug technology
Nutraceutical market, scope and growth: Herbal drug technology
Lokesh Patil
 
Comparative structure of adrenal gland in vertebrates
Comparative structure of adrenal gland in vertebratesComparative structure of adrenal gland in vertebrates
Comparative structure of adrenal gland in vertebrates
sachin783648
 
Structural Classification Of Protein (SCOP)
Structural Classification Of Protein (SCOP)Structural Classification Of Protein (SCOP)
Structural Classification Of Protein (SCOP)
aishnasrivastava
 
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
Scintica Instrumentation
 
Richard's entangled aventures in wonderland
Richard's entangled aventures in wonderlandRichard's entangled aventures in wonderland
Richard's entangled aventures in wonderland
Richard Gill
 
Citrus Greening Disease and its Management
Citrus Greening Disease and its ManagementCitrus Greening Disease and its Management
Citrus Greening Disease and its Management
subedisuryaofficial
 
4. An Overview of Sugarcane White Leaf Disease in Vietnam.pdf
4. An Overview of Sugarcane White Leaf Disease in Vietnam.pdf4. An Overview of Sugarcane White Leaf Disease in Vietnam.pdf
4. An Overview of Sugarcane White Leaf Disease in Vietnam.pdf
ssuserbfdca9
 
EY - Supply Chain Services 2018_template.pptx
EY - Supply Chain Services 2018_template.pptxEY - Supply Chain Services 2018_template.pptx
EY - Supply Chain Services 2018_template.pptx
AlguinaldoKong
 
insect taxonomy importance systematics and classification
insect taxonomy importance systematics and classificationinsect taxonomy importance systematics and classification
insect taxonomy importance systematics and classification
anitaento25
 
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Ana Luísa Pinho
 
Lateral Ventricles.pdf very easy good diagrams comprehensive
Lateral Ventricles.pdf very easy good diagrams comprehensiveLateral Ventricles.pdf very easy good diagrams comprehensive
Lateral Ventricles.pdf very easy good diagrams comprehensive
silvermistyshot
 
GBSN- Microbiology (Lab 3) Gram Staining
GBSN- Microbiology (Lab 3) Gram StainingGBSN- Microbiology (Lab 3) Gram Staining
GBSN- Microbiology (Lab 3) Gram Staining
Areesha Ahmad
 

Recently uploaded (20)

Hemoglobin metabolism_pathophysiology.pptx
Hemoglobin metabolism_pathophysiology.pptxHemoglobin metabolism_pathophysiology.pptx
Hemoglobin metabolism_pathophysiology.pptx
 
Mammalian Pineal Body Structure and Also Functions
Mammalian Pineal Body Structure and Also FunctionsMammalian Pineal Body Structure and Also Functions
Mammalian Pineal Body Structure and Also Functions
 
In silico drugs analogue design: novobiocin analogues.pptx
In silico drugs analogue design: novobiocin analogues.pptxIn silico drugs analogue design: novobiocin analogues.pptx
In silico drugs analogue design: novobiocin analogues.pptx
 
NuGOweek 2024 Ghent - programme - final version
NuGOweek 2024 Ghent - programme - final versionNuGOweek 2024 Ghent - programme - final version
NuGOweek 2024 Ghent - programme - final version
 
Cancer cell metabolism: special Reference to Lactate Pathway
Cancer cell metabolism: special Reference to Lactate PathwayCancer cell metabolism: special Reference to Lactate Pathway
Cancer cell metabolism: special Reference to Lactate Pathway
 
What is greenhouse gasses and how many gasses are there to affect the Earth.
What is greenhouse gasses and how many gasses are there to affect the Earth.What is greenhouse gasses and how many gasses are there to affect the Earth.
What is greenhouse gasses and how many gasses are there to affect the Earth.
 
Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...
Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...
Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...
 
general properties of oerganologametal.ppt
general properties of oerganologametal.pptgeneral properties of oerganologametal.ppt
general properties of oerganologametal.ppt
 
Nutraceutical market, scope and growth: Herbal drug technology
Nutraceutical market, scope and growth: Herbal drug technologyNutraceutical market, scope and growth: Herbal drug technology
Nutraceutical market, scope and growth: Herbal drug technology
 
Comparative structure of adrenal gland in vertebrates
Comparative structure of adrenal gland in vertebratesComparative structure of adrenal gland in vertebrates
Comparative structure of adrenal gland in vertebrates
 
Structural Classification Of Protein (SCOP)
Structural Classification Of Protein (SCOP)Structural Classification Of Protein (SCOP)
Structural Classification Of Protein (SCOP)
 
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
 
Richard's entangled aventures in wonderland
Richard's entangled aventures in wonderlandRichard's entangled aventures in wonderland
Richard's entangled aventures in wonderland
 
Citrus Greening Disease and its Management
Citrus Greening Disease and its ManagementCitrus Greening Disease and its Management
Citrus Greening Disease and its Management
 
4. An Overview of Sugarcane White Leaf Disease in Vietnam.pdf
4. An Overview of Sugarcane White Leaf Disease in Vietnam.pdf4. An Overview of Sugarcane White Leaf Disease in Vietnam.pdf
4. An Overview of Sugarcane White Leaf Disease in Vietnam.pdf
 
EY - Supply Chain Services 2018_template.pptx
EY - Supply Chain Services 2018_template.pptxEY - Supply Chain Services 2018_template.pptx
EY - Supply Chain Services 2018_template.pptx
 
insect taxonomy importance systematics and classification
insect taxonomy importance systematics and classificationinsect taxonomy importance systematics and classification
insect taxonomy importance systematics and classification
 
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
 
Lateral Ventricles.pdf very easy good diagrams comprehensive
Lateral Ventricles.pdf very easy good diagrams comprehensiveLateral Ventricles.pdf very easy good diagrams comprehensive
Lateral Ventricles.pdf very easy good diagrams comprehensive
 
GBSN- Microbiology (Lab 3) Gram Staining
GBSN- Microbiology (Lab 3) Gram StainingGBSN- Microbiology (Lab 3) Gram Staining
GBSN- Microbiology (Lab 3) Gram Staining
 

Google BigQuery for analysis of scientific datasets: Interactive exploration and analysis of the data using KNIME Analytics Platform

  • 1. © 2019 KNIME AG. All Rights Reserved. Google BigQuery for analysis of scientific datasets: Interactive exploration and analysis of the data using KNIME Analytics Platform Greg Landrum Martyna Pawletta Jeanette Prinz greg.landrum@knime.com @dr_greg_landrum
  • 2. © 2019 KNIME AG. All Rights Reserved. 2 Acknowledgements • Steve Boyer (Collabra) • Lutz Weber (OntoChem) • Ian Wetherbee (Google)
  • 3. © 2019 KNIME AG. All Rights Reserved. 3 Google BigQuery? • A giant collection of tables that I can query with SQL • If the tables share common keys, I can do interesting things Might be an oversimplification. ☺
  • 4. © 2019 KNIME AG. All Rights Reserved. 4 An aside: searching vs exploring a.k.a. why I’m enthusiastic about this project
  • 5. © 2019 KNIME AG. All Rights Reserved. 5 An aside: searching vs exploring
  • 6. © 2019 KNIME AG. All Rights Reserved. 6 An aside: searching vs exploring
  • 7. © 2019 KNIME AG. All Rights Reserved. 7 An aside: searching vs exploring • There are definitely arguments for specialized interfaces that are tailored to make answering a particular question super efficient and easy • But! There are times when I’m still trying to figure out exactly what the question is • For this it’s nice to have a giant pile of data and a general purpose tool for exploring it
  • 8. © 2019 KNIME AG. All Rights Reserved. 8 What we’re going to do here • Do some exploration of the scientific data that’s now in BigQuery… • … with KNIME
  • 9. © 2019 KNIME AG. All Rights Reserved. 9 Workflow part 1
  • 10. © 2019 KNIME AG. All Rights Reserved. 10 Workflow part 2
  • 11. © 2019 KNIME AG. All Rights Reserved. 11 The first database queries
  • 12. © 2019 KNIME AG. All Rights Reserved. 12 Picking the disease/condition
  • 13. © 2019 KNIME AG. All Rights Reserved. 13 Results
  • 14. © 2019 KNIME AG. All Rights Reserved. 14 Compound classes
  • 15. © 2019 KNIME AG. All Rights Reserved. 15
  • 16. © 2019 KNIME AG. All Rights Reserved. 16