SlideShare a Scribd company logo
1 of 36
Download to read offline
Overview of
Machine Learning for Molecules and
Materials Workshop @ NIPS 2017
NIPS2017 @PFN
Jan. 21st 2018
Preferred Networks, Inc.
Kenta Oono oono@preferred.jp
Kenta Oono (@delta2323_)
• Preferred Networks (PFN), Engineer
• MSc. in mathematics
• 2014.10 - Present: PFN
• Role
– Biology project
– Chainer developer
– Chainer Chemistry developer
Workshop overview
• 15 invited talks, 22 posters, 3 sponsors
• Session titles
– Introduction to Machine Learning and Chemistry
– Machine Learning Applications in Chemistry
– Kernel Learning with Structured Data
– Deep Learning Approaches
• Areas of interest
– ML + (Quantum) Chemistry / ML + Quantum Physics / Material Informatics
– DL : Vinyals (DeepMind), Duvenaud (Google), Smola (Amazon)
Why materials and molecules?
• Material informatics
– Material genome initiative
– MI2I project (NIMS)
• Drug discovery
– Big pharmas’ investment
– IPAB drug discovery contest
https://medium.com/the-ai-lab/artificial-intelligence-in-drug-discovery-is-
overhyped-examples-from-astrazeneca-harvard-315d69a7f863
Chemical prediction - Two approaches
• Quantum simulation
– Theory-based approach
– e.g. DFT (Density Functional Theory)
J Precision is guaranteed
L High calculation cost
• Machine learning
– Data-based approach
– e.g. Graph convolution
J Low cost, high speed calculation
L Hard to guarantee precision
“Neural message passing for quantum chemistry”Justin et al
Hardness of learning with molecules
• How to represent molecules?
– Discrete and structured nature of molecules
– 2D and 3D information
• Vast search space (~10**60)
Topics
• Molecule generation with VAE
• Graph convolution
MOLECULE
GENERATION WITH VAE
Molecule generation
Prediction Generation
Solvable Solvable
SMILES
A format of encoding molecules in text.
Simple solution: Treat a molecule as a sequential data and apply NLP techniques.
OC[C@@H](O1)[C@@H](O)[C@H]
(O)[C@@H](O)[C@@H](O)1
Variational AutoEncoder (VAE) [Kingma+13][Rezende+14]
Kingma, D. P., & Welling, M. (2013). Auto-
encoding variational bayes. arXiv preprint
arXiv:1312.6114.
Rezende, D. J., Mohamed, S., & Wierstra, D.
(2014). Stochastic backpropagation and
approximate inference in deep generative
models. arXiv preprint arXiv:1401.4082.
• Variational inference
• Use NN as an inference model.
• Train in end-to-end manner with backpropagation.
• Extension to RNN encoder/decoder [Fabius+15]
https://www.slideshare.net/KentaOono/vaetype-
deep-generative-models
z
x
z
x
approximate
Inference model
qφ(z | x)
Generative model
pθ (z | x)
Molecule generation with VAE (CVAE) [Gómez-Bombarelli+16]
• Encode and decode molecules represented
as SMILE with VAE.
• Latent representation can be used for semi-
supervised learning.
• We can use learned models to find
molecule with desired property by
optimizing representation in latent space
and decode it.
L generated molecules are not guaranteed to be
valid syntactically.
Gómez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J. M., Sánchez-Lengeling, B., Sheberla, D., ... & Aspuru-
Guzik, A. (2016). Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science.
Grammar VAE (GVAE) [Kusner+17]
Kusner, M. J., Paige, B., & Hernández-Lobato, J. M. (2017). Grammar Variational
Autoencoder. arXiv preprint arXiv:1703.01925.
• Generate sequence of production
rules of syntax of SMILES
• Generated molecules are
guaranteed to be valid
syntactically.
Encode
Decode
• Represent SMILES syntax as CFG
• Convert a molecule to a parse tree
to get a sequence of production
rules.
• Feed the sequence to RNN-VAE.
L generated molecules are not guaranteed
to be valid semantically.
Syntax-Directed VAE (SDVAE) Best paper award
• Use attribute grammar to guarantee
that generated molecules are both
syntactically and semantically valid.
• Generate attributes stochastically
(stochastic lazy attributes) for on-the-
fly semantic check.
← Simplified schematic view
(Note: Bottom up semantic check for
explanation)
http://www.quantum-
machine.org/workshops/nips2017/assets/pdf/sd
vae_workshop_camera_ready.pdf
https://openreview.net/forum?id=SyqShMZRb
Discussion
• Is SMILES appropriate as an input representation?
– Input representation is not unique (e.g. CC#C and C#CC represent same molecule).
– Molecule representation is not guaranteed to be invariant to relabeling (i.e. permutation of
indexes) of molecules.
– SMILES is not natural language. Can we justify to apply NLP techniques?
• Synthesizability is not considered.
Related papers
• Extension of VAE
– Semi-supervised Continuous Representation of Molecules
– Learning Hard Quantum Distributions With Variational Autoencoders
• Seq2seq models
– “Found in translation”: Predicting Outcomes of Complex Organic Chemistry Reactions Using
Neural Sequence-to-sequence Models
• Molecule generation
– Learning a Generative Model for Validity in Complex Discrete Structure
– ChemTS: de novo molecular generation with MCTS and RNN (for rollout)
GRAPH CONVOLUTION
ALGORITHMS
Extended Connectivity Fingerprint (ECFP)
Convert molecule into fixed length bit representation
J Pros
• Calculation is fast
• Show presence of particular substructures
L Cons
• Bit collision
– Two (or more) different substructure features
could be represented by the same bit position
• Task-independent featurizer
https://chembioinfo.com/2011/10/30/revisiting-
molecular-hashed-fingerprints/
https://docs.chemaxon.com/display/docs/Extended
+Connectivity+Fingerprint+ECFP
How graph convolution works
Graph convolution
Convolution kernel depends on Graph structure
Image
class label
Chemical
property
CNN on image
Unified view of graph convolution
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017). Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212.
Update Readout
v
w
hw
evw
hv
mv
mv
mv
mv
hv
y
Many message-passing algorithms (NFP, GGNN, Weave) are formulated as the
iterative application of Update function and Readout function [Gilmer et al. 17].
Aggregates neighborhood information and
updates node representations.
Aggregates all node representations
and updates the final output.
Neural Fingerprint (NFP) [Duvenaud+15]
Atom feature embedding
Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T.,
Aspuru-Guzik, A.,&Adams, R. P. (2015). Convolutional networks on graphs
for learning molecular fingerprints. In Advances in neural information
processing systems (pp. 2224-2232).
HCNOS
Neural Fingerprint (NFP)
Update
hnew
3= σ ( W2(h3+h2+h4) )
hnew
7= σ ( W3(h7+h6+h8+h9) )
Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T.,
Aspuru-Guzik, A.,&Adams, R. P. (2015). Convolutional networks on graphs
for learning molecular fingerprints. In Advances in neural information
processing systems (pp. 2224-2232).
Neural Fingerprint (NFP)
Readout
h7
h8
R = ∑i softmax (Whi)
h6
h1
h2
h3
h4
h5
h9 h10
ECFP and NFP
[Duvenaud+15] Fig.2
Comparison between graph convolution networks
NFP GGNN Weave SchNet
How to extract
atom features
Man-made or
Embed
Man-made or
Embed
Man-made or
Embed
Man-made or
Embed
Graph convolution
strategy
Adjacent
atoms only
Adjacent
atoms only
All atom-atom
pairs
All atom-atom
pairs
How to represent
connection
information
Degree Bond type
Man-made
pair features
(bond type,distance etc.)
Distance
End-to-end Learning of Graph Neural Networks
for Molecular Representation [Tsubaki+17]
1. Embed r-radius subgraphs
2. Update node and vertex representations
3. Use LSTM to capture long-term dependency in vertices and edges
4. Readout the final output with self-attention mechanism
Best paper award
https://www.dropbox.com/s/ujzuj2kd2nyz348/tsubaki_nips2017.pdf
Extension to semi-supervised learning [Hai+17]
Compute representations of subgraphs inductively with
neural message passing (→)
Optimize the representation in unsupervised manner in
the same way as Paragraph vector (↓)
Nguyen, H., Maeda, S. I.,&Oono, K. (2017).
Semi-supervised learning of hierarchical
representations of molecules using neural
message passing.arXiv preprint
arXiv:1711.10168.
Workshop paper
Chainer Chemistry (http://chainer-chemistry.readthedocs.io/)
Chainer extension library for Biology and Chemistry
FileParser (SDF, CSV) Loader (QM 9, Tox 21)
Graph convolution NN
(NFP, GGNN, SchNet, Weave)
Preprocessing
Example
Multitask
learning with
QM9 / Tox21
Model
Layer
Dataset
Pretrained
Model
Feature extractor
(TBD)
GraphLinear, EmbedAtomID
Basic information
Release:12/14/2017, Version: v0.1.0, License: MIT, Language: Python
Discussion
• Is message passing neural network general enough to formulate many
graph convolution algorithms?
• How can we incorporate 3D information to graph convolution algorithms (e.g.
Chirality).
Other topics (DNN models)
• CNN models
– ChemNet: A Transferable and Generalizable Deep Neural Network for
Small-molecule Property Prediction
– Ligand Pose Optimization With Atomic Grid-based Convolutional Neural
Networks
• Other DNN models
– Deep Learning for Prediction of Synergistic Effects of Anti-cancer Drugs
– Deep Learning Yields Virtual Assays
– Neural Network for Learning Universal Atomic Forces
Other topics
• Chemical synthesis
– Automatically Extracting Action Graphs From Materials Science
Synthesis Procedures
– Marwin Segler’s talk: Planning Chemical Syntheses with Neural
Networks and Monte Carlo Tree Search
• Bayesian optimization
– Bayesian Protein Optimization
– Constrained Bayesian Optimization for Automatic Chemical Design
Segler, M. H., Preuss, M.,&Waller, M. P. (2017). Learning to Plan Chemical
Syntheses. arXiv preprint arXiv:1708.04202.
Summary
• Data-driven approach for understanding molecules are being paid attention
in material informatics, quantum chemistry, and quantum physics fields.
• Recent advances of :
– Molecule generation with VAE
– Learning graph-structured data with graph convolution algorithms.
BACKUP
Chainer Chemistry (http://chainer-chemistry.readthedocs.io/)
Chainer extension library for Biology and Chemistry
Basic information
release:12/14/2017, version: v0.1.0, license: MIT, language: Python
Features
• State-of-the-art deep learning neural network models (especially graph
convolutions) for chemical molecules (NFP, GGNN, Weave, SchNet etc.)
• Preprocessors of molecules tailored for these models
• Parsers for several standard file formats (CSV, SDF etc.)
• Loaders for several well-known datasets (QM9, Tox21 etc.)
Example: HOMO prediction with QM9 dataset
# Dataset preprocessing (for NFP Network)
preprocessor = preprocess_method_dict['nfp']()
dataset = D.get_qm9(preprocessor, labels='homo’)
# Cache dataset for second use
NumpyTupleDataset.save('input/nfp_homo/data.npz', dataset)
train, val = split_dataset_random(dataset, first_size=10000)
# Build model and use as an ordinary Chain
model = GraphConvPredictor(NFP(16, 16, 4), MLP(16, 1))
Overview of Machine Learning for Molecules and Materials Workshop @ NIPS2017

More Related Content

What's hot

【論文紹介】How Powerful are Graph Neural Networks?
【論文紹介】How Powerful are Graph Neural Networks?【論文紹介】How Powerful are Graph Neural Networks?
【論文紹介】How Powerful are Graph Neural Networks?Masanao Ochi
 
Graph Neural Network - Introduction
Graph Neural Network - IntroductionGraph Neural Network - Introduction
Graph Neural Network - IntroductionJungwon Kim
 
Machine Learning for Chemical Sciences
Machine Learning for Chemical SciencesMachine Learning for Chemical Sciences
Machine Learning for Chemical SciencesIchigaku Takigawa
 
Introduction to Graph neural networks @ Vienna Deep Learning meetup
Introduction to Graph neural networks @  Vienna Deep Learning meetupIntroduction to Graph neural networks @  Vienna Deep Learning meetup
Introduction to Graph neural networks @ Vienna Deep Learning meetupLiad Magen
 
第一原理計算と密度汎関数理論
第一原理計算と密度汎関数理論第一原理計算と密度汎関数理論
第一原理計算と密度汎関数理論dc1394
 
Graph Representation Learning
Graph Representation LearningGraph Representation Learning
Graph Representation LearningJure Leskovec
 
Machine Learning for Molecular Graph Representations and Geometries
Machine Learning for Molecular Graph Representations and GeometriesMachine Learning for Molecular Graph Representations and Geometries
Machine Learning for Molecular Graph Representations and GeometriesIchigaku Takigawa
 
Materials informatics
Materials informaticsMaterials informatics
Materials informaticsskalidindi7
 
[DL輪読会]Energy-based generative adversarial networks
[DL輪読会]Energy-based generative adversarial networks[DL輪読会]Energy-based generative adversarial networks
[DL輪読会]Energy-based generative adversarial networksDeep Learning JP
 
PFP:材料探索のための汎用Neural Network Potential_中郷_20220422POLセミナー
PFP:材料探索のための汎用Neural Network Potential_中郷_20220422POLセミナーPFP:材料探索のための汎用Neural Network Potential_中郷_20220422POLセミナー
PFP:材料探索のための汎用Neural Network Potential_中郷_20220422POLセミナーMatlantis
 
Deep learning for medical imaging
Deep learning for medical imagingDeep learning for medical imaging
Deep learning for medical imaginggeetachauhan
 
Machine learning for materials design: opportunities, challenges, and methods
Machine learning for materials design: opportunities, challenges, and methodsMachine learning for materials design: opportunities, challenges, and methods
Machine learning for materials design: opportunities, challenges, and methodsAnubhav Jain
 
SchNet: A continuous-filter convolutional neural network for modeling quantum...
SchNet: A continuous-filter convolutional neural network for modeling quantum...SchNet: A continuous-filter convolutional neural network for modeling quantum...
SchNet: A continuous-filter convolutional neural network for modeling quantum...Kazuki Fujikawa
 
GraphSage vs Pinsage #InsideArangoDB
GraphSage vs Pinsage #InsideArangoDBGraphSage vs Pinsage #InsideArangoDB
GraphSage vs Pinsage #InsideArangoDBArangoDB Database
 
論文紹介資料「Quantum Deep Field : Data-Driven Wave Function ...」
論文紹介資料「Quantum Deep Field : Data-Driven Wave Function ...」論文紹介資料「Quantum Deep Field : Data-Driven Wave Function ...」
論文紹介資料「Quantum Deep Field : Data-Driven Wave Function ...」DaikiKoge
 
PFP:材料探索のための汎用Neural Network Potential - 2021/10/4 QCMSR + DLAP共催
PFP:材料探索のための汎用Neural Network Potential - 2021/10/4 QCMSR + DLAP共催PFP:材料探索のための汎用Neural Network Potential - 2021/10/4 QCMSR + DLAP共催
PFP:材料探索のための汎用Neural Network Potential - 2021/10/4 QCMSR + DLAP共催Preferred Networks
 
グラフ構造データに対する深層学習〜創薬・材料科学への応用とその問題点〜 (第26回ステアラボ人工知能セミナー)
グラフ構造データに対する深層学習〜創薬・材料科学への応用とその問題点〜 (第26回ステアラボ人工知能セミナー)グラフ構造データに対する深層学習〜創薬・材料科学への応用とその問題点〜 (第26回ステアラボ人工知能セミナー)
グラフ構造データに対する深層学習〜創薬・材料科学への応用とその問題点〜 (第26回ステアラボ人工知能セミナー)STAIR Lab, Chiba Institute of Technology
 
Discovering new functional materials for clean energy and beyond using high-t...
Discovering new functional materials for clean energy and beyond using high-t...Discovering new functional materials for clean energy and beyond using high-t...
Discovering new functional materials for clean energy and beyond using high-t...Anubhav Jain
 

What's hot (20)

【論文紹介】How Powerful are Graph Neural Networks?
【論文紹介】How Powerful are Graph Neural Networks?【論文紹介】How Powerful are Graph Neural Networks?
【論文紹介】How Powerful are Graph Neural Networks?
 
Graph Neural Network - Introduction
Graph Neural Network - IntroductionGraph Neural Network - Introduction
Graph Neural Network - Introduction
 
Machine Learning for Chemical Sciences
Machine Learning for Chemical SciencesMachine Learning for Chemical Sciences
Machine Learning for Chemical Sciences
 
Introduction to Graph neural networks @ Vienna Deep Learning meetup
Introduction to Graph neural networks @  Vienna Deep Learning meetupIntroduction to Graph neural networks @  Vienna Deep Learning meetup
Introduction to Graph neural networks @ Vienna Deep Learning meetup
 
第一原理計算と密度汎関数理論
第一原理計算と密度汎関数理論第一原理計算と密度汎関数理論
第一原理計算と密度汎関数理論
 
Graph Representation Learning
Graph Representation LearningGraph Representation Learning
Graph Representation Learning
 
Machine Learning for Molecular Graph Representations and Geometries
Machine Learning for Molecular Graph Representations and GeometriesMachine Learning for Molecular Graph Representations and Geometries
Machine Learning for Molecular Graph Representations and Geometries
 
Materials informatics
Materials informaticsMaterials informatics
Materials informatics
 
[DL輪読会]Energy-based generative adversarial networks
[DL輪読会]Energy-based generative adversarial networks[DL輪読会]Energy-based generative adversarial networks
[DL輪読会]Energy-based generative adversarial networks
 
PFP:材料探索のための汎用Neural Network Potential_中郷_20220422POLセミナー
PFP:材料探索のための汎用Neural Network Potential_中郷_20220422POLセミナーPFP:材料探索のための汎用Neural Network Potential_中郷_20220422POLセミナー
PFP:材料探索のための汎用Neural Network Potential_中郷_20220422POLセミナー
 
Deep learning for medical imaging
Deep learning for medical imagingDeep learning for medical imaging
Deep learning for medical imaging
 
Graph LSTM解説
Graph LSTM解説Graph LSTM解説
Graph LSTM解説
 
Machine learning for materials design: opportunities, challenges, and methods
Machine learning for materials design: opportunities, challenges, and methodsMachine learning for materials design: opportunities, challenges, and methods
Machine learning for materials design: opportunities, challenges, and methods
 
SchNet: A continuous-filter convolutional neural network for modeling quantum...
SchNet: A continuous-filter convolutional neural network for modeling quantum...SchNet: A continuous-filter convolutional neural network for modeling quantum...
SchNet: A continuous-filter convolutional neural network for modeling quantum...
 
GraphSage vs Pinsage #InsideArangoDB
GraphSage vs Pinsage #InsideArangoDBGraphSage vs Pinsage #InsideArangoDB
GraphSage vs Pinsage #InsideArangoDB
 
論文紹介資料「Quantum Deep Field : Data-Driven Wave Function ...」
論文紹介資料「Quantum Deep Field : Data-Driven Wave Function ...」論文紹介資料「Quantum Deep Field : Data-Driven Wave Function ...」
論文紹介資料「Quantum Deep Field : Data-Driven Wave Function ...」
 
Deep Learning for Computer Vision: Medical Imaging (UPC 2016)
Deep Learning for Computer Vision: Medical Imaging (UPC 2016)Deep Learning for Computer Vision: Medical Imaging (UPC 2016)
Deep Learning for Computer Vision: Medical Imaging (UPC 2016)
 
PFP:材料探索のための汎用Neural Network Potential - 2021/10/4 QCMSR + DLAP共催
PFP:材料探索のための汎用Neural Network Potential - 2021/10/4 QCMSR + DLAP共催PFP:材料探索のための汎用Neural Network Potential - 2021/10/4 QCMSR + DLAP共催
PFP:材料探索のための汎用Neural Network Potential - 2021/10/4 QCMSR + DLAP共催
 
グラフ構造データに対する深層学習〜創薬・材料科学への応用とその問題点〜 (第26回ステアラボ人工知能セミナー)
グラフ構造データに対する深層学習〜創薬・材料科学への応用とその問題点〜 (第26回ステアラボ人工知能セミナー)グラフ構造データに対する深層学習〜創薬・材料科学への応用とその問題点〜 (第26回ステアラボ人工知能セミナー)
グラフ構造データに対する深層学習〜創薬・材料科学への応用とその問題点〜 (第26回ステアラボ人工知能セミナー)
 
Discovering new functional materials for clean energy and beyond using high-t...
Discovering new functional materials for clean energy and beyond using high-t...Discovering new functional materials for clean energy and beyond using high-t...
Discovering new functional materials for clean energy and beyond using high-t...
 

Similar to Overview of Machine Learning for Molecules and Materials Workshop @ NIPS2017

A semantic framework and software design to enable the transparent integratio...
A semantic framework and software design to enable the transparent integratio...A semantic framework and software design to enable the transparent integratio...
A semantic framework and software design to enable the transparent integratio...Patricia Tavares Boralli
 
deeplearningpresentation-180625071236.pptx
deeplearningpresentation-180625071236.pptxdeeplearningpresentation-180625071236.pptx
deeplearningpresentation-180625071236.pptxJeetDesai14
 
MADICES Mungall 2022.pptx
MADICES Mungall 2022.pptxMADICES Mungall 2022.pptx
MADICES Mungall 2022.pptxChris Mungall
 
Natural language processing and transformer models
Natural language processing and transformer modelsNatural language processing and transformer models
Natural language processing and transformer modelsDing Li
 
012517 ResumeJH Amex DS-ML
012517 ResumeJH Amex DS-ML012517 ResumeJH Amex DS-ML
012517 ResumeJH Amex DS-MLJeremy Hadidjojo
 
Machine Learning and Reasoning for Drug Discovery
Machine Learning and Reasoning for Drug DiscoveryMachine Learning and Reasoning for Drug Discovery
Machine Learning and Reasoning for Drug DiscoveryDeakin University
 
Towards reproducibility and maximally-open data
Towards reproducibility and maximally-open dataTowards reproducibility and maximally-open data
Towards reproducibility and maximally-open dataPablo Bernabeu
 
AI for automated materials discovery via learning to represent, predict, gene...
AI for automated materials discovery via learning to represent, predict, gene...AI for automated materials discovery via learning to represent, predict, gene...
AI for automated materials discovery via learning to represent, predict, gene...Deakin University
 
20090219 The case for another systems biology modelling environment
20090219 The case for another systems biology modelling environment20090219 The case for another systems biology modelling environment
20090219 The case for another systems biology modelling environmentJonathan Blakes
 
Mpp Rsv 2008 Public
Mpp Rsv 2008 PublicMpp Rsv 2008 Public
Mpp Rsv 2008 Publiclab13unisa
 
NS-CUK Seminar: H.B.Kim, Review on "metapath2vec: Scalable representation le...
NS-CUK Seminar: H.B.Kim,  Review on "metapath2vec: Scalable representation le...NS-CUK Seminar: H.B.Kim,  Review on "metapath2vec: Scalable representation le...
NS-CUK Seminar: H.B.Kim, Review on "metapath2vec: Scalable representation le...ssuser4b1f48
 
Continuous modeling - automating model building on high-performance e-Infrast...
Continuous modeling - automating model building on high-performance e-Infrast...Continuous modeling - automating model building on high-performance e-Infrast...
Continuous modeling - automating model building on high-performance e-Infrast...Ola Spjuth
 
Advanced machine learning for metabolite identification
Advanced machine learning for metabolite identificationAdvanced machine learning for metabolite identification
Advanced machine learning for metabolite identificationDai-Hai Nguyen
 
Interactive Analysis of Large-Scale Sequencing Genomics Data Sets using a Rea...
Interactive Analysis of Large-Scale Sequencing Genomics Data Sets using a Rea...Interactive Analysis of Large-Scale Sequencing Genomics Data Sets using a Rea...
Interactive Analysis of Large-Scale Sequencing Genomics Data Sets using a Rea...Dominic Suciu
 
00 Automatic Mental Health Classification in Online Settings and Language Emb...
00 Automatic Mental Health Classification in Online Settings and Language Emb...00 Automatic Mental Health Classification in Online Settings and Language Emb...
00 Automatic Mental Health Classification in Online Settings and Language Emb...Duke Network Analysis Center
 

Similar to Overview of Machine Learning for Molecules and Materials Workshop @ NIPS2017 (20)

A semantic framework and software design to enable the transparent integratio...
A semantic framework and software design to enable the transparent integratio...A semantic framework and software design to enable the transparent integratio...
A semantic framework and software design to enable the transparent integratio...
 
mlss
mlssmlss
mlss
 
deeplearningpresentation-180625071236.pptx
deeplearningpresentation-180625071236.pptxdeeplearningpresentation-180625071236.pptx
deeplearningpresentation-180625071236.pptx
 
MADICES Mungall 2022.pptx
MADICES Mungall 2022.pptxMADICES Mungall 2022.pptx
MADICES Mungall 2022.pptx
 
Natural language processing and transformer models
Natural language processing and transformer modelsNatural language processing and transformer models
Natural language processing and transformer models
 
012517 ResumeJH Amex DS-ML
012517 ResumeJH Amex DS-ML012517 ResumeJH Amex DS-ML
012517 ResumeJH Amex DS-ML
 
Machine Learning and Reasoning for Drug Discovery
Machine Learning and Reasoning for Drug DiscoveryMachine Learning and Reasoning for Drug Discovery
Machine Learning and Reasoning for Drug Discovery
 
Deep learning presentation
Deep learning presentationDeep learning presentation
Deep learning presentation
 
Towards reproducibility and maximally-open data
Towards reproducibility and maximally-open dataTowards reproducibility and maximally-open data
Towards reproducibility and maximally-open data
 
BioNLPSADI
BioNLPSADIBioNLPSADI
BioNLPSADI
 
AI for automated materials discovery via learning to represent, predict, gene...
AI for automated materials discovery via learning to represent, predict, gene...AI for automated materials discovery via learning to represent, predict, gene...
AI for automated materials discovery via learning to represent, predict, gene...
 
20090219 The case for another systems biology modelling environment
20090219 The case for another systems biology modelling environment20090219 The case for another systems biology modelling environment
20090219 The case for another systems biology modelling environment
 
Mpp Rsv 2008 Public
Mpp Rsv 2008 PublicMpp Rsv 2008 Public
Mpp Rsv 2008 Public
 
NS-CUK Seminar: H.B.Kim, Review on "metapath2vec: Scalable representation le...
NS-CUK Seminar: H.B.Kim,  Review on "metapath2vec: Scalable representation le...NS-CUK Seminar: H.B.Kim,  Review on "metapath2vec: Scalable representation le...
NS-CUK Seminar: H.B.Kim, Review on "metapath2vec: Scalable representation le...
 
Cv long
Cv longCv long
Cv long
 
CV_10/17
CV_10/17CV_10/17
CV_10/17
 
Continuous modeling - automating model building on high-performance e-Infrast...
Continuous modeling - automating model building on high-performance e-Infrast...Continuous modeling - automating model building on high-performance e-Infrast...
Continuous modeling - automating model building on high-performance e-Infrast...
 
Advanced machine learning for metabolite identification
Advanced machine learning for metabolite identificationAdvanced machine learning for metabolite identification
Advanced machine learning for metabolite identification
 
Interactive Analysis of Large-Scale Sequencing Genomics Data Sets using a Rea...
Interactive Analysis of Large-Scale Sequencing Genomics Data Sets using a Rea...Interactive Analysis of Large-Scale Sequencing Genomics Data Sets using a Rea...
Interactive Analysis of Large-Scale Sequencing Genomics Data Sets using a Rea...
 
00 Automatic Mental Health Classification in Online Settings and Language Emb...
00 Automatic Mental Health Classification in Online Settings and Language Emb...00 Automatic Mental Health Classification in Online Settings and Language Emb...
00 Automatic Mental Health Classification in Online Settings and Language Emb...
 

More from Kenta Oono

Minimax statistical learning with Wasserstein distances (NeurIPS2018 Reading ...
Minimax statistical learning with Wasserstein distances (NeurIPS2018 Reading ...Minimax statistical learning with Wasserstein distances (NeurIPS2018 Reading ...
Minimax statistical learning with Wasserstein distances (NeurIPS2018 Reading ...Kenta Oono
 
Comparison of deep learning frameworks from a viewpoint of double backpropaga...
Comparison of deep learning frameworks from a viewpoint of double backpropaga...Comparison of deep learning frameworks from a viewpoint of double backpropaga...
Comparison of deep learning frameworks from a viewpoint of double backpropaga...Kenta Oono
 
深層学習フレームワーク概要とChainerの事例紹介
深層学習フレームワーク概要とChainerの事例紹介深層学習フレームワーク概要とChainerの事例紹介
深層学習フレームワーク概要とChainerの事例紹介Kenta Oono
 
20170422 数学カフェ Part2
20170422 数学カフェ Part220170422 数学カフェ Part2
20170422 数学カフェ Part2Kenta Oono
 
20170422 数学カフェ Part1
20170422 数学カフェ Part120170422 数学カフェ Part1
20170422 数学カフェ Part1Kenta Oono
 
情報幾何学の基礎、第7章発表ノート
情報幾何学の基礎、第7章発表ノート情報幾何学の基礎、第7章発表ノート
情報幾何学の基礎、第7章発表ノートKenta Oono
 
GTC Japan 2016 Chainer feature introduction
GTC Japan 2016 Chainer feature introductionGTC Japan 2016 Chainer feature introduction
GTC Japan 2016 Chainer feature introductionKenta Oono
 
On the benchmark of Chainer
On the benchmark of ChainerOn the benchmark of Chainer
On the benchmark of ChainerKenta Oono
 
Tokyo Webmining Talk1
Tokyo Webmining Talk1Tokyo Webmining Talk1
Tokyo Webmining Talk1Kenta Oono
 
VAE-type Deep Generative Models
VAE-type Deep Generative ModelsVAE-type Deep Generative Models
VAE-type Deep Generative ModelsKenta Oono
 
Common Design of Deep Learning Frameworks
Common Design of Deep Learning FrameworksCommon Design of Deep Learning Frameworks
Common Design of Deep Learning FrameworksKenta Oono
 
Introduction to Chainer and CuPy
Introduction to Chainer and CuPyIntroduction to Chainer and CuPy
Introduction to Chainer and CuPyKenta Oono
 
Stochastic Gradient MCMC
Stochastic Gradient MCMCStochastic Gradient MCMC
Stochastic Gradient MCMCKenta Oono
 
Chainer Contribution Guide
Chainer Contribution GuideChainer Contribution Guide
Chainer Contribution GuideKenta Oono
 
2015年9月18日 (GTC Japan 2015) 深層学習フレームワークChainerの導入と化合物活性予測への応用
2015年9月18日 (GTC Japan 2015) 深層学習フレームワークChainerの導入と化合物活性予測への応用 2015年9月18日 (GTC Japan 2015) 深層学習フレームワークChainerの導入と化合物活性予測への応用
2015年9月18日 (GTC Japan 2015) 深層学習フレームワークChainerの導入と化合物活性予測への応用 Kenta Oono
 
Introduction to Chainer (LL Ring Recursive)
Introduction to Chainer (LL Ring Recursive)Introduction to Chainer (LL Ring Recursive)
Introduction to Chainer (LL Ring Recursive)Kenta Oono
 
日本神経回路学会セミナー「DeepLearningを使ってみよう!」資料
日本神経回路学会セミナー「DeepLearningを使ってみよう!」資料日本神経回路学会セミナー「DeepLearningを使ってみよう!」資料
日本神経回路学会セミナー「DeepLearningを使ってみよう!」資料Kenta Oono
 
提供AMIについて
提供AMIについて提供AMIについて
提供AMIについてKenta Oono
 
Chainerインストール
ChainerインストールChainerインストール
ChainerインストールKenta Oono
 
Caffeインストール
CaffeインストールCaffeインストール
CaffeインストールKenta Oono
 

More from Kenta Oono (20)

Minimax statistical learning with Wasserstein distances (NeurIPS2018 Reading ...
Minimax statistical learning with Wasserstein distances (NeurIPS2018 Reading ...Minimax statistical learning with Wasserstein distances (NeurIPS2018 Reading ...
Minimax statistical learning with Wasserstein distances (NeurIPS2018 Reading ...
 
Comparison of deep learning frameworks from a viewpoint of double backpropaga...
Comparison of deep learning frameworks from a viewpoint of double backpropaga...Comparison of deep learning frameworks from a viewpoint of double backpropaga...
Comparison of deep learning frameworks from a viewpoint of double backpropaga...
 
深層学習フレームワーク概要とChainerの事例紹介
深層学習フレームワーク概要とChainerの事例紹介深層学習フレームワーク概要とChainerの事例紹介
深層学習フレームワーク概要とChainerの事例紹介
 
20170422 数学カフェ Part2
20170422 数学カフェ Part220170422 数学カフェ Part2
20170422 数学カフェ Part2
 
20170422 数学カフェ Part1
20170422 数学カフェ Part120170422 数学カフェ Part1
20170422 数学カフェ Part1
 
情報幾何学の基礎、第7章発表ノート
情報幾何学の基礎、第7章発表ノート情報幾何学の基礎、第7章発表ノート
情報幾何学の基礎、第7章発表ノート
 
GTC Japan 2016 Chainer feature introduction
GTC Japan 2016 Chainer feature introductionGTC Japan 2016 Chainer feature introduction
GTC Japan 2016 Chainer feature introduction
 
On the benchmark of Chainer
On the benchmark of ChainerOn the benchmark of Chainer
On the benchmark of Chainer
 
Tokyo Webmining Talk1
Tokyo Webmining Talk1Tokyo Webmining Talk1
Tokyo Webmining Talk1
 
VAE-type Deep Generative Models
VAE-type Deep Generative ModelsVAE-type Deep Generative Models
VAE-type Deep Generative Models
 
Common Design of Deep Learning Frameworks
Common Design of Deep Learning FrameworksCommon Design of Deep Learning Frameworks
Common Design of Deep Learning Frameworks
 
Introduction to Chainer and CuPy
Introduction to Chainer and CuPyIntroduction to Chainer and CuPy
Introduction to Chainer and CuPy
 
Stochastic Gradient MCMC
Stochastic Gradient MCMCStochastic Gradient MCMC
Stochastic Gradient MCMC
 
Chainer Contribution Guide
Chainer Contribution GuideChainer Contribution Guide
Chainer Contribution Guide
 
2015年9月18日 (GTC Japan 2015) 深層学習フレームワークChainerの導入と化合物活性予測への応用
2015年9月18日 (GTC Japan 2015) 深層学習フレームワークChainerの導入と化合物活性予測への応用 2015年9月18日 (GTC Japan 2015) 深層学習フレームワークChainerの導入と化合物活性予測への応用
2015年9月18日 (GTC Japan 2015) 深層学習フレームワークChainerの導入と化合物活性予測への応用
 
Introduction to Chainer (LL Ring Recursive)
Introduction to Chainer (LL Ring Recursive)Introduction to Chainer (LL Ring Recursive)
Introduction to Chainer (LL Ring Recursive)
 
日本神経回路学会セミナー「DeepLearningを使ってみよう!」資料
日本神経回路学会セミナー「DeepLearningを使ってみよう!」資料日本神経回路学会セミナー「DeepLearningを使ってみよう!」資料
日本神経回路学会セミナー「DeepLearningを使ってみよう!」資料
 
提供AMIについて
提供AMIについて提供AMIについて
提供AMIについて
 
Chainerインストール
ChainerインストールChainerインストール
Chainerインストール
 
Caffeインストール
CaffeインストールCaffeインストール
Caffeインストール
 

Recently uploaded

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FIDO Alliance
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfFIDO Alliance
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...FIDO Alliance
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2DianaGray10
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaCzechDreamin
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxAbida Shariff
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxDavid Michel
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCzechDreamin
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
Buy Epson EcoTank L3210 Colour Printer Online.pdf
Buy Epson EcoTank L3210 Colour Printer Online.pdfBuy Epson EcoTank L3210 Colour Printer Online.pdf
Buy Epson EcoTank L3210 Colour Printer Online.pdfEasyPrinterHelp
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlPeter Udo Diehl
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationZilliz
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024Stephanie Beckett
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka DoktorováCzechDreamin
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfChristopherTHyatt
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsUXDXConf
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?Mark Billinghurst
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsStefano
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge
 

Recently uploaded (20)

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Buy Epson EcoTank L3210 Colour Printer Online.pdf
Buy Epson EcoTank L3210 Colour Printer Online.pdfBuy Epson EcoTank L3210 Colour Printer Online.pdf
Buy Epson EcoTank L3210 Colour Printer Online.pdf
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG Evaluation
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering Teams
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 

Overview of Machine Learning for Molecules and Materials Workshop @ NIPS2017

  • 1. Overview of Machine Learning for Molecules and Materials Workshop @ NIPS 2017 NIPS2017 @PFN Jan. 21st 2018 Preferred Networks, Inc. Kenta Oono oono@preferred.jp
  • 2. Kenta Oono (@delta2323_) • Preferred Networks (PFN), Engineer • MSc. in mathematics • 2014.10 - Present: PFN • Role – Biology project – Chainer developer – Chainer Chemistry developer
  • 3. Workshop overview • 15 invited talks, 22 posters, 3 sponsors • Session titles – Introduction to Machine Learning and Chemistry – Machine Learning Applications in Chemistry – Kernel Learning with Structured Data – Deep Learning Approaches • Areas of interest – ML + (Quantum) Chemistry / ML + Quantum Physics / Material Informatics – DL : Vinyals (DeepMind), Duvenaud (Google), Smola (Amazon)
  • 4. Why materials and molecules? • Material informatics – Material genome initiative – MI2I project (NIMS) • Drug discovery – Big pharmas’ investment – IPAB drug discovery contest https://medium.com/the-ai-lab/artificial-intelligence-in-drug-discovery-is- overhyped-examples-from-astrazeneca-harvard-315d69a7f863
  • 5. Chemical prediction - Two approaches • Quantum simulation – Theory-based approach – e.g. DFT (Density Functional Theory) J Precision is guaranteed L High calculation cost • Machine learning – Data-based approach – e.g. Graph convolution J Low cost, high speed calculation L Hard to guarantee precision “Neural message passing for quantum chemistry”Justin et al
  • 6. Hardness of learning with molecules • How to represent molecules? – Discrete and structured nature of molecules – 2D and 3D information • Vast search space (~10**60)
  • 7. Topics • Molecule generation with VAE • Graph convolution
  • 10. SMILES A format of encoding molecules in text. Simple solution: Treat a molecule as a sequential data and apply NLP techniques. OC[C@@H](O1)[C@@H](O)[C@H] (O)[C@@H](O)[C@@H](O)1
  • 11. Variational AutoEncoder (VAE) [Kingma+13][Rezende+14] Kingma, D. P., & Welling, M. (2013). Auto- encoding variational bayes. arXiv preprint arXiv:1312.6114. Rezende, D. J., Mohamed, S., & Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082. • Variational inference • Use NN as an inference model. • Train in end-to-end manner with backpropagation. • Extension to RNN encoder/decoder [Fabius+15] https://www.slideshare.net/KentaOono/vaetype- deep-generative-models z x z x approximate Inference model qφ(z | x) Generative model pθ (z | x)
  • 12. Molecule generation with VAE (CVAE) [Gómez-Bombarelli+16] • Encode and decode molecules represented as SMILE with VAE. • Latent representation can be used for semi- supervised learning. • We can use learned models to find molecule with desired property by optimizing representation in latent space and decode it. L generated molecules are not guaranteed to be valid syntactically. Gómez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J. M., Sánchez-Lengeling, B., Sheberla, D., ... & Aspuru- Guzik, A. (2016). Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science.
  • 13. Grammar VAE (GVAE) [Kusner+17] Kusner, M. J., Paige, B., & Hernández-Lobato, J. M. (2017). Grammar Variational Autoencoder. arXiv preprint arXiv:1703.01925. • Generate sequence of production rules of syntax of SMILES • Generated molecules are guaranteed to be valid syntactically. Encode Decode • Represent SMILES syntax as CFG • Convert a molecule to a parse tree to get a sequence of production rules. • Feed the sequence to RNN-VAE. L generated molecules are not guaranteed to be valid semantically.
  • 14. Syntax-Directed VAE (SDVAE) Best paper award • Use attribute grammar to guarantee that generated molecules are both syntactically and semantically valid. • Generate attributes stochastically (stochastic lazy attributes) for on-the- fly semantic check. ← Simplified schematic view (Note: Bottom up semantic check for explanation) http://www.quantum- machine.org/workshops/nips2017/assets/pdf/sd vae_workshop_camera_ready.pdf https://openreview.net/forum?id=SyqShMZRb
  • 15. Discussion • Is SMILES appropriate as an input representation? – Input representation is not unique (e.g. CC#C and C#CC represent same molecule). – Molecule representation is not guaranteed to be invariant to relabeling (i.e. permutation of indexes) of molecules. – SMILES is not natural language. Can we justify to apply NLP techniques? • Synthesizability is not considered.
  • 16. Related papers • Extension of VAE – Semi-supervised Continuous Representation of Molecules – Learning Hard Quantum Distributions With Variational Autoencoders • Seq2seq models – “Found in translation”: Predicting Outcomes of Complex Organic Chemistry Reactions Using Neural Sequence-to-sequence Models • Molecule generation – Learning a Generative Model for Validity in Complex Discrete Structure – ChemTS: de novo molecular generation with MCTS and RNN (for rollout)
  • 18. Extended Connectivity Fingerprint (ECFP) Convert molecule into fixed length bit representation J Pros • Calculation is fast • Show presence of particular substructures L Cons • Bit collision – Two (or more) different substructure features could be represented by the same bit position • Task-independent featurizer https://chembioinfo.com/2011/10/30/revisiting- molecular-hashed-fingerprints/ https://docs.chemaxon.com/display/docs/Extended +Connectivity+Fingerprint+ECFP
  • 19. How graph convolution works Graph convolution Convolution kernel depends on Graph structure Image class label Chemical property CNN on image
  • 20. Unified view of graph convolution Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017). Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212. Update Readout v w hw evw hv mv mv mv mv hv y Many message-passing algorithms (NFP, GGNN, Weave) are formulated as the iterative application of Update function and Readout function [Gilmer et al. 17]. Aggregates neighborhood information and updates node representations. Aggregates all node representations and updates the final output.
  • 21. Neural Fingerprint (NFP) [Duvenaud+15] Atom feature embedding Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T., Aspuru-Guzik, A.,&Adams, R. P. (2015). Convolutional networks on graphs for learning molecular fingerprints. In Advances in neural information processing systems (pp. 2224-2232). HCNOS
  • 22. Neural Fingerprint (NFP) Update hnew 3= σ ( W2(h3+h2+h4) ) hnew 7= σ ( W3(h7+h6+h8+h9) ) Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T., Aspuru-Guzik, A.,&Adams, R. P. (2015). Convolutional networks on graphs for learning molecular fingerprints. In Advances in neural information processing systems (pp. 2224-2232).
  • 23. Neural Fingerprint (NFP) Readout h7 h8 R = ∑i softmax (Whi) h6 h1 h2 h3 h4 h5 h9 h10
  • 25. Comparison between graph convolution networks NFP GGNN Weave SchNet How to extract atom features Man-made or Embed Man-made or Embed Man-made or Embed Man-made or Embed Graph convolution strategy Adjacent atoms only Adjacent atoms only All atom-atom pairs All atom-atom pairs How to represent connection information Degree Bond type Man-made pair features (bond type,distance etc.) Distance
  • 26. End-to-end Learning of Graph Neural Networks for Molecular Representation [Tsubaki+17] 1. Embed r-radius subgraphs 2. Update node and vertex representations 3. Use LSTM to capture long-term dependency in vertices and edges 4. Readout the final output with self-attention mechanism Best paper award https://www.dropbox.com/s/ujzuj2kd2nyz348/tsubaki_nips2017.pdf
  • 27. Extension to semi-supervised learning [Hai+17] Compute representations of subgraphs inductively with neural message passing (→) Optimize the representation in unsupervised manner in the same way as Paragraph vector (↓) Nguyen, H., Maeda, S. I.,&Oono, K. (2017). Semi-supervised learning of hierarchical representations of molecules using neural message passing.arXiv preprint arXiv:1711.10168. Workshop paper
  • 28. Chainer Chemistry (http://chainer-chemistry.readthedocs.io/) Chainer extension library for Biology and Chemistry FileParser (SDF, CSV) Loader (QM 9, Tox 21) Graph convolution NN (NFP, GGNN, SchNet, Weave) Preprocessing Example Multitask learning with QM9 / Tox21 Model Layer Dataset Pretrained Model Feature extractor (TBD) GraphLinear, EmbedAtomID Basic information Release:12/14/2017, Version: v0.1.0, License: MIT, Language: Python
  • 29. Discussion • Is message passing neural network general enough to formulate many graph convolution algorithms? • How can we incorporate 3D information to graph convolution algorithms (e.g. Chirality).
  • 30. Other topics (DNN models) • CNN models – ChemNet: A Transferable and Generalizable Deep Neural Network for Small-molecule Property Prediction – Ligand Pose Optimization With Atomic Grid-based Convolutional Neural Networks • Other DNN models – Deep Learning for Prediction of Synergistic Effects of Anti-cancer Drugs – Deep Learning Yields Virtual Assays – Neural Network for Learning Universal Atomic Forces
  • 31. Other topics • Chemical synthesis – Automatically Extracting Action Graphs From Materials Science Synthesis Procedures – Marwin Segler’s talk: Planning Chemical Syntheses with Neural Networks and Monte Carlo Tree Search • Bayesian optimization – Bayesian Protein Optimization – Constrained Bayesian Optimization for Automatic Chemical Design Segler, M. H., Preuss, M.,&Waller, M. P. (2017). Learning to Plan Chemical Syntheses. arXiv preprint arXiv:1708.04202.
  • 32. Summary • Data-driven approach for understanding molecules are being paid attention in material informatics, quantum chemistry, and quantum physics fields. • Recent advances of : – Molecule generation with VAE – Learning graph-structured data with graph convolution algorithms.
  • 34. Chainer Chemistry (http://chainer-chemistry.readthedocs.io/) Chainer extension library for Biology and Chemistry Basic information release:12/14/2017, version: v0.1.0, license: MIT, language: Python Features • State-of-the-art deep learning neural network models (especially graph convolutions) for chemical molecules (NFP, GGNN, Weave, SchNet etc.) • Preprocessors of molecules tailored for these models • Parsers for several standard file formats (CSV, SDF etc.) • Loaders for several well-known datasets (QM9, Tox21 etc.)
  • 35. Example: HOMO prediction with QM9 dataset # Dataset preprocessing (for NFP Network) preprocessor = preprocess_method_dict['nfp']() dataset = D.get_qm9(preprocessor, labels='homo’) # Cache dataset for second use NumpyTupleDataset.save('input/nfp_homo/data.npz', dataset) train, val = split_dataset_random(dataset, first_size=10000) # Build model and use as an ordinary Chain model = GraphConvPredictor(NFP(16, 16, 4), MLP(16, 1))