2019年9月に開催された日本バイオインフォマティクス学会2019年年会 第8回生命医薬情報学連合大会（IIBMP2019）のスポンサーセッション講演資料です。 ・鈴木脩司「Preferred Networksについて」 ・大野健太「深層学習フレームワーク Chainer」 ・菅原洋平「深層学習初学者向け無料オンライン教材の紹介」 ・中郷孝祐「化学・生物学分野のための深層学習ライブラリ ChainerChemistry」 ・太田健「ハイパーパラメータ自動最適化フレームワーク Optuna」

1. Preferred Networks :
P

2. 1. Preferred Networks
2. Chainer
3.
4. Chainer Chemistry
5. Optuna

4. 11
PFN careers

5. Preferred Networks(PFN)
Automobile
Factory
Robot
Manufacturing
Bio /
Healthcare
Personal
Robot
Creative
2014 3
N 270 2019 7
NTT Fanuc Toyota DYHD
JXTG
•
•

6. ● Mechanical Engineering
● Electrical Engineering
● Navigation
● Manipulation
● Embedded Systems
● SLAM / 3D Reconstruction
● Computer Vision
● Simulation
● Reinforcement Learning
● Data Collection / Annotation
● Speech Recognition
● Operating System
● Natural Language Processing(NLP)
● Human–Computer Interaction(HCI)
(Machine
Learning)
(Deep
Learning)
● User Interface (UI)
● Computer Network
● Motion Planning
● IoT data analysis
● Generative models
● 3D computer
graphics (CG)
● Full-stack dev.
● Bioinformatics
● Biology/Chemistry
● Medical Images
Researcher & Engineer
+ α

7. ● 2017 12
DL
MN-1/1b/2
2560 GPU

8. 2019
●
2018
●
● ODSC East 2018, the Open Source Data Science
Project Award
● CEATEC Award
2017
●
● Japan-U.S. Innovation Awards Emerging Leader
Award
● FT ArcelorMittal Boldness in Business Awards
2016
● JEITA
● Forbes JAPAN’s CEO OF THE YEAR 2016
1

11. (PFN Visual Inspection)
•

12. Crypko

13. 14
PFDeNA
(PFN DeNA )
14

14. RSNA Pneumonia Detection Challenge
Kaggle X
X

15. IT
• IT
•
– (Sirtuin 1)

16. Chainer
Preferred Networks
(oono@preferred.jp)
2019 9 10
8 IIBMP2019

17. ü NN Python
→ NN
ü Define-by-Run NN
→ Python
→ NN
ü CuPy NumPy-like GPU
→ CPU/GPU agnostic
Speed up research and development of deep learning and its applications.

18. Awards
ODSC East 2018
Open Source Data Science Project
2018

19. https://www.youtube.com/watch?v=yNc5N1MOOt4 https://www.youtube.com/watch?v=lGOjchGdVQs

20. Wikipedia: Electrocardiography
AI
AI

21. Design networks Train and evaluate
networks
Dataset

22. Neural Network as a Computational Graph
• NN
• 2 DAG
y = x1 * x2
z = y - x3
x1 mul suby
x3
z
x2

23. Define-By-Run
1: x = Variable(np.array(3))
3
5*2 -1
- 4
**2
x z
6
9
3
x
y
3 5*2 -1
x
62: y = 2 * x - 1
3: z = x ** 2 - y
y

24. 9
!, "
For # = 1, … , ##)*+,)#-.:
Get a minibatch (01, 21) (1 = 1, … , #4,)5ℎ7#8*)
9 = ∑;<=
#>?@ABCDEF
∇" !(0;, 2;; ")
" ← " – K9
Output " • L
• θ
Stochastic Gradient Descent, SGD

25. Backpropagation.
10
y = x1 * x2
ga * z * a ga ∇a z
x1 mul suby
x3
z
x2
z = y - x3
gx1 = gy * x2
gx2 = gy * x1
gz = 1
gy = gz
gx3 = -gz
gzgy
gx3
gx1
gx2

26. Multi Layer Perceptron; MLP
x Linear
W1 b1
h1 ReLU a1
Linear
W2 b2
h2
Soft
max
prob
Cross
Entropy
loss
t
11
1
2

27. 12
class MLP(chainer.Chain):
def __init__(self):
super(MLP, self).__init__()
with self.init_scope():
self.fc1 = L.Linear(None, 100)
self.fc2 = L.Linear(100, 10)
def forward(self, x):
a1 = F.relu(self.fc1(x))
h2 = self.fc2(a1)
return h2
model = MLP()
model = L.Classifier(model)
x Linear
W1 b1
h1 ReLU
a1 Linear
W2 b2
h2

28. # Setup Trainer
trainer = training.Trainer(
updater, stop_trigger=(100, 'epoch’))
trainer.extend(extensions.ProgressBar())
trainer.run()
# Parameter Updater
opt = optimizers.SGD(lr=0.01)
opt.setup(model)
updater = training.StandardUpdater(it, opt)
13
# Dataset Preparation
dataset = [(x1, t1), (x2, t2), ...]
it = iterators.SerialIterator(dataset)
Example
https://github.com/pfnet/chainer/tree/master/examples
• (__len__ __getitem__
)
•
• Optimizer, Updater
•
• Extension Trainer
• Extension

29. Caffemodel
Caffe .caffemodel
VGG16, 19, GoogLeNet, ResNet50, 101, 152
model = links.ResNet50Layers(pretrained_model=‘auto')
image = np.random.uniform(0, 1, (224, 224, 3)).astype(np.float32)
image = prepare(image)
with chainer.using_config('train', False):
with chainer.using_config('enable_backprop', False):
feature = model.extract([image], layers=['pool5'])
• NoneImageNet ResNet
•

30. ChainerCV
• Detection: FasterRCNNVGG16, SDD300, 512, YOLOv2, v3
• Semantic Segmentation: SegNetBasic, PSPNetResNet101
• Instance Segmentation: FCISResNet101
• Classification: SEResNet50, 101, 152, SEResNeXt50, 101

31. CuPy GPU
ü NumPy API
ü NVIDIA GPU
cuBLAS, cuDNN, cuRAND, cuSPARSE, and NCCL
ü
ElementwiseKernel, ReductionKernel
To GPU
To CPU

32. ChainerMN
optimizer = chainer.optimizers.MomentumSGD()
optimizer = chainermn.DistributedOptimizer(
chainer.optimizers.MomentumSGD())
ResNet-50 ImageNet
ImageNet 1024GPU, 15
[Akiba et al., 17] [Tokui et al. KDD19]

33. Chainer Family
Chainer UI Chainer Chemistry

34. • Chianer define-by-run
•
Caffe, Theano
• NN Chainer
https://chainer.org

35. Preferred Networks
(suga@preferred.jp)
2019 9 10
8 IIBMP2019

36. 1.
u ~ DL ~
2.
u AI
u Chainer Tutorial

37. 1.
u ~ DL ~
2.
u AI
u Chainer Tutorial

38. PFN
Humanoid Robot
Consumer Industrial
Cloud
Device
PhotoGame
Text
Speech
Infrastructure
Factory Robot
Automotive
Smart City
Industry4.0
Industrial IoT
Healthcare

39. …

40. DL (1)
DeepVariant ⇒ CNN
(Shanrong Zhao et al., Cloud Computing for Next-Generation Sequencing Data Analysis, 2017 )
(Poplin, Ryan et al., “A universal SNP and small-indel variant caller using deep neural networks.”, 2018 )
•
• DeepVariant (https://github.com/google/deepvariant)
Pileup image

41. DL (2)
AlphaFold
CASP (Critical Assessment of
techniques for protein Structure
Prediction)
2
(http://predictioncenter.org/casp13/doc/CAS
P13_Abstracts.pdf )

42. PFN

43. 1.
u ~ DL ~
2.
u AI
u Chainer Tutorial

44. AI
• AI AI
• https://japan-medical-ai.github.io/medical-ai-course-materials/
• GitHub https://github.com/japan-medical-ai/medical-ai-course-materials
• AI AI https://www.japan-medical-ai.org/?page_id=26
•
• Python Google Colaboratory
1.
2.
3.

45. 1)
• Google Colaboratory Web
• Google
• GPU
1. https://colab.research.google.com/
2. Colaboratory ( )
3. GPU
4.
5.
Colab

46. 2)
•
•
•
•
•
•
•
•
•
•
•
• NumPy
•
• Scikit-learn
• Chainer
1
•
•
•
2
• /
• NumPy
• Scikit-learn
3
•
• NumPy
4 Deep Learning
•
• Chainer

47. 3)
(Sequential regulatory activity prediction across chromosomes with convolutional neural networks, D. R. Kelly and et al., 2018)
5 MRI
• MRI
6
•
- “ ”
- ” ”
- ” ”
( )
7 DL
• DNA
8
• /

48. 1.
u ~ DL ~
2.
u AI
u Chainer Tutorial

49. Chainer Tutorial
•
• Python
1.
2.
3. AI
•
• AI
• Python Pandas Matplotlib

50. Chainer Tutorial
• 2019 4 10
• (8 )
• 101 views
• 13 users
• AI
( )
• Python
• Cupy
• Pandas
• Matplotlib
•
• …

51. •
•
• AI
• Chainer Tutorial

52. Chainer Chemistry
Preferred Networks
2019 9 10
8 (IIBMP2019)

53. Chainer Chemistry: Chainer
• Deep learning ( Graph Convolution)
–
https://github.com/pfnet-research/chainer-chemistry

54. Chainer Chemistry: Chainer
• OSS
– https://github.com/pfnet-research/chainer-chemistry

55. ● Data driven
● Deep learning
○ Graph Convolution
● Chainer Chemistry
○
○
●
○
○

56. →
A 1.8
B 8.8
C 3.2
D
E
F

57. →
D 4.4
E 6.3
F 10.5
A 1.8
B 8.8
C 3.2

58. VS
DFT (Density Functional Theory) MD
→ Pros: ( )
Cons:
→ Pros:
Cons: /
“Neural message passing for quantum chemistry” Justin et al

59. Deep
learning
Graph Convolution

60. Graph convolution ~ ~
CNN: Convolutional Neural Network
class label
Chemical
property
Graph convolutional network

61. Graph convolutional Neural Network
Graph convolution
Graph
Conv
Graph
Conv
Graph
Conv
Graph
Conv
Graph
Readout
Linear Linear
Graph convolutional Neural Network
Graph Graph→

62. ( )
C
N
O
1.0 0.0 0.0 6.0 1.0
atom type
0.0 1.0 0.0 7.0 1.0
0.0 0.0 1.0 8.0 1.0
charge
chirality
Man-made features
Molecular Graph Convolutions: Moving Beyond Fingerprints
Steven Kearnes, Kevin McCloskey, Marc Berndl, Vijay Pande, Patrick Riley arXiv:1603.00856

63. Graph Convolution: ( )
Graph Convolution
Han Altae-Tran, Bharath Ramsundar, Aneesh S. Pappu, & Vijay Pande (2017). Low Data Drug
Discovery with One-Shot Learning. ACS Cent. Sci., 3 (4)

64. Graph Readout: ( )
Han Altae-Tran, Bharath Ramsundar, Aneesh S. Pappu, & Vijay Pande (2017). Low Data Drug
Discovery with One-Shot Learning. ACS Cent. Sci., 3 (4)

65. Graph convolutional Neural Network
Graph convolution
Graph
Conv
Graph
Conv
Graph
Conv
Graph
Conv
Graph
Readout
Linear Linear
Graph convolutional Neural Network
Graph Graph→

66. Chainer Chemistry

67. ●
○
●
○ Graph Convolution
●
○ Imbalanced
○

68. File Parser
(SDF file, CSV file) QM 9, Tox21, molnet dataset
Graph convolution NN
GraphConvolution (Upate)
GraphReadout
Preprocessing
(NFP, GGNN, SchNet)
Example
Train and prediction
with
QM9/tox21/molnet
dataset
Model
Layer/Function
Dataset
Pretrained
Model
(TBD)
Preprocessor (Feature Extractor)

69. Dataset introduction - tox21
# of Dataset: Train 11757, Validation 295, Test 645
Label - Following 12 types of toxity is included:
'NR-AR', 'NR-AR-LBD', 'NR-AhR', 'NR-Aromatase', 'NR-ER', 'NR-ER-LBD',
'NR-PPAR-gamma', 'SR-ARE', 'SR-ATAD5', 'SR-HSE', 'SR-MMP', 'SR-p53'
Example:
SMILES:
C(=O)C1(O)Cc2c(O)c3c(c(O)c2C(OC2CC
(N)C(O)C(C)O2)C1)C(=O)c1c(O)cccc1C3
=O
LABEL: [ 0 1 -1 1 -1 1 -1 -1 1 -1 1 1]
SMILES:
CCCOc1ccc(C(=O)CCN2CCCCC2)cc1.Cl
LABEL: [ 0 0 0 -1 1 0 0 -1 -1 -1 0 0]
SMILES:
CCOP(=S)(OCC)SC(CCl)N1C(=O)c2cccc
c2C1=O
LABEL: [ 0 0 1 0 1 1 0 1 0 0 -1 -1]
SMILES: O=c1c(O)c(-
c2ccc(O)cc2)oc2cc(O)cc(O)c12
LABEL: [ 0 0 1 -1 1 1 -1 0 0 0 1 0]
2948 3895 6558 7381

70. Dataset introduction - molnet
http://moleculenet.ai/datasets-1
Chainer Chemistry molecutenet
molnet:
• PubChem PDBbind
•

71. Dataset introduction - molnet
http://moleculenet.ai/datasets-1

72. Chainer Chemistry
As of 2019/9/9 from https://github.com/pfnet-research/chainer-chemistry#supported-models
• NFP: Neural Fingerprint [2, 3]
• GGNN: Gated Graph Neural Network [4, 3]
• WeaveNet [5, 3]
• SchNet [6]
• RSGCN: Renormalized Spectral Graph Convolutional Network [10]
• RelGCN: Relational Graph Convolutional Network [14]
• GAT: Graph Attention Networks [15]
• GIN: Graph Isomorphism Networks [17]
• MPNN: Message Passing Neural Networks [3]
• Set2Set [19]
• Graph Warp Module (GWM) [18]-attached models
… →

74. 4 IT (2017 )
• Sirtuin 1
250
•
• 400
200
• 2-stage
– Thermal Shift Assay
– Inhibitory assay → IC50 measurement Sirtuin 1
http://www.ipab.org/eventschedule/contest/contest4

75. Ours Average
(18 teams in total)
1st screening (TSA) 23 / 200 (11.5%) 69 / 3559 (1.9 %)
2nd screening (IC50) 1 5
We found one hit compound and
won one of Grand prize (IPAB )

76. Kaggle Molecular property prediction
Kaggle:
CHAMPS
NMR Coupling constant
https://www.kaggle.com/toshik/schnet-starter-kit

77. •
• JXTG
… and more!

79. BayesGrad
“BayesGrad: Explaining Predictions of Graph Convolutional Networks”
Akita et al., ICONIP 2018
https://arxiv.org/abs/1807.01985
https://github.com/pfnet-research/bayesgrad
Dropout Saliency

80. GWM
“Graph Warp Module: an Auxiliary Module for Boosting the Power of Graph
Neural Networks in Molecular Graph Analysis” Ishiguro et al.
https://arxiv.org/abs/1902.01020
https://github.com/pfnet-research/chainer-chemistry
Graph Convolution Neural Network

81. GraphNVP
“GraphNVP: An Invertible Flow Model for Generating Molecular Graphs”
Madhawa et al.
https://arxiv.org/abs/1905.11600
https://github.com/pfnet-research/graph-nvp
Flow

82. Optuna
Preferred Networks
2019 9 10
8 ( IIBMP2019 )

83. ●
● :
○ Define-by-Run API
○
○
● Chainer
2
https://optuna.org

84. ●
● Optuna
● :
○ Define-by-Run
○
○
●
3

86. 5
● :
○
○ :
● :
○

87. :
•
•
6
×
PASCAL VOC2007

88. 7

89. 8
● SGD
● Momentum SGD
● Adam
● etc

90. 9
● SGD
● Momentum SGD
○ learning rate (LR) = ?
○ momentum = ?
● Adam
● etc

91. :
• LR: 0.1
• Dropout: 0.5
… !
Accuracy: 0.6
Trial 1
?
• LR: 0.01
• Dropout: 0.0
… !
Accuracy: 0.5
Trial 2
• LR: 0.05
• Dropout: 0.3
… !
Accuracy: 0.8
Trial 3
10

92. :
• LR: 0.1
• Dropout: 0.5
… !
Accuracy: 0.6
Trial 1
?
• LR: 0.01
• Dropout: 0.0
… !
Accuracy: 0.5
Trial 2
?
• LR: 0.05
• Dropout: 0.3
… !
Accuracy: 0.8
Trial 3
11

93. Optuna

94. 13
●
○ : f(X) → y
■ X:
■ y:
○ : argmin f(X)
■ X f(X)
■ f(X) X
f

95. : (x - 2)2
14

96. : (x - 2)2
15

97. : (x - 2)2
16
trial

98. : (x - 2)2
17
(SuggestAPI)

99. : (x - 2)2
18

100. : (x - 2)2
19
study

101. : (x - 2)2
20

102. : (x - 2)2
21
● :
○ (x): 1.99016
○ : 0.00009

103. : (x - 2)2
22
● :
○ TPE: Tree-structured Parzen Estimator
○
○

104. Define-by-Run

105. Define-by-Run API
24
● Define-by-Run:
○
○ : Chainer, PyTorch, TensorFlow Eager
●

106. :
25
0, 5, 3, 8, …
MNIST ( )
From: https://en.wikipedia.org/wiki/MNIST_database

107. 26

108. 27

109. Define-by-Run
28
:
● Python
●

111. (pruning)
30
•

112. 31
• SVHN CNN
•
(Successive Halving)
SVHN: The Street View Housing Numbers Dataset
CNN: Convolutional Neural Network

113. 32

115. 34
•
•

116. • study :
○ study_name: study
○ storage: (RDB) URL
⇒ study
35

117. 36
RDB
Storage
(e.g., MySQL)
RDBURLStudy

119. Kaggle: Open Images Challenge 2018
38
•
• NMW
mAP
○ NMW: Non-Maximum Weighted suppression
○ mAP: mean Average Precision

120. : HPL
39
• HPL (High Performance Linpack)
○
○ (MN-1b)
• :
○ E.g., LU Broadcast
○
⇒ Optuna

122. 41
•
• Optuna
•

123. Make your optimization optimized!
42