representation learning on irregularly structured input data such as graphs, point clouds, and manifolds (aka geometric deep learning)

1. Graph deep learning
aka geometric deep learning
(as of 20190919)
, Review papers
workshop
Representation learning on irregularly structured input
data such as graphs, point clouds, and manifolds

2. (as of 2019/9/19)
(Ullmann Nauty VF2 etc)
Discriminative/Contrast
/ /
etc
Weisfeiler-Lehman Extended
Connectivity Fingerprint(ECFP)
( etc)
Graph Embedding (DeepWalk LINE node2vec etc)
Graph Convolutions: Laplacian-based (Kipf & Welling or De errard )
Neural Fingerprint (Duvenaud ) Weave Module (Kearns ) etc
Scarselli Gated Graph Neural Networks
GraphSAGE(Hamilton )
Message Passing Neural Networks/MPNN(Gilmer+)
Non-local neural networks/NLNN(Wang+) Graph Networks(Battaglia+)
Graph Isomorphism Network (GIN) GNN
Attention Transformer ( )

3. http://tkipf.github.io/misc/SlidesCambridge.pdf

4. https://arxiv.org/pdf/1812.04202.pdf

5. (by )
http://www-erato.ist.hokudai.ac.jp/lecture2011/material/lecture-washio-2.pdf
https://www.slideshare.net/itakigawa/94-71669875
(< > - )
https://doi.org/10.11509/isciesci.60.3_107
https://www.slideshare.net/itakigawa/ss-79556659
Graph mining: procedure, application to drug discovery and recent advances
https://doi.org/10.1016/j.drudis.2012.07.016
( )
https://www.slideshare.net/itakigawa/ss-151297814

6. 1. Geometric deep learning: going beyond Euclidean data
(Bronstein+ IEEE Signal Processing Magazine, 2017)
https://doi.org/10.1109/MSP.2017.2693418
2. Representation Learning on Graphs: Methods and Applications
(Halilton+ IEEE Data Engineering Bulletin, 2017)
http://sites.computer.org/debull/A17sept/p52.pdf
3. Graph embedding techniques, applications, and performance: A survey
(Goyal+ Knowledge-Based Systems, 2018)
https://doi.org/10.1016/j.knosys.2018.03.022
4. Relational inductive biases, deep learning, and graph networks
(Battaglia+ arXiv:1806.01261, 2018)
https://arxiv.org/abs/1806.01261
5. Deep Learning on Graphs: A Survey
(Zhang+ arXiv:1812.04202, 2018)
https://arxiv.org/abs/1812.04202
6. Attention Models in Graphs: A Survey
(Lee+ arXiv:1807.07984, 2018)
https://arxiv.org/abs/1807.07984
7. A comprehensive survey on graph neural networks
(Wu+ arXiv:1901.00596, 2019)
https://arxiv.org/abs/1901.00596
8. Graph Neural Networks: A Review of Methods and Applications
(Zhou+ arXiv:1812.08434 , 2019)
https://arxiv.org/abs/1812.08434
(review papers)

7. Websites/Workshops
• NeurIPS2019 Workshop on Graph Representation Learning
https://grlearning.github.io/
• ICLR2019 Workshop on Representation Learning on Graphs and Manifolds (RLGM)
https://rlgm.github.io/overview/
• ICML2019 Workshop on Learning and Reasoning with Graph-Structured Representations
https://graphreason.github.io
• KDD2019 Workshop on Mining and Learning with Graphs (MLG)
http://www.mlgworkshop.org/2019/
• KDD2019 Workshop on Deep Learning on Graphs (DLG)
https://dlg2019.bitbucket.io
• SDM2019 Workshop on Deep Learning on Graphs
https://sites.google.com/view/graph-representation-workshop/
• Tutorials (geometricdeeplearning.com)
http://geometricdeeplearning.com/#portfolio
• Dive into Deep Learning
https://www.d2l.ai
• Practical Deep Learning for Coders
https://course.fast.ai
• Interpretable Machine Learning
https://christophm.github.io/interpretable-ml-book/
Online books (not graph-speci c)

8. List of papers ( )
• http://geometricdeeplearning.com/#bibliography
• https://paperswithcode.com/task/graph-neural-network
• Literature of Deep Learning for Graphs
https://github.com/DeepGraphLearning/LiteratureDL4Graph
• Awesome resources on Graph Neural Networks
https://github.com/nnzhan/Awesome-Graph-Neural-Networks
• Awesome Graph Classification
https://github.com/benedekrozemberczki/awesome-graph-classification
• Must-read papers on GNN
https://github.com/thunlp/GNNPapers
SOTA
• Browse > Graphs
https://paperswithcode.com/area/graphs
• Browse > Graphs > Graph Classification
https://paperswithcode.com/task/graph-classification
paperswithcode.com

9. Graph Nets library
https://github.com/deepmind/graph_nets
Pytorch geometric
https://github.com/rusty1s/pytorch_geometric
Deep Graph Library (DGL)
https://github.com/dmlc/dgl
Libraries/Frameworks
Chainer Chemistry
https://github.com/pfnet-research/chainer-chemistry
Kaggle:
• CHAMPS (CHemistry And Mathematics in Phase Space) competition
https://www.kaggle.com/c/champs-scalar-coupling/overview
• #1 Solution - hybrid
https://www.kaggle.com/c/champs-scalar-coupling/discussion/106575#latest-628267
• #2 solution 🤖 Quantum Uncertainty 🤖
https://www.kaggle.com/c/champs-scalar-coupling/discussion/106468#latest-618474
• 3rd solution - BERT in chemistry - End to End is all you need
https://www.kaggle.com/c/champs-scalar-coupling/discussion/106572#latest-620382

10. Attention Transformer
Google :
RNN(LSTM/GRU) CNN
RNN CNN Attention
Transformer !?
Google BERT
OpenAI GPT-2
Attention ...
( )
mX
i=1
↵ifi(x)
<latexit sha1_base64="GtBCcjWksxUuedJZIIM2Dm3iIpM=">AAACyHichVFNL8RQFD3qa3wPNhKbYULYTF6HhEgkwkasfA0SQ9PWG170K21nGM1sbCT+gIUViYWILT/Axh+w8BPEksTGwm2niSC4TfvOO/ee+87r1RxDeD5jjzVSbV19Q2Oiqbmlta29I9nZteLZRVfnOd02bHdNUz1uCIvnfOEbfM1xuWpqBl/VdmfC/GqJu56wrWW/7PANU922REHoqk+UkuzLe0VTCcSkXNk0U3nVcHZURaQKihjKa2awXxlWkmmWYVGkfgI5BmnEMW8n75HHFmzoKMIEhwWfsAEVHj3rkMHgELeBgDiXkIjyHBU0k7ZIVZwqVGJ36btNu/WYtWgf9vQitU6nGPS6pExhgD2wS/bC7tkVe2Lvv/YKoh6hlzKtWlXLHaXjuGfp7V+VSauPnU/Vn559FDAeeRXk3YmY8BZ6VV86OHlZmlgcCAbZOXsm/2fskd3RDazSq36xwBdP//CjkZff/1iYjytohPL3gf0EK9mMPJLJLoymp6bjYSbQi34M0cTGMIVZzCNHJxzhGje4leYkR9qTytVSqSbWdONLSIcfaXWkfQ==</latexit>
mX
i=1
↵i(x)fi(x)
<latexit sha1_base64="4Gy7Jf+OdRdINqIV17gPixBjMOc=">AAAC0HichVE9LwRRFD3G9/paNBLNxobQbN4gIRLJhkbJrkVimcyMt7yYr8zMbjDZiFajVKhIFKLWUmj8AcX+BFGSaBTuzE4iCO5k5p137j33nTdXcwzh+YzVGqTGpuaW1rb2REdnV3dPsrdvxbPLrs4Lum3Y7pqmetwQFi/4wjf4muNy1dQMvqrtzof51Qp3PWFby/6+wzdMddsSJaGrPlFKcqzolU0lELNyddNMFVXD2VEVMVrUzGCvOpYqfWIlmWYZFkXqJ5BjkEYci3byAUVswYaOMkxwWPAJG1Dh0bMOGQwOcRsIiHMJiSjPUUWCtGWq4lShErtL323arcesRfuwpxepdTrFoNclZQrD7JFdsRf2wK7ZE3v/tVcQ9Qi97NOq1bXcUXqOB/Jv/6pMWn3sfKr+9OyjhOnIqyDvTsSEt9Dr+srB6Ut+JjccjLAL9kz+z1mN3dMNrMqrfrnEc2d/+NHIy+9/LMzHFTRC+fvAfoKV8Yw8kRlfmkxn5+JhtmEQQxiliU0hiwUsokAnnOAGt7iTctKedCgd1UulhljTjy8hHX8An3GnuQ==</latexit>
(variation )
attention =
context
ICML2019 Tutorial https://icml.cc/media/Slides/icml/2019/halla(10-09-15)-10-15-45-4343-a_tutorial_on.pdf
GLUE SOTA
SQuAD SOTA
CMU XLNet
Microsoft MT-DNN
2018/10/18
2019/01/31
(!)
2019/02/14
2019/06/19

11. DL (Kaggler !?)
Meta learning (Learning to Learn) Metric Learning
• Siamise Networks/Contrastive Loss
• Triplet network/Triplet Loss
• L2 Softmax Loss
• ArcFace / CosFace / SphereFace Loss
• AdaCos Loss
• Matching Networks (Vinyals+ NIPS2016)
• Prototypical Networks (Snell+ NIPS2017)
• MAML (Finn+ ICML2017)
• SNAIL (Mishra+ ICLR2018)
• TADAM (Oreshkin+ NeurIPS2018)
• MTL (CVPR2019)
n-shot k-way learning ( / or )
n /k (e.g. zero-shot/one-shot/few-shot learning)
semi-supervised learning, con dent learning (noisy labels), imitation learning,
multi-task/multi-label, generative learning, adversarial learning, normalization, ...
CNN techniques
FishNet, Squeeze-Exitation(SE), GeM pooling, CBAM, Shake-Shake, Cyclic pooling, ...
Loss
CELoss vs BCE Loss, Focal Loss, Lovasz Loss, Dice Loss, macro F1 score, top-k BCE, ...
Ensembling / LR scheduling / HPO / NAS
FGE (Fast Geometric Ensemble), SWA (Stochastic Weight Averaging), ...