1. May 13th, 2019
Tatsuya Shirakawa
Seeing Unseens with Machine Learning
⾒えていないものを⾒出す機械学習
Tech-on MeetUp#06 — What can “AI (I)” do?

2. Tatsuya Shirakawa
2
ABEJA, Inc. (Researcher)
- Deep Learning (CV, Graph, NLP, )
- Machine Learning
Github https://github.com/TatsuyaShirakawa
NTT Data Mathematical Systems Inc.
- Mathematical Optimization
- Machine Learning / Deep Learning
Math.
Tech blog http://tech-blog.abeja.asia/
- 異空間への埋め込み！Poincare Embeddingsが拓く表現学習の新展開
- 機は熟した！グラフ構造に対するDeep Learning、Graph Convolutionのご紹介
- 双曲空間でのMachine Learningの最近の進展
- より良い機械学習のためのアノテーションの機械学習
Now

3. Researchers at ABEJA
3
1.
2.
3.
4.
先に⾒つける
シンプルに解く
先に失敗する
• 最新テクノロジーのキャッチアップ
• 技術視点を交えた新しいビジネス構想
• 独⾃技術の開発・検証
• ⾼難易度タスクのコアロジックの構築
• 技術ソリューションの提案
• プロダクトの根本的な精度改善
• アイデアの検証
• 既存のやり⽅/考え⽅の再検討
視点を与える

4. AIの特性ふりかえり
4
MLについて⾔いたいこと
AI
ML
DL

5. Daniel Kehneman
There are two modes of thought
System 1（勘・直感）
fast, instinctive and emotional
System 2（論理的思考）
Slower, more deliberative, and more logical
5
MLはコッチ

6. 6(Andrew Ng, “AI Transformation Playbook”)
AIは運⽤することで
改善する

7. AIは⼈間の代替ではない
7
Human AI
同じ作業を続ける ✔
スケールさせる ✔
未知な状況への適応 ✔
複雑な作業 ✔
適材適所が⼤事

8. Today’s Talk
1. Software 2.0
2. Bigger, Deeper and Better
3. Discovery 2.0
8
1. Software 2.0
2. Bigger, Deeper and Better
3. Discovery 2.0

9. 9

10. Software 2.0
Software 1.0 — Write a program that works
- Explicit instructions to the computer which identifies a specific point in
program space with some desirable behavior
Software 2.0 — Find a desirable program that fits to data
- A rough skelton of the code (e.g. NNs) that identifies a subset of program
space to search
- Search this space for a program that works
10

11. Why Software 2.0?
"it is significantly easier to collect the data (or more generally, identify a
desirable behavior) than to explicitly write the program”
11

12. Dogs and Cats
Why dogs are dogs and cats are cats?
12

13. Gender, Age, Recognition is Not Trivial !
13

14. Paradigm Change
Things which is hard to define/code can be learn implicitly from data
14
00110110110
11010101010
11010111011
10110110100
01010111011
Coding Learn
Software

15. Today’s Talk
1. Software 2.0
2. Bigger, Deeper and Better
3. Discovery 2.0
15

16. Bigger, Deeper and Better
16
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018.9)
BigGAN — 巨⼤な計算リソースで学習された巨⼤なモデルで⾼解像度画像の⽣成に成功。
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (2018.11)
GPipe — 巨⼤なNNを効率的に学習するための分散学習ライブラリ。ImageNetで新SOTA。
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018.10)
BERT — 巨⼤なモデルを巨⼤なデータで教師なしすることで⾔語理解系タスクにたいする強⼒な初期モデルを獲得
Language Models are Unsupervised Multitask Learners (2019.2)
GPT-2 — 巨⼤な⾔語モデルをクリーンで巨⼤なデータで学習し、⽂書⽣成系タスクをゼロショットで⾼精度にこなせるモデルを獲得

17. BigGAN — Class Conditionalな⾼解像度画像⽣成
既存のSOTA⼿法（SA-GAN）に対して、バッチサイズやチャンネル数を増やし、各種⼯夫を加え
ることで、512x512のClass Conditionalな⾼精度画像⽣成に成功。既存SOTAを⼤きく上回るスコ
アを達成。
17“Large Scale GAN Training for High Fidelity Natural Image Synthesis ”

18. GPipe — 巨⼤なNNの学習に最適化された分散学習ライブラリ
複数のGPUを活⽤してForward/Backward計算をスケーラブルかつ効率的に⾏うライブラリ。こ
れを⽤いて学習された巨⼤なモデルはImageNetで新たなSOTAを達成。
18(“GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism”)

19. BERT — ⾔語理解系タスクでの強⼒な教師なし事前学習⼿法
強⼒なモデル（BERT）を教師なしで構成できる下記の2タスクで事前学習することで
さまざまな⾔語理解系タスクでSOTAを⼤幅更新
19
The cat [MASK] on the mat
sat
1. 単語の⽳埋め
GLUE test results (論⽂より）
1. The man went to [MASK] store
2. He bought a gallon [MASK] milk
→ IsNext / NotNext?
2. ⼆⽂が連続⽂かどうかの判定

20. GPT-2 — クリーンで多様なデータで学習された巨⼤な⾔語モデル
信頼性の⾼いWebページをクローリングして得たクリーンで多様なコーパス（WebText）上で
強⼒な⾔語モデル（GPT-2）を教師なし学習（尤度最⼤化）。
⽂書⽣成系のさまざまなタスクのZero-shot学習でSOTAを更新
20

21. 21
Winning Way
Larger (Cleaner) Datasets
+ Deeper Neural Networks

22. Today’s Talk
1. Software 2.0
2. Bigger, Deeper and Better
3.Discovery 2.0
22

23. Can You See Gender/Age from Ears?
23
0 10 20 30 40 50 60 70 80
Age

24. Can You See Gender, Age and BMI from Eyes (Fundus)?
How about Heart / Brain Diseases?
24
0 10 20 30 40 50 60 70 80
Age

25. DNNs Can See Gender/Age from Ears
D. Yaman+, “Age and Gender Classification from Ear Images”, IWBF2018
25
Age: 18-28 / 29-38 / 39-48 / 49-58 / 58-68+

26. DNNs Can See Gender, Age, BMI and even
Brain/Heart Diseases from Eyes
R. Poplin+, “Prediction of cardiovascular risk factors from retinal fundus
photographs via deep learning”, Nature Biomedical Engineering 2018
26

27. What happens?
DNNs can see what human cannot see or recognize.
27
…

28. Discovery 2.0
Discovery 1.0 — Fully Utilizing Domain Knowledge
- explicit construction of hypothesis is constructed mainly from
domain knowledge or deep understanding of the domain
Discovery 2.0 — Seeing by Training
- capture some aspects of data by training models on it
- not new but should be emphasized again
28
※ serendipity could be another source of discovery :)

29. Discovery 2.0 — Seeing by Training
1. Seeing Predictability / Correlation
29

30. Discovery 2.0 — Seeing by Training
1. Seeing Predictability / Correlation
Beyond human imagination
- Every data should be connect to create new connections
Correlation first
- Correlation finding is the first goal
- Causality should be checked post-hook if possible
Relatively cheap to apply if data exists
- Models should have weak domain dependence (e.g. NNs)
30

31. Graph Convolutional Neural Networks (GCNNs)
A specific type of neural networks which is
designed for processing connectivity of data well
31
Tech blog http://tech-blog.abeja.asia/
- 異空間への埋め込み！Poincare Embeddingsが拓く表現学習の新展開
- 機は熟した！グラフ構造に対するDeep Learning、Graph Convolutionのご紹介
- 双曲空間でのMachine Learningの最近の進展
- より良い機械学習のためのアノテーションの機械学習

32. Task Relations — Taskonomy
Relation = Transferability
A. R. Zamir+, “Taskonomy: Disentangling Task Transfer Learning”, CVPR2018
32
Autoencoding
Object Class.
Scene Class.
Curvature
Denoising
Occlusion Edges
Egomotion
Cam. Pose (fix)
2D Keypoint
3D Keypoint
Cam. Pose (nonfix)
Matching
Reshading
Distance
Z-Depth
Normals
Layout
2.5D Segm.
2D Segm.
Semantic Segm.
Vanishing Pts.
Novel Task 1
Novel Task 2
Novel Task 3
Autoencoding
Object Class.
Scene Class.
Curvature
Denoising
Occlusion Edges
Egomotion
Cam. Pose (fix)
2D Keypoint
3D Keypoint
Cam. Pose (nonfix)
Matching
Reshading
Distance
Z-Depth
Normals
Layout
2.5D Segm.
2D Segm.
Semantic Segm.
Vanishing Pts.
Novel Task 1
Novel Task 2
Novel Task 3
https://storage.googleapis.com/taskonomy_slides/taskonomy_slides.html

33. Discovery 2.0 — Seeing by Training
1. Seeing Predictability / Correlation
2. Representation Learning / Embeddings
33
T. Mikolov+, “Distributed representation of words and phrases and their
compositionality, NeurIPS2013
https://github.com/facebookresearch/poincare-embeddings

34. Hyperbolic Space
• Manifolds with positive constant sectional curvature
• Tree structure is naturally aligned in the space
→ automatic tree structure detection!
34
Tech blog http://tech-blog.abeja.asia/
- 異空間への埋め込み！Poincare Embeddingsが拓く表現学習の新展開
- 機は熟した！グラフ構造に対するDeep Learning、Graph Convolutionのご紹介
- 双曲空間でのMachine Learningの最近の進展
- より良い機械学習のためのアノテーションの機械学習
「異空間散歩！双曲空間を歩いてみよう。」

35. Hyperbolic Embeddings
35
[M. Nickel+]
Poincaré Embeddings ~
17’ 05 18’ 04
[C. D. Sa+]
Representation Tradeoff ~
(Near-exact tree embs., h-MDS)
[O. Ganea+]
Hyperbolic Entailment Cones
(Poincare embs. + Order embs.)
18’ 05
[C. Gulcehre+]
Hyperbolic Attention Networks
(Einstein Mid. Point)
[O. Ganea+]
Hyperbolic Neural Networks
[M. Nickel+]
Learning Continuous Hierarchies
in the Lorentz Model~
[A. Tifrea+]
Poincaré Glove ~
(Poincaré Glove)
18’ 06 18’ 10 …19’ 2
[R. Suzuki+]
Hyperbolic Disk Embeddings
[A. Gu+]
Mixed-Curvature
Representations
18’ 9

36. Mixed-Curvature Representations
A. Gu+, “Learning Mixed-Curvature Representations in Products of model Spaces”, ICLR2019
ユークリッド空間、球⾯、双曲空間の積空間への埋め込みを構成することで、
様々な（断⾯）曲率の空間への埋込を可能にした。
36

37. 37
データの構造が
⾒えた！

38. Discovery 2.0 — Seeing by Training
1. Seeing Predictability / Correlation
2. Representation Learning / Embeddings
※ Off course, domain/scientific knowledge is crucial for efficient/meaningful exploration
38

39. What can AI (I) do?
39
00110110110
11010101010
11010111011
10110110100
01010111011
Coding Learn
Software
Discovery
Dom
ain
Know
ledge
Learn
Software 1.0 Software 2.0
Discovery 1.0
Discovery 2.0

40. What can AI (I) do?
40
Psychology
x
AI

41. ⼈格⼼理学（Personality Psychology）
Personality Psychology is a scientific study which aims to show how people
are individually different due to psychological forces (wikipedia).
41
Personality Traits（特性）
Features
Personality Types（類型）
Clustering / Classification
あなたは◯◯タイプ！
ex) ex)

42. Big 5（Five Factor Model, FFM）
42
1. Openness（経験への開放性）
is a general appreciation for art, emotion, adventure,
unusual ideas, imagination, curiosity, and variety of experience
2. Conscientiousness（誠実性）
is a tendency to display self-discipline, act dutifully, and strive
for achievement against measures or outside expectations
3. Extraversion（外向性）
is characterized by breadth of activities (as opposed to depth), surgency
from external activity/situations, and energy creation from external means
4. Agreeableness（協調性）
trait reflects individual differences in general concern for social harmony
5. Neuroticism（神経症的傾向）
is the tendency to experience negative emotions, such as anger, anxiety, or depression
(wikipedia)

43. Big5はすごい！
• さまざまな研究で提案されたパーソナリティ特性との相関性が⾼い（事実上の
デファクトスタンダード）
• 英語辞書中のパーソナリティに関する単語と既存パーソナリティテストの結果
を総合して作られた、けっこうデータ駆動な作られ⽅
• 双⼦の研究によると、Big5の変動のだいたい50%は遺伝で、残りの50%は環境
で決まっている（分散分析）
• Big5のうち、Agreeablenessをのぞいた4つは、年をとってもあまり変化しない
43

44. What can I do?
1. Personality embeddings (in hyperbolic spaces?)
2. Multimodal analysis (facial expressions, psychological measures, )
3. More efficient assessor (like Akinator?)
脳科学、遺伝学、進化論、⼼理学、…、機械学習の融合領域！
44

45. 45
Annotation Driven AI