5月22日、研究所内勉強会の担当回だった時の資料(一部変更)。
画像説明文生成や真相学習を利用した画像生成など、以下のオーラル発表5本を紹介。
1. Show and Tell: A Neural Image Caption Generator
2. Long-term Recurrent Convolutional Networks for Visual Recognition and Description
3. Deep Visual-Semantic Alignments for Generating Image Descriptions
4. Deep Neural Networks are Easilly Fooled: High Confidence Predictions for Unrecognizable Images
5. Understanding Deep Image Representation by Inverting Them
Frontiers of Vision and Language: Bridging Images and Texts by Deep LearningYoshitaka Ushiku
Slide used on 11/11/2017 for the keynote in International Conference on Document Analysis and Recognition Workshop on Machine Learning.
(ICDAR WML 2017, https://icdarwml.wixsite.com/icdarwml2017)
This is a translated and updated version of https://www.slideshare.net/YoshitakaUshiku/deep-learning-73499744, which is written in Japanese.
Recognize, Describe, and Generate: Introduction of Recent Work at MILYoshitaka Ushiku
In English. (日本語解説文は下にあります。)
Some pagees and a Japanese version of this slide are used in
2017/04/29 The 11th Machine Learning 15minutes!
2017/05/11 GPU Technology Conference 2017@San Jose
This slide introduces the recent work of Machine Intelligence Laboratory (MIL), University of Tokyo, which I belong to as a lecturer.
2017/04/29 第11回 Machine Learning 15minutes!
2017/05/11 GPU Technology Conference 2017@San Jose
にて一部もしくは日本語版を使用。
2017年現在牛久が講師として所属している東京大学 Machine Intelligence Laboratory (MIL) においての、最近の研究成果をまとめたものです。
セル生産方式におけるロボットの活用には様々な問題があるが,その一つとして 3 体以上の物体の組み立てが挙げられる.一般に,複数物体を同時に組み立てる際は,対象の部品をそれぞれロボットアームまたは治具でそれぞれ独立に保持することで組み立てを遂行すると考えられる.ただし,この方法ではロボットアームや治具を部品数と同じ数だけ必要とし,部品数が多いほどコスト面や設置スペースの関係で無駄が多くなる.この課題に対して音𣷓らは組み立て対象物に働く接触力等の解析により,治具等で固定されていない対象物が組み立て作業中に運動しにくい状態となる条件を求めた.すなわち,環境中の非把持対象物のロバスト性を考慮して,組み立て作業条件を検討している.本研究ではこの方策に基づいて,複数物体の組み立て作業を単腕マニピュレータで実行することを目的とする.このとき,対象物のロバスト性を考慮することで,仮組状態の複数物体を同時に扱う手法を提案する.作業対象としてパイプジョイントの組み立てを挙げ,簡易な道具を用いることで単腕マニピュレータで複数物体を同時に把持できることを示す.さらに,作業成功率の向上のために RGB-D カメラを用いた物体の位置検出に基づくロボット制御及び動作計画を実装する.
This paper discusses assembly operations using a single manipulator and a parallel gripper to simultaneously
grasp multiple objects and hold the group of temporarily assembled objects. Multiple robots and jigs generally operate
assembly tasks by constraining the target objects mechanically or geometrically to prevent them from moving. It is
necessary to analyze the physical interaction between the objects for such constraints to achieve the tasks with a single
gripper. In this paper, we focus on assembling pipe joints as an example and discuss constraining the motion of the
objects. Our demonstration shows that a simple tool can facilitate holding multiple objects with a single gripper.
【DLゼミ】XFeat: Accelerated Features for Lightweight Image Matchingharmonylab
公開URL:https://arxiv.org/pdf/2404.19174
出典:Guilherme Potje, Felipe Cadar, Andre Araujo, Renato Martins, Erickson R. ascimento: XFeat: Accelerated Features for Lightweight Image Matching, Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
概要:リソース効率に優れた特徴点マッチングのための軽量なアーキテクチャ「XFeat(Accelerated Features)」を提案します。手法は、局所的な特徴点の検出、抽出、マッチングのための畳み込みニューラルネットワークの基本的な設計を再検討します。特に、リソースが限られたデバイス向けに迅速かつ堅牢なアルゴリズムが必要とされるため、解像度を可能な限り高く保ちながら、ネットワークのチャネル数を制限します。さらに、スパース下でのマッチングを選択できる設計となっており、ナビゲーションやARなどのアプリケーションに適しています。XFeatは、高速かつ同等以上の精度を実現し、一般的なラップトップのCPU上でリアルタイムで動作します。
2. 自己紹介
2014.3 博士(情報理工学)、東京大学
2014.4~2016.3 NTT CS研 研究員
2016.4~2018.9 東京大学 講師 (原田・牛久研究室)
2016.9~ 産業技術総合研究所 協力研究員
2016.12~2018.9 国立国語研究所 共同研究員
2018.10~ オムロンサイニックエックス株式会社
Principal Investigator
[Ushiku+, ACMMM 2012]
[Ushiku+, ICCV 2015]
画像キャプション生成 主観的な感性表現を持つ
画像キャプション生成
動画の特定区間と
キャプションの相互検索
[Yamaguchi+, ICCV 2017]
A guy is skiing with no shirt on
and yellow snow pants.
A zebra standing in a field with
a tree in the dirty background.
[Shin+, BMVC 2016]
A yellow train on the tracks near
a train station.
9. ユーザー生成コンテンツの爆発的増加
特にコンテンツ投稿・共有サービスでは…
• Facebookに画像が2500億枚 (2013年9月時点)
• YouTubeにアップロードされる動画
1分間で計400時間分 (2015年7月時点)
Pōhutukawa blooms this
time of the year in New
Zealand. As the flowers
fall, the ground
underneath the trees look
spectacular.
画像/動画と
関連する文章の対
→大量に収集可能
これらの背景から…
つぎのような様々な取り組みが!
10. 画像キャプション生成
Group of people sitting
at a table with a dinner.
Tourists are standing on
the middle of a flat desert.
[Ushiku+, ICCV 2015]
11. 動画キャプション生成
A man is holding a box of doughnuts.
Then he and a woman are standing next each other.
Then she is holding a plate of food.
[Shin+, ICIP 2016]
22. どれがどれくらい良いキャプションなのか?
CoSMoS [Ushiku et al., ICCV 2015]
Group of people sitting at a table with a dinner.
Corpus-Guided [Yang et al., EMNLP 2011]
Three people are showing the bottle on the street
Midge [Mitchel et al., EACL 2012]
people with a bottle at the table
アンケートによる比較:相対的な良さの評価
• 毎回ほかの手法と比較してもらわなければならない
• 絶対的なキャプションの良さの評価がほしい
23. 定量評価指標
機械翻訳では…
• テスト文に複数の参照訳が付随(通常5文)
• これらの参照訳と近い訳文が「良い」
• 既存の評価指標(BLEUやMETEOR、ROUGEなど)
• キャプション生成の評価指標(CIDErやSPICEなど)
One jet lands at an airport while another takes off next to it.
Two airplanes parked in an airport.
Two jets taxi past each other.
Two parked jet airplanes facing opposite directions.
two passenger planes on a grassy plain
キャプション生成の評価でも同様の流れ
PASCAL Sentenceの画像と参照キャプションの例
35. 問題の発展:キャプション列生成
アルバムのような系列画像に対して
The family
got
together for
a cookout.
They had a
lot of
delicious
food.
The dog
was happy
to be there.
They had a
great time
on the
beach.
They even
had a swim
in the water.
[Park+Kim, NIPS 2015][Huang+, NAACL 2016]
36. 問題の発展:キャプション列生成
A man is holding a box of doughnuts.
Then he and a woman are standing next each other.
Then she is holding a plate of food.
[Shin+, ICIP 2016]
シーンの切り替わる動画に対して
37. 問題の発展:キャプション列生成
A boat is floating on the water near a mountain.
And a man riding a wave on top of a surfboard.
Then he on the surfboard in the water.
[Shin+, ICIP 2016]
42. VQA: Visual Question Answering
• ビジュアル質問応答を分野として確立
– ベンチマークデータセットの提供
– ベースとなるパイプラインでの実験
• ポータルサイトも運営
– http://www.visualqa.org/
– 国際コンペティションも開催
[Antol+, ICCV 2015]
What color are her eyes?
What is the mustache made of?
48. VQA Challenge
コンペティション参加チームの解答例から
Q: Why is there snow on one
side of the stream and clear
grass on the other?
GT A: shade
Machine A: yes
Q: Is the hydrant painted a new
color?
GT A: yes
Machine A: no
54. Visual Question Generation
• 視覚的質問生成
(VQG)の提案
質問生成は画像
キャプションの
生成か検索を検討
• Qに対する要求
– その質問から会話が始まるような質問
– 画像を見てわかるような質問ではだめ
✗ How many horses
are in the field?
✓ Who won the race?
[Mostafazadeh+, ACL 2016]
55. 未知物体についてのVQG
画像認識器が知らない物体: 人から教わりたい
• 質問なら何でもいいわけじゃない
• 「なにこれ?」のような曖昧な質問だと…
回答も「物体」のように曖昧になりそう
• 学習して自動生成できた質問の例
What is the
woman
holding in
her right
hand?
What type
of shirt is
the man
wearing?
What in
on the
man’s
lap?
?
[Uehara+, ECCV 2018]
62. GuessWhat?!
連続するYes/No型のVQAデータ
Is it a person? No
Is it an item being worn or held? Yes
Is it a snowboard? Yes
Is it the red one? No
Is it the one being held by the Yes
person in blue?
Is it a cow? Yes
Is it the big cow in the middle? No
Is the cow on the left? No
On the right? Yes
First cow near us? Yes
[de Vries+, CVPR 2017]
63. GuessWhat?! の概要
• Questioner
– Guesserが何を見ているのかを知るために質問
• Guesser
– 自分が見ているものに応じてYes/Noで応答
• MS COCOを利用
– 画像数 64,000
– 対話 135,000
– 質問 673,000 Is it a vase? Yes
Is it partially visible? No
Is it in the left corner? No
Is it the turquoise and Yes
purple one?
[de Vries+, CVPR 2017]
65. Visual Dialog (VisDial)
Questioner Answerer
A couple of people
in the snow on skis.
What are their genders?
Are they both adults?
Do they wear goggles?
Do they have hats on?
Are there any other people?
What color is man’s hat?
Is it snowing now?
What is woman wearing?
Are they smiling?
Do you see trees?
1 man 1 woman
Yes
Looks like sunglasses
Man does
No
Black
No
Blue jacket and black pants
Yes
Yes
[Das+, CVPR 2017]
70. MNIST Dialog
VisDialのMNIST版
• 4x4のMNIST画像(白黒)に
– 文字色5種、背景色5種、スタイル2種を適用
– 画像数 50,000
– 対話 3 dialogs/image
– 質問 10 Q&As/dialog
How many 9’s are there in the image? four
How many brown digits are there among them? one
What is the background color of the digit at the left of it? white
What is the style of the digit? flat
What is the color of the digit at the left of it? blue
What is the number of the blue digit? 4
Are there other blue digits? two
[Seo+, NIPS 2017]
75. その他にも…
• Image-Grounded Conversation [Mostafazadeh+,
IJCNLP 2017]
– VisDialはVQAベース
No
Any huge pumpkins?
No
Do you see trees?
No
Do you see anyone?
That is possible
Do you think it's for Halloween?
Possibly
Is this at a farm?
No
Is the photo close up?
Yes
Is the photo in color?
76. その他にも…
• Image-Grounded Conversation [Mostafazadeh+,
IJCNLP 2017]
– VisDialはVQAベース
– 雑談の様な対話を収集
Place near my house is getting
ready for Halloween a little early.
Don't you think Halloween
should be year-round, though?
That'd be fun since it's
my favorite holiday!
It‘s my favorite holiday as well!
I never got around to carving a
pumpkin last year even though I
bought one.
Well, it's a good thing that they are
starting to sell them early this year!
77. その他にも…
• DialEdit [Ramesh+, 2018]
• Video Scene-Aware Dialog Data [Hori+, 2018]
– Dialog System Technology Challenge (DSTC) 7
– VisDialの動画バージョン
93. SL-Pretrained(教師あり事前学習のみ)
Questioner Answerer
A couple of people
in the snow on skis.
2
I can’t tell
I can’t tell, they are far away
I can’t tell
Yes
Yes
Yes
Yes
Yes
Yes
How many people?
Male or female?
What are they wearing?
What color are skis?
Are they wearing goggles?
Are they wearing goggles?
Are they wearing goggles?
Are they wearing goggles?
Are they wearing hats?
Are they wearing goggles?
94. RL-full-QAf(提案手法)
Questioner Answerer
A couple of people
in the snow on skis.
Are people male or female or both?
What are they wearing?
Is it snowing?
Can you see any trees?
Can you see any buildings in background?
Can you see any trees?
Can you see any buildings in background?
Can you see any trees?
Can you see any buildings in background?
Does it look like they are in park?
I think 1 is male, can’t see other 1 ...
They are all wearing snow pants and jackets
It does not appear to be snowing at moment
Yes, there are lot of trees in background
No, I can not
Yes, there are lot of trees in background
No, I do not see any buildings
Yes, there are lot of trees in background
No , I do not see any buildings
It does not appear to be
You can easily know the answer because the official site still has the information about ILSVRC 2012.
Yes, the 1st team with deep learning achieved 15% error, the 2nd team without deep learning achieved 26% error … and if you scroll down this web page, the members of the second team are shown in a table. There seems to be several guys in the second team, and now please remember this name. It is hard to pronounce. Yoshitaka Ushiku.
You can easily know the answer because the official site still has the information about ILSVRC 2012.
Yes, the 1st team with deep learning achieved 15% error, the 2nd team without deep learning achieved 26% error … and if you scroll down this web page, the members of the second team are shown in a table. There seems to be several guys in the second team, and now please remember this name. It is hard to pronounce. Yoshitaka Ushiku.