paper review of "Scene Completion Using Millions of Photographs" https://www.youtube.com/watch?v=sNSo_CSWBmE

1. IMAGE COMPLETION
Mark Chang

2. Scene Completion Using Millions of
Photographs
• James Hays & Alexei A. Efros
• Carnegie Mellon University
• ACM Transactions on Graphics (SIGGRAPH 2007)
• http://graphics.cs.cmu.edu/projects/scene-completion/

3. Scene Completion Using Millions of
Photographs
照⽚片 輸⼊入 輸出
類似圖⽚片圖庫
影像融合影像檢索

4. 使⽤用圖庫
⽤用圖⽚片中的其他地⽅方填補 ⽤用不同圖⽚片填補
原圖 輸⼊入影像

5. 影像檢索
找到符合語意的圖⽚片找到不符合語意的圖⽚片

6. 影像融合
圖⽚片1
圖⽚片2 融合成功
融合失敗

7. 詳細過程介紹
• 圖庫取得：Flickr Image
• 影像檢索：Gist Descriptor
• 影像融合：Graph Cut & Poisson Blending

8. 圖庫取得
• Photo from Flickr.com
• 2.3 million unique images
• 396 gigabytes of JPEG compressed data

9. 影像檢索
• 取出影像特徵（Gist Descriptor）
• 找尋特徵較接近的影像（Euclidean Distance）
Euclidean
Distance
Gist
Descriptor

10. 情景（Scene）
• 情景相近：圖⽚片內容物語意相近
• 不需要特別辨識個別物體。
⼤大致情境 詳細內容（不符語意）

11. Modeling the Scene

12. Fourier Transform
X(!1, !2) =
1X
n1= 1
1X
n2= 1
x(n1, n2)e j(!1n1+!2n2)

13. Fourier Transform

14. Receptive Fields

15. Gist Descriptor

16. Gist Descriptor

17. 影像遮罩

18. 影像檢索
• 以量取勝
• 照⽚片越多，越能找到適⽤用的照⽚片

19. 影像融合
融合⾊色調和材質
Poisson Blending
得出適當的邊界
Graph Cut

20. Graph Cut
合併後的圖⽚片 圖⽚片接合處圖⽚片1 圖⽚片2

21. Graph Cut
cost(s, t, A, B) = |A(s) B(s)| + |A(t) B(t)|
s
A(s) A(t)
B(t)B(s)
t

22. Graph Cut
cost(s, t, A, B) = |A(s) B(s)| + |A(t) B(t)|
A(s)
A(t)
B(t)
B(s)
A(s)
A(t)
B(t)
B(s)
cut
overlap

23. Graph Cut
cost =
X
p
Cd(p, L(p)) +
X
p,q
Ci(p, q, L(p), L(q))
L(p) =
(
patch
exist
(
C(p, patch) = (k ⇤ Dist(p, hole))3
C(p, exist) = 1 if p in hole
hole
exist patch

24. Graph Cut
cost(s, t, A, B) = |A(s) B(s)| + |A(t) B(t)|
原本的 graph-cut：融合失敗
轉換到 gradient domain：融合成功
圖⽚片1 圖⽚片2
cost(s, t, A, B) = |rA(s, t) rB(s, t)|

25. Poisson Blending

26. Poisson Blending
• v : Gradient of a region in an image
• g : Selected region of source
• f* : Known pixel values in domain S
• f : Unknown pixel values in domain Ω
• Ω : Region g placed on domain S
• ∂Ω : Boundaries
min
ZZ
⌦
|rf v|2
with f|@⌦ = f⇤
|@⌦

27. Poisson Blending : 1D Example
min((f2 f1 1)2
+ (f3 f2 ( 1))2
+ (f4 f3 2)2
+ (f5 f4 ( 1))2
+ (f6 f5 ( 1))2
)
min(2f2
2 2f2f3 16f2 + 2f2
3 + 6f3 2f3f4 + 2f2
4 2f4f5 6f4 + 2f2
5 2f5 + 59)
with f1 = 6, f6 = 1
min
ZZ
⌦
|rf v|2
with f|@⌦ = f⇤
|@⌦

28. Poisson Blending : 1D Example
dQ
df2
= 4f2 2f3 16
dQ
df3
= 2f2 + 4f3 2f4 + 6
dQ
df3
= 2f3 + 4f4 2f5 6
dQ
df3
= 2f4 + 4f5 2
8
>>><
>>>:
f2 = 6
f3 = 4
f4 = 5
f5 = 3
2
6
6
4
4 2 0 0
2 4 2 0
0 2 4 2
0 0 2 4
3
7
7
5
2
6
6
4
f2
f3
f4
f5
3
7
7
5 =
2
6
6
4
16
6
6
2
3
7
7
5
source target

29. 實驗結果
• 成功案例
• 失敗案例

30. 成功案例⼀一

31. 成功案例⼆二

32. 成功案例三

33. 失敗案例⼀一：邊界太明顯

34. 失敗案例⼆二：找錯影像

35. 失敗案例三：未補完原圖內容

36. ⽤用原圖材質填補，效果⽐比較好
原圖 ⽤用原圖材質填補 ⽤用其他圖⽚片填補

37. 近期相關研究
• Deep Learning
• Supervised Semantics-Preserving Deep Hashing for Large-Scale
Image Search
http://arxiv.org/pdf/1507.00101.pdf

38. 延伸應⽤用
• 全景圖產⽣生器
• 透視圖產⽣生器

39. 全景圖產⽣生器
Automatic Generation of An Infinite Panorama
http://cybertron.cg.tu-berlin.de/pdci09/infinite_panorama/
http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj_final/www/lisachan/

40. 透視圖產⽣生器
http://parosky.net/projects/undresser.html
http://parosky.net/assets/data/anoron2013w.pdf
画像補完技術による衣服の除去

41. 延伸閱讀
• Scene Completion
• James Hays & Alexei A. Efros. Scene Completion Using Millions of
Photographs.
• GIST Descriptor
• Aude Oliva & Antonio Torralba. Modeling the Shape of the Scene: A
Holistic Representation of the Spatial Envelope.
• Aude Oliva & Antonio Torralba. Building the gist of a scene: the role of
global image features in recognition.
• Graph Cut
• Vivek Kwatra, Arno Schodl, Irfan Essa, Greg Turk and Aaron Bobick.
Graphcut Textures: Image and Video Synthesis Using Graph Cuts.
• Poisson Blending
• Patrick Perez, Michel Gangnet and Andrew Blake. Poisson Image
Editing.
• http://eric-yuan.me/poisson-blending

42. 圖⽚片來源
• James Hays & Alexei A. Efros. Scene Completion Using Millions of Photographs.
• Aude Oliva & Antonio Torralba. Modeling the Shape of the Scene: A Holistic
Representation of the Spatial Envelope.
• Aude Oliva & Antonio Torralba. Building the gist of a scene: the role of global image
features in recognition.

43. 圖⽚片來源
• Vivek Kwatra, Arno Schodl, Irfan Essa, Greg Turk and Aaron Bobick. Graphcut
Textures: Image and Video Synthesis Using Graph Cuts.
• Patrick Perez, Michel Gangnet and Andrew Blake. Poisson Image Editing.
• http://eric-yuan.me/poisson-blending
• http://d.hatena.ne.jp/LM-7/20090629/1246282979