1. Image and Video
Phylogeny Reconstruction
Filipe de Oliveira Costa
Advisor: Prof. Anderson Rocha
Co-advisor: Prof. Zanoni Dias

2. Outline
• Introduction
• Image Phylogeny
• Video Phylogeny
• Conclusions
2

3. Introduction
3

4. Multimedia document
Modified versions
Source: Google Images
Dark side of the moon internet
4

5. Near-duplicate detection
5
Near-duplicate
detection

6. Multimedia Phylogeny
6

7. Multimedia Phylogeny
7

8. 8
Multimedia Phylogeny
Phylogeny tree
Phylogeny forest

9. Main Objective
Design and develop solutions that allow us to identify
the structure (phylogeny tree / forest) that represents
the generation process overtime of images and videos
9

10. Contributions
• Image phylogeny
• Approaches to image phylogeny forest
reconstruction
• New dissimilarity measures for image phylogeny
• Video Phylogeny
• Phylogeny reconstruction for misaligned and
compressed video sequences
10

11. Image Phylogeny

12. Image Phylogeny
• Divided into two basic steps
• Dissimilarity calculation
• Phylogeny Reconstruction
12

13. Dissimilarity Calculation
Itgt
d(Isrc, Itgt)
13
d(Isrc, Itgt) = min
T!2T
|Itgt T!(Isrc)| point-wise comparison L.
Isrc

14. Dissimilarity Calculation
Geometric matching
14
Itgt
Isrc

15. Dissimilarity Calculation
Color and compression matching
Isrc Itgt
R
G
B
Compression
15

16. Dissimilarity Calculation
Point-wise comparison
(Mean Squared Error)
I0
src
L
16
Itgt

17. Phylogeny Reconstruction
A B C D E
A - 2 1 3 4
B 4 - 1 2 6
C 3 4 - 1 2
D 4 7 4 - 4
E 5 7 8 3 -
Dissimilarity
matrix
17
A
C
E D
B
Phylogeny tree
2
2
1
1

18. Phylogeny Reconstruction
18
• Oriented Kruskal (OK) [Dias et al. (2010)]
• Based on the classic Kruskal's Minimum
Spanning Tree algorithm
• Adapted for oriented graphs
• Optimum Branching (OB) [Edmonds (1957)]
• Always returns the optimum tree

19. • Image Phylogeny Forests Reconstruction
F. O. Costa, M. Oikawa, Z. Dias, S. Goldenstein, and A. Rocha. Image phylogeny forests
reconstruction. IEEE Transactions on Information Forensics and Security (TIFS), vol.9, no.
10, pp. 1533–1546, 2014
• New dissimilarity measures
F. O. Costa, A. Oliveira, P. Ferrara, Z. Dias, S. Goldenstein, and A. Rocha. New dissimilarity
measures for image phylogeny reconstruction. Submitted to Springer Pattern Analysis and
Applications, 2016
Image Phylogeny
Contributions
19

20. Image Phylogeny Forests
Reconstruction
20

21. 21
Image Phylogeny Forests
Reconstruction
Phylogeny tree
Phylogeny forest

22. Image Phylogeny Forests
Reconstruction
Dias et al.(2013b): Automatic Oriented Kruskal (AOK)
• Adaptation of the OK algorithm for forests
• Select the edges in crescent order
• Stops selecting edges according to a threshold
22

23. Hypothesis #1
Using exact approaches for phylogeny forests
reconstruction problem is at least as effective
than using heuristic-based approaches
23

24. Image Phylogeny Forests
Reconstruction
We designed and developed an algorithm for forests
reconstruction
• Automatic Optimum Branching (AOB)
• Adaptation of the OB algorithm
24

25. Automatic Optimum Branching
(AOB)
Optimum
Branching
25
Dissimilarity matrix
M 0 1 2 3 4 5 6 7 8 9
0 - 52 29 57 43 55 24 52 49 26
1 50 - 45 35 60 68 40 42 63 53
2 42 48 - 56 42 54 28 52 50 29
3 56 23 55 - 65 67 50 22 66 58
4 45 61 35 67 - 29 42 63 43 52
5 25 69 65 70 27 - 48 68 45 65
6 26 50 30 52 39 50 - 50 48 48
7 51 32 49 30 61 67 49 - 62 55
8 52 65 55 69 24 30 49 64 - 26
9 25 54 26 59 50 61 30 56 30 -
17
3
6
8
4
5
2
9
0
40
2322
26
26
26
30
24
29

26. Step X Y d(X,Y) Diff
1 3 7 22 - -
2 3 1 23 1 -
3 8 4 24 1 1,41
4 9 2 26 2 2,00
5 6 0 26 0 3,42
6 0 9 26 0 3,58
7 4 5 29 3 3,52
8 9 8 30 1 4,68
9 1 6 40 10 5,53
Automatic Optimum Branching
(AOB)
aob = 2.026
17
3
6
8
4
5
2
9
0
40
2322
26
26
26
30
24
29
aob ⇥

27. Automatic Optimum Branching
(AOB)
aob = 2.027
17
3
6
8
4
5
2
9
0
2322
26
26
26
30
24
29
Forest: [6, 3, 9, 3, 8, 4, 6, 3, 9, 0]
Cost: 206
Dissimilarity matrix
M 0 1 2 3 4 5 6 7 8 9
0 - 52 29 57 43 55 24 52 49 26
1 50 - 45 35 60 68 40 42 63 53
2 42 48 - 56 42 54 28 52 50 29
3 56 23 55 - 65 67 50 22 66 58
4 45 61 35 67 - 29 42 63 43 52
5 25 69 65 70 27 - 48 68 45 65
6 26 50 30 52 39 50 - 50 48 48
7 51 32 49 30 61 67 49 - 62 55
8 52 65 55 69 24 30 49 64 - 26
9 25 54 26 59 50 61 30 56 30 -

28. M 1 3 7 0 2 4 5 6 8 9
1 - 35 42 - - - - - - -
3 23 - 22 - - - - - - -
7 32 30 - - - - - - - -
0 - - - - 29 43 55 24 49 26
2 - - - 42 - 42 54 28 50 29
4 - - - 45 35 - 29 42 43 52
5 - - - 25 65 27 - 48 45 65
6 - - - 26 30 39 50 - 48 48
8 - - - 52 55 24 30 49 - 26
9 - - - 25 26 50 61 30 30 -
Extended Automatic Optimum
Branching (E-AOB)
28
17
3
6
8
4
5
2
9
0
Dissimilarity matrix

29. Extended Automatic Optimum
Branching (E-AOB)
Forest: [5, 3, 9, 3, 8, 4, 0, 3, 9, 0]
Cost: 204
29
17
3 6
8
4
5
2
9
0
2322
26
26
26
30
24
29
M 1 3 7 0 2 4 5 6 8 9
1 - 35 42 - - - - - - -
3 23 - 22 - - - - - - -
7 32 30 - - - - - - - -
0 - - - - 29 43 55 24 49 26
2 - - - 42 - 42 54 28 50 29
4 - - - 45 35 - 29 42 43 52
5 - - - 25 65 27 - 48 45 65
6 - - - 26 30 39 50 - 48 48
8 - - - 52 55 24 30 49 - 26
9 - - - 25 26 50 61 30 30 -
Dissimilarity matrix

30. Image Phylogeny Forests
Reconstruction
• Main approaches
• AOK: Greedy approach
• AOB/E-AOB: Exact approach
30

31. Hypothesis #2
It is possible to lead to complementary
properties of different forest reconstruction
algorithm and combine them for improving the
quality of the phylogeny forest reconstruction
31

32. Fusion approach
32
M
Additive noise
100
variations

33. Fusion approach
Tree count
V = [1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5]
Median(V) = (2 + 2)/2 = 2
100
variations
…
AOK
0
35
1
4
7
8
2
9
6
100
variations
…
AOB
0
3 5
1
4
7
8
2
9
6
100
variations
…
E-AOB
0
35
1
4 7
8
2
9
6
33

34. Fusion approach
100
variations
…
AOK
0
35
1
4
7
8
2
9
6
100
variations
…
AOB
0
3 5
1
4
7
8
2
9
6
100
variations
…
E-AOB
0
35
1
4 7
8
2
9
6
Sum of votes of each node
[260, 250, 90, 30, 35, 24, 0, 0, 0, 0]
Roots {0, 1}
34

35. 0
351
4 7
8
2
9
6
Fusion approach
100
variations
…
AOK
0
35
1
4
7
8
2
9
6
100
variations
…
AOB
0
3 5
1
4
7
8
2
9
6
M 0 1 2 3 4 5 6 7 8 9
0 - - 0 270 0 286 0 0 0 0
1 - - 0 0 255 0 0 280 0 0
2 - - - 0 0 0 130 0 270 300
3 - - 280 - 0 0 0 0 30 0
4 - - 0 0 - 0 0 20 0 0
5 - - 20 0 0 - 0 0 0 0
6 - - 0 0 0 0 - 0 0 0
7 - - 0 0 10 0 0 - 0 0
8 - - 0 0 0 0 0 0 - 0
9 - - 0 0 0 0 170 0 0 -
100
variations
…
E-AOB
0
35
1
4 7
8
2
9
6
Final forest
[0, 1, 3, 0, 1, 0, 2, 1, 2, 2]
35

36. Experiments
36

37. Datasets
37
• 36,000 test cases
• [2..10] trees
• Ground truth is
available
• One Camera and
Multiple Cameras

38. Datasets
38
• Transformations
• Crop
• Resize
• Brightness, contrast
and gamma correction
• Rotation
• JPEG Compression

39. 10
6
7
9
8
25
3
11 7
Evaluation
39
2
8 9 103
65
4
39
4
Ground truth Reconstructed forest
Roots

40. 3 10
6
71
2
8
95
1 7
Evaluation
40
2
8 9 103
65
4
40
4
Ground truth Reconstructed forest
Edges

41. 10
6
71
2 9
3
1 7
Evaluation
41
2
8 9 103
65
4
41
8
5
4
Ground truth Reconstructed forest
Leaves

42. 7
10
6
1
83
25
1 7
Evaluation
Ancestry
42
2
8 9 103
65
4
42
9
4
Ground truth Reconstructed forest

43. Results - One Camera
43
2 3 4 5 6 7 8 9 10
# of trees
0.65
0.7
0.75
0.8
0.85
0.9
0.95
Score
Root - One Camera
AOK
AOB
E-AOB
FUSION
2 3 4 5 6 7 8 9 10
# of trees
0.65
0.7
0.75
0.8
0.85
0.9
0.95
Score
Ancestry - One Camera
AOK
AOB
E-AOB
FUSION
Error Reduction (E-AOB vs. AOK): 19% Error Reduction (E-AOB vs. AOK): 18%
Error Reduction (FUSION vs. AOK): 17% Error Reduction (Fusion vs. AOK): 17%

44. Results - Multiple Cameras
44
2 3 4 5 6 7 8 9 10
# of trees
0.65
0.7
0.75
0.8
0.85
0.9
0.95
Score
Root - Multiple Cameras
AOK
AOB
E-AOB
FUSION
2 3 4 5 6 7 8 9 10
# of trees
0.65
0.7
0.75
0.8
0.85
0.9
0.95
Score
Ancestry - Multiple Cameras
AOK
AOB
E-AOB
FUSION
Error Reduction (E-AOB vs. AOK): 20% Error Reduction (E-AOB vs. AOK): 19%
Error Reduction (FUSION vs. AOK): 28% Error Reduction (Fusion vs. AOK): 21%

45. New dissimilarity measures
for image phylogeny reconstruction
45

46. New dissimilarity measures
for image phylogeny reconstruction
• Previous work: Mean Squared Error (MSE)
• Problems
• Few information about the content of the
images
• Small misaligned caused on mapping can affect
the results
46

47. Hypothesis #3
The comparison of the distributions of the
pixel values of two near-duplicate images is
more effective for the dissimilarity calculation
than their point-wise comparison
47

48. Gradient point-wise comparison
MSE
Gradient
Computation
I0
src Itgt
48
Gradient
Computation

49. Mutual Information comparison
49
MI(I0
src, Itgt)
I0
src Itgt
MI(I0
src, Itgt) = H(Itgt) H(Itgt|I0
src)
H(X) = Ex[log(P(X))]

50. Comparing gradients with
mutual information
Mutual
Information
I0
src Itgt
MSE
50
X
Gradient
Computation
Gradient
Computation

51. Color matching
• Previous work
• Normalization based on mean and standard
deviation of another image
• Problem
• Unexpected Artifacts
• New color matching approach
• Histogram matching [Gonzales (2007)]
51

52. Histogram color matching
52
0 20 40 60 80 100 120 140 160 180 200 220 240 255
i
0
0.025
0.05
0.075
0.1
0.125
0.15
p(i)
Histogram - SRC
0 20 40 60 80 100 120 140 160 180 200 220 240 255
i
0
0.025
0.05
0.075
0.1
0.125
0.15
p(i)
Histogram - TGT
Histogram

53. 0 20 40 60 80 100 120 140 160 180 200 220 240 255
i
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
CDF(i)
CDF - SRC
0 20 40 60 80 100 120 140 160 180 200 220 240 255
i
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
CDF(i)
CDF - TGT
99 114
Histogram color matching
53
200 238
20 22
Cumulative Distribution Function

54. Histogram color matching
54
0 20 40 60 80 100 120 140 160 180 200 220 240 255
i
0
0.025
0.05
0.075
0.1
0.125
0.15
p(i)
Histogram - TGT
0 20 40 60 80 100 120 140 160 180 200 220 240 255
i
0
0.025
0.05
0.075
0.1
0.125
0.15
p(i)
Result of historgram matching
Resultant histogram

55. Color matching approaches
Result: Mean/STD Result: Histogram
Isrc Itgt
Color
Matching
Result: Mean/STD maching Result: Histogram maching
55

56. Experiments
56

57. Experimental setup
• Reconstruction
• Extended AOB
• Evaluation
• Roots, edges, leaves and ancestry
57

58. Roots - One Camera
Accuracy
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
Number of trees
1 2 3 4 5 6 7 8 9 10
MSE
GRAD
MINF
GRMI
HGMI
Results
58
Roots - Multiple Cameras
Accuracy
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
Number of trees
1 2 3 4 5 6 7 8 9 10
MSE
GRAD
MINF
GRMI
HGMI
Error Reduction (HGMI vs. MSE): 53% Error Reduction (HGMI vs. MSE): 47%

59. Results
59
Ancestry - One Camera
Accuracy
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
Number of trees
1 2 3 4 5 6 7 8 9 10
MSE
GRAD
MINF
GRMI
HGMI
Error Reduction (HGMI vs. MSE): 60% Error Reduction (HGMI vs. MSE): 58%
Ancestry - Multiple Cameras
Accuracy
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
Number of trees
1 2 3 4 5 6 7 8 9 10
MSE
GRAD
MINF
GRMI
HGMI

60. Qualitative results
Ellen DeGeneres’ Selfie
60
Group bGroup a
Group c
Group d
Group e

61. a09
a00 a11
a01 a02 a04 a05 a06 a07 a08
a03a10
a12
b03
b01 b06
b04
b02 c07
b05
c01
c08
c02 c05 c09 c10
c06
c03 c04
d08
d01 d07
d03 d02 d06 d05
d04
e01
e02 e03 e04 e05 e06
e07
61
Group b
Group a
Group c
Group d
Group e
Qualitative results
Ellen DeGeneres’ Selfie

62. a09
a00 a11
a01 a02 a04 a05 a06 a07 a08
a03a10
a12
b03
b01 b06
b04
b02 c07
b05
c01
c08
c02 c05 c09 c10
c06
c03 c04
d08
d01 d07
d03 d02 d06 d05
d04
e01
e02 e03 e04 e05 e06
e07
62
Group b
Group a
Group c
Group d
Group e
Qualitative results
Ellen DeGeneres’ Selfie

63. a09
a00 a11
a01 a02 a04 a05 a06 a07 a08
a03a10
a12
b03
b01 b06
b04
b02 c07
b05
c01
c08
c02 c05 c09 c10
c06
c03 c04
d08
d01 d07
d03 d02 d06 d05
d04
e01
e02 e03 e04 e05 e06
e07
63
Group b
Group a
Group c
Group d
Group e
Qualitative results
Ellen DeGeneres’ Selfie

64. a09
a00 a11
a01 a02 a04 a05 a06 a07 a08
a03a10
a12
b03
b01 b06
b04
b02 c07
b05
c01
c08
c02 c05 c09 c10
c06
c03 c04
d08
d01 d07
d03 d02 d06 d05
d04
e01
e02 e03 e04 e05 e06
e07
64
Group b
Group a
Group c
Group d
Group e
Qualitative results
Ellen DeGeneres’ Selfie

65. Video Phylogeny

66. Video Phylogeny
• More challenging problem than Image Phylogeny
• Now we have a new dimension: time
• Videos can be misaligned
• Videos can have different compression parameters
66

67. Hypothesis #4
Temporal alignment is paramount for a proper
video phylogeny reconstruction process
67

68. • Phylogeny reconstruction for misaligned and compressed
video sequences
F. O. Costa, S. Lameri, P. Bestagini, Z. Dias, A. Rocha, M. Tagliassachi, S. Tubaro. Phylogeny
reconstruction for misaligned and compressed video sequences. IEEE International
Conference on Image Processing (ICIP), 2015
• Hashing-based Temporal Alignment for Video Phylogeny
Reconstruction
F. O. Costa, P. Bestagini, S. Tubaro, Z. Dias, A. Rocha. Hashing-based Temporal Alignment
for Video Phylogeny Reconstruction. Submitted to IEEE WIFS, 2016
Video Phylogeny
Contributions
68

69. Phylogeny reconstruction for misaligned
and compressed video sequences
69

70. Video dissimilarity calculation
• Temporal alignment
• Frame registration
• Color matching
• Coding matching (controlled)
• Comparison (MSE)
70

71. Temporal alignment
71
V
1 2 3 4 5 6 7 8 9 10 … n-2 n-1 n
y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 yn-2 yn-1 yn
yi = Average of luminance of V(i)
LDV = [y2 - y1, y3 - y2, … , yn-1 - yn-2, yn - yn-1 ]
Difference of luminance

72. 50 100 150 200 250
0
i
L
50 100 150 200 250
0
5
10
i
LDV(i)
V = Vsrc
V = Vtgt
−200 −100 0 100 200
0
0.2
0.4
0.6
i
phase-corr.
Video Alignment
72
Phase-correlation: ˆi = arg maxi F 1
h F[LDVsrc ]·F[LDVtgt ]⇤
|F[LDVsrc ]·F[LDVtgt ]⇤|
i
(i)

73. Comparison
• We perform two kinds of comparison
• Average of MSE for the correspondent frames of
two videos
• Parenthood matrix
73

74. Parenthood matrix
74
Frequency
of the edges
Parenthood
Matrix
Optimum
branching
(MAX)
Final phylogeny tree
…M1 M2 M3 M4 M5 M6 Mn-1 Mn
V3 …
V1 …
V2
…
V4 …
V5
…

75. Experiments
75

76. Dataset
76
Available at https://media.xiph.org/video/derf/
•200 trees with 10 videos
•100 - No clip
•100 - Clip at the Beginning/end of the
stream
•Probability of clip: 50%
•CODEC: MPEG2, MPEG4, H264
•Crop, resize, bright, contrast, coding
and temporal clipping

77. Experimental Setup
• Phylogeny Reconstruction
• OB algorithm
• Evaluation metrics
• Root, edges, leaves,
ancestry and depth
77
1
2
3
5
4
4
23
5
1
Reconstructed treeGround truth
Depth = 1

78. Results #1
78
Verifying alignment and coding matching
Root Depth Edges Leaves Ancestry
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Score
Results - Dataset without temporal clipping
Without temporal alignment With temporal alignment With temporal alignment and compression matching
Root Depth Edges Leaves Ancestry
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Score
Results - Dataset with temporal clipping
Without alignment With alignment With alignment and compression matching

79. Root Depth Edges Leaves Ancestry
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Score
Results - Dataset with temporal clipping
A - NO COD
P - NO COD
A - COD
P - COD
Root Depth Edges Leaves Ancestry
Results - Dataset without temporal clipping
Results #2
79
Comparison: Average of MSE (A) x Parenthood matrix (P)
Root Depth Edges Leaves A
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Score
Results - Dataset with temporal clipping
A - NO COD
P - NO COD
A - COD
P - COD
Root Depth Edges Leaves Ancestry
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Score
Results - Dataset without temporal clipping
A - NO COMPRESSION MATCHING P - NO COMPRESSION MATCHING
A - WITH COMPRESSION MATCHING P - WITH COMPRESSION MATCHING

80. Hashing-based temporal alignment
for video phylogeny
80

81. • Near-duplicate matching
• Near-duplicate extraction
• Near-duplicate alignment
Hashing-based temporal
alignment
81

82. Step 1
Near-duplicate matching
• Groups of 64 frames
• All frames are resized to 32 x 32 pixel
82
1 frame
Temporal axis
……
1 2 3 4 5 6 7 8 9 10 59 60 61 62 63 64 65 66 67 68

83. Step 1
Near-duplicate matching
83
64
coefficients
32 coefficients
32coefficients
DCT coefficients
4 x 4 x 4
DCT coefficients
…
3D Discrete Cosine Transform

84. Step 1
Near-duplicate matching
• Binary hash
• According to the median of DCT
• Distance matrix
• Hamming distance of two binary hashes of
different videos
84

85. Step 2
Near-duplicate extraction
85
0 30 60 90 120 150 180 210 236
0
20
40
60
80
100
120
140
160
180
200
0 30 60 90
0
20
40
60
80
100
120
140
160
180
200
20 150 180 210 236 0 30 60 90 120 150 180 210 236
0
20
40
60
80
100
120
140
160
180
200
0 30 60 90
0
20
40
60
80
100
120
140
160
180
200
20 150 180 210 236 0 30 60 90 120 150 180 210 236
0
20
40
60
80
100
120
140
160
180
200
0 30 60 90
0
20
40
60
80
100
120
140
160
180
200
20 150 180 210 236 0 30 60 90 120 150 180 210 236
0
20
40
60
80
100
120
140
160
180
200
BinarizationOpeningExtraction

86. V1 V2
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Sequences without temporal alignment
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Alignment by phase correlation
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Hashing-based temporal alignment
86
Step 3
Near-duplicate alignment

87. V1 V2
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Sequences without temporal alignment
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Alignment by phase correlation
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Hashing-based temporal alignment
V1 V2
0 20 40 60 80 100 120 140
-3
-2
-1
0
1
2
3
Diff.ofluminance
Sequences witho
3
Alignment by
87
Step 3
Near-duplicate alignment

88. V1 V2
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Sequences without temporal alignment
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Alignment by phase correlation
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Hashing-based temporal alignment
88
Step 3
Near-duplicate alignment0 20 40 60 80 100 120 140
-3
-2
-1
Diff.
0 20 40 60 80 100 120 140
-3
-2
-1
0
1
2
3
Diff.ofluminance
Alignment by

89. V1 V2
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Sequences without temporal alignment
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Alignment by phase correlation
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
-3
-2
-1
0
1
2
3
Diff.ofluminance
Hashing-based temporal alignment
89
Step 3
Near-duplicate alignment
0 20 40 60 80 100 120 140
-3
-2
-1
0
Diff.oflu
-3
-2
-1
0
1
2
3
Diff.ofluminance
Hashing-based

90. Experiments
90

91. Experiments
•Dataset
•300 trees with 10 videos
•100 - No clip
•100 - Clip at the Beginning/end of the stream
•100 - Clip anywhere
91

92. Results
92
Diff. of luminance vs. Hashing-based alignment
Difference of luminance Hashing-based
Root Depth Edges Leaves Ancestry Root Depth Edges Leaves Ancestry
No
coding
matching
No clip 0,630 0,500 0,772 0,832 0,674 0,650 0,440 0,777 0,828 0,702
Clip
border 0,710 0,35 0,788 0,826 0,724 0,710 0,330 0,807 0,846 0,740
Clip any 0,620 0,560 0,724 0,758 0,628 0,670 0,450 0,755 0,789 0,683
With
coding
matching
No clip 0,870 0,140 0,839 0,859 0,812 0,690 0,400 0,778 0,831 0,711
Clip
border 0,840 0,170 0,863 0,901 0,834 0,660 0,400 0,780 0,824 0,722
Clip any 0,720 0,390 0,801 0,828 0,696 0,660 0,470 0,765 0,810 0,696

93. Conclusion
93

94. Conclusion
• New solutions for multimedia phylogeny
• Robust methods for phylogeny forest reconstruction
• More robust dissimilarity measures
• Gradient + mutual information
• Good results for video phylogeny when dealing with
misaligned and compressed videos
94

95. Future work
• Other transformations
• Blurring, sharpness, content insertion…
• Videos with different frame rates
• Video dissimilarity calculation using time information
• Robust methods to reliably estimate the compression
parameters directly from the video
95

96. Acknowledgements
• Examiners
• Prof. Anderson Rocha and Prof. Zanoni Dias
• RECOD - IC/Unicamp
• ISPG - Politecnico di Milano
• FAPESP
• CAPES
96

97. Thanks!
97