Presents a novel and general method for the detection, rectification and segmentation of imaged coplanar repeated patterns. The only assumption made of the scene geometry is that repeated scene elements are mapped to each other by planar Euclidean transformations. The class of patterns covered is broad and includes nearly all commonly seen, planar, man-made repeated patterns. In addition, novel linear constraints are used to reduce geometric ambiguity between the rectified imaged pattern and the scene pattern. Rectification to within a similarity of the scene plane is achieved from one rotated repeat, or to within a similarity with a scale ambiguity along the axis of symmetry from one reflected repeat. A stratum of constraints is derived that gives the necessary configuration of repeats for each successive level of rectification. A generative model for the imaged pattern is inferred and used to segment the pattern with pixel accuracy. Qualitative results are shown on a broad range of image types on which state-of-the-art methods fail.

1. Detection, Rectification and
Segmentation of Co-planar Repeated
Patterns
James Pritts
Ondrej Chum and Jiri Matas
Center for Machine Perception (CMP)
Czech Technical University in Prague
Faculty of Electrical Engineering
Department of Cybernetics

2. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Introduction
 Repetitive patterns are ubiquitous in images
 Unless considered, they usually decrease vision algorithm performance
 Seek a model-based approach to precisely locate and segment
general co-planar repeats
2/25

3. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
 GOAL: Create a short-list of non-
random matches to query image
 Because of “burstiness”, repeated
elements are over-counted:
 High frequency words from
repeats skew scoring
 Co-occurring features are not
independent
Image form H. Jegou and Matthijs Douze, On the burstiness of
visual elements. In CVPR, 2009.
Problems with Repetitions: Image Retrieval
Query
Match??
3/25

4. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Problems with Repetitions: Stereo Matching
 Cannot disambiguate tentative correspondences
 F-estimate invalid
 Epipolar constraint provides only weak spatial verification
(even with good F)
4/25
mismatched
mismatched

5. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Prior Work
 Detecting repeats is well studied
 Rectification is nearly universal (vanishing
lines)
 Exploit some constraint that is valid in
rectified space
 Lattice
 Schaffalitzky, F., Img. Vis. Comp. 2000
 Lattice, Axial Symmetry
 Wu et al, CVPR 2011
 Symmetry
 Hong et al, IJCV 2004
 Congruency
 Liebowitz et al, CVPR 1998
 Aiger et al, Comp. Graph. Forum 2012
5/25

6. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
The State of the Art
 TILT: Zhang et al., IJCV 2012.
 Find homography minimizing
image rank
 Manual cueing of pattern required
 Fails with significant perspective
warp, occlusions or if repeats are
sparse
 Aiger et al, Comp. Graph. Forum 2012
 Joint maximization of congruent
line segments has no
convergence guarantee
 Systems of rational equations
sampled by Hough transform
(slow)
6/25

7. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Problem Formulation
 Task: Segment imaged co-planar repeats with pixel level accuracy
 Some scene element repeats on a plane
 Need not have any regularity or be densely sampled
 Work without image structure modulo the repeat
- A common assumption is the existence of vanishing lines in the image that can be used to rectify the
scene plane
 Work when repeats cover only a small part of the image
 Fully automated: no cueing is required
 Can segment pattern to pixel level accuracy
 Assumptions
 Repeated scene elements are coplanar
 Scene elements can be mapped to each other by rigid transforms
 Imaged by perspective camera
 Scene element is repeated at least 3 times
7/25

8. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Proposed method
 A method for detection, precise alignment and segmentation of general
co-planar repeated patterns
8/25

9. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Proposed method
 A method for detection, precise alignment and segmentation of general
co-planar repeated patterns
8/25

10. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Proposed method
 A method for detection, precise alignment and segmentation of general
co-planar repeated patterns
8/25

11. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Intra-Image Feature Correspondence
 Extremal regions (MSERs)
detected for local representation of
images
 Local Affine Frames (LAFs)
derived from extremal regions to
concisely capture local geometry
 Affine frames described by SIFTs
T(Ax)=AT(x)
Affine covariance
11/25

12. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Feature Correspondence to Clusters
 Cluster: set of LAFs that are photometrically consistent.
12/25

13. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
From Clusters to Repeats: Rectification
 Spatial verification of photometric clustering is needed
 Perspective and affine imaging does not preserve scale or congruency
 Need general rectification method for rigidly transformed repeats
13/25
image rectified

14. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Rectification Stratum
 Translated and rotated co-planar
pattern
affinity similarity
similarity w
scale ambiguity
14/25

15. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Rectification Stratum
 Translated and rotated co-planar
pattern
 Translation: Affine rectification
affinity similarity
similarity w
scale ambiguity
14/25

16. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Rectification Stratum
 Translated and rotated co-planar
pattern
 Translation: Affine rectification
 Rotation: Similarity rectification
affinity similarity
similarity w
scale ambiguity
14/25

17. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Rectification Stratum
 Translated and rotated co-planar
pattern
 Translation: Affine rectification
 Reflection: Similarity with scale
ambiguity along reflection axis
 Rotation: Similarity rectification
affinity similarity
similarity w
scale ambiguity
14/25

18. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Affine Rectification (Chum et al [3])
 Assumption: repeated elements in real world are equiareal.
 Constraint: images of repeated elements should be equiareal after affine
rectification.
Source image
Coordinates and scales are known
Destination image
Only scales are known (no positions)
H
Result: unit area ratio, but
not necessarily equal angles
and extent length ratios
18/25

19. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Similarity Rectification
 Assumption: repeated elements in real world have equal extent lengths.
 Constraint: images of repeated elements should have equal extent lengths.
 Result: Equal area ratios, relative extent lengths preserved, equal angles
Rotation Reflection
Imaged
Scene
2 LAFS needed 1 LAF needed
19/25

20. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Affinity removal with reflections
20/25

21. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Affinity removal with reflections
20/25

22. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Rectification Results
22/25

23. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Repeats to Motifs
 Repeat: photometrically consistent cluster of local affine frames (LAFs)
that satisfies scale constraint
 Motif: is a collection of repeats that are spatially coherent
 Instance: An occurrence of the motif in the pattern
 Goal: Estimate a motif and set of transforms between
23/25

24. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Motif Estimation
 Open Problem: Formulate cost that balances model complexity and motif
cardinality
24/25

25. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Greedy Motif Construction
25/25

26. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Greedy Motif Construction
25/25

27. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Generative Model
 Generate the imaged pattern from the motif
motif
27/25

28. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Generative Model
 Generate the imaged pattern from the motif
 Estimate pattern and rectification from image
 SIFTs extracted from image and clustered
 Rectification estimated from linear constraints
 Clusters verified against scale constraint to make repeats
 Geometric hashing of LAFS in rectified space to construct motif
motif
27/25

29. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Generative Model
 Generate the imaged pattern from the motif
 Estimate pattern and rectification from image
 SIFTs extracted from image and clustered
 Rectification estimated from linear constraints
 Clusters verified against scale constraint to make repeats
 Geometric hashing of LAFS in rectified space to construct motif
 Refine pattern, rectification and lens distortion by minimizing pattern re-
projection error
motif
27/25

30. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Motif Construction
28/25

31. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Motif Construction
28/25

32. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Cows
30/25

33. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Cows
30/25

34. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Multiple motifs
34/25

35. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Multiple motifs
34/25

36. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Multiple motifs
34/25

37. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Future Work
 Seek join estimation of photometric clustering and rectification
 Sequential estimation is error prone, especially for multiple planes
 Failure modes
 Infer rectification jointly from more constraints
 Broaden the class of images from which patterns can be extracted
 Model complexity cost to principally estimate number of planes
 Formulate optimization problem for motif construction
 Integrate into image retrieval engine
37/25

38. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Conclusions
 Demonstrated effectiveness of new linear constraints
 valid for nearly all man-made patterns
 effective in a fast RANSAC framework
 Improved the state-of-the-art (TILT, Aiger et al):
 Rectifies patterns that are: a small part of the image, of low texture
 Localizes pattern automatically
 Affine distortion can be removed with as few as 1 repeat
 Explicitly model the pattern
 Segmentation of pattern at pixel-level
 SIFT variance decreased by using refined pattern to resample image
 Multiple motifs can be used to jointly optimize rectification
38/25

39. 3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Questions
Thanks for your attention
***Cosegmentations performed by J. Cech, J. Matas, and M. Perdoch.
Efficient sequential correspondence selection by cosegmentation. In CVPR,
2008.
39/25