The document discusses graph cut-based optimization techniques for computer vision problems. It describes how image labelling problems can be formulated as energy minimization problems over random fields with complex dependencies between labels. Solving such problems directly is difficult, so the document proposes transforming them into equivalent maximum flow problems on graphs, which can then be solved efficiently using the Ford-Fulkerson algorithm. This allows exploiting graph cuts to optimize random fields for applications like foreground/background segmentation.
Swin Transformer가 최근 오브젝트디텍션 그리고 Semantic segmentation분야에서의 성능이 가장 좋은 모델 중 하나로
주목 받고 있습니다.
Swin Transformer nlp분야에서 많이 쓰이는 트랜스포머를 비전 분야에 적용한 모델로 Hierarchical feature maps과
Window-based Self-attention의 특징적입니다 Swin Transformer는 작년 구글에서 제안된 방법인
비전 트랜스포머의 한계점을 개선한 모델이라고 보시면 됩니다
트랜스포머의 한계란.. ㄷㄷ 이네요
이미지 처리팀의 김선옥님이 자세한 리뷰 도와주셨습니다!!
오늘도 많은 관심 미리 감사드립니다!!
https://youtu.be/L3sH9tjkvKI
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...Simplilearn
This presentation on Convolutional neural network tutorial (CNN) will help you understand what is a convolutional neural network, hoe CNN recognizes images, what are layers in the convolutional neural network and at the end, you will see a use case implementation using CNN. CNN is a feed forward neural network that is generally used to analyze visual images by processing data with grid like topology. A CNN is also known as a "ConvNet". Convolutional networks can also perform optical character recognition to digitize text and make natural-language processing possible on analog and hand-written documents. CNNs can also be applied to sound when it is represented visually as a spectrogram. Now, lets deep dive into this presentation to understand what is CNN and how do they actually work.
Below topics are explained in this CNN presentation(Convolutional Neural Network presentation)
1. Introduction to CNN
2. What is a convolutional neural network?
3. How CNN recognizes images?
4. Layers in convolutional neural network
5. Use case implementation using CNN
Simplilearn’s Deep Learning course will transform you into an expert in deep learning techniques using TensorFlow, the open-source software library designed to conduct machine learning & deep neural network research. With our deep learning course, you’ll master deep learning and TensorFlow concepts, learn to implement algorithms, build artificial neural networks and traverse layers of data abstraction to understand the power of data and prepare you for your new role as deep learning scientist.
Why Deep Learning?
It is one of the most popular software platforms used for deep learning and contains powerful tools to help you build and implement artificial neural networks.
Advancements in deep learning are being seen in smartphone applications, creating efficiencies in the power grid, driving advancements in healthcare, improving agricultural yields, and helping us find solutions to climate change. With this Tensorflow course, you’ll build expertise in deep learning models, learn to operate TensorFlow to manage neural networks and interpret the results.
And according to payscale.com, the median salary for engineers with deep learning skills tops $120,000 per year.
You can gain in-depth knowledge of Deep Learning by taking our Deep Learning certification training course. With Simplilearn’s Deep Learning course, you will prepare for a career as a Deep Learning engineer as you master concepts and techniques including supervised and unsupervised learning, mathematical and heuristic aspects, and hands-on modeling to develop algorithms. Those who complete the course will be able to:
Learn more at: https://www.simplilearn.com/
What is a superpixel?
This presentation describes Superpixel algorithms such as watershed, mean-shift, SLIC, BSLIC (SLIC superpixels based on boundary term)
references:
[1] Luc Vincent and Pierre Soille. Watersheds in digital spaces: An efficient algorithm based on immersion simulations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(6):583–598, 1991.
[2] D. Comaniciu and P. Meer. Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5):603–619, May 2002.
[3] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk, SLIC Superpixels Compared to State-of-the-art Superpixel Methods, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, num. 11, p. 2274 - 2282, May 2012.
[4] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk, SLIC Superpixels, EPFL Technical Report no. 149300, June 2010.
[5] Hai Wang, Xiongyou Peng, Xue Xiao, and Yan Liu, BSLIC: SLIC Superpixels Based on Boundary Term, Symmetry 2017, 9(3), Feb 2017.
Swin Transformer가 최근 오브젝트디텍션 그리고 Semantic segmentation분야에서의 성능이 가장 좋은 모델 중 하나로
주목 받고 있습니다.
Swin Transformer nlp분야에서 많이 쓰이는 트랜스포머를 비전 분야에 적용한 모델로 Hierarchical feature maps과
Window-based Self-attention의 특징적입니다 Swin Transformer는 작년 구글에서 제안된 방법인
비전 트랜스포머의 한계점을 개선한 모델이라고 보시면 됩니다
트랜스포머의 한계란.. ㄷㄷ 이네요
이미지 처리팀의 김선옥님이 자세한 리뷰 도와주셨습니다!!
오늘도 많은 관심 미리 감사드립니다!!
https://youtu.be/L3sH9tjkvKI
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...Simplilearn
This presentation on Convolutional neural network tutorial (CNN) will help you understand what is a convolutional neural network, hoe CNN recognizes images, what are layers in the convolutional neural network and at the end, you will see a use case implementation using CNN. CNN is a feed forward neural network that is generally used to analyze visual images by processing data with grid like topology. A CNN is also known as a "ConvNet". Convolutional networks can also perform optical character recognition to digitize text and make natural-language processing possible on analog and hand-written documents. CNNs can also be applied to sound when it is represented visually as a spectrogram. Now, lets deep dive into this presentation to understand what is CNN and how do they actually work.
Below topics are explained in this CNN presentation(Convolutional Neural Network presentation)
1. Introduction to CNN
2. What is a convolutional neural network?
3. How CNN recognizes images?
4. Layers in convolutional neural network
5. Use case implementation using CNN
Simplilearn’s Deep Learning course will transform you into an expert in deep learning techniques using TensorFlow, the open-source software library designed to conduct machine learning & deep neural network research. With our deep learning course, you’ll master deep learning and TensorFlow concepts, learn to implement algorithms, build artificial neural networks and traverse layers of data abstraction to understand the power of data and prepare you for your new role as deep learning scientist.
Why Deep Learning?
It is one of the most popular software platforms used for deep learning and contains powerful tools to help you build and implement artificial neural networks.
Advancements in deep learning are being seen in smartphone applications, creating efficiencies in the power grid, driving advancements in healthcare, improving agricultural yields, and helping us find solutions to climate change. With this Tensorflow course, you’ll build expertise in deep learning models, learn to operate TensorFlow to manage neural networks and interpret the results.
And according to payscale.com, the median salary for engineers with deep learning skills tops $120,000 per year.
You can gain in-depth knowledge of Deep Learning by taking our Deep Learning certification training course. With Simplilearn’s Deep Learning course, you will prepare for a career as a Deep Learning engineer as you master concepts and techniques including supervised and unsupervised learning, mathematical and heuristic aspects, and hands-on modeling to develop algorithms. Those who complete the course will be able to:
Learn more at: https://www.simplilearn.com/
What is a superpixel?
This presentation describes Superpixel algorithms such as watershed, mean-shift, SLIC, BSLIC (SLIC superpixels based on boundary term)
references:
[1] Luc Vincent and Pierre Soille. Watersheds in digital spaces: An efficient algorithm based on immersion simulations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(6):583–598, 1991.
[2] D. Comaniciu and P. Meer. Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5):603–619, May 2002.
[3] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk, SLIC Superpixels Compared to State-of-the-art Superpixel Methods, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, num. 11, p. 2274 - 2282, May 2012.
[4] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk, SLIC Superpixels, EPFL Technical Report no. 149300, June 2010.
[5] Hai Wang, Xiongyou Peng, Xue Xiao, and Yan Liu, BSLIC: SLIC Superpixels Based on Boundary Term, Symmetry 2017, 9(3), Feb 2017.
More companies in the process of recruitment, play more emphasis in the topic of numbers in numerical aptitude. Especially for AMCAT aspirants this is very much useful.
Three View Self Calibration and 3D ReconstructionPeter Abeles
Presentation which goes over techniques and math involved in self calibration a camera and performing a dense reconstruction from only three views. This is a challenging problem that I think many people will find interesting. Source code and pre-built applications are provided.
This presentation displays the applications of CNNs, a quick review about Neural Networks and their drawbacks, the convolution process, padding, striding, convolution over volume, types of layers in CNN, max pool layer, fully connected layer, and lastly the famous CNNs, LetNet-5, AlexNet, VGG-16, ResNet and GoogLeNet.
Divide and Conquer Algorithms - D&C forms a distinct algorithm design technique in computer science, wherein a problem is solved by repeatedly invoking the algorithm on smaller occurrences of the same problem. Binary search, merge sort, Euclid's algorithm can all be formulated as examples of divide and conquer algorithms. Strassen's algorithm and Nearest Neighbor algorithm are two other examples.
Histogram equalization is a method in image processing of contrast adjustment using the image's histogram. Histogram equalization can be used to improve the visual appearance of an image. Peaks in the image histogram (indicating commonly used grey levels) are widened, while the valleys are compressed.
Here is a basic Linear Algebra review for the class of Machine Learning. This is actually becoming a new class in the mathematics of Intelligent Systems, there I will be teaching stuff in
1.- Linear Algebra - From the basics to the Cayley-Hamilton Theorem with applications
2.- Mathematical Analysis - from set to the Reimann Integral
3.- Topology - Mostly in Hilbert Spaces
4.- Optimization - Convex functions, KKT conditions, Duality Theory, etc.
The stuff is going to be interesting...
More companies in the process of recruitment, play more emphasis in the topic of numbers in numerical aptitude. Especially for AMCAT aspirants this is very much useful.
Three View Self Calibration and 3D ReconstructionPeter Abeles
Presentation which goes over techniques and math involved in self calibration a camera and performing a dense reconstruction from only three views. This is a challenging problem that I think many people will find interesting. Source code and pre-built applications are provided.
This presentation displays the applications of CNNs, a quick review about Neural Networks and their drawbacks, the convolution process, padding, striding, convolution over volume, types of layers in CNN, max pool layer, fully connected layer, and lastly the famous CNNs, LetNet-5, AlexNet, VGG-16, ResNet and GoogLeNet.
Divide and Conquer Algorithms - D&C forms a distinct algorithm design technique in computer science, wherein a problem is solved by repeatedly invoking the algorithm on smaller occurrences of the same problem. Binary search, merge sort, Euclid's algorithm can all be formulated as examples of divide and conquer algorithms. Strassen's algorithm and Nearest Neighbor algorithm are two other examples.
Histogram equalization is a method in image processing of contrast adjustment using the image's histogram. Histogram equalization can be used to improve the visual appearance of an image. Peaks in the image histogram (indicating commonly used grey levels) are widened, while the valleys are compressed.
Here is a basic Linear Algebra review for the class of Machine Learning. This is actually becoming a new class in the mathematics of Intelligent Systems, there I will be teaching stuff in
1.- Linear Algebra - From the basics to the Cayley-Hamilton Theorem with applications
2.- Mathematical Analysis - from set to the Reimann Integral
3.- Topology - Mostly in Hilbert Spaces
4.- Optimization - Convex functions, KKT conditions, Duality Theory, etc.
The stuff is going to be interesting...
Rising from non-existence a few short years ago, Node.js is already attracting the accolades and disdain enjoyed and endured by the Ruby and Rails community just a short time ago. It overtook Rails as the most popular Github repository in 2011 and was selected by InfoWorld for the Technology of the Year Award in 2012. This presentation explains the basic theory and programming model central to Node's approach and will help you understand the resulting benefits and challenges it presents. You can also watch this presentation at http://bit.ly/1362UGA
Automation of Discovery Technology Lab WorkflowsAvetis Ghukasyan
This presentation is a quick overview of all of my projects for Cubist Pharmaceuticals while I was working as an intern. I have worked on 5 projects for automating discovery technologies lab workflows and all of them are described in full detail in this presentation.
Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...Guozhang Wang
We present Apache Kafka’s core design for stream processing, which relies on its persistent log architecture as the storage and inter-processor communication layers to achieve correctness guarantees. Kafka Streams, a scalable stream processing client library in Apache Kafka, defines the processing logic as read process-write cycles in which all processing state updates and result outputs are captured as log appends. Idempotent and transactional write protocols are utilized to guarantee exactly once semantics. Furthermore, revision-based speculative processing is employed to emit results as soon as possible while handling out-of-order data. We also demonstrate how Kafka Streams behaves in practice with large-scale deployments and performance insights exhibiting its flexible and low-overhead trade-offs.
TMPA-2017: Tools and Methods of Program Analysis
3-4 March, 2017, Hotel Holiday Inn Moscow Vinogradovo, Moscow
Dl-Check: Dynamic Potential Deadlock Detection Tool for Java Programs
Nikita Koval, Dmitry Tsitelov, Roman Elizarov, Devexperts
For video follow the link: https://youtu.be/uyQvsxVL_TI
Would like to know more?
Visit our website:
www.tmpaconf.org
www.exactprosystems.com/events/tmpa
Follow us:
https://www.linkedin.com/company/exactpro-systems-llc?trk=biz-companies-cym
https://twitter.com/exactpro
Hadoop at Bloomberg:Medium data for the financial industryMatthew Hunt
Overview of big data systems origins, strengths, weaknesses, and uses at Bloomberg for solving "medium data" timeseries issues. Medium data requires modest clusters with consistently low latency and high availability, and Bloomberg has driven changes to core Hadoop components to address.
DocValues aka. Column Stride Fields in Lucene 4.0 - By Willnauer Simonlucenerevolution
See conference video - http://www.lucidimagination.com/devzone/events/conferences/revolution/2011
Lucene 4.0 is on its way to deliver a tremendous amount of new features and improvements. Beside
Real-Time Search & Flexible Indexing DocValues aka. Column Stride Fields is one of the “next
generation” features. DocValues enable Lucene to efficiently store and retrieve type-safe Document
& Value pairs in a column stride fashion either entirely memory resident random access or disk
resident iterator based without the need to un-invert fields. Its final goal is to provide a
independently update-able per document storage for scoring, sorting or even filtering. This talk will
introduce the current state of development, implementation details, its features and how DocValues
have been integrated into Lucene’s Codec API for full extendability.
Lucene 4.0 is on its way to deliver a tremendous amount of new features and improvements. Beside Real-Time Search & Flexible Indexing DocValues aka. Column Stride Fields is one of the "next generation" features
An Introduction to Basics of Search and Relevancy with Apache SolrLucidworks (Archived)
The open source Apache Solr open source search engine provides powerful, versatile search application development technology so you to take full control of your search needs. Solr’s rich interfaces and convenient server packaging of the underlying Apache Lucene search libraries into web service interfaces, and near limitless customizability let you take control of your search. From e-commerce to content management and endless variations in between, Solr is the right tool at the right time to turn ever growing volume and variety of data and documents to the advantage of your business.http://www.lucidimagination.com/blog/2009/12/01/webinar-an-introduction-to-basics-of-search-and-relevancy-with-apache-solr/
It was the talk, titled "Graph-Tool: The Efficient Network Analyzing Tool for Python", at PyCon APAC 2014 [1] and PyCon SG 2014 [2]. It introduces you to Graph-Tool by mass code snippets.
[1] https://tw.pycon.org/2014apac
[2] https://pycon.sg/
Nearest neighbor models are conceptually just about the simplest kind of model possible. The problem is that they generally aren’t feasible to apply. Or at least, they weren’t feasible until the advent of Big Data techniques. These slides will describe some of the techniques used in the knn project to reduce thousand-year computations to a few hours. The knn project uses the Mahout math library and Hadoop to speed up these enormous computations to the point that they can be usefully applied to real problems. These same techniques can also be used to do real-time model scoring.
PLNOG19 - Krzysztof Szarkowicz - RIFT i nowe pomysły na routingPROIDEA
Istniejący zestaw protokołów dynamicznego routingu był pierwotnie nastawiony w kierunku sieci o nieregularnej topologii i niskim stopniu połączeń między routerami. Kilka prób to dostosowania tych protokołów do potrzeb sieci w centrum danych zostało podjętych już wcześniej. Najbardziej udana próba z rozszerzeniem BGP (RFC4271, RFC7938) została podjęta nie tyle ze względu na właściwość BGP aby rozwiązać problem, ale raczej postrzeganej zdolności BGP do zmodyfikowania "szybciej" i trudnościami z protokołami link-state w wypełnieniu niektórych szczególnych wymagań.Patrząc na problem przez samą soczewkę istotnych wymagań, optymalne podejście nie wydaje się być prostą modyfikacją protokołów link-state (distributed computation - obliczenia rozdzielne) albo protokołów distance-vector (diffused computation - obliczenia rozproszone), ale raczej mieszanką obu. Mówiąc językiem potocznym jako “link-state w kierunku kręgosłupa (spine)” i “distance-vector w kierunku liści (leafs)”. Ten nowy protokół nazywa się RIFT (Routing In Fat Trees) i jest tematem tej prezentacji. Nowey protokół może być szybko zintegrowany z Junos, poprzez Programmable RPD (PRPD) APIs, nie wymagając żadnych zmian w kodzie Junos.
Challenging Web-Scale Graph Analytics with Apache Spark with Xiangrui MengDatabricks
Graph analytics has a wide range of applications, from information propagation and network flow optimization to fraud and anomaly detection. The rise of social networks and the Internet of Things has given us complex web-scale graphs with billions of vertices and edges. However, in order to extract the hidden gems within those graphs, you need tools to analyze the graphs easily and efficiently.
At Spark Summit 2016, Databricks introduced GraphFrames, which implemented graph queries and pattern matching on top of Spark SQL to simplify graph analytics. In this talk, you'll learn about work that has made graph algorithms in GraphFrames faster and more scalable. For example, new implementations like connected components have received algorithm improvements based on recent research, as well as performance improvements from Spark DataFrames. Discover lessons learned from scaling the implementation from millions to billions of nodes; compare its performance with other popular graph libraries; and hear about real-world applications.
Challenging Web-Scale Graph Analytics with Apache SparkDatabricks
Graph analytics has a wide range of applications, from information propagation and network flow optimization to fraud and anomaly detection. The rise of social networks and the Internet of Things has given us complex web-scale graphs with billions of vertices and edges. However, in order to extract the hidden gems within those graphs, you need tools to analyze the graphs easily and efficiently.
At Spark Summit 2016, Databricks introduced GraphFrames, which implemented graph queries and pattern matching on top of Spark SQL to simplify graph analytics. In this talk, you’ll learn about work that has made graph algorithms in GraphFrames faster and more scalable. For example, new implementations like connected components have received algorithm improvements based on recent research, as well as performance improvements from Spark DataFrames. Discover lessons learned from scaling the implementation from millions to billions of nodes; compare its performance with other popular graph libraries; and hear about real-world applications.
Similar to CVPR2012: Tutorial: Graphcut-based Optimisation for Computer Vision (20)
Embracing GenAI - A Strategic ImperativePeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Synthetic Fiber Construction in lab .pptxPavel ( NSTU)
Synthetic fiber production is a fascinating and complex field that blends chemistry, engineering, and environmental science. By understanding these aspects, students can gain a comprehensive view of synthetic fiber production, its impact on society and the environment, and the potential for future innovations. Synthetic fibers play a crucial role in modern society, impacting various aspects of daily life, industry, and the environment. ynthetic fibers are integral to modern life, offering a range of benefits from cost-effectiveness and versatility to innovative applications and performance characteristics. While they pose environmental challenges, ongoing research and development aim to create more sustainable and eco-friendly alternatives. Understanding the importance of synthetic fibers helps in appreciating their role in the economy, industry, and daily life, while also emphasizing the need for sustainable practices and innovation.
Honest Reviews of Tim Han LMA Course Program.pptxtimhan337
Personal development courses are widely available today, with each one promising life-changing outcomes. Tim Han’s Life Mastery Achievers (LMA) Course has drawn a lot of interest. In addition to offering my frank assessment of Success Insider’s LMA Course, this piece examines the course’s effects via a variety of Tim Han LMA course reviews and Success Insider comments.
Model Attribute Check Company Auto PropertyCeline George
In Odoo, the multi-company feature allows you to manage multiple companies within a single Odoo database instance. Each company can have its own configurations while still sharing common resources such as products, customers, and suppliers.
How to Make a Field invisible in Odoo 17Celine George
It is possible to hide or invisible some fields in odoo. Commonly using “invisible” attribute in the field definition to invisible the fields. This slide will show how to make a field invisible in odoo 17.
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
2. Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
3. Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
4. Image Labelling Problems
Assign a label to each image pixel
Geometry Estimation Image Denoising Object Segmentation Depth Estimation
Sky
Building
Tree
Grass
6. Image Labelling Problems
• Labellings highly structured
• Labels highly correlated with very complex dependencies
• Neighbouring pixels tend to take the same label
• Low number of connected components
• Classes present may be seen in one image
• Geometric / Location consistency
• Planarity in depth estimation
• … many others (task dependent)
7. Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
8. Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
• Whole labelling should be formulated as one optimisation
problem
9. Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
• Whole labelling should be formulated as one optimisation
problem
• Number of pixels up to millions
• Hard to train complex dependencies
• Optimisation problem is hard to infer
10. Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
• Whole labelling should be formulated as one optimisation
problem
• Number of pixels up to millions
• Hard to train complex dependencies
• Optimisation problem is hard to infer
Vision is hard !
11. Image Labelling Problems
• You can
either
• Change the subject from the Computer
Vision to the History of Renaissance Art
Vision is hard !
12. Image Labelling Problems
• You can
either
• Change the subject from the Computer
Vision to the History of Renaissance Art
or
• Learn everything about Random Fields and
Graph Cuts
Vision is hard !
14. Foreground / Background Estimation
Data term Smoothness term
Data term
Estimated using FG / BG
colour models
Smoothness term
where
Intensity dependent smoothness
16. Foreground / Background Estimation
Data term Smoothness term
How to solve this optimisation problem?
17. Foreground / Background Estimation
Data term Smoothness term
How to solve this optimisation problem?
• Transform into min-cut / max-flow problem
• Solve it using min-cut / max-flow algorithm
18. Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Applications
• Conclusions
19. Max-Flow Problem
source
5 9
4
2 2
3 5
2 3
1 1
6 5
3
2
6 8 5
Task : sink
Maximize the flow from the sink to the source such that
1) The flow it conserved for each node
2) The flow for each pipe does not exceed the capacity
21. Max-Flow Problem
source flow from the
flow from node i source
5 9 to node j capacity
4
2 2 set of edges
3 5
2 3
1 1
6 5
3
2
conservation of flow set of nodes
6 8 5
sink
22. Max-Flow Problem
source
9 Ford & Fulkerson algorithm (1956)
5
4
2 2 Find the path from source to sink
3 5
2 3 While (path exists)
1 1
6 5 flow += maximum capacity in the path
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 8 5
End
sink
23. Max-Flow Problem
source
9 Ford & Fulkerson algorithm (1956)
5
4
2 2 Find the path from source to sink
3 5
2 3 While (path exists)
1 1
6 5 flow += maximum capacity in the path
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 8 5
End
sink
24. Max-Flow Problem
source
9 Ford & Fulkerson algorithm (1956)
5
4
2 2 Find the path from source to sink
3 5
2 3 While (path exists)
1 1
6 5 flow += maximum capacity in the path
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 8 5
End
sink
flow = 3
25. Max-Flow Problem
source
9 Ford & Fulkerson algorithm (1956)
5-3
4
2+ 2 Find the path from source to sink
3 5
3-3
2 3 While (path exists)
1 6-3 1
flow += maximum capacity in the path
+3 5
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 8-3 5
End
sink
flow = 3
26. Max-Flow Problem
source
9 Ford & Fulkerson algorithm (1956)
2
4
5 2 Find the path from source to sink
5
2 3 While (path exists)
1 1
3 flow += maximum capacity in the path
3 5
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 5 5
End
sink
flow = 3
27. Max-Flow Problem
source
9 Ford & Fulkerson algorithm (1956)
2
4
5 2 Find the path from source to sink
5
2 3 While (path exists)
1 1
3 flow += maximum capacity in the path
3 5
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 5 5
End
sink
flow = 3
28. Max-Flow Problem
source
9 Ford & Fulkerson algorithm (1956)
2
4
5 2 Find the path from source to sink
5
2 3 While (path exists)
1 1
3 flow += maximum capacity in the path
3 5
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 5 5
End
sink
flow = 6
29. Max-Flow Problem
source
9-3 Ford & Fulkerson algorithm (1956)
2
4
5 2 Find the path from source to sink
5
2 +3 3-3 While (path exists)
1 1
3 5 flow += maximum capacity in the path
3
3-3 Build the residual graph (“subtract” the flow)
2
+3 Find the path in the residual graph
6 5-3 5
End
sink
flow = 6
30. Max-Flow Problem
source
6 Ford & Fulkerson algorithm (1956)
2
4
5 2 Find the path from source to sink
5
2 3 While (path exists)
1 1
3 flow += maximum capacity in the path
3 5
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 2 5
End
sink
flow = 6
31. Max-Flow Problem
source
6 Ford & Fulkerson algorithm (1956)
2
4
5 2 Find the path from source to sink
5
2 3 While (path exists)
1 1
3 flow += maximum capacity in the path
3 5
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 2 5
End
sink
flow = 6
32. Max-Flow Problem
source
6 Ford & Fulkerson algorithm (1956)
2
4
5 2 Find the path from source to sink
5
2 3 While (path exists)
1 1
3 flow += maximum capacity in the path
3 5
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 2 5
End
sink
flow = 11
33. Max-Flow Problem
source
6-5 Ford & Fulkerson algorithm (1956)
2
4
5 2+5 Find the path from source to sink
5-5
2 3 While (path exists)
1 1+5
3 flow += maximum capacity in the path
3 5-5
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 2 5-5
End
sink
flow = 11
34. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
2
4
5 Find the path from source to sink
7
2 3 While (path exists)
1 6
3 flow += maximum capacity in the path
3
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 2
End
sink
flow = 11
35. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
2
4
5 Find the path from source to sink
7
2 3 While (path exists)
1 6
3 flow += maximum capacity in the path
3
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 2
End
sink
flow = 11
36. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
2
4
5 Find the path from source to sink
7
2 3 While (path exists)
1 6
3 flow += maximum capacity in the path
3
3 Build the residual graph (“subtract” the flow)
2
Find the path in the residual graph
6 2
End
sink
flow = 13
37. Max-Flow Problem
source
2-2 1 Ford & Fulkerson algorithm (1956)
4
5 Find the path from source to sink
7
2-2 While (path exists)
+2 3
1 6
3 flow += maximum capacity in the path
+2 3
3 Build the residual graph (“subtract” the flow)
2-2
Find the path in the residual graph
6 2-2
End
sink
flow = 13
38. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
4
5 Find the path from source to sink
7
2 3 While (path exists)
1 6
3 flow += maximum capacity in the path
2 3
3 Build the residual graph (“subtract” the flow)
Find the path in the residual graph
6
End
sink
flow = 13
39. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
4
5 Find the path from source to sink
7
2 3 While (path exists)
1 6
3 flow += maximum capacity in the path
2 3
3 Build the residual graph (“subtract” the flow)
Find the path in the residual graph
6
End
sink
flow = 13
40. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
4
5 Find the path from source to sink
7
2 3 While (path exists)
1 6
3 flow += maximum capacity in the path
2 3
3 Build the residual graph (“subtract” the flow)
Find the path in the residual graph
6
End
sink
flow = 15
41. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
4-2
5 Find the path from source to sink
7
2 3 While (path exists)
1 3-2 6 flow += maximum capacity in the path
2-2 3+
32 Build the residual graph (“subtract” the flow)
+2
Find the path in the residual graph
6-2
End
sink
flow = 15
42. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
2
5 Find the path from source to sink
7
2 3 While (path exists)
1 6
1 flow += maximum capacity in the path
5
3 Build the residual graph (“subtract” the flow)
2
4 Find the path in the residual graph
End
sink
flow = 15
43. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
2
5 Find the path from source to sink
7
2 3 While (path exists)
1 6
1 flow += maximum capacity in the path
5
3 Build the residual graph (“subtract” the flow)
2
4 Find the path in the residual graph
End
sink
flow = 15
44. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
2
5 Find the path from source to sink
7
2 3 While (path exists)
1 6
1 flow += maximum capacity in the path
5
3 Build the residual graph (“subtract” the flow)
2
4 Find the path in the residual graph
End
sink
flow = 15
45. Max-Flow Problem
source
1 Ford & Fulkerson algorithm (1956)
2
5 Why is the solution globally optimal ?
7
2 3
1 6
1
5
2 3
4
sink
flow = 15
46. Max-Flow Problem
S source
1 Ford & Fulkerson algorithm (1956)
2
5 Why is the solution globally optimal ?
7
2 3
1 6
1 1. Let S be the set of reachable nodes in the
5
3 residual graph
2
4
sink
flow = 15
47. Max-Flow Problem
S5 source
9 Ford & Fulkerson algorithm (1956)
4
2 2 Why is the solution globally optimal ?
3 5
2 3
1 1
6 5 1. Let S be the set of reachable nodes in the
3 residual graph
2
2. The flow from S to V - S equals to the sum
6 8 5
of capacities from S to V – S
sink
48. Max-Flow Problem
A5 source
9 Ford & Fulkerson algorithm (1956)
4
2 2 Why is the solution globally optimal ?
3 5
2 3
1 1
6 5 1. Let S be the set of reachable nodes in the
residual graph
3
2 2. The flow from S to V - S equals to the sum of
6 5 capacities from S to V – S
8
3. The flow from any A to V - A is upper bounded
sink by the sum of capacities from A to V – A
49. Max-Flow Problem
source
5/5 8/9 Ford & Fulkerson algorithm (1956)
2/4
0/2 0/2 Why is the solution globally optimal ?
5/5
2/2 3/3
0/1 0/1 3/3
5/6 5/5 1. Let S be the set of reachable nodes in the
residual graph
3/3
0/2 2. The flow from S to V - S equals to the sum of
2/6 5/5 capacities from S to V – S
8/8
3. The flow from any A to V - A is upper bounded
sink by the sum of capacities from A to V – A
4. The solution is globally optimal
flow = 15
Individual flows obtained by summing up all paths
50. Max-Flow Problem
source
5/5 8/9 Ford & Fulkerson algorithm (1956)
2/4
0/2 0/2 Why is the solution globally optimal ?
3/3 5/5
2/2
0/1 0/1 3/3
5/6 5/5 1. Let S be the set of reachable nodes in the
residual graph
3/3
0/2 2. The flow from S to V - S equals to the sum of
2/6 5/5 capacities from S to V – S
8/8
3. The flow from any A to V - A is upper bounded
sink by the sum of capacities from A to V – A
4. The solution is globally optimal
flow = 15
51. Max-Flow Problem
source
Ford & Fulkerson algorithm (1956)
1000 1000
Order does matter
1
1000 1000
sink
52. Max-Flow Problem
source
Ford & Fulkerson algorithm (1956)
1000 1000
Order does matter
1
1000 1000
sink
53. Max-Flow Problem
source
Ford & Fulkerson algorithm (1956)
1000 999
Order does matter
1
999 1000
sink
54. Max-Flow Problem
source
Ford & Fulkerson algorithm (1956)
1000 999
Order does matter
1
999 1000
sink
55. Max-Flow Problem
source
Ford & Fulkerson algorithm (1956)
999 999
Order does matter
1
999 999
sink
56. Max-Flow Problem
source
Ford & Fulkerson algorithm (1956)
999 999
Order does matter
• Standard algorithm not polynomial
1
• Breath first leads to O(VE2)
• Path found in O(E)
999 999 • At least one edge gets saturated
• The saturated edge distance to the
sink source has to increase and is at
most V leading to O(VE)
(Edmonds & Karp, Dinic)
57. Max-Flow Problem
source
Ford & Fulkerson algorithm (1956)
999 999
Order does matter
• Standard algorithm not polynomial
1
• Breath first leads to O(VE2)
(Edmonds & Karp)
999 999 • Various methods use different algorithm
to find the path and vary in complexity
sink
58. Min-Cut Problem
source
5 9
4
2 2
3 5
2 3
1 1
6 5
3
2
6 8 5
Task : sink
Minimize the cost of the cut
1) Each node is either assigned to the source S or sink T
2) The cost of the edge (i, j) is taken if (i∈S) and (j∈T)
84. Min-Cut Problem
min cut
C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3)
S source
1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)
x1 2 x2 + 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5)
5
7
+ 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5)
x3 3
2
1 6
1 cost = 0
x4 5 x6 cost = 0
2 3
x5
4 • All coefficients positive
sink T • Must be global minimum
S – set of reachable nodes from s
85. Min-Cut Problem
min cut
C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3)
S source
1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)
x1 2 x2 + 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5)
5
7
+ 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5)
x3 3
2
1 6
1
x4 5 x6 cost = 0
2 3
x5
4 • All coefficients positive
sink T • Must be global minimum
T – set of nodes that can reach t
(not necesarly the same)
86. Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
87. Markov and Conditional RF
• Markov / Conditional Random fields model
probabilistic dependencies of the set of random
variables
88. Markov and Conditional RF
• Markov / Conditional Random fields model conditional
dependencies between random variables
• Each variable is conditionally independent of all other
variables given its neighbours
89. Markov and Conditional RF
• Markov / Conditional Random fields model conditional
dependencies between random variables
• Each variable is conditionally independent of all other
variables given its neighbours
• Posterior probability of the labelling x given data D is :
partition function cliques potential functions
90. Markov and Conditional RF
• Markov / Conditional Random fields model conditional
dependencies between random variables
• Each variable is conditionally independent of all other
variables given its neighbours
• Posterior probability of the labelling x given data D is :
partition function cliques potential functions
• Energy of the labelling is defined as :
91. Markov and Conditional RF
• The most probable (Max a Posteriori (MAP)) labelling
is defined as:
92. Markov and Conditional RF
• The most probable (Max a Posteriori (MAP)) labelling
is defined as:
• The only distinction (MRF vs. CRF) is that in the CRF
the conditional dependencies between variables
depend also on the data
93. Markov and Conditional RF
• The most probable (Max a Posteriori (MAP)) labelling
is defined as:
• The only distinction (MRF vs. CRF) is that in the CRF
the conditional dependencies between variables
depend also on the data
• Typically we define an energy first and then pretend
there is an underlying probabilistic distribution there,
but there isn’t really (Pssssst, don’t tell anyone)
94. Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
104. Graph Cut based Inference
After summing up :
Let : Then :
Equivalent st-mincut problem is :
105. Foreground / Background Estimation
Data term Smoothness term
Data term
Estimated using FG / BG
colour models
Smoothness term
where
Intensity dependent smoothness
107. Graph Cut based Inference
Extendable to (some) Multi-label CRFs
• each state of multi-label variable encoded using
multiple binary variables
• the cost of every possible cut must be the same as
the associated energy
• the solution obtained by inverting the encoding
Encoding
multi-label energy Build Obtain solution
Graph Cut
Graph solution
108. Dense Stereo Estimation
• For each pixel assigns a disparity label : z
• Disparities from the discrete set {0, 1, .. D}
Left Camera Image Right Camera Image Dense Stereo Result
109. Dense Stereo Estimation
Data term
Left image Shifted right
feature image feature
Left Camera Image Right Camera Image
112. Dense Stereo Estimation
source
sour
∞ ce ∞
Unary Potential
xi0 xk0
∞ ∞ xk1 xk2
. .
∞ ∞ The cost of every cut should be
equal to the corresponding
. . energy under the encoding
∞ ∞
xiD xkD
cost = +
sink
Ishikawa PAMI03
113. Dense Stereo Estimation
source
sour
∞ ce ∞
Unary Potential
xi0 xk0
∞ ∞ xk1 xk2
cost = +
. .
∞ ∞ The cost of every cut should be
equal to the corresponding
. . energy under the encoding
∞ ∞
xiD xkD
sink
Ishikawa PAMI03
114. Dense Stereo Estimation
source
sour
∞ ce ∞
Unary Potential
xi0 xk0
∞ ∞
cost = ∞
xk1 xk2
. .
∞ ∞ The cost of every cut should be
equal to the corresponding
. . energy under the encoding
∞ ∞
xiD xkD
sink
Ishikawa PAMI03
115. Dense Stereo Estimation
source
sour
∞ ce ∞
Pairwise Potential
K
xi0 xk0
∞ ∞ xk1 xk2
cost = + +K
K
. .
∞ ∞ The cost of every cut should be
K equal to the corresponding
. . energy under the encoding
∞ ∞
K
xiD xkD
sink
Ishikawa PAMI03
116. Dense Stereo Estimation
source
sour
∞ ce ∞
Pairwise Potential
K
xi0 xk0
∞ ∞ xk1 xk2
K
. .
∞ ∞ The cost of every cut should be
K equal to the corresponding
. . energy under the encoding
∞ ∞
K
xiD xkD
cost = + +DK
sink
Ishikawa PAMI03
117. Dense Stereo Estimation
source
sour
ce ∞ Extendable to any convex cost
∞
xi0 xk0
∞ ∞ xk1 xk2 Convex function
. .
∞ ∞
. .
∞ ∞
xiD xkD See [Ishikawa PAMI03] for more details
sink
118. Graph Cut based Inference
Move making algorithms
• Original problem decomposed into a series of subproblems
solvable with graph cut
• In each subproblem we find the optimal move from the
current solution in a restricted search space
Encoding
Initial solution
Propose Build Update solution
Graph Cut
move Graph solution
119. Graph Cut based Inference
Example : [Boykov et al. 01]
αβ-swap
– each variable taking label α or β can change its label to α or β
– all αβ-moves are iteratively performed till convergence
α-expansion
– each variable either keeps its old label or changes to α
– all α-moves are iteratively performed till convergence
Transformation function :
120. Graph Cut based Inference
Sufficient conditions for move making algorithms :
(all possible moves are submodular)
αβ-swap : semi-metricity
Proof:
121. Graph Cut based Inference
Sufficient conditions for move making algorithms :
(all possible moves are submodular)
α-expansion : metricity
Proof:
122. Object-class Segmentation
Data term Smoothness term
Data term
Discriminatively trained classifier
Smoothness term
where
125. Object-class Segmentation
grass building
grass
Original Image Initial solution Building expansion
sky
building
grass
Sky expansion
126. Object-class Segmentation
grass building
grass
Original Image Initial solution Building expansion
sky sky
tree
building building
grass grass
Sky expansion Tree expansion
127. Object-class Segmentation
grass building
grass
Original Image Initial solution Building expansion
sky sky sky
tree tree building
building building aeroplane
grass grass grass
Sky expansion Tree expansion Final Solution
128. Graph Cut based Inference
Range-(swap) moves [Veksler 07]
– Each variable in the convex range can change its label to any
other label in the convex range
– all range-moves are iteratively performed till convergence
Range-expansion [Kumar, Veksler & Torr 1]
– Expansion version of range swap moves
– Each varibale can change its label to any label in a convex
range or keep its old label
129. Dense Stereo Reconstruction
Left Camera Image Right Camera Image Dense Stereo Result
Data term
Same as before
Smoothness term
130. Dense Stereo Reconstruction
Left Camera Image Right Camera Image Dense Stereo Result
Data term
Same as before
Smoothness term
Convex range Truncation
132. Graph Cut based Inference
Original Image Initial Solution After 1st expansion
133. Graph Cut based Inference
Original Image Initial Solution After 1st expansion
After 2nd expansion
134. Graph Cut based Inference
Original Image Initial Solution After 1st expansion
After 2nd expansion After 3rd expansion
135. Graph Cut based Inference
Original Image Initial Solution After 1st expansion
After 2nd expansion After 3rd expansion Final solution
136. Graph Cut based Inference
• Extendible to certain classes of binary higher order energies
• Higher order terms have to be transformable to the pairwise submodular
energy with binary auxiliary variables zC
Higher order term Pairwise term
Encoding
Initial solution
Propose Transf. to Update solution
Graph Cut
move pairwise subm. solution
137. Graph Cut based Inference
• Extendible to certain classes of binary higher order energies
• Higher order terms have to be transformable to the pairwise submodular
energy with binary auxiliary variables zC
Higher order term Pairwise term
Example :
Kolmogorov ECCV06, Ramalingam et al. DAM12
138. Graph Cut based Inference
Enforces label consistency of the clique as a weak constraint
Input image Pairwise CRF Higher order CRF
Kohli et al. CVPR07
139. Graph Cut based Inference
If the energy is not submodular / graph-representable
– Overestimation by a submodular energy
– Quadratic Pseudo-Boolean Optimisation (QPBO)
Encoding
Initial solution solution
Propose Transf. to Update
Graph Cut
move pairwise subm. solution
140. Graph Cut based Inference
Overestimation by a submodular energy (convex / concave)
– We find an energy E’(t) s.t.
– it is tight in the current solution E’(t0) = E(t0)
– Overestimates E(t) for all t E’(t) ≥ E(t)
– We replace E(t) by E’(t)
– The moves are not optimal, but quaranteed to converge
– The tighter over-estimation, the better
Example :
Encoding
Initial solution
Propose Overestim. by Update solution
Graph Cut
move pairwise subm. solution
141. Graph Cut based Inference
Quadratic Pseudo-Boolean Optimisation
– Each binary variable is encoded using two binary
_ _
variables xi and xi s.t. xi = 1 - xi
142. Graph Cut based Inference
Quadratic Pseudo-Boolean Optimisation
source
xi xj
_ _ _
xi xj
– Energy submodular in xi and xi
_
– Solved by dropping the constraint xi = 1 - xi
_
– If xi = 1 - xi the solution for xi is optimal
sink
143. Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
144. Motivation
• To propose formulations enforcing certain structure
properties of the output useful in computer vision
– Label Consistency in segments as a weak constraint
– Label agreement over different scales
– Label-set consistency
– Multiple domains consistency
• To propose feasible Graph-Cut based inference method
145. Structures in CRF
• Taskar et al. 02 – associative potentials
• Woodford et al. 08 – planarity constraint
• Vicente et al. 08 – connectivity constraint
• Nowozin & Lampert 09 – connectivity constraint
• Roth & Black 09 – field of experts
• Woodford et al. 09 – marginal probability
• Delong et al. 10 – label occurrence costs
146. Object-class Segmentation Problem
• Aims to assign a class label for each pixel of an image
• Classifier trained on the training set
• Evaluated on never seen test images
147. Pairwise CRF models
Standard CRF Energy for Object Segmentation
Local context
Cannot encode global consistency of labels!!
Ladický, Russell, Kohli, Torr ECCV10
148. Encoding Co-occurrence
[Heitz et al. '08]
Co-occurrence is a powerful cue [Rabinovich et al. ‘07]
• Thing – Thing
• Stuff - Stuff
• Stuff - Thing
[ Images from Rabinovich et al. 07 ] Ladický, Russell, Kohli, Torr ECCV10
149. Encoding Co-occurrence
[Heitz et al. '08]
Co-occurrence is a powerful cue [Rabinovich et al. ‘07]
• Thing – Thing
• Stuff - Stuff
• Stuff - Thing
Proposed solutions :
1. Csurka et al. 08 - Hard decision for label estimation
2. Torralba et al. 03 - GIST based unary potential
3. Rabinovich et al. 07 - Full-connected CRF
[ Images from Rabinovich et al. 07 ] Ladický, Russell, Kohli, Torr ECCV10
152. Desired properties
1. No hard decisions
Incorporation in probabilistic framework
Unlikely possibilities are not completely ruled out
Ladický, Russell, Kohli, Torr ECCV10
153. Desired properties
1. No hard decisions
2. Invariance to region size
Ladický, Russell, Kohli, Torr ECCV10
154. Desired properties
1. No hard decisions
2. Invariance to region size
Cost for occurrence of {people, house, road etc .. }
invariant to image area
Ladický, Russell, Kohli, Torr ECCV10
155. Desired properties
1. No hard decisions
2. Invariance to region size
The only possible solution :
L(x)={ , , }
Local context Global context
Cost defined over the
assigned labels L(x)
Ladický, Russell, Kohli, Torr ECCV10
156. Desired properties
1. No hard decisions
2. Invariance to region size
3. Parsimony – simple solutions preferred
L(x)={ aeroplane, tree, flower,
building, boat, grass, sky }
L(x)={ building, tree, grass, sky }
Ladický, Russell, Kohli, Torr ECCV10
157. Desired properties
1. No hard decisions
2. Invariance to region size
3. Parsimony – simple solutions preferred
4. Efficiency
Ladický, Russell, Kohli, Torr ECCV10
158. Desired properties
1. No hard decisions
2. Invariance to region size
3. Parsimony – simple solutions preferred
4. Efficiency
a) Memory requirements as O(n) with the image size and
number or labels
b) Inference tractable
Ladický, Russell, Kohli, Torr ECCV10
159. Previous work
• Torralba et al.(2003) – Gist-based unary potentials
• Rabinovich et al.(2007) - complete pairwise graphs
• Csurka et al.(2008) - hard estimation of labels present
Ladický, Russell, Kohli, Torr ECCV10
160. Related work
• Zhu & Yuille 1996 – MDL prior
• Bleyer et al. 2010 – Surface Stereo MDL prior
• Hoiem et al. 2007 – 3D Layout CRF MDL Prior
C(x) = K |L(x)|
• Delong et al. 2010 – label occurence cost
C(x) = ΣLKLδL(x)
Ladický, Russell, Kohli, Torr ECCV10
161. Related work
• Zhu & Yuille 1996 – MDL prior
• Bleyer et al. 2010 – Surface Stereo MDL prior
• Hoiem et al. 2007 – 3D Layout CRF MDL Prior
C(x) = K |L(x)|
• Delong et al. 2010 – label occurence cost
C(x) = ΣLKLδL(x)
All special cases of our model
Ladický, Russell, Kohli, Torr ECCV10
164. Inference for Co-occurence
Move Energy
Decomposition to α-dependent and α-independent part
α-independent α-dependent
Ladický, Russell, Kohli, Torr ECCV10
165. Inference for Co-occurence
Move Energy
Decomposition to α-dependent and α-independent part
Either α or all labels in the image after the move
Ladický, Russell, Kohli, Torr ECCV10
167. Inference for Co-occurence
Move Energy
non-submodular
Non-submodular energy overestimated by E'(t)
• E'(t) = E(t) for current solution
• E'(t) ≥ E(t) for any other labelling
Ladický, Russell, Kohli, Torr ECCV10
168. Inference for Co-occurence
Move Energy
non-submodular
Non-submodular energy overestimated by E'(t)
• E'(t) = E(t) for current solution
• E'(t) ≥ E(t) for any other labelling
Occurrence - tight
Ladický, Russell, Kohli, Torr ECCV10
169. Inference for Co-occurence
Move Energy
non-submodular
Non-submodular energy overestimated by E'(t)
• E'(t) = E(t) for current solution
• E'(t) ≥ E(t) for any other labelling
Co-occurrence overestimation
Ladický, Russell, Kohli, Torr ECCV10
170. Inference for Co-occurence
Move Energy
non-submodular
Non-submodular energy overestimated by E'(t)
• E'(t) = E(t) for current solution
• E'(t) ≥ E(t) for any other labelling
General case
[See the paper]
Ladický, Russell, Kohli, Torr ECCV10
171. Inference for Co-occurence
Move Energy
non-submodular
Non-submodular energy overestimated by E'(t)
• E'(t) = E(t) for current solution
• E'(t) ≥ E(t) for any other labelling
Quadratic representation
Ladický, Russell, Kohli, Torr ECCV10
172. Potential Training for Co-occurence
Generatively trained Co-occurence potential
Approximated by 2nd order representation
Ladický, Russell, Kohli, Torr ICCV09
173. Results on MSRC data set
Comparisons of results with and without co-occurence
Input Image Without With Input Image Without With
Co-occurence Co-occurence Co-occurence Co-occurence
Ladický, Russell, Kohli, Torr ICCV09
174. Pairwise CRF models
Pixel CRF Energy for Object-class Segmentation
Data term Smoothness term
Input Image Pairwise CRF Result
Shotton et al. ECCV06
175. Pairwise CRF models
Pixel CRF Energy for Object-class Segmentation
Data term Smoothness term
• Lacks long range interactions
• Results oversmoothed
• Data term information may be insufficient
176. Pairwise CRF models
Segment CRF Energy for Object-class Segmentation
Data term Smoothness term
Input Image Unsupervised Shi, Malik PAMI2000,
Comaniciu, Meer PAMI2002,
Segmentation Felzenschwalb, Huttenlocher, IJCV2004,
177. Pairwise CRF models
Segment CRF Energy for Object-class Segmentation
Data term Smoothness term
Input Image Pairwise CRF Result
Batra et al. CVPR08, Yang et al. CVPR07, Zitnick et al. CVPR08,
Rabinovich et al. ICCV07, Boix et al. CVPR10
178. Pairwise CRF models
Segment CRF Energy for Object-class Segmentation
Data term Smoothness term
• Allows long range interactions
• Does not distinguish between instances of objects
• Cannot recover from incorrect segmentation
179. Pairwise CRF models
Segment CRF Energy for Object-class Segmentation
Data term Smoothness term
• Allows long range interactions
• Does not distinguish between instances of objects
• Cannot recover from incorrect segmentation
180. Robust PN approach
Segment CRF Energy for Object-class Segmentation
Pairwise CRF Segment
consistency
term
Input Image Higher order CRF
Kohli, Ladický, Torr CVPR08
181. Robust PN approach
Segment CRF Energy for Object-class Segmentation
Pairwise CRF Segment
consistency
term
Kohli, Ladický, Torr CVPR08
182. Robust PN approach
Segment CRF Energy for Object-class Segmentation
Pairwise CRF Segment
consistency
term
No dominant
label
Kohli, Ladický, Torr CVPR08
183. Robust PN approach
Segment CRF Energy for Object-class Segmentation
Pairwise CRF Segment
consistency
term
Dominant label l
Kohli, Ladický, Torr CVPR08
184. Robust PN approach
Segment CRF Energy for Object-class Segmentation
Pairwise CRF Segment
consistency
term
• Enforces consistency as a weak constraint
• Can recover from incorrect segmentation
• Can combine multiple segmentations
• Does not use features at segment level
Kohli, Ladický, Torr CVPR08
185. Robust PN approach
Segment CRF Energy for Object-class Segmentation
Pairwise CRF Segment
consistency
term
Transformable to pairwise graph with auxiliary variable
Kohli, Ladický, Torr CVPR08
186. Robust PN approach
Segment CRF Energy for Object-class Segmentation
Pairwise CRF Segment
consistency
term
where
Ladický, Russell, Kohli, Torr ICCV09
187. Robust PN approach
Segment CRF Energy for Object-class Segmentation
Pairwise CRF Segment
consistency
term
Input Image Higher order CRF Ladický, Russell, Kohli, Torr ICCV09
188. Associative Hierarchical CRFs
Can be generalized to Hierarchical model
• Allows unary potentials for region variables
• Allows pairwise potentials for region variables
• Allows multiple layers and multiple hierarchies
Ladický, Russell, Kohli, Torr ICCV09
189. Associative Hierarchical CRFs
AH CRF Energy for Object-class Segmentation
Pairwise CRF Higher order term
Ladický, Russell, Kohli, Torr ICCV09
190. Associative Hierarchical CRFs
AH CRF Energy for Object-class Segmentation
Pairwise CRF Higher order term
Higher order term recursively defined as
Segment Segment Segment higher
unary term pairwise term order term
Ladický, Russell, Kohli, Torr ICCV09
191. Associative Hierarchical CRFs
AH CRF Energy for Object-class Segmentation
Pairwise CRF Higher order term
Higher order term recursively defined as
Segment Segment Segment higher
unary term pairwise term order term
Why is this generalisation useful?
Ladický, Russell, Kohli, Torr ICCV09
192. Associative Hierarchical CRFs
Let us analyze the case with
• one segmentation
• potentials only over segment level
Ladický, Russell, Kohli, Torr ICCV09
193. Associative Hierarchical CRFs
Let us analyze the case with
• one segmentation
• potentials only over segment level
1) Minimum is segment-consistent
Ladický, Russell, Kohli, Torr ICCV09
194. Associative Hierarchical CRFs
Let us analyze the case with
• one segmentation
• potentials only over segment level
2) Minimum is segment-consistent
3) Cost of every segment-consistent CRF equal to the
cost of pairwise CRF over segments
Ladický, Russell, Kohli, Torr ICCV09
195. Associative Hierarchical CRFs
Let us analyze the case with
• one segmentation
• potentials only over segment level
Equivalent to pairwise CRF over segments!
Ladický, Russell, Kohli, Torr ICCV09
196. Associative Hierarchical CRFs
• Merges information over multiple scales
• Easy to train potentials and learn parameters
• Allows multiple segmentations and hierarchies
• Allows long range interactions
• Limited (?) to associative interlayer connections
Ladický, Russell, Kohli, Torr ICCV09
197. Inference for AH-CRF
Graph Cut based move making algorithms [Boykov et al. 01]
α-expansion transformation function for base layer
Ladický (thesis) 2011
198. Inference for AH-CRF
Graph Cut based move making algorithms [Boykov et al. 01]
α-expansion transformation function for base layer
α-expansion transformation function for auxiliary layer
(2 binary variables per node)
Ladický (thesis) 2011
200. Inference for AH-CRF
Interlayer connection between the base layer and the first auxiliary layer:
The move energy is:
where
Ladický (thesis) 2011
201. Inference for AH-CRF
The move energy for interlayer connection between the base and auxiliary layer is:
Can be transformed to binary submodular pairwise function:
Ladický (thesis) 2011
202. Inference for AH-CRF
The move energy for interlayer connection between auxiliary layers is:
Can be transformed to binary submodular pairwise function:
Ladický (thesis) 2011
208. Potential training for Object class Segmentation
Pixel unary potential
Unary likelihoods based on spatial configuration (Shotton et al. ECCV06)
Classifier trained using boosting
Ladický, Russell, Kohli, Torr ICCV09
209. Potential training for Object class Segmentation
Segment unary potential
Classifier trained using boosting (resp. Kernel SVMs)
Ladický, Russell, Kohli, Torr ICCV09
217. Dense Stereo Reconstruction
Unary Potential
Disparity = 0
Unary Cost dependent on the similarity of
patches, e.g.cross correlation
218. Dense Stereo Reconstruction
Unary Potential
Disparity = 5
Unary Cost dependent on the similarity of
patches, e.g.cross correlation
219. Dense Stereo Reconstruction
Unary Potential
Disparity = 10
Unary Cost dependent on the similarity of
patches, e.g.cross correlation
220. Dense Stereo Reconstruction
Pairwise Potential
• Encourages label consistency in adjacent pixels
• Cost based on the distance of labels
Linear Truncated Quadratic Truncated
222. Dense Stereo Reconstruction
Does not work for Road Scenes !
Different brightness Patches can be matched to any
in cameras other patch for flat surfices
223. Dense Stereo Reconstruction
Does not work for Road Scenes !
Different brightness Patches can be matched to any
in cameras other patch for flat surfices
Could object recognition for road scenes help?
Recognition of road scenes is relatively easy
224. Joint Dense Stereo Reconstruction
and Object class Segmentation
• Object class and 3D location are
mutually informative
∞
• Sky always in infinity
(disparity = 0)
sky
225. Joint Dense Stereo Reconstruction
and Object class Segmentation
• Object class and 3D location are
mutually informative
• Sky always in infinity
(disparity = 0)
• Cars, buses & pedestrians
have their typical height
sky building car
226. Joint Dense Stereo Reconstruction
and Object class Segmentation
• Object class and 3D location are
mutually informative
• Sky always in infinity
(disparity = 0)
• Cars, buses & pedestrians
have their typical height
• Road and pavement on the
ground plane
sky building car road
227. Joint Dense Stereo Reconstruction
and Object class Segmentation
• Object class and 3D location are
mutually informative
• Sky always in infinity
(disparity = 0)
• Cars, buses & pedestrians
have their typical height
• Road and pavement on the
ground plane
• Buildings and pavement on
the sides
sky building car road
228. Joint Dense Stereo Reconstruction
and Object class Segmentation
• Object class and 3D location are
mutually informative
• Sky always in infinity
(disparity = 0)
• Cars, buses & pedestrians
have their typical height
• Road and pavement on the
ground plane
• Buildings and pavement on
the sides
• Both problems formulated as CRF
• Joint approach possible? sky building car road
229. Joint Formulation
• Each pixels takes label zi = [ xi yi ] ∈ L1 × L2
• Dependency of xi and yi encoded as a unary and pairwise
potential, e.g.
• strong correlation between x = road, y = near ground plane
• strong correlation between x = sky, y = 0
• Correlation of edge in object class and disparity domain
230. Joint Formulation
Unary Potential
Object layer
Joint unary links
Disparity layer
• Weighted sum of object class, depth and joint potential
• Joint unary potential based on histograms of height
231. Joint Formulation
Pairwise Potential
Object layer
Joint pairwise links
Disparity layer
• Object class and depth edges correlated
• Transitions in depth occur often at the object boundaries
233. Inference
• Standard α-expansion
• Each node in each expansion move keeps its old label
or takes a new label [xL1, yL2],
• Possible in case of metric pairwise potentials
234. Inference
• Standard α-expansion
• Each node in each expansion move keeps its old label
or takes a new label [xL1, yL2],
• Possible in case of metric pairwise potentials
Too many moves! ( |L1| |L2| )
Impractical !
235. Inference
• Projected move for product label space
• One / Some of the label components remain(s)
constant after the move
• Set of projected moves
• α-expansion in the object class projection
• Range-expansion in the depth projection
236. Dataset
• Leuven Road Scene dataset
• Contained Left camera Right camera Object GT Disparity GT
• 3 sequences
• 643 pairs of images
• We labelled
• 50 training + 20 test images
• Object class (7 labels)
• Disparity (100 labels)
• Publicly available
• http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip
237. Qualitative Results
• Large improvement for dense stereo estimation
• Minor improvement in object class segmentation
239. Summary
• We proposed :
• New models enforcing higher order structure
• Label-consistency in segments
• Consistency between scale
• Label-set consistency
• Consistency between different domains
• Graph-Cut based inference methods
• Source code http://cms.brookes.ac.uk/staff/PhilipTorr/ale.htm
There is more to come !
Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
Many different approaches have been proposed for dealing with co-occurrence . In order to justify our approach, I'll run through some basic criteria that we think a co-occurrence potential should satisfy. Firstly, text...
Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
There are 3 existing approaches to co-occurrence. Describe None Of them, exhibit invariance, and there is a direct trade off between the use of a global or dynamic cost function, that adapts to the current labelling and computational efficiency.
There are 3 existing approaches to co-occurrence. Describe None Of them, exhibit invariance, and there is a direct trade off between the use of a global or dynamic cost function, that adapts to the current labelling and computational efficiency.
There are 3 existing approaches to co-occurrence. Describe None Of them, exhibit invariance, and there is a direct trade off between the use of a global or dynamic cost function, that adapts to the current labelling and computational efficiency.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.