A brief introduction to extracting information from images


Published on

Lecture at the 2011 European Summer School on Information Retrieval (ESSIR 2011). 31st August 2011.

Published in: Education, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Reuters got in some trouble because of image manipulation recently, and this resulted in a backlash in the press. There is a blog “photoshop disasters” with many examples of tampering; here are just a few...
  • This is a case of image tampering in an image published Reuters and later withdrawn by Reuters. The image depicts Beirut after an Israeli air strike. The tampering makes the scene look worse than it perhaps was. The use of the clone tool is quite evident however.August 2006: This photograph by Adnan Hajj, a Lebanese photographer, showed thick black smoke rising above buildings in the Lebanese capital after an Israeli air raid. The Reuters news agency initially published this photograph on their web site and then withdrew it when it became evident that the original image had been manipulated to show more and darker smoke. "Hajj has denied deliberately attempting to manipulate the image, saying that he was trying to remove dust marks and that he made mistakes due to the bad lighting conditions he was working under", said Moira Whittle, the head of public relations for Reuters. "This represents a serious breach of Reuters' standards and we shall not be accepting or using pictures taken by him." A second photograph by Hajj was also determined to have been doctored.** The picture on the left was created around 1864 - it is supposed to depict Ulysses S. Grant in front of his troops not far from here, at City Point Virginia. Unfortunately, this is an example of an early forgery; the rider on the horse, is actually Major General McCook. McCook and his horse have been superimposed on the image an image of confederate prisoners at Fishers Hill, and Grant’s head on top of this!circa 1864: This print purports to be of General Ulysses S. Grant in front of his troops at City Point, Virginia, during the American Civil War. Some very nice detective work by researchers at the Library of Congress revealed that this print is a composite of three separate prints: (1) the head in this photo is taken from a portrait of Grant; (2) the horse and body are those of Major General Alexander M. McCook; and (3) the background is of Confederate prisoners captured at the battle of Fisher's Hill, VA.
  • So, images can be tampered with, but is there any way to detect this automatically? There is a whole research field based around the idea of forensic techniques. Here are two examples of the kind of automatic forensic processing that is possibleCloning parts of images to hide something is common. In this case the original picture showed George bush on a lectern. Automatic analysis is able to detect the manipulations.
  • A brief introduction to extracting information from images

    1. 1. A brief introduction to extracting information from images<br />Jonathon Hare<br />University of Southampton<br />
    2. 2. What can images tell us?<br />How are images represented in digital computers<br />How do we extract information from images<br />Examples of some different extraction techniques<br />Analogies with text<br />Free software!<br />Contents<br />
    3. 3. Images CAN…<br />the main roles of images in the communications process<br />
    4. 4. Attract attention and make documents more appealing<br />
    5. 5.
    6. 6. Convey opinions and emotional messages<br />
    7. 7.
    8. 8. Convey information for documenting a claim<br />
    9. 9.
    10. 10. Representation and Understanding<br />how a computer “sees”<br />
    11. 11. digital image Representation<br /> 87 91 85 ...<br /> 86 86 81 ...<br /> 88 85 84 ...<br />... ... ... ...<br />137 145 144 ...<br />153 150 137 ...<br />148 139 123 ...<br />... ... ... ...<br /> 89 91 89 ...<br /> 84 88 90 ...<br /> 88 87 90 ...<br />... ... ... ...<br />
    12. 12. Understanding An Image<br />
    13. 13. Feature Extraction<br />f(x)<br />Feature extraction is the process of extracting “descriptors” from an image. <br />Descriptors describe some aspect of the image content.<br />Typically, a descriptor is a numerical vector called a “feature vector”, however other forms of descriptor are possible.<br />
    14. 14. Higher-level features<br />Directly interpretable by humans<br />i.e. the number of faces in the image<br />Either hand-crafted or trained with machine learning techniques<br />Lower-level features<br />Much more abstract; convey a notion of the image content<br />i.e. the colour distribution of the image<br />IMAGE Feature Morphology<br />
    15. 15. Example High-Level Features<br />faces, composition & photoshop disasters<br />
    16. 16. The detection of faces in an image is a very useful feature for inferring information about an image<br />Face detection is the first step of face recognition<br />The most popular face detection algorithm is the “Viola-Jones” detector<br />Conceptually simple<br />Uses machine learning; Requires training (slow).<br />Very fast detection<br />High-level features: face Detection<br />
    17. 17. Viola-Jones face Detection<br />Bank of filters. Consider all possible position, scale and type parameters<br />(very large numbers of features)<br />For each feature create a simple (weak) binary classifier (a stump)<br />Use ADABOOST to select the informative features <br />P. Viola, M. Jones, Robust Real-Time Face Detection, IJCV, Vol. 57(2), 2004. (first version appeared at CVPR 2001)<br />
    18. 18. Viola-Jones face Detection<br />P. Viola, M. Jones, Robust Real-Time Face Detection, IJCV, Vol. 57(2), 2004. (first version appeared at CVPR 2001)<br />
    19. 19. Photographers use the “rule-of-thirds” to improve the composition of their photos.<br />The basic idea is to place main subjects at roughly one-third of the horizontal or vertical dimension of the photograph. <br />High-level features: Composition<br />
    20. 20. High-level features: Composition<br />It is possible to design features that look for the presence of composition using the rule-of thirds<br />image<br />saliency map<br />segments + saliency map<br />distance to closest power-point<br />area of segment * saliency of segment<br />Che-HuaYeh, Yuan-Chen Ho, Brian A. Barsky, and Ming Ouhyoung. "Personalized Photograph Ranking and Selection System". In ACM Multimedia 2010, pages 211–220, October 2010.<br />
    21. 21. High-level features: Tampering<br />
    22. 22. High-level features: Tampering<br />
    23. 23. High-level features: Tampering<br />A Political Advertisement for George W. Bush<br />Automatic cloning detection (“copy-move” forgery)<br />
    24. 24. Example LOW-Level Features<br />colour histograms, segments and sift<br />
    25. 25. Global features describe the content of an entire image<br />One of the simplest global features is the “Global RGB Colour Histogram”<br />Quantise each pixel into a discrete number of colours and then build a histogram.<br />Low-Level Features: Global<br />
    26. 26. Global features are useful for some tasks, but in many cases are not powerful enough<br />Local features attempt to overcome this by breaking the image into smaller parts from which to extract features<br />Three primary techniques for splitting up the image<br />Low-level features: Local<br />segmentation<br />salient regions &<br />interest points<br />grids & blocks<br />
    27. 27. Salient interest regions and their associated features are currently the most popular way of describing an image content.<br />Extracting image features using interest regions is a two-part process:<br />Find regions<br />Extract feature to describe region properties<br />Typically, the resultant image feature will have a variable length, dependent on the number of regions<br />Salient interest regions<br />
    28. 28. Important regions portray:<br />Repeatability<br />Saliency<br /> Corners and blobs have these qualities<br />Detectable using various techniques<br />Difference of Gaussian - corners<br />Harris corner detector - corners<br />MSER - blobs<br />Salient interest region Location<br />corners<br />blobs<br />
    29. 29. Good region descriptors portray:<br />Resilience to image transforms<br />Compactness<br />Emphasise different image characteristics:<br />Pixel intensities, colour, texture, edges etc.<br />Common descriptors include:<br />SIFT: histogram of edge orientation<br />Shape context: histogram of edge location<br />Salient interest region descriptors<br />
    30. 30. SIFT: Scale Invariant Feature Transform<br />
    31. 31. Analogies with text<br />introducing the visual bag-of-words<br />
    32. 32. In the computer vision community over recent years it has become popular to model the content of an image in a similar way to a “bag-of-terms” in textual document analysis.<br />Bags of Visual Words<br />
    33. 33. Features localised by a robust region detector and described by a local descriptor such as SIFT.<br />A vocabulary of exemplar feature-vectors is learnt.<br />Traditionally through k-means clustering.<br />Local descriptors can then be quantised to discrete visual terms by finding the closest exemplar in the vocabulary.<br />BoVW using local features<br />
    34. 34. BOVW models have many applications<br />Auto-annotation and object recognition<br />Concept classification<br />Large-scale indexing<br />Applications of BOVW<br />
    35. 35. open-source tools for image analysis and indexing<br />introducing openimaj & imageterrier<br />
    36. 36. http://www.openimaj.org<br />Open-source (BSD Licence) libraries and tools for multimedia (image, video, sound) analysis and information extraction<br />Implemented in Java; use with any JVM language<br />Implementations of all the techniques mentioned in this tutorial<br />Scalability of extraction using Hadoop with the included tools<br />
    37. 37. http://www.imageterrier.org<br />Extension to the Terrier retrieval system to allow indexing of images<br />Collections and documents that read data produced from image feature extractors.<br />New indexers and supporting classes to make compressed augmented inverted indices for visual term data.<br />New distance measures implemented as WeightingModels.<br />Geometric re-ranking implemented as DocumentScoreModifiers.<br />Command-line tools for indexing and searching.<br />Freely available under the Mozilla Licence<br />