Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

"Image and Video Summarization," a Presentation from the University of Washington

541 views

Published on

For the full video of this presentation, please visit:
http://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/dec-2016-member-meeting-uofw

For more information about embedded vision, please visit:
http://www.embedded-vision.com

Professor Jeff Bilmes of the University of Washington delivers the presentation "Image and Video Summarization" at the December 2016 Embedded Vision Alliance Member Meeting. Bilmes provides an overview of the state of the art in image and video summarization.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

"Image and Video Summarization," a Presentation from the University of Washington

  1. 1. Bigness Info. in Data Image Summarization Video Summarization End Image and Video Summarization Jeffrey A. Bilmes Professor Departments of Electrical Engineering & Computer Science and Engineering University of Washington, Seattle http://melodi.ee.washington.edu/~bilmes and Visiting Scientist, Google Research Wednesday, Dec 7th, 2016 J. Bilmes Image and Video Summarization — 12/7/2016 page 1 / 35
  2. 2. Bigness Info. in Data Image Summarization Video Summarization End Outline 1 Bigness 2 Information In Data 3 Image Summarization 4 Video Summarization 5 End J. Bilmes Image and Video Summarization — 12/7/2016 page 2 / 35
  3. 3. Bigness Info. in Data Image Summarization Video Summarization End Summarization What is summarization? Why do we need summarization? J. Bilmes Image and Video Summarization — 12/7/2016 page 4 / 35
  4. 4. Bigness Info. in Data Image Summarization Video Summarization End Bigness J. Bilmes Image and Video Summarization — 12/7/2016 page 5 / 35
  5. 5. Bigness Info. in Data Image Summarization Video Summarization End Water H2O molecules small (n-body) medium (fluid dynamics, viscosity, compressibility), large (global weather systems, meteorology). Same underlying molecular collision events! J. Bilmes Image and Video Summarization — 12/7/2016 page 6 / 35
  6. 6. Bigness Info. in Data Image Summarization Video Summarization End Neurons Neurons small (neural spike trains, population coding) medium (intelligence, consciousness, psychology) large (society, social choice, wisdom of the crowd) Same underlying electrical and chemical impulses. J. Bilmes Image and Video Summarization — 12/7/2016 page 7 / 35
  7. 7. Bigness Info. in Data Image Summarization Video Summarization End Bigger is Different “More is Different”, P.W. Anderson, 1972 (Nobel laureate). “The ability to reduce everything to simple fundamental laws does not imply the ability to start from those laws and reconstruct the universe.” “. . . alterations of being . . . are not only the transition of one magnitude into another, but a transition from quantity into quality,” Hegel, The Science of Logic, 1816 J. Bilmes Image and Video Summarization — 12/7/2016 page 8 / 35
  8. 8. Bigness Info. in Data Image Summarization Video Summarization End Big Data is Different Data Hypothesis: extremely large data sets offer qualitatively different capabilities than small data sets. Evidence: Image Completion (Hays & Efros, 2007) “our initial experiments . . . on a dataset of ten thousand images were very discouraging. However, increasing the image collection to two million yielded a qualitative leap in performance” J. Bilmes Image and Video Summarization — 12/7/2016 page 9 / 35
  9. 9. Bigness Info. in Data Image Summarization Video Summarization End Modern Times Big Data is Big and Getting Even Bigger. Sensors & Devices Social Media VoIP Enterprise Data Now A Few Years From Now A Few Years Ago VolumeofData 2.5 quintillion bytes (2.5 million terabytes) of data/day (IBM) >90% of the world’s data has been created in the last two years. J. Bilmes Image and Video Summarization — 12/7/2016 page 10 / 35
  10. 10. Bigness Info. in Data Image Summarization Video Summarization End Big Data in Machine Learning Statistics, Machine Learning, and Artificial Intelligence (AI) “There’s no data like more data”, Computational Consequences: Massive computational resource demands! Research opportunities to address new computational challenges 1 systems programming, parallel and distributed computing, network topologies, efficient databases, GPUs. 2 Examples: map reduce, Hadoop, GraphLab, HaLoop, Greenplum, Asterix, Spark, SystemML, MLBase, Myria, etc. J. Bilmes Image and Video Summarization — 12/7/2016 page 11 / 35
  11. 11. Bigness Info. in Data Image Summarization Video Summarization End The Two Foundations of Big Data? Data StatisticalSignificance ParallelComputingSystems ?????????????? Large Data StatisticalSignificance Larg J. Bilmes Image and Video Summarization — 12/7/2016 page 12 / 35
  12. 12. Bigness Info. in Data Image Summarization Video Summarization End Goal: Data to Information Data is: Streaming Torrential Relentless Multi-modal Mostly Unstructured Sensors/Actuators Redundant High Dimensional Distributed J. Bilmes Image and Video Summarization — 12/7/2016 page 13 / 35
  13. 13. Bigness Info. in Data Image Summarization Video Summarization End Big Data is Different Data: A Proposition Hypothesis: extremely large data sets offer qualitatively different capabilities than small data sets. Problem: Big data sets are big, unwieldy, computationally challenging, and often highly redundant. Research Quest: Can statistical predictions and actions be made cost effectively using the right data management strategies? Yes, by reducing redundancy. J. Bilmes Image and Video Summarization — 12/7/2016 page 14 / 35
  14. 14. Bigness Info. in Data Image Summarization Video Summarization End How to identify and measure redundancy? J. Bilmes Image and Video Summarization — 12/7/2016 page 15 / 35
  15. 15. Bigness Info. in Data Image Summarization Video Summarization End How to identify and measure redundancy? J. Bilmes Image and Video Summarization — 12/7/2016 page 16 / 35
  16. 16. Bigness Info. in Data Image Summarization Video Summarization End Measuring Information in Data What is information? Information Theory (entropy, mutual information) ⇔ probability distributions. Kolmogorov Complexity ⇔ algorithms & models of computation. Information measures over non-random data samples (e.g., images). f( ) = f( , ) < f( , ) < f( , ) J. Bilmes Image and Video Summarization — 12/7/2016 page 18 / 35
  17. 17. Bigness Info. in Data Image Summarization Video Summarization End Measuring Information in Data Diminishing returns: The more you have, the less valuable is anything you don’t have. f( )f( )- f( ) - f( ) ≥ J. Bilmes Image and Video Summarization — 12/7/2016 page 19 / 35
  18. 18. Bigness Info. in Data Image Summarization Video Summarization End Example: Number of Colors of Balls in Urns Consider an urn containing colored balls. Given a set S of balls, f (S) counts the number of distinct colors. Initial value: 2 (colors in urn). New value with added blue ball: 3 Initial value: 3 (colors in urn). New value with added blue ball: 3 Submodularity : Incremental Value of Object Diminishes in a Larger Context (diminishing returns). Thus, f is submodular. J. Bilmes Image and Video Summarization — 12/7/2016 page 20 / 35
  19. 19. Bigness Info. in Data Image Summarization Video Summarization End As the data set size grow . . . There is no data like more data ⇒ more data is like no more data. From Andrew Ng’s Stanford machine learning class, 2011 J. Bilmes Image and Video Summarization — 12/7/2016 page 21 / 35
  20. 20. Bigness Info. in Data Image Summarization Video Summarization End As the data set sizes grow . . . 0.70 0.75 0.80 0.85 0.90 0.95 1.00 0.1 1 10 100 1000 Millions of Words TestAccuracy Memory-Based Winnow Perceptron Naïve Bayes Banko & Brill 2001 (Riccardi & Hakkani-Tür, 2005, Speech Recognition) (Callison-Burch&Bloodgood, 2010, Machine Translation) Tong & Koller, 2001 (Soon, Ng, Lim, 2001, Coreference Resolution) 50 100 150 200 250 Number of Training Examples 0 0.2 0.4 0.6 0.8 1 Accuracy (Kadie, 1995, Generic Classification) Sentiment Tutorial, Chris Potts, Stanford Ling., 2011 78 80 82 84 86 88 90 92 700 1.750 3.500 7.000 10.500 14.000 17.500 21.000 Overallaccuracy(%) Training set size SVMs DTs ( Kavzoglu & Colkesen, 2012, Image Classification) Speed, memory, attention, problem solving playing game Luminosity http://www.lumosity.com/blog/how-much-and-how-often-should-i-train/ J. Bilmes Image and Video Summarization — 12/7/2016 page 22 / 35
  21. 21. Bigness Info. in Data Image Summarization Video Summarization End Submodularity and Learning Curves Proposition Let V = {1, 2, . . . , n} be a finite ground set, and let f : 2V → R be a set function. If for all permutations σ of V , we have that for all i ≤ j: f (σj |Si−1) ≥ f (σj |Sj−1) (1) with Si = {σ1, σ2, . . . , σi }, then f is submodular. Learning curves might not be exactly submodular, but submodularity seems a reasonable model. J. Bilmes Image and Video Summarization — 12/7/2016 page 23 / 35
  22. 22. Bigness Info. in Data Image Summarization Video Summarization End What is Summarization? 1 Start with a massive data set (images, videos, etc.), set V . 2 Identify a good (submodular) information function f (by hand or by machine learning) that represents information in V . 3 Find a subset A ⊆ V of a given size that retains as much information as possible. 4 Luckily, this normally exponential time problem can be done computationally efficiently!! J. Bilmes Image and Video Summarization — 12/7/2016 page 24 / 35
  23. 23. Bigness Info. in Data Image Summarization Video Summarization End Modern Image Collections Many images, also that have a higher level gestalt than just a few. J. Bilmes Image and Video Summarization — 12/7/2016 page 26 / 35
  24. 24. Bigness Info. in Data Image Summarization Video Summarization End Image Summarization Task: Summarize collection of images by representative subset of the images Applications: Summarizing your holiday pictures. Summarizing image search results Efficient browsing of image collections Video frame summarization Difficulties: No high level ⇓ J. Bilmes Image and Video Summarization — 12/7/2016 page 27 / 35
  25. 25. Bigness Info. in Data Image Summarization Video Summarization End Image Summarization - Data Collection Data Statistics 14 image collections with 100 pictures each ∼ 400 human summaries for every image collection, via Amazon Turk, about 5500 summaries total! Example collections: J. Bilmes Image and Video Summarization — 12/7/2016 page 28 / 35
  26. 26. Bigness Info. in Data Image Summarization Video Summarization End Image Summarization Whole collection: 3 best summaries: 3 medium summaries: 3 worst summaries: J. Bilmes Image and Video Summarization — 12/7/2016 page 29 / 35
  27. 27. Bigness Info. in Data Image Summarization Video Summarization End Image Summarization Typical Results - Learnt mixture using Max-Margin f(∅) = 0 f(V ) = 1 Greedy Min Average Pruned Random Max of Learned Mixture Average Pruned Human Greedy Max Average Pruned Random Average Pruned Human J. Bilmes Image and Video Summarization — 12/7/2016 page 30 / 35
  28. 28. Bigness Info. in Data Image Summarization Video Summarization End Real-Time Running Online Video Summarization Live Video Feed Most recent representative video snippet (repeating) Next most recent representative video snippet (repeating) Third most recent representative video snippet (repeating) Fourth most recent representative video snippet (repeating) Least recent representative video snippet (repeating) ... J. Bilmes Image and Video Summarization — 12/7/2016 page 32 / 35
  29. 29. Bigness Info. in Data Image Summarization Video Summarization End Real-Time Running Online Video Summarization J. Bilmes Image and Video Summarization — 12/7/2016 page 33 / 35
  30. 30. Bigness Info. in Data Image Summarization Video Summarization End The End The End: Thank you! ++ + + f(A) f(B) f(A ∪ B) = f(Ar ) +f(C) + f(Br ) ≥ ≥ = f(A ∩ B) f(A ∩ B) = f(Ar ) + 2f(C) + f(Br ) J. Bilmes Image and Video Summarization — 12/7/2016 page 35 / 35

×