2008_12 ISM2008 Semantics Meets UX


Published on

ISM2008 presentation: "Semantics Meets UX: Mediating Intelligent Indexing of Consumers’ Multimedia Collections for Multifaceted Visualization and Media Creation" by Hibino, Loui, Wood, Fryer and Cerosaletti (Eastman Kodak Company)

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

2008_12 ISM2008 Semantics Meets UX

  1. 1. Semantics Meets UX: Mediating Intelligent Indexing of Consumers’ Multimedia Collections for Multifaceted Visualization and Media Creation Stacie Hibino, Alex Loui, Mark D. Wood, Sam Fryer, Cathleen Cerosaletti Eastman Kodak Company {hibino, alexander.loui, mdw, samuel.fryer, cathleen.cerosaletti}@kodak.com ISM 2008 Conference Presentation 16 December 2008
  2. 2. Overview Problem Statement and Approach SSDF Architecture Semantic Indexing: Automatically Derived Metadata Inferencing Engine Workflow and User Experience: Browse and Search GUI Based on Semantic Indexing Media Creation Summary ©Eastman Kodak Company, 2008 2
  3. 3. Problem Statement and Approach Problem • Increasing size of consumer photo and video clip collections • Desire by users to experience their collections, not organize them Approach • Semantic System Demonstration Framework (SSDF) – Flexible and extensible framework – Combines multiple semantic indexing algorithms for consumer photo and video clip collections into one integrated system • Koi, an SSDF desktop client application – Creates a user experience designed to mediate and leverage the intelligent indexing incorporated in SSDF • Together, Koi and SSDF empower users to experience their personal multimedia in novel and sophisticated ways ©Eastman Kodak Company, 2008 3
  4. 4. SSDF Architecture … Client Client Network Web Server … Asset Query Upload Servlet Processor Operating System System Manager Semantic Indexer(s) Asset Database Triple Inference Store Store Engine ©Eastman Kodak Company, 2008 4
  5. 5. Semantic Indexing Overview Goal • Create rich semantic descriptions of multimedia content Technology Needs • Detection and recognition of faces and people • Detection and recognition of typical consumer events • Detection of location of an event • Search and retrieve images and videos based on content • Sorting of images and videos based on specific image characteristics • Classification of scene for image retrieval ©Eastman Kodak Company, 2008 5
  6. 6. People Recognition • Unconstrained imaging conditions and pose • Repeated examples of same small set under different conditions • Family members spanning all ages Facial Detect Faces Features Feature Extraction Recognize Faces Classification Results Face Madeline Alicia Database ©Eastman Kodak Company, 2008 6
  7. 7. Event Clustering and Auto-Labeling Unorganized collection Use Date, Time, Metadata, and Image Content Analysis to Automatically Group and Label Organized event grouping with label Label May 28, 2002 Super event Label Memorial Day ©Eastman Kodak Company, 2008 7
  8. 8. CBIR Workflow Primitive Features: color histogram, Histogram of color composition, etc. ROI Primitive Features Sorted Compare Display List Feature Sets from global, Image tiles, and Database tile groups ©Eastman Kodak Company, 2008 8
  9. 9. Image Value Index (IVI) To automatically compute a set of probabilistic measures of image content and semantic characteristics comprising the following elements Measures based on technical quality (sharpness Technical and contrast) Measures based on aesthetic quality Aesthetic Measures based on social significance of people Social in image (via face recognition and social relationship) Measures based on occasion of image acquisition Event Measures based on user responses and image Usage use ©Eastman Kodak Company, 2008 9
  10. 10. Inferencing Engine A1 ContainsPerson Soccer Higher order reasoning CapturedOn supported by modeling 2008-12-15 Likes people and asset metadata as P1 HasIVI a semantic network HasGender ParentOf • Enables inference of 4 Female relationships between people P2 • Enables intelligent, semi- HasEventType ParentOf automated media creation Sports Semantic network SpouseOf represented using RDF data P3 model HasName ParentOf Inferencing based on Prolog- Alice like query language P4 HasGender Female ©Eastman Kodak Company, 2008 10
  11. 11. Workflow and User Experience Overview The Koi client user experience was designed to mediate the results of automated indexing by • Presenting multiple views of the same data and corresponding interactions to leverage strengths of individual and combined algorithm results • Supporting multifaceted browsing • Enabling user correction in a way that is not disruptive to users’ current activities ©Eastman Kodak Company, 2008 11
  12. 12. Screen Layout quick filters current view navigation bar notifications current filter sort order main viewing area side menus ©Eastman Kodak Company, 2008 12
  13. 13. Koi: Set Views Drifting View Calendar Views – All Years • Displays 12−20 media at a time, drifting horizontally across – Single Year screen – Month over Multiple Years • Size, speed, and vertical – Single Month placement of media are – Day View randomly determined • Users can navigate between • Video clips auto play without calendar views by clicking a audio as they drift across the year, month, or day screen, adding surprise Grid View Slide Show View • Standard “light table” layout for • Presents a slide show of the efficiently viewing many media currently filtered set of assets ©Eastman Kodak Company, 2008 13
  14. 14. Set View: Calendar View – All Years ©Eastman Kodak Company, 2008 14
  15. 15. Set View: Calendar View – Month Over Yrs ©Eastman Kodak Company, 2008 15
  16. 16. Koi: Pop-Up Views 1. Single Picture View Expanded view of image or video clip 2. Details View Extracted metadata with links for facet-based navigation 3. Connection View Related media across four dimensions (People, Event, Image Similarity, Scene Type), visually depicting Single Picture View connections between media Connection View Details View ©Eastman Kodak Company, 2008 16
  17. 17. Pop-Up View: Single Picture Details Extracted metadata with links for facet-based navigation ©Eastman Kodak Company, 2008 17
  18. 18. Pop-Up View: Connection View Displays related media by: People, Event, Image Similarity, Scene Type Visually depicts connections between media • Supports connection-based browsing (click outer media to place in center) ©Eastman Kodak Company, 2008 18
  19. 19. Koi: Other Views People Views All People Shows one face thumbnail per people cluster, each cluster initially set by people-clustering algorithm Family Displays family relationships Edit Person Enables users to enter in profile and relationship information for individuals Event View Displays assets by event, using event-clustering algorithm Everyday View (for Groups ) Summarizes group-based media, activities (e.g., rating, notes), and presence ©Eastman Kodak Company, 2008 19
  20. 20. Media Creation and Notification System-Generated Media Creation • System looks for opportunities to create composited media “stories” • Story generation triggered by date or recent uploads – Date example: Mother’s Day is in a week; system automatically produces an album featuring mother and kids – Event example: Pictures recently uploaded from a sporting event; system produces sport-themed multimedia creation • Triggers use inferencing engine Notification of New Stories • RSS-based notification mechanism • Notifier provides user with story name, click to preview ©Eastman Kodak Company, 2008 20
  21. 21. Summary SSDF architecture and Koi client provide a new approach to a flexible, semantically aware client-server architecture System supports multifaceted search, browse, and creation based on automatically extracted and algorithmically derived metadata Koi employs a novel user interface for searching and browsing collections SSDF architecture provides an extensible platform for adding additional intelligence Potential future work includes evolving existing algorithms to increased performance levels as well as integrating new user-tested algorithms ©Eastman Kodak Company, 2008 21
  22. 22. Backup Slides ©Eastman Kodak Company, 2008 22
  23. 23. Set View: Grid View ©Eastman Kodak Company, 2008 23
  24. 24. Search Menu ©Eastman Kodak Company, 2008 24