Downloaded 14 times
Uploaded byLynda Hardman
Creating Media Stories, SSSW July 13
The document discusses the creation of stories using media and the importance of metadata for user interaction and multimedia applications. It emphasizes the need for mechanisms to identify and annotate media assets to enhance user experience in accessing and interpreting information. Multiple examples of application areas and processes, such as the Cewe color photobook and the Voxpopuli system for generating video documentaries, are presented to illustrate these concepts.
More Related Content
Similar to Creating Media Stories, SSSW July 13
Creating Media Stories, SSSW July 13
- 1. Creating stories withmedia Lynda Hardman http://www.cwi.nl/~lynda CWI, Information Access UvA, Institute for Informatics http://www.flickr.com/photos/iboy/4528401870/
- 2. Video collection: What doyou think of war as a solution? 2 http://www.flickr.com/photos/soldiersmediacenter/1183836576
- 4. Interactive Media Access •Users need support for – finding desired content – in one or more media types – for their specific task • We need to be aware that there is more than the information “expressed'' by the media asset itself, e.g. – when/where the media was captured – the intended purpose of the creator – the context in which the media asset was created 4
- 5. We need toenable • the processing of information-bearing content • of one or more media types • that can be interpreted by end users End-users are primarily interested in • the meaning conveyed by a combination of media assets • interacting further with the media – as part of complex “search” task – passing it on to someone else in media “chain” We don’t care about the media! 5http://www.flickr.com/photos/benheine/4687572408/
- 6. How can weget this to work? We need mechanisms – for identifying (part of) an individual media asset – for associating metadata with an identified fragment – for agreeing on the meaning of metadata – that enable larger meaningful structures to be composed, identified and annotated 6http://www.flickr.com/photos/jannem/3312115991
- 7. Outline of talk •Information processes in which media and metadata play a role –“canonical processes of media production” • VoxPopuli system, demonstrating high level user interaction enabled by media and metadata • Study of information needs for videos that are able to support processes of opinion-forming and decision-making 7
- 8. Workflow for Multimedia Applications •Identify and define a number of canonical processes of media production • Community effort 8 –2005: Dagstuhl seminar – 2005: ACM MM Workshop on Multimedia for Human Communication –2008: Multimedia Systems Journal Special Issue (core model and companion system papers) editors: Frank Nack, ZeljkoObrenovicand Lynda Hardman http://link.springer.com/journal/530/14/6/
- 9. CanonicalReduced to thesimplest and most significant form possible, without loss of generality
- 10. Example application areas •multimedia feature extraction systems • professional news production systems (VRT) • new media art • hyper-video production • photo book production (CeWe) • ambient multimedia systems with complex sensory networks
- 11. Overview of CanonicalProcesses 11
- 12. Example: CeWe ColorPhotoBook • Application for authoring digital photo books • Automatic selection, sorting and ordering of photos – Context analysis methods: e.g., timestamp, annotation – Content analysis methods: e.g., color histograms, edge detection • Customized layout and background 12 http://www.cewe-photobook.com
- 13. CeWe Color PhotoBookProcesses My winter ski holidays with my friends 13 Premeditate Construct Message
- 14. CeWe Color PhotoBookProcesses • Media assets are captured, generated or transformed Create
- 15. CeWe Color PhotoBookProcesses 15 Annotate
- 16. CeWe Color PhotoBookProcesses 16
- 17. CeWe Color PhotoBookProcesses 17 Organize Query
- 18. landscape portrait 18 Organise using domainannotations skiing holiday
- 19. CeWe Color PhotoBookProcesses 19 Publish Distribute
- 20. CeWe Color PhotoBookProcesses 20
- 21.
- 22. Creating Stories withMedia • Long term goal to find and present information to end-users – In a way that is useful to them • We understand how to design information interfaces by hand. – How can metadata help us in giving more flexible access to media collections? • We can link media assets to existing linked data, and use this to improve presentation, e.g. by – Selecting a sub-set – Grouping, ordering and linking media assets – Influencing the (visual) presentation 22
- 23. How can annotationshelp? What can be expressed explicitly? – the message to be conveyed – objects that are depicted in a media asset – domain information (e.g., art, painter) – human communication roles (discourse) What can they be used for? – disambiguating query terms – grouping similar items for conveying topic breadth – visualizing items for presentation, e.g. timeline, map – finding similar items – … 23
- 24. VoxPopuli: Generating video documentaries fromannotated media repositories Stefano Bocconi, Frank Nack(CWI, Amsterdam)
- 25. Video Documentaries onthe Web • Traditional video authoring: there is only one final version, what is shown is the choice of the author/editor • Proposed video sequence creation: – Annotate the video material – Show automatically what the user asks to see, using presentation forms a film editor would use 25
- 26. Video material • Interviewwith America video footage with interviews and background material about the opinion of American people after 9- 11www.interviewwithamerica.com • Filming 27-10-2001 to 01-11-2001 in Stamford (CT), NewYork(NY), Boston(MA) and Cleveland(OH) 26 Premeditate
- 27. Example: What doyou think of the war in Afghanistan? “I am never a fan of military action, in the big picture I don’t think it is ever a good thing, but I think there are circumstances in which I certainly can’t think of a more effective way to counter this sort of thing...” 27 http://www.flickr.com/photos/soldiersmediacenter/1183836576
- 28. The annotations Rhetorical – RhetoricalStatement (mostly verbal, but visual also possible) – Argumentation model: Toulmin model Descriptive – Question asked – Interviewee (social) – Filmic next slide 28 Annotate
- 29. Filmic annotations Continuity, e.g. –camera movement none, pan left/right, shaking, tilt up/down, zoom in/out – framing continuity close-up, medium shot, long shot – gaze direction of speaker left, centre, right – lighting conditions – background sound 29 Continuity, e.g. – camera movement none, pan left/right, shaking, tilt up/down, zoom in/out – framing continuity close-up, medium shot, long shot – gaze direction of speaker left, centre, right – lighting conditions – background sound Annotate
- 30. Statement encoding 3-part statement: –<subject><modifier><predicate> – E.g. “warbest solution” Thesaurus (pre Wordnet) containing: – Terms (155) – Relations between terms: similar(72), opposite(108), generalization(10), specialization(10) e.g. war opposite diplomacy 30 Annotate 3-part statement: – <subject><modifier><predicate> – E.g. “warbest solution” Thesaurus (pre Wordnet) containing: – Terms (155) – Relations between terms: similar(72), opposite(108), generalization(10), specialization(10) e.g. war opposite diplomacy
- 31. Connect statements • Usingthe thesaurus, generate related statements and query the repository “war best solution”, “diplomacy best solution”, “war not solution” • Create a graph of related statements – nodes are the statements (corresponding to video segments) – edges are either support or contradict 31
- 32. Statement Graph 32 = support =contradict war best solution war not solution diplomacy best solution
- 33. Toulmin model 33 ClaimData Qualifier Warrant Backing Condition Concession 57 Claims,16 Data, 4 Concessions, 3 Warrants, 1 Condition
- 34. Toulmin in example 34 Claim Concession Claimcontradict support Claim I am not a fan of military action War has never solved anything Two billions dollar bombs on tents I cannot think of a more effective solution weaken Annotate
- 35. What do youthink of the war in Afghanistan? 35 I am not a fan of military actions War has never solved anything I cannot think of a more effective solution Two billions dollar bombs on tents
- 36.
- 37.
- 38. VoxPopuli Conclusions • Automaticgeneration of video interviews augmented with supporting and/or contradicting material • The user can determine the subject and the bias of the presentation • The documentarist can add material and let the system generate new documentaries 38
- 39. Pointer & Acknowledgments •More on VoxPopuli at: http://www.cwi.nl/~media/demo/VoxPopuli/ • VoxPopuli was funded by the Dutch national ToKeN I2RP and CHIME projects 39
- 40. User information needsfor environmental opinion-forming and decision-making in linked-enriched video Ana Carina Palumbo University of Amsterdam Technical University of Eindhoven
- 41. The message • Informcitizens about environmental issues • Scenario: users form opinions while watching videos (from TV or internet) • Goal: specify information that should be captured in annotations
- 42. Method • Expert interviews –environmental governance – video production and broadcasting • User survey – information users say they need – 215 participants • User experiment – information users really select – 6 participants
- 43. High-level overview ofresults
- 44. Environmental video conclusions •We know what to annotate in environmental videos to support opinion-forming • Some annotations can be done automatically (e.g. subjects and concepts, location) • Others need to be manually annotated • Challenges: – level of objectivity and subjectivity – trustworthiness of sources. EuroITV '13
- 45. What are mymessages? • Annotations associated with media assets can be used for different stages of interactive access, not just searching • Annotations can be added by hand, linked automatically or automatically extracted • The intended message can be made explicit (more annotations) • Media content and associated annotations can be passed among systems • We need community agreement on how to do this (e.g. canonical processes) • Users can be given much richer and more flexible access to (annotated) media content, but… • we need to store annotations & media in a reusable way 45 45
- 46. Acknowledgements 46 Raphaël Troncy Ana Carina Palumbo Jaccovan Ossenbruggen Stefano Bocconi Frank Nack Andre Fialho
- 47. 47 Literature • Special Issueon Canonical Processes of Media Production http://link.springer.com/journal/530/14/6/ http://www.cwi.nl/~media/projects/canonical/ • Lynda Hardman, ZeljkoObrenovic, Frank Nack, Brigitte Kerhervé and Kurt Piersol: Canonical Processes of Semantically Annotated Media Production. Multimedia Systems Journal, 14(6), 2008 • Philipp Sandhaus, Sabine Thieme and Susanne Boll: Canonical Processes in Photo Book Production. Multimedia Systems Journal, 14(6), 2008 • Stefano Bocconi, Frank Nack and Lynda Hardman: Automatic generation of matter-of-opinion video documentaries. Journal of Web Semantics, 6(2), p139-150, 2008. • Ana Carina Palumbo and Lynda Hardman: User information needs for environmental opinion-forming and decision-making in link-enriched video In EuroITV '13, pp 85-88
Editor's Notes
- #5 [[PICTURE OF MONA LISA, MORE RECENT POLITICAL FIGURE crowd taksim square]]While user task oriented, we need to remember that a picture is not WYSIWYG…Some of these aspects can be captured in associated metadata.
- #6 In some sense we don’t even care about the media for its own sake.We care about the story or the information conveyed by the media. (the narrative in a broad sense of the word)E.g., in the context of a complex search task; or passing the media along to someone else in the “media chain”
- #7 These mechanisms are necessary, but aren't enough:We need toolsto create houses – butthey don't tell us how to be architects.Similarly, if we want to communicate using media, then we need to firstunderstand what it is we want to communicate and then how we are able to communicate it.Raphaël Troncy, EURECOM, has done a lot of work on these components.
- #11 The rule for contribution was: there needed to be an already-implemented system. This should then be described in terms of the canonical processes.
- #15 Many thanks to Raphaël Troncy, EURECOM, for the use of the photos.After a process of premeditation, however short or long, at some point a media asset is created. Some device or another is used to collect images or sound for a period of time, be it photo or video camera, scanner, sound recorder, heart-rate monitor, MRI etc. A media asset may already exist, and the creation process is the transformation of some exist asset to a new one (e.g. to make the sky look brighter). (The media asset may even be sensor data.)Images can be created with image editing programs, or generated by transforming one or more existing images. We are not interested in the method of creation per se. If the method is considered as significant, however, then this information should be recorded as part of the annotation.
- #41 So, if I am claiming that we can explicitly model the message, then how do we go about doing this?In this case we study the goals of the communication process itself. In some sense, we are putting ourselves into the seat of the documentary maker. Given infinite resources, then what annotations would we like to have.
- #44 Most requested types of information:Factual, exact, unbiased and objective data – Neutral peopleSafety, security, health implications: risks, outcomes, consequencesAdvantages and disadvantages: Costs and benefitsAlternatives: Level of community needWhen we have these annotations, we can create a VoxPopuli type system.
- #45 Next step is to work out the associated costs and benefits for these different annotations.Once we have them, we can incorporate them in to the Linked Open Data cloud. We need to do them once per genre, but not once per “program”.Ana Carina Palumbo and Lynda Hardman. User information needs for environmental opinion-forming and decision-making in link-enriched video EuroITV. http://diglib.project.cwi.nl:2076/10.1145/2465958.2465973
- #46 These interactive grouping, story telling at least as important as only searchCheck the slide for details