Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Model-Driven Design of Audiovisual Indexing Processes for Search Apps.


Published on

As the Web becomes a platform for multimedia content fruition, audiovisual search assumes a central role in providing users with the content most adequate to their information needs. A key issue for enabling audiovisual search is extracting indexable knowledge from opaque media. Such a process is heavily constrained by scalability and performance issues and must be able to flexibly incorporate specialized components for educing selected features from media elements. This paper shows how the use of a model-driven approach can help designers specify multimedia indexing processes, verify properties of interest in such processes, and generate the code that orchestrates the components, so as to enable rapid prototyping of content analysis processes in presence of evolving requirements.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Model-Driven Design of Audiovisual Indexing Processes for Search Apps.

  1. 1. Model-Driven Design of Audiovisual Indexing Processes for Search-Based Applications Alessandro Bozzon, Marco Brambilla , Piero Fraternali CBMI 2009, June 5 th , Chania, Crete.
  2. 2. Agenda <ul><li>Introduction </li></ul><ul><li>Search-based applications (SBAs) </li></ul><ul><li>The Content Provision and Annotation (CPA) process </li></ul><ul><ul><li>Definition </li></ul></ul><ul><ul><li>Verification of properties </li></ul></ul><ul><li>Conclusions </li></ul>
  3. 3. Introduction <ul><li>Search has become the prominent paradigm for information seeking across both online spaces and enterprises. </li></ul><ul><li>Search-Based Applications (SBAs) </li></ul><ul><li>searching over heterogeneous data constitutes the predominant user interaction paradigm </li></ul><ul><li>Search Engine </li></ul><ul><li>canned applications </li></ul><ul><li>fixed behavior </li></ul><ul><li>homogeneous data source </li></ul><ul><li>simple content processing flows </li></ul><ul><li>basic query flows </li></ul><ul><li>SBA </li></ul><ul><li>tailor-made solution, depending on varying data and user's needs nature </li></ul><ul><li>search engine s are part of a complex system </li></ul><ul><ul><li>data source integration </li></ul></ul><ul><ul><li>content analysis </li></ul></ul><ul><ul><li>querying </li></ul></ul><ul><ul><li>Web-mediated social interactions </li></ul></ul>
  4. 4. Motivations <ul><li>SBAs are complex application integrating: </li></ul><ul><ul><li>complex front-end: query expression, result presentation, personalization, adaptation </li></ul></ul><ul><ul><li>complex back-end: content provisioning, annotation, indexing, distributed query execution </li></ul></ul><ul><li>SBAs embody data-intensive and process-intensive tasks </li></ul><ul><ul><ul><li>content items , queries , and result lists are first-class objects, subject to a life cycle </li></ul></ul></ul><ul><ul><ul><li>Content Provisioning and Annotation process ( CAI process) </li></ul></ul></ul><ul><ul><ul><ul><li>indexation of contents coming from the application data sources (data retrieval from external sources, transformation or aggregation, analysis, indexation) </li></ul></ul></ul></ul><ul><ul><ul><li>Query and Result Presentation process ( QRP process) </li></ul></ul></ul><ul><ul><ul><ul><li>operations related to query analysis, execution, orchestration and result-set composition </li></ul></ul></ul></ul><ul><ul><ul><li>User Interaction process ( UI process) </li></ul></ul></ul><ul><ul><ul><ul><li>the way users interact with the application's functionalities ( statement of information needs, result navigation, social interactions) </li></ul></ul></ul></ul>
  5. 5. Thesis <ul><li>Current practices lack complete tools and methodologies </li></ul><ul><li>rely either on very simple models or on the programming skills of the developers </li></ul><ul><ul><li>low separation of concerns among involved actors </li></ul></ul><ul><ul><li>low productivity in the implementation phase </li></ul></ul><ul><ul><li>problems in managing and maintaining applications over time </li></ul></ul><ul><li>SBA development demands for the same evolution in methods and tools that has characterized in the recent past the progress of Software Engineering for Web applications </li></ul><ul><ul><li>clear separation of concerns among the involved actors </li></ul></ul><ul><ul><li>central roles of models as key development artifacts </li></ul></ul><ul><ul><li>automatic code generation, etc. </li></ul></ul>SBA Model Driven Development Framework
  6. 6. The contribution <ul><li>A modeling framework </li></ul><ul><li>an intuitive Rich Process Model to specify the schema of the CPA process for a given application, </li></ul><ul><li>use of formal methods to investigate properties of interest of his design </li></ul><ul><li>code generation techniques to implement and deploy the process </li></ul>
  7. 7. Background <ul><li>Information Retrieval </li></ul><ul><ul><li>” ... the process of finding contents (e.g., textual documents, multimedia items, etc.) that satisfy an information need from within large collections (usually stored on computers)...” </li></ul></ul><ul><li>Business Process Design </li></ul><ul><ul><li>representing processes (of heterogeneous nature) in terms of related, structured activities or tasks that produce a specific service or product </li></ul></ul><ul><ul><li>several proposals for visual modeling languages (e.g., UML, YAML, BPMN ) </li></ul></ul><ul><li>Model Driven Web Engineering </li></ul><ul><ul><li>raise of abstraction (separation of platform independent and platform dependent concerns) in Web application design and development </li></ul></ul><ul><ul><li>use of models (and model transformations) as the key artifacts for application developments </li></ul></ul><ul><ul><li>several proposals (e.g., UML, Hera, OOHDM, UWE, W2000, WebML ) </li></ul></ul><ul><li>Formal methods </li></ul><ul><ul><li>Formal semantics to models </li></ul></ul><ul><ul><li>Automatic verification </li></ul></ul>
  8. 8. Background Models BPMN WebML <ul><li>Business Process Model Notation </li></ul><ul><li>Activities </li></ul><ul><li>FlowsConstraints (OR-XOR-AND gateways) </li></ul><ul><li>Artifacts (Data Objects and data associations) </li></ul><ul><li>Events </li></ul><ul><li>Web Modeling Language </li></ul><ul><li>Domain ModelsNavigation Models ( content publication and manipulation, link behavioral semantics) </li></ul><ul><li>Service composition and orchestration Models ( Web service invocation and publication, XML management) </li></ul>
  9. 9. Modeling Approach <ul><li>Process Models: BPMN (CAI, QRP), Interaction Pattern Composition (UI) </li></ul><ul><li>Domain data and process metadata: ER/UML </li></ul><ul><li>Application Models: WebML </li></ul><ul><li>Model To Model Transformation: Java / ATL </li></ul><ul><li>Model To Code Transformation: Java </li></ul>
  10. 10. CPA Process - Domain Model <ul><li>Content model : the objects that relate to the Content Items indexed by a search application </li></ul><ul><li>Annotation model : structure of the annotations associated with searchable Content Items during the indexing process </li></ul><ul><li>Usage model : user, roles, permissions </li></ul><ul><li>Index model : </li></ul><ul><li>abstraction for the actual physical implementation of search engine indexes </li></ul>
  11. 11. CPA Process - Process and Application design <ul><li>Coarse CPA process model </li></ul><ul><ul><li>Content Registration </li></ul></ul><ul><ul><li>Content Analysis </li></ul></ul><ul><ul><li>Content Indexation </li></ul></ul><ul><li>Fine-grained process model </li></ul><ul><ul><li>Analysis of audiovisual content trough face recognition and identification technologies </li></ul></ul><ul><li>Application model </li></ul><ul><ul><li>Invocation algorithm of annotation technologies </li></ul></ul>Refinement M2M Transformation M2T Transformation <ul><li>Running CPA process </li></ul><ul><ul><li>Console trace of the working annotation technology </li></ul></ul><ul><ul><li>Process advancement control UI </li></ul></ul>Full Size Video
  12. 12. CPA - Examples of complex processes Analysis of audiovisual content Analysis of audio-visual content with textual annotations
  13. 13. Search-specific extensions to the process model (BPMN*) <ul><li>BPMN </li></ul><ul><ul><li>does not provide support for: </li></ul></ul><ul><ul><ul><li>process data </li></ul></ul></ul><ul><ul><ul><li>data flows and dependencies among activities </li></ul></ul></ul><ul><ul><li>does not convey domain-specific information about the modeled process for activity’s business logic </li></ul></ul><ul><li>Formalized extensions to the BPMN language, introducing: </li></ul><ul><ul><li>typed attributes (content, annotation, process) </li></ul></ul><ul><ul><li>typed activities </li></ul></ul><ul><ul><li>activity properties </li></ul></ul><ul><ul><ul><li>type of produced and consumed data </li></ul></ul></ul><ul><ul><ul><li>type of enforced operations (classified w.r.t. their input/output) </li></ul></ul></ul><ul><ul><li>finer-grained data flows </li></ul></ul><ul><ul><ul><li>mapping of links with guard conditions and parameters </li></ul></ul></ul>
  14. 14. BPMN* > WebML transformation <ul><li>More detailed transformation rules </li></ul><ul><ul><li>finer-grained Application Model, needing less (or no) refinement by the designer </li></ul></ul><ul><ul><ul><li>typed activities enables reusable PIM models </li></ul></ul></ul><ul><ul><ul><li>data dependencies are specified at an higher level </li></ul></ul></ul><ul><ul><li>less errors in Application Model design </li></ul></ul><ul><ul><li>fastest SBA development </li></ul></ul>ATL Transformation Example of ATL BPMD-WebML transformation rule for activities
  15. 15. Verification of properties for CPA processes <ul><li>Assessment of CPA process outputs </li></ul><ul><ul><li>Algebraic representation </li></ul></ul><ul><ul><ul><li>Expression of an activity depend on the type (number and type of input/output) of the activity and on the schema (activity pre-condition , mutual exclusions or indetermination in setting activity inputs) of the process </li></ul></ul></ul><ul><ul><li>The expression evaluation requires assigning the semantics to each activity </li></ul></ul><ul><ul><ul><li>functions from input to output values, described extensively, as tabular expression (or ground-truth) </li></ul></ul></ul><ul><li>Variation of activities’ business logics (or in the process schema) produces variations in the representativeness of annotations </li></ul><ul><ul><li>assessment of produced annotation quality, validation against expected outcomes, compare outcomes of different processes </li></ul></ul><ul><li>indexAnnotation( </li></ul><ul><li>AggregateAnnotation( </li></ul><ul><ul><li>AnalyzeAudio(TranscodeAudioVideo( </li></ul></ul><ul><ul><ul><li>RetrieveVideo()).audioOut), </li></ul></ul></ul><ul><ul><li>AnalyzeFaces(TranscodeAudioVideo( </li></ul></ul><ul><ul><ul><li>RetrieveVideo()).videoOut))) </li></ul></ul></ul>
  16. 16. Implementation <ul><li>Extensions to a commercial tool for the automatic generation of Web applications </li></ul><ul><li>Creation of a Web-based BPMN editor for the specification of the CAI and QRP process models </li></ul><ul><li>Set of model to model and model to text transformations </li></ul>
  17. 17. Experiences <ul><li>PHAROS ( P latform for searc H of A udiovisual R esources across O nline S paces) project </li></ul><ul><ul><li>First-hand experience on a large scale SBA application </li></ul></ul><ul><ul><ul><li>Adoption of models in the platform design </li></ul></ul></ul><ul><ul><ul><ul><li>Domain model </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Query and result presentation process </li></ul></ul></ul></ul><ul><ul><ul><ul><li>User interaction process </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Experiments with Content Analysis and </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Indexing process </li></ul></ul></ul></ul><ul><ul><li>Developer evaluation </li></ul></ul>Pros Cons Complete set of design dimensions High-quality prototypes Quick prototyping cycles Fast development time Lack of reverse transformations from application to process models Availability of only simple guard condition expressions
  18. 18. Conclusions and future work <ul><li>A modeling framework </li></ul><ul><li>an intuitive and visual design of the CPA process, </li></ul><ul><ul><li>Helps the initial design of the process </li></ul></ul><ul><ul><li>Helps for its evolution when requirements change </li></ul></ul><ul><li>the extensibility of the model, based on a plugin approach </li></ul><ul><li>the amenability to automatic verification </li></ul><ul><ul><li>for checking properties </li></ul></ul><ul><ul><li>for estimating the characteristics of a specific CPA configuration </li></ul></ul><ul><li>the availability of well-established code generation technology </li></ul><ul><ul><li>great improvement in productivity </li></ul></ul><ul><li>Future work </li></ul><ul><li>extension of the validation capabilities </li></ul><ul><li>industrial implementation </li></ul>
  19. 19. Thank You! Marco Brambilla [email_address]