The document describes the MICO broker, an orchestration framework developed as part of the MICO project to extract metadata from multimedia content using linked data extractors. The MICO broker was initially implemented using RabbitMQ but had limitations. A new version uses Apache Camel for workflow execution, supports extractor registration and configuration, and allows semi-automatic workflow creation based on extractor dependencies and input/output types. The goal is to support distributed, error-handling processing of content and provide administration feedback on performance.

1. © Fraunhofer IDMT
Patrick Aichroth, Marcel Sieland, Luca Cuccovillo, Thomas Köllmer
The MICO Broker: An Orchestration Framework
for Linked Data Extractors
Fraunhofer IDMT
Thomas Köllmer
thomas.koellmer@idmt.fraunhofer.de

2. © Fraunhofer IDMT 2
The MICO Project
 http://www.mico-project.eu
 3.4 Million Euro Project
 7 Partners: Salzburg Research, Fraunhofer IDMT, Oxford University,
University of Passau, Umeå University, Zaizi, InsideOut10
 Duration: 36 Months (end: 10/2016)
 Goal of MICO is to develop a platform to analyse “media in context
” …
 ...by orchestrating different content extraction tools that can work
simultaneously or in a sequence and …..
 ...provide valuable metadata to third party applications.

3. © Fraunhofer IDMT 3
MICO and the MICO broker
MICO: EU project to develop a platform for distributed analysis of
multimedia content, based on Linked Data:
 media (textual, visual, audio) extractors
 persistence (binary + metadata/annotations)
 extractor orchestration (“broker”)
 common metadata model (MMM, Anno4J)
 cross-media querying (SPARQL/MM)
 cross-media reco (using all data sources, content-based + CF-based)

4. © Fraunhofer IDMT 4
MICO and the MICO broker

5. © Fraunhofer IDMT 5
An example workflow / pipeline

6. © Fraunhofer IDMT 6
An example workflow / pipeline

7. © Fraunhofer IDMT 7
An example workflow / pipeline

8. © Fraunhofer IDMT 8
 Initial broker version
 implemented on top of RabbitMQ
 accessible via MICO Event API (Java and C++)
 content item input queue: receiving new content
 content item reply-to queue: providing extractor results
 registry queue: extractor service registration
 Initial RabbitMQ approach proved to be quite stable and robust, but:
 orchestration via simple mime-types: unintended processing and loops
 Other features desired → new Broker version needed
Broker challenges and requirements (1/4)

9. © Fraunhofer IDMT 9
Broker challenges and requirements (2/4)
 Extractor dependencies and parameters
 support for extractor configuration
 support for extractor “modes”
 support for more than one extractor input and output
 support for extractor versioning
 Workflow creation
 support for extractor dependency checking
 simple workflow creation

10. © Fraunhofer IDMT 10
 Workflow execution
 support for several workflows per MICO instance
 support for error handling
 support for workflow status tracking and logging
 support for Enterprise Integration Pattern (EIP) support, e.g. routing,
aggregation, splitting
 support for dynamic routing for context-aware processing
Broker challenges and requirements (3/4)

11. © Fraunhofer IDMT 11
Broker challenges and requirements (4/4)

12. © Fraunhofer IDMT 12
 Extractor information is provided at different times (during vs. after packaging)
and by different people (dev, other dev, admin)
→ dedicated registration REST service
 Extractor I/O typing knows different layers:
 mime-types, e.g. image/png,
 syntactic types, e.g. image region,
 semantic types, e.g. face region
 All three must match!
New MICO broker: Design Choices (1/2)

13. © Fraunhofer IDMT 13
New MICO broker: Design Choices (2/2)
 Existing Linked Data sources and the MICO metadata model (MMM) to be
reused as far as possible; types should be identified via URIs
 Apache Camel for workflow execution: EIP and broad protocol support
 Support for semi-automatic workflow creation via dedicated GUI/App
 Support for admin feedback
 extractors performance
 pipeline performance
 Support for content sets, and assignment of content sets to workflows

14. © Fraunhofer IDMT 14
New MICO broker: components & overall process

15. © Fraunhofer IDMT 15
New MICO broker: data model

16. © Fraunhofer IDMT 16
New MICO broker: data model

17. © Fraunhofer IDMT 17
New MICO broker: semi-automatic workflow creation
 Manual workflow creation: complicated (depends on extractor
interdependencies and constraints, content, use case etc.)
 Now: Semi-automatic approach uses data model (mime types, syntactic
and semantic types)
 Output: Camel route

18. © Fraunhofer IDMT 18
Example Camel route

19. © Fraunhofer IDMT 19
Example Camel route

20. © Fraunhofer IDMT 20
New MICO broker: status and outlook
 Part of the final MICO release (late 2016)
 Current broker status:
 Mostly completed
 Partially integrated into the platform release
 Basic functionalities released under Apache OSS
 Future functionalities planned as premium features:
 Advanced semi-automatic workflow creation
 Advanced distributed process management
 Advanced security features

21. © Fraunhofer IDMT
Patrick Aichroth, Marcel Sieland, Luca Cuccovillo, Thomas Köllmer
The MICO Broker: An Orchestration Framework
for Linked Data Extractors
Fraunhofer IDMT
Ενχαριδτώ!
Questions?

22. © Fraunhofer IDMT
Patrick Aichroth, Marcel Sieland, Luca Cuccovillo, Thomas Köllmer
The MICO Broker: An Orchestration Framework
for Linked Data Extractors
Fraunhofer IDMT
Thank you!
Questions?

23. © Fraunhofer IDMT 23
New MICO broker: data model

24. © Fraunhofer IDMT 24
New MICO broker: data model

25. © Fraunhofer IDMT 25
New MICO broker: data model

26. © Fraunhofer IDMT 26
New MICO broker: data model