A Real-time Collaboration-enabled Mobile Augmented Reality System with Semantic Multimedia


Published on

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A Real-time Collaboration-enabled Mobile Augmented Reality System with Semantic Multimedia

  1. 1. A Real-time Collaboration-enabled Mobile Augmented Reality System with Semantic Multimedia Dejan Kovachev, Gökhan Aksakali & Ralf Klamma RWTH Aachen University, Germany Advanced Community Information Systems (ACIS) kovachev@dbis.rwth-aachen.deLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-1 This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
  2. 2. Advanced Community Information Systems (ACIS) Responsive Web Engineering Community Web Analytics Open Visualization Community and Information Simulation Systems Community Community Support AnalyticsLehrstuhl Informatik 5 Requirements(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-2 Engineering
  3. 3. Agenda  Motivation and scenarios  Conceptual approach  System design and implementation  Evaluation  Related Work  Conclusions and Future WorkLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-3
  4. 4. Motivating Scenario Documentation of Ghazni WallLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-4
  5. 5. Digital Documentation in Cultural Heritage  Ghazni, Afghanistan, capital of Islamic Culture 2013Lehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-5 http://arch.rwth-aachen.de/cms/Architektur/Forschung/Forschungsprojekte/Cultural_Heritage/~cqdn/Ghazni_2013/lidx/1/
  6. 6. Workflow of Documentation 1- images and videos 2- tag basic metadata On-site Documentation Expert On site Architecture Remote expert (Historian) Expert Collaborative 8-view repository Multimedia Cloud Metadata Local Workforce RepositoryLehrstuhl Informatik 5 MPEG-7,(Information Systems) CIDOC CRM Prof. Dr. M. Jarke I5-KAKl-1012-6
  7. 7. Increase Awareness of Cultural Heritage  Multimedia overlays on video stream Metadata  Interaction via position and 3D Captured time movement Taken location Creation time Provenance ….  Still missing Ancient Helmet – real-time collab. – semanticLehrstuhl Informatik 5 metadata(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-7
  8. 8. XMPP-base Mobile Multimedia Collaboration (XMMC) Goals: How to enable  Collaborative digital documentation of historical sites with rich multimedia and metadata using inexpensive hardware  Increase of cultural heritage awareness Prototype: XMMC  Mobile real-time multimedia collaboration system with - integrated AR multimedia browser - standardized metadata for semantic multimedia and annotations (MPEG-7)Lehrstuhl Informatik 5(Information Systems) - Coordination, field note taking, instant geo-tagged multimedia Prof. Dr. M. Jarke I5-KAKl-1012-9 acquisition and sharing
  9. 9. XMMC Use Cases Share Acquire multimedia multimedia Content Producer <<extend>> Add POI Annotate multimedia Synchronize with other clients Collaborator Chat Call LAS MPEG-7 Browse services AR <<include>> <<include>> Display the Consumer Select a POI detailsLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-10
  10. 10. Concurrent Editing of XML  Most metadata standards use XML format – Used by standards like MPEG-7, CIDOC CRM, SVG  XML suited for concurrent editing of metadata – Hierarchal structure more efficient then linear  Concurrency Problems – Divergence, Causality Violation, Intention Violation Approach Architecture Algorithms Examples Pessimistic centralized locking, turn taking SVN,CVS Optimistic P2P OT, treeOPT, CMAX Google Operational [Gerlicher 2007] Wave Transformation,Lehrstuhl Informatik 5(Information Systems) CEFX+ Prof. Dr. M. Jarke I5-KAKl-1012-11
  11. 11. Consistency Maintenance Algorithm for XML (CMAX) [Gerlicher 2007]  Out-of-the-box lightweight concurrent editing of XML files – CEFX+ [Voigt 2009] based on CMAX  Optimistic approach mainly inspired from OT – First execute locally then propagate  Direct addressing with <cefx-uid>  Concurrent editing of the same node – Delegates conflict resolution to user  State Vector – Operation count for every participant  History Buffer – Previous operationsLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-12
  12. 12. eXtensible Messaging and Presence Protocol (XMPP)  Classic SOAs like HTTP, SOAP are inadequate for real-time purposes  Good fit for real-time communication  Easily customizable & extendable – Open XML-based standard  Existing extensions – File transfer, multi-user chat, PubSub  Library and server support – Smack(Java), OpenfireLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-13
  13. 13. XMMC Architecture Client side Server Side Client Side XmmC Client XmmC Server Media Acquisition Collab. Modules XMPP AR Multimedia Relational Annotation Module Module DB AR Browser Module Metadata LAS- Annotation Media Media Collab. AR MPEG7 MPEG-7 Catalog Store Editing Service Integration Services Service Service Service CEFX+ Service XMPP Connection XMPP Client (Smack) implemented Layer (XCL) modified previously available XMPP XMPP Client (aSmack) XMPP XMPP Server (Openfire) Camera WLAN GPS Compass ACCELLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-14
  14. 14. Technical Contributions  Adaptation of CEFX+ for multimedia annotation – New operations: editing and deleting of nodes – Later join, leave collaboration session  XMPP-based AR browser by extending Mixare [http://www.mixare.org/] – POI awareness, ARML-based, mobile content provider  Custom XMPP IQ stanzas for multimedia sharing and AR  Integration with MPEG-7 metadata servicesLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-15
  15. 15. Collaborative Annotation Mobile Client-2 Mobile Client-1 Collaborative Collaborative MPEG-7 Mpeg-7 Annotation Editing Integration <message id="8Dpbh-26" Service Service Services Module to="xmmc@merian.informatik.rwth-aachen.de/Smack" from="3b90aa319d44@conference.merian...de/test_1" <jabber:iq:rpc> joinSession() joinSession() type="groupchat"> <jabber:iq:rpc> SessionData SessionData <body> <basetype type="time" cefx:uid="101"> loadDocument() <jabber:iq:rpc> loadDocument() <name cefx:uid="201"> <jabber:iq:rpc> DocumentData DocumentData Second World War</name> <date cefx:uid="301"> <jabber:iq:rpc> joinSession() … 1997-09-24T00:00:00:000F1000+01:00 <jabber:iq:rpc> DocumentData </date> updateSemantic <groupchat>Mess </basetype> Message (Operation) <groupchat> executeOperation() Semanaticbasetype age (Operation) BaseTypes() </body> Service() … <x xmlns="jabber:x:cefx#ins" Ack Ack p="100" ci="1" leaveSession() <jabber:iq:rpc> sv="0,1" ba="1" fn="null"/> <jabber:iq:rpc> leaveSession() </message>Lehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-16
  16. 16. XMPP-based Augmented Reality Browser Contemporary mobile AR Browsers 3rd Party GPS data Content Provider Provider Server Get POIs Wikipedia, SOAP/RESTful Data Flicker, Facebook, etc. XmmC GPS data XMPP XmmC Create POI Server Create POI Server Edit POI Metadata Edit POI Metadata Event Message PubSub Publish Event Node Data Get POIs Get POIs XMPP XMPPLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-17
  17. 17. Lehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-18
  18. 18. Technical Evaluation Collaborative Annotation  Remote operation execution – Average = 412ms – Stan. Derivation = 209ms  Conflict resolution – Resolved the possible conflicts – with 900ms latency and1,2 KB/s bandwidth network – Still has open issues regarding undo and redo operations AR Browser  The positions of closed POIs are not very accurate due toLehrstuhl Informatik 5 - Inaccuracy in GPS signal, compass, accelerometer(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-20 - Presence of nearby metal objects causes inaccurate results
  19. 19. Technical Evaluation Energy consumption  XMPP – Generates verbose XML streams – Requires constant open sockets – Community works for Distribution of Stanza Types improvement for mobile networks  AR browser consumes 4 times more battery than collaborative annotationLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-21
  20. 20. User Evaluation  Field study with 7 participants – various mobile devices, Android versions and screen sizes  Tasks – acquisition of multimedia artifacts in a historical site – editing basic metadata – collaborative annotation – browsing with AR  Questionnaire – collect subjective qualitative measures and user experienceLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-22
  21. 21. Evaluation Session Results  Majority of the participants observed real-time updates on the annotations (85%)  Participants mostly satisfied with the user interface and found it user-friendly and responsive  MPEG-7metadata service integration was successful  The prototype increased cultural heritage awareness of the participantsLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-23
  22. 22. Related Work Real-time Bi-directional Mobile AR Multimedia System Mobility Light-weight Collaborative Comm. Browser Metadata Editing Mobilis ++ ++ - - ++ + (TU Dresden, 2009) Mobile Campfire ++ - - + - ++ (RWTH Aachen, 2010) Junction ++ + - ++ - - (Stanford, 2010) Google Wave - ++ - - ++ - (Google Inc, 2009) Novell Vibe Cloud - ++ - - ++ + (Novell, 2011) CEFX+ + ++ - + ++ - (London Arts U., 2007) Layar ++ - ++ - - - (Layar, 2009) Mixare ++ - + ++ - - (Mixare.org, 2010 ) Wikitude Architect ++ - + + - - (Wikitude, 2009) XmmC (This work) ++ ++ + ++ ++ ++Lehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-24
  23. 23. Conclusions  XmmC is capable of storing and sharing various multimedia types and sizes  Concurrent XML file editing with mobile clients  Collaborative annotation in digital documentation of historical sites and increasing cultural heritage awareness  XMPP fits well for mobile real-time multimedia collaboration context  However, AR and XMPP consumes batteryLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-25
  24. 24. Future Work  Further multimedia types – Videos, 3D objects, visual formats  Improved AR browser with 3D rendering and visual location mapping  Integration with real-time video streaming services, mobile cloud computing  Community support – User centric features, e.g. trust and securityLehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-26
  25. 25. Thanks for your attention! Q&ALehrstuhl Informatik 5(Information Systems) Prof. Dr. M. Jarke I5-KAKl-1012-27