Embedded Media Markers (EMMs) are printed marks that link multimedia content to specific locations on paper documents. EMMs use visual features to identify document patches when photographed by a cell phone. EMMs improve usability by guiding users to augmented media content and are more computationally efficient than other feature-based approaches. The printed marks can identify document regions despite scaling, rotation, or partial occlusion and work for various document types like text and images. An evaluation of over 2000 printed documents linked to multimedia showed that EMMs could accurately retrieve the linked content except in one case where the document patch had too few distinctive features.

1. Embedded Media Markers: Marks on Paper that Signify Associated Media Qiong Liu, Chunyuan Liao, Lynn Wilcox, Anthony Dunnigan, Bee Liew FX Palo Alto Laboratory, Palo Alto, CA, USA

2. Motivation & Vision Multimedia and digital content displayed on cell phone User takes a picture of a document patch Motivation: Combine advantages of paper and cell phones Vision: Use Embedded Media Markers (EMMs) to link multimedia to paper patch EMMs are meaningful marks on paper Use visual features inside EMM to identify document patch Slide 2

3. What is EMM Embedded Media Markers (EMM) are optical-filter-like overlaid marks printed on paper documents. Printed marks (EMMs) that improve usability and recognition accuracy Computationally efficient features (FIT) that identify document patches Slide 3 Media type (video) EMM-signified document location EMM-signified document patch

4. EMM Patch Identification Patch identification is based on image local feature (FIT) Similar to SIFT features. More efficient. Robust to scaling, rotation and partial occlusion FIT features work on a broad range of document types Western/Asian text, images, photos Slide 4

5. EMM Operation Flowchart Compute feature points Take a snapshot of EMM Match feature points against the document database Find the matched document patch Retrieve the linked video Play the video on the phone

6. Document Identification Methods Barcode QR-code Data Glyphs Anoto Pattern BWC EMM Marker-based Technologies Recognizable Visual Markers (e.g. Barcode, QR-Code) Nearly Invisible Fiduciary Markers (e.g. DataGlyphs, Anoto) Content-based Technologies Text-words-based Features (e.g. BWC) Image-based Features (e.g. SIFT, FIT) EMM is a hybrid approach Provide users multimedia cues and interaction guidance Save computation and improve performance Minimize the interference with document content SIFT

7. EMM Design Guidelines Major Entities User Friendly – guide users for media finding and query. Document Friendly –minimize changes to document content and format. Machine Friendly – save computation resources for document identification process.

8. EMM Components’ Functions feature-boundary-circle media-type-icon Anchor point Top View of a 3-level EMM Model Feature boundary mark – framing feature capture Media type mark Anchor point – associate media with an accurate location. Graphical effects – attract attention and help separate content and marker.

9. Semi-automatically arrange EMM based on feature distribution Criterion 1 – Minimize author’s effort. Author only selects an EMM anchor point. Criterion 2 – Minimize machine resources used for patch query. Find a small feature boundary, and index the small number of keypoints inside the feature boundary. Criterion 3 – Minimize the EMM interference to document. Minimize the number of keypoints overlaid by an EMM. EMM Authoring Tool

10. Parameters for EMM Placement Anchor point location (m,n) – provided by a user Feature-boundary-circle center (X,Y) Feature-boundary-circle radius R Media-type-circle center (x,y)

11. Parameter Optimization EMM boundary optimization Find the center (X,Y) and the minimum radius R such that The number of keypoints in the circle exceeds the threshold to ensure patch identification accuracy The user selected anchor point is contained in the boundary Media type icon placement Select the media-type-icon center (x,y) so that the icon covers the minimum number of keypoints

12. Parameter Optimization Speedup Counting the number of keypoints in a circle is time consuming O(2N). N is the number of keypoints in a page. Speedup Inscribed square to estimate the number of keypoint in a circle Use the cumulative feature-point histogram IABCD =IC-IB-ID+IA Optimization of media type circle is similar A B IP P C D B A Boundary Circle C D

13. Video Slide 13 http://www.youtube.com/watch?v=oub-2b6wQIY

14. 2188 EMMs in the database At randomly selected 2188 document locations of2188 pages 110 EMM Overlaid Page Printouts 90 1-up + 20 4-up on letter-size paper 110 Queries Either in an office or open patio w/natural lighting conditions EMM Evaluation in Real Life

15. The Missed One Include many less distinctive keypoints Correctly identified after position adjustment Slide 15

16. Related Work University of Oldenberg “Bookmarkr” Link photobook picture to digital photo. Use SIFT features ACM Multimedia 08 Ricoh “HotPaper” Link personal media to text. Use BWC (Brick Wall Coding) Only works on Western text ACM Multimedia 08

17. Document ID Technology Comparison

18. Technology Comparison - cont * SP-code requires high-res printer and scanner

19. Summary EMMs are nearly transparent iconic marks printed on paper document to link various multimedia to specific document locations EMMs are robust to scaling, rotation and partial occlusion, and applicable to a broad range of document types Unlike barcodes, EMMs do not change document layouts and are less visually obtrusive Unlike other feature-based approaches, EMMs explicitly signify the media-augmented document regions and guide user interaction EMM can improve the efficiency of computation

20. Thank You