Presentation for a class at Polytechnique Montreal. First halve focuses on presentation of the platform, second halve focuses on presentation of algorithms.

1. ICRA: Intelligent Platform for Collaboration and Interaction
Lukas Tencer and Marta Reznakova
Synchromedia Laboratory
École de Technologie Supérieure

2. Outline
 Introduction
 Collaboration
 Context and definitions
 Characteristics of collaborative work
 Application domain and functionality
 Architecture of collaborative work
 ICRA
 Retrieval
 New retrieval paradigms
 Sketch-based retrieval
 Photo-based retrieval
 Interaction
 Gesture recognition
 Online learning
 Fuzzy Models, ART Network

3. Synchromedia Laboratory
 Director: Professeur Mohamed CHERIET
 Mission: Develop an advanced network infrastructure that integrates
various aspects of multimodal perceptual information to support
collaborative work
 Main Projets
 Image Processing and historic documents
 Handwriting recognition
 Indian Ocean World (IOW): Image Enhancement, retrieval, interaction
 Virtualization and Green IT, GreenStar and Green Telco Cloud
 Open positions and Web Address
 http://www.synchromedia.ca

4. COLLABORATION

5. Context and definitions
 CSCW: Computer-supported collaborative work
 Allows multiple people to work together through a technological
infrastructure
 Groupware: software which aims to foster collaborative work mode
 Why CSCW?
 Globalization: Linking people and build virtual teams
 Teleworking and Mobility
 Saves time: Booking meeting rooms, travel
 Increased productivity through the use of appropriate tools
 Reduces hierarchical barriers and strengthens team spirit
 Better management and sharing of knowledge

6. Knowledge: core and proximal development
• In community core knowledge of individuals overlap and border areas
coincide with core areas of others.
Source: Lewis R. (1995). « Editorial : Professional learning », Journal of Computer Assisted Learning, 11 (4), 193-195.

7. The basic structure of an activity
Source: Kuutti K. (1996). « Activity Theory as a Potential Framework for Human-Computer Interaction Research », in Context
and consciousness : Activity theory and human computer interaction (ed. B.A. Nardi) p. 17-44. Cambridge, MA : MIT Press.

8. Features / Functions of groupware
 Communication
 Exchange of information between users
 Coordination
 Arrangement of tasks / Managing User Roles
 Production
 Editing Documents
 Whiteboards Communication
Coordination Production

9. Application areas and features
Areas of application
 Games
 Workflow
 Teaching and research
 Shared working spaces
 ConnectNow (Adobe)
 WebEx (Cisco)
 BigBlueButton (Open source)
 Some features
 Chat
 Audio and video
 Viewing and editing documents
 Whiteboard
 Sharing of programs

10. Concepts and constraints associated with groupware
 Concepts
 Group
 Session
 Communication and sharing
 Consistency
 Group consciousness
 Security and privat life
 Identity theft
 Upload of personal documents
 Constraints
 WYSIWIS: What You See Is What I See, Real-time synchronization
 Response time and almost zero delay

11. Architecture: Distribution of the different
components of groupware
Source: Jörg Roth, A taxonomy for synchronous groupware architectures

12. Advantages / Disadvantages
of distributed vs.
centralized architecture??

13. Distributed vs. Centralized Architecture
 Centralized architecture
 Ease of implementation
 Higher response time
 Distributed architecture
 Implementation complexity
 Fast synchronization of the data model
 Centralized architecture is almost always more efficient than
distributed architecture, regarding audio and video streaming! Why?

14. Architecture: Models according toPatterson 1/4
Different levels of consistency states
Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications

15. Architecture: Models according toPatterson 2/4
Consistency of applications after the states sharing
Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications

16. Architecture: Models according toPatterson 3/4
Consistency of applications after the synchronization state
Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications

17. Architecture: Models according toPatterson 4/4
Hybrid architecture
Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications

18. Case: Groupware and Software Engineering
Source: Carl Cook, Neville Churcher, Constructing Real-Time Collaborative Software Engineering
Tools Using CAISE, an Architecture for Supporting Tool Development

19. Case: Groupware and Software Engineering
Source: Carl Cook, Neville Churcher, Constructing Real-Time Collaborative Software Engineering
Tools Using CAISE, an Architecture for Supporting Tool Development

20. Case: Groupware and Software Engineering
Source: Carl Cook, Neville Churcher, Constructing Real-Time Collaborative Software Engineering
Tools Using CAISE, an Architecture for Supporting Tool Development

21. Problems linked to this
architecture??

22. Collaborative editing and resolution of
conflicts
Source: Abdessamad Imine, Flexible Concurrency Control for Real-time Collaborative Editors

23. Solutions??

24. Collaborative editing and resolution of
conflicts
• Maintain a history of changes and identify dependencies to treat
Source: Abdessamad Imine, Flexible Concurrency Control for Real-time Collaborative Editors

26. History: Phase 1
 Tele-education project
 Technology deployed on the network of the Université du Québec
(VLAN linking ETS, Téluq-Mtl, Qc-Téluq, UQAM)
 Collection of collaborative tools: chat, whiteboard, audio / video,
virtual laboratory

27. History: Phase 1

28. History: Phase 2
 The technological infrastructure is extended to connect the
Concordia and Waterloo, via networks RISQ, CA * net 4 and
Internet2.

29. History: Phase 2
 New Vision
 Document Sharing
 Sharing desktop applications
 New Infrastracture
 Multimedial room at Synchromedia

30. Multimedial room at Synchromedia

31. ICRA

32. ICRA: Snapshots

33. ICRA: Snapshots

34. ICRA: Snapshots

35. ICRA: Use case

37. ICRA: Processes

39. ICRA: Architecture of N
Layers

41. ICRA: video encoding
 Using the MPEG-4 H.264
 10/15 frames per second
 Consistent image quality (80%) with variable flow
 Opportunity to improve performance by adjusting the image quality.

42. ICRA: audio encoding
 Used codec: Speex
 Open-source and free
 Sampling rate
 8 kHz, 16 kHz, 32 kHz
 Bit-rate
 2.15 kbps to 44 kbps
 Quality
 Variable between 0 and 10
 Lossy data compression
 Variable Bit-Rate VBR
 Allows the codec to adapt its speed according to the difficulty of encoding
(eg vowels require greater throughput for a given quality)
 Detection of voice and discontinuing of transmission

43. ICRA: Future versions and Research
 Whiteboard
 Sharing desktop applications
 Image processing
 Handwriting recognition and handwritten annotations
 Document Retrieval
 Gesture Interaction
Research

44. Demo

45. Retrieval

46. Document retrieval for collaborative work
 Motivation: To collaborate based on large document collections, we
should be able to retrieve these documents first
 Main components of document retrieval system:
 Feature extraction
 Distance metric
 Semantic Representation
 Indexing
 Retrieval
 User Feedback
 Query refinement
 Our vision: Explore new paradigms for document retrieval

47. New retrieval paradigms for documents
 Sketch-based document retrieval
 Mission: Based on user’s sketch, retrieve document’s which include
graphic similar to user’s sketch
 Photo-based document retrieval
 Mission: Based on visual example (mobile phone picture) retrieve
relevant documents

48. Sketch-based document retrieval
 Mission: Based on user’s sketch, retrieve document’s which include
graphic similar to user’s sketch

49. SBIR challenges
 Preprocessing
 Sketch representation
 Invariance
 Semantic vs. Visual Similarity
 User Feedback / Query refinement
 Result Visualization
f(Q
,I)

50. SBIR representation and invariance
 Gradient-based techniques:
HoG, EHD, SIFT
 Angular Techniques
Angular partitioning

51. SBIR similarity
 L1, L2, Entropy, what next?
 Neighbourhood-based distance (Graph Transduction)

52. (Yang et al. 2008) (Bai, Yang, Latecki, Liu, & Tu, 2010)
52
SBIR Graph Transduction

53. 53
SBIR Indexing and Retrieval
 Representation for indexing
 Bag-of-words
 Topic Model - LSI
 Spatial pyramid matching
 Effective indexing, structure of index?
 Vector space
 Inverted index
 Hash table
 Combine Visual and Semantic similarity
 RankBoost algorithm
 User-specific model

54.  Index Representation:
 Vector Space
 Bag-of-words + inverted index
 Constellation Models
 Retrieved Results:
 Best n-results
 Clustering of Results
 Feedback:
 Implicit: Select relevant images
 Explicit: Temporal information
Query refinement
54
SBIR final system

55. Photo-based retrieval
 Input: Photo from your mobile camera, database of document
 Output: Matching Document

56. Photo-based retrieval approach
 Distribution of connected components in local neighbourhood

57. Interaction

58. Interaction by handwritten
gestures
 Where to use?
 Tablets, whiteboards, smartphones, ...
 Simple shapes, letters, ...
 Perform some action using a connected gesture

59. Interaction by handwritten
gestures
 Why to use?
 Simplify the mouse/keyboard access
 Natural interaction
 Fast interaction

60. Interaction by handwritten
gestures
 Example
 [x, y] coordinates, pressure, time, penUps, penDowns, ...
 Ordered sequences

61. Handwritten gesture recognition
 To recognize gestures (or anything else), the ‘how to’
needs to be learned

62. Handwritten gesture recognition
 To recognize gestures (or anything else), the ‘how to’
needs to be learned
 Problem?
 For user-friendly and adaptive system no predefinition
should be needed. ->Where to get the database if it is
not known? How to learn our system? -> online
learning

63. Online learning
 For each new gesture:
 Recognize the gesture
 Perform learning of the system (or a model)
 Go back to next gesture
 Data (gestures) are treated sequentially (incremental
learning)
 Usually no revisit of old data is allowed (== forget the
processed data)

64. Online learning
 Problems?
 Low accuracy at the beginning <- learning from few
examples
 Difficult generalization, since the data are not seen as a
whole package; future data are not known

65. Online learning
 User-friendly applications:
 No predefinition of classes, their number, ...
 Starting from a scratch
=> System must be capable of learning all the
parameters it is using, if any.

66. Our solution
 Fuzzy models -> applying fuzzy logic into simple
neural networks
 Rules: IF a is ‘P’ THEN ‘B’
 Taking the opinion of all rules
 How to generate the rules?

67. Our solution
 ART (Adaptive Resonance Theory)
 Incremental clustering (unsupervised) method for
detecting similarities
 Starting from a scratch
=> can generate rules for fuzzy models

68. Questions?
Director of the Research Laboratory Synchromedia
Mohamed CHERIET
mohamed.cheriet@etsmtl.ca