This document proposes a visual-haptic multiplexing scheme for teleoperation over constant bitrate communication links. Large video packets were blocking haptic transmission, so the scheme plans force sample transmission ahead of time. It allocates bandwidth to video and haptics together in the same packets. This reduces haptic delay variability to zero milliseconds compared to 1-204 milliseconds without multiplexing. The multiplexing approach controls transmission delay for force signals while reducing variability. Future work will extend this to variable bitrate links and integrate with time delay compensation.

1. A Visual-Haptic Multiplexing Scheme for
Teleoperation over Constant-Bitrate
Communication Links
Eurohaptics 2014
Versailles, France, June 24 – 26
Burak Cizmeci
burak.cizmeci@tum.de
Eckehard Steinbach
eckehard.steinbach@tum.de
Chair of Media Technology
Technische Universität München

2. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Overview
• Introduction
• Visual-Haptic Multiplexing
• Experimental Results
• Conclusion & Future Work
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 2

3. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Introduction: Teleoperation
 High haptic sample rate: 1 kHz  1000 packets/s in both directions.
 Perceptual deadband haptic data reduction  irregular transmission events.
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 3
position/velocity position/velocity
Network
video and force video and force
operator teleoperator
Hinterseer et al. “Perception-based data reduction and transmission of haptic data in TPTA systems„ IEEE Trans. on Signal Proc., 2008
Ferrel and Sheridan, “Supervisory Control of Remote Manipulation„ IEEE Spectrum, 1967

4. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Problem definition
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 4
 Packet switched transmission.
 Constant bitrate transmission (CBR).
 Large video packets block the haptic transmission.
 Problem: How to bring down the delay on haptics?
 Solution: A look ahead based planning on force samples.

5. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Overview
• Introduction
• Visual-Haptic Multiplexing
• Experimental Results
• Conclusion & Future Work
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 5

6. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Visual-Haptic Multiplexing Algorithm
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 6

7. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Visual-Haptic Multiplexing Algorithm
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 7

8. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Visual-Haptic Multiplexing Algorithm
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 8

9. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Visual-Haptic Multiplexing Algorithm
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 9

10. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Overview
• Introduction
• Visual-Haptic Multiplexing
• Experimental Results
• Conclusion & Future Work
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 10

11. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Experimental Setup
Force
Dimension
Omega 6
Apposite
Netropy N60
KUKA LWR
JR3 Force
sensor
Peg-in-hole task
• Haptic systems: KUKA LWR, Force Dimension Omega 6.
• Video system: Nvidia GeForce GTX 570 GPU, Nvidia H.264 codec API.
• Network emulation: real-time testing with Apposite Tech. Netropy N60.
• Time machine: real-time Linux time server to measure signal latencies.
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 11

12. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Parameter settings
 CBR channel of 528 kbps without network delay, jitter and packet loss.
 Video stream at 440 kbps constant bitrate mode.
 Multiplexing buffersize 15 ms.
 Irregular force transmission, 45 samples/s on average.
Performance evaluation cases:
 Teleoperation without multiplexing: First come first serve (FCFS).
 Teleoperation with proposed multiplexing algorithm.
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 12

13. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Results
 Demo video available on youtube:
http://youtu.be/aMJVsAPy708
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 13
Mode
Video delay (ms) Force delay (ms)
Min. Max. Mean Std.
dev.
Min. Max. Mean Std.
dev.
MUX 25 348 65 50 17 17 17 0
FCFS 23 260 55 32 1 204 44 36

14. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Conclusion & Future Work
 A visual-haptic multiplexing for teleoperation over CBR links.
 A controllable transmission delay on force signal.
 The variability of the delay on force signal is reduced to 0 ms.
 Extension to variable bitrate links.
 Integration with time delay compensation architectures.
 Quality of Experience & Task Performance.
Future Work:
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 14

15. Technische Universität München
Institute for Media Technology
Prof. Dr.-Ing. Eckehard Steinbach
Thank you
burak.cizmeci@tum.de
Burak Cizmeci: Visual-Haptic Multiplexing for Teleoperation over CBR communication Links02.07.2014 15
Thanks to people involved:
Rahul Chaudhari, Xiao Xu and Nicolas Alt
 Demo video available on youtube:
http://youtu.be/aMJVsAPy708