Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Terminal mode architecture

7,405 views

Published on

More info at: http://www.nokia.com/terminalmode

  • Be the first to comment

Terminal mode architecture

  1. 1. Terminal Mode Technical Architecture Release Candidate v0.9 This document is an integral part of the Terminal Mode Specification, and together with “TmSer- verDevice:1 Device Template, version 0.9”, “TmApplicationServer:1 Service Template, version 0.9”, “TmClientProfile:1 Service Template, version 0.9”, “TmClientDevice:1 Device Template, ver- sion 0.9”, “TmApplicationClient:1 Service Template, version 0.9”, “TmConnectionManager:1 Ser- vice Template, version 0.9” define the Specification version 0.9. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswa- gen AG. All rights reserved. The copyright holders hereby grant you the right to make verbatim copies of the Specification and to make available and distribute such copies. You may not distribute, make available or copy only parts of the Specification, nor modify or create any derivative works of the Specification. No patent rights or other rights not expressly stated as granted, are granted herein. The above copyright notice and these terms and the following disclaimer must be retained in all copies of the Specification. THE SPECIFICATION IS PROVIDED “AS IS”. THE COPYRIGHT HOLDERS GIVE NO WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF ANY THIRD PARTY INTELLECTUAL PROPERTY RIGHTS REGARDING THE SPECIFICATION. Document editor: Jörg Brakensiek Jorg.Brakensiek@nokia.com Nokia Inc. 955 Page Mill Road 94304 Palo Alto, CA USA Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  2. 2. -2- List of Contributors Name Affiliation Dennis Fernahl Carmeq (for Volkswagen AG) Jörg Brakensiek Nokia Corporation Holger Grandy BMW AG Kari Kostiainen Nokia Corporation Keun-Young Park Nokia Corporation Mark Beckmann Volkswagen AG Martin Fesefeldt Volkswagen AG Martti Ala-Rantala Nokia Corporation Matthias Benesch Daimler AG Michael Wolf Daimler AG N. Asokan Nokia Corporation Raja Bose Nokia Corporation Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  3. 3. -3- TABLE OF CONTENTS TABLE OF CONTENTS .............................................................................................................................. 3 LIST OF FIGURES ....................................................................................................................................... 5 LIST OF TABLES ......................................................................................................................................... 7 TERMS AND ABBREVIATIONS ............................................................................................................... 9 1 ABOUT ................................................................................................................................................ 10 2 INTRODUCTION TO TERMINAL MODE .................................................................................... 11 3 CONNECTIVITY PROTOCOL STACK......................................................................................... 13 3.1 PHYSICAL & LINK LAYER ............................................................................................................. 13 3.1.1 Universal Serial Bus (USB) ..................................................................................................... 13 3.1.2 Wireless Local Area Networks (WLAN)................................................................................... 14 3.2 NETWORKING AND TRANSPORT LAYER ........................................................................................ 14 3.2.1 USB Networking ...................................................................................................................... 15 3.2.2 WLAN Networking ................................................................................................................... 16 3.2.3 Transport Layer ....................................................................................................................... 16 3.3 SESSION & APPLICATION LAYER .................................................................................................. 16 4 AUTHENTICATION AND SECURITY .......................................................................................... 17 4.1 HOST AUTHENTICATION MECHANISMS......................................................................................... 17 4.1.1 USB Networking ...................................................................................................................... 17 4.1.2 WLAN Networking ................................................................................................................... 17 4.2 CONFIDENTIALITY AND INTEGRITY MECHANISMS ........................................................................ 17 4.2.1 USB Networking ...................................................................................................................... 17 4.2.2 WLAN Networking ................................................................................................................... 17 4.3 DEVICE IDENTIFICATION MECHANISM .......................................................................................... 17 5 DISPLAY OUTPUT AND CONTROL INPUT ............................................................................... 19 5.1 CONNECTION SETUP ..................................................................................................................... 20 5.2 TRADITIONAL VNC PROTOCOL PHASES ....................................................................................... 21 5.2.1 Handshaking Phase ................................................................................................................. 21 5.2.2 Initialization Phase .................................................................................................................. 22 5.2.3 Framebuffer Update and Event Phase ..................................................................................... 23 5.3 VNC TERMINAL MODE EXTENSION MESSAGES ........................................................................... 26 5.3.1 Display Configuration Messages ............................................................................................. 28 5.3.2 Event Configuration Messages ................................................................................................ 31 5.3.3 Event Mapping Messages ........................................................................................................ 34 5.3.4 Key Event Listing Message ...................................................................................................... 36 5.3.5 Virtual Keyboard Trigger Messages ........................................................................................ 39 5.3.6 Device Status Messages ........................................................................................................... 41 5.3.7 Content Attestation Messages .................................................................................................. 44 5.3.8 Framebuffer Blocking Notification .......................................................................................... 47 5.4 ADDITIONAL ENCODINGS AND PSEUDO ENCODINGS..................................................................... 49 5.4.1 Terminal Mode Pseudo Encoding ............................................................................................ 49 5.4.2 Context Information Pseudo Encoding .................................................................................... 49 6 AUDIO OUTPUT AND INPUT......................................................................................................... 51 6.1 RTP PACKET STRUCTURE AND HEADER DEFINITION.................................................................... 52 Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  4. 4. -4- 6.2 RTP AUDIO PAYLOAD DEFINITION ............................................................................................... 53 6.2.1 16 Bit Audio Payload (Mono) .................................................................................................. 53 6.2.2 16 Bit Audio Payload (Stereo) ................................................................................................. 54 6.3 ESTABLISHING THE RTP CONNECTION ......................................................................................... 54 6.4 RECOMMENDATION FOR CLIENT AND SERVER IMPLEMENTATION ................................................ 55 6.5 INTEROPERABILITY WITH BLUETOOTH.......................................................................................... 56 6.5.1 Bluetooth Profiles relevant for Terminal Mode ....................................................................... 56 6.5.2 Interoperability States –Terminal Mode Server Perspective ................................................... 56 6.5.3 Interoperability States –Terminal Mode Client Perspective .................................................... 58 7 SERVICE NEGOTIATION FRAMEWORK .................................................................................. 61 7.1 UPNP USAGE MODELS ................................................................................................................. 61 7.1.1 2-Box Pull Model ..................................................................................................................... 61 7.1.2 2-Box Push Model.................................................................................................................... 61 7.1.3 3-Box Model............................................................................................................................. 62 7.2 UPNP DEVICE ARCHITECTURE ..................................................................................................... 63 7.3 TMSERVERDEVICE:1 DEVICE ....................................................................................................... 63 7.3.1 TmApplicationServer:1 Service ............................................................................................... 63 7.3.2 TmClientProfile:1 Service ....................................................................................................... 64 7.4 TMCLIENTDEVICE:1 DEVICE ........................................................................................................ 64 8 AUDIO LINK SELECTION .............................................................................................................. 66 8.1 AUDIO LINK OPTIONS ................................................................................................................... 66 8.2 AUDIO LINK SELECTION ............................................................................................................... 67 8.2.1 LaunchApplication (AppID, ProfileID) ................................................................................... 67 8.2.2 TerminateApplication (AppID, ProfileID) ............................................................................... 69 8.2.3 GetApplicationStatus (AppID, ProfileID) ................................................................................ 70 9 DEVICE ATTESTATION ................................................................................................................. 71 9.1 DEVICE ATTESTATION LAUNCH .................................................................................................... 71 9.2 DEVICE ATTESTATION TERMINATION ........................................................................................... 72 9.3 DEVICE ATTESTATION PROTOCOL ................................................................................................ 72 10 REFERENCES .................................................................................................................................... 78 APPENDIX A – EVENT MAPPING ......................................................................................................... 80 KNOB SHIFT AND ROTATE EVENTS ............................................................................................................ 80 KEY EVENT MAPPING ................................................................................................................................ 81 APPENDIX B – APPLICATION CONTEXT INFORMATION ............................................................ 85 TRUST LEVEL ............................................................................................................................................. 85 APPLICATION CATEGORIES ........................................................................................................................ 85 CONTENT CATEGORIES .............................................................................................................................. 86 CONTENT RULES ........................................................................................................................................ 86 Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  5. 5. -5- LIST OF FIGURES Figure 1: Terminal Mode Concept ............................................................................................................ 11 Figure 2: Terminal Mode Networking and Transport Stack................................................................. 15 Figure 3: Session Layer............................................................................................................................... 16 Figure 4: Terminal Mode VNC Setup ...................................................................................................... 19 Figure 5: VNC Connection Setup ............................................................................................................. 20 Figure 6: VNC Handshaking Phase ......................................................................................................... 21 Figure 7: Initialization Phase ..................................................................................................................... 22 Figure 8: Framebuffer Update Phase ....................................................................................................... 23 Figure 9: Server and Client Display Configuration ............................................................................... 28 Figure 10: VNC Server Options for non-scalable Clients ...................................................................... 30 Figure 11: Server and Client Event Configuration ................................................................................. 31 Figure 12: Example Event Mapping Message Flow ............................................................................... 34 Figure 13: Key Event Listing Messages ................................................................................................... 36 Figure 14: Key Event Listing – Incremental Updates ............................................................................ 37 Figure 15: Key Event Listing – Non-Incremental Updates ................................................................... 38 Figure 16: Example Virtual Keyboard Trigger Message Flow ............................................................. 39 Figure 17: Example Device Status Message Flow .................................................................................. 41 Figure 18: Example Content Attestation Message Flow ........................................................................ 44 Figure 19: Example Framebuffer Blocking Notification Message Flow .............................................. 47 Figure 20: Context Information (Example) .............................................................................................. 49 Figure 21: Audio Setup .............................................................................................................................. 51 Figure 22 Sequence for RTP connection: RTP Server by TM Server .................................................... 55 Figure 23: Bluetooth / Terminal Mode IOP States – Terminal Mode Server Perspective ................ 57 Figure 24: Bluetooth / Terminal Mode IOP States – Terminal Mode Client Perspective ................. 59 Figure 25: General UPnP Device Architecture........................................................................................ 61 Figure 26: 2-Box Pull Model ...................................................................................................................... 61 Figure 27: 2-Box Push Model .................................................................................................................... 62 Figure 28: 3-Box Model .............................................................................................................................. 62 Figure 29: Terminal Mode UPnP Service Architecture.......................................................................... 63 Figure 30: Terminal Mode Client Device Architecture .......................................................................... 64 Figure 30: Message Flow – Launch Audio Link ..................................................................................... 68 Figure 31: Message Flow – Terminate Audio Link ................................................................................ 69 Figure 32: Message Flow – Launch Device Attestation Protocol ......................................................... 71 Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  6. 6. -6- Figure 33: Message Flow – Terminate Device Attestation Protocol .................................................... 72 Figure 34: Device Attestation certification infrastructure ..................................................................... 72 Figure 35: Device and software attestation protocol overview ............................................................ 73 Figure 36: Coordinate System for Knob Events ...................................................................................... 80 Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  7. 7. -7- LIST OF TABLES Table 1: Bandwidth Requirements vs. Display Update Rate ................................................................ 13 Table 2: Requirements for Handshaking Phase Messages .................................................................... 22 Table 3: Requirements for Initialization Phase Messages ..................................................................... 23 Table 4: Requirements for Framebuffer Update and Event Phase Messages ..................................... 24 Table 5: VNC Extension Type Message Structure .................................................................................. 26 Table 6: New Extension Types for Terminal Mode Messages .............................................................. 26 Table 7: Server Display Configuration Message..................................................................................... 29 Table 8: Client Display Configuration Message ..................................................................................... 29 Table 9: Server Event Configuration Message ........................................................................................ 32 Table 10: Client Event Configuration Message ....................................................................................... 33 Table 11: Event Mapping Message ........................................................................................................... 34 Table 12: Event Mapping Request Message ............................................................................................ 34 Table 13: Key Event Listing Message ....................................................................................................... 37 Table 14: Key Event Listing Request Message ........................................................................................ 38 Table 15: Virtual Keyboard Trigger Message ......................................................................................... 40 Table 16: Virtual Keyboard Trigger Request Message .......................................................................... 40 Table 17: Device Status Message .............................................................................................................. 42 Table 18: Device Status Request Message ............................................................................................... 42 Table 19: Terminal Mode Server Device Status Default Values ........................................................... 43 Table 20: Content Attestation Response Message .................................................................................. 45 Table 21: Content attestation signature content ..................................................................................... 45 Table 22: Content Attestation Request Message ..................................................................................... 46 Table 23: Framebuffer Blocking Notification Message .......................................................................... 47 Table 24: Blocked Rectangle from Framebuffer Update ........................................................................ 48 Table 25: Additional VNC Encodings ...................................................................................................... 49 Table 26: Context Information Pseudo Encoding ................................................................................... 50 Table 27: RTP Packet Header Definition ................................................................................................. 52 Table 28: RTP Payload Format – 16 Bit Mono ......................................................................................... 54 Table 29: RTP Payload Format – 16 Bit Stereo ........................................................................................ 54 Table 30: IOP Transition (from Terminal Mode Server perspective)................................................... 58 Table 31: IOP Transition (from Terminal Mode Client perspective) ................................................... 60 Table 32: UPnP Negotiation for Audio Selection ................................................................................... 67 Table 33: Device Attestation – attestationRequest elements ................................................................. 74 Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  8. 8. -8- Table 34: Device Attestation – Component List ..................................................................................... 74 Table 35: Device Attestation – attestationResponse Elements .............................................................. 75 Table 36: Device Attestation – Possible Attestation Result Values ...................................................... 76 Table 37: Knob Shift and Rotate Configuration Settings ....................................................................... 80 Table 38: Key Event Mapping ................................................................................................................... 84 Table 39: Trust Level .................................................................................................................................. 85 Table 40: Application Categories .............................................................................................................. 86 Table 41: Content Categories..................................................................................................................... 86 Table 42: Content Rules to tackle Driver Distraction ............................................................................. 87 Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  9. 9. -9- TERMS AND ABBREVIATIONS A2DP Bluetooth Advanced Audio Distribution Profile ARP Address Resolution Protocol BT Bluetooth CDC Communications Device Class; specified from USB Device Working Group CE Consumer Electronics; CE devices are referred to as mobile devices within this specification DHCP Dynamic Host Configuration Protocol ECM Ethernet Control Model; part of the CDC device class HFP Bluetooth Hands-free Profile HSP Bluetooth Handset Profile HMI Human Machine Interface" HU Head-unit (this term is used interchangeably with the Terminal Mode client) HS Head-set IP Internet Protocol NCM Network Control Model; part of the CDC device class RFB Remote Framebuffer RTP Real-time Transport Protocol TCP Transmission Control Protocol TM Terminal Mode UDP User Datagram Protocol UI User Interface UPnP Universal Plug and Play USB Universal Serial Bus VNC Virtual Network Computing Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  10. 10. - 10 - 1 ABOUT This document specifies an interface for enabling remote user interaction of a mobile device via another device. This specification is written having a car head-unit to interact with the mobile de- vice in mind, but it will similarly apply for other devices, which do provide a colored display, audio input/output and user input mechanisms. This document is aimed at people going to design and develop compliant solutions. The docu- ment will provide all necessary interface functionality and requirements to implement a fully compliant device, on both the mobile device and the head-unit side. The specification lists a series of requirements, either explicitly or within the text, which are man- datory elements for a compliant solutions. Recommendations are given, to ensure optimal usage and to provide suitable performance. All recommendations are optional. The document will focus on the interface functionality, its parameters and protocols only. It does not provide any guidelines for implementing the protocol. If there is a reference towards an im- plementation, this is of informative nature only. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  11. 11. - 11 - 2 INTRODUCTION TO TERMINAL MODE The Terminal Mode provides a concept for integrating the mobile device (hereinafter referred to as the “Terminal Mode Server”) and the vehicle head-unit (hereinafter referred to as the “Terminal Mode Client”). In a Terminal Mode context, the control and interaction of applications and services running on the mobile device will be replicated into the car environment. Diverting display and audio output to the car head-unit come together with receiving key and voice control input from it are the main interaction streams, as shown in the following figure. Applications User Speaker Content Display & Services Input & Micro Display Control Consumer Automotive Electronics Head Unit Device Audio / Voice Internet Figure 1: Terminal Mode Concept The result is a concept somewhere between running the applications natively in the mobile phone or in the car unit. From the user experience point of view it can offer "the best of the both worlds" where the large variety of mobile phone applications is complemented and enhanced by the car system providing convenient and safe means for using (i.e. controlling) these applications. It is easier to add new consumer electronic functionalities into the vehicle environment via a mobile device than integrating them into the car infotainment system. In any case, the usage of those applications will become more convenient if the same device with the same content stored in it can be used in all the different environments from home to car, and providing Internet connectivity at the same time. On the other hand, the large displays of the car units can enhance the user experience from what the mobile device can offer by itself. In addition the mobile device typically provides the latest technologies, from radio connectivity, to multimedia codecs. At the same time, the openness of the platforms, allows delivery of new applications and services at any time. There are no standard methods currently defined for Terminal Mode connectivity. However, when creating the required solutions, technologies provided by existing open, non-proprietary standards - like USB, TCP/IP, VNC, UPnP etc. - should be used as the basis. The needed additional elements should then be developed and agreed in cooperation between the related industry sectors. The car systems comprise of several different methods for user interaction, like individual keys, rotating knobs, touch screen and even voice-activated control. For proper interoperability, the control method towards the mobile device should be the same regardless of the actual input mechanism on the car side. Furthermore, to ensure that Terminal Mode does provide Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  12. 12. - 12 - interoperability independent of any application, even legacy ones, it hooks into low-level abstraction. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  13. 13. - 13 - 3 CONNECTIVITY PROTOCOL STACK The connectivity between the terminal mode server and client is the basis to provide interopera- bility between both. The Connectivity stack is specified in the following, starting from the low layer and going up the protocol stack. It is not the objective of this specification to provide a detailed overview of the different protocols. Instead this document should highlight the components and parameter required to ensure proper connectivity. The connectivity solution is built purely on existing wireless and wired standards. Therefore detailed information is available in the respective documents. 3.1 Physical & Link Layer In principle this specification does not intend to limit the use of any wireless and wired technolo- gy. Nevertheless the connectivity solution must provide reasonable high bandwidth. Minimum bandwidth on link layer cannot be given, as the user experience depends on the networking & transport layer performance, as well as on the parameter of the display (resolution and color for- mat). The following table gives some indication of the required bandwidth on the display level, i.e. on top of any transport mechanism. These values assume non-incremental, uncompressed updates. Full Display Example: QVGA Example: QHD Example: WVGA Example: WVGA Update / s 320 x 240 x 4 640 x 360 x 4 800 x 480 x 2 800 x 480 x 4 2 614 000 Byte/s 1 843 200 Byte/s 1 536 000 Byte/s 3 072 000 Byte/s 5 1 536 000 Byte/s 4 608 000 Byte/s 3 840 000 Byte/s 7 680 000 Byte/s 10 3 072 000 Byte/s 9 216 000 Byte/s 7 680 000 Byte/s 15 360 000 Byte/s 20 6 144 000 Byte/s 18 432 000 Byte/s 15 360 000 Byte/s 30 720 000 Byte/s Table 1: Bandwidth Requirements vs. Display Update Rate Wired technologies do have advantages with regard to achievable bandwidth and security over wireless technologies. In addition wired USB nowadays provides charging capabilities and is in- deed the preferred charging interface in the mobile industry. 3.1.1 Universal Serial Bus (USB) USB provides a high-bandwidth connection while allowing charging of the mobile device at the same time. Requirement: The USB host must at least support USB 2.0 high-speed. To simplify the user intervention on the terminal mode server, it should set the right USB profile automatically1, once connected to the terminal mode client. To facilitate the automatic personality selection, the USB host should send a specific identification message to the device, prior configur- ing the device, according the following format. bmRequestType = 0x40 bRequest = 0xF0 1 A USB personality may include multiple USB device classes, which can be then used from the USB host simultaneously. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  14. 14. - 14 - wValue[1] = Terminal Mode major version (e.g. 1) wValue[2] = Terminal Mode minor version (e.g. 0) wIndex = USB Host vendorID wLength = 0 Data = None The message should be sent before set configuration, since the phone may have wrong personali- ty loaded before that2. It must also be understood, that as a result of this, if wrong personality is loaded, the phone will disconnect and reconnect again with a new personality loaded. This will restart the enumeration3. Requirement: The USB device must recognize Terminal Mode request message and must select the respective USB personality. Requirement: If the Terminal Mode client does not send the described identification mes- sage, the mobile device must present the user with a list of available USB per- sonalities. The Terminal Mode specification does not attempt to specify, which other USB profiles should be available under the Terminal Mode personality, and which USB personalities are available and supported from the device. But to support multiple USB profiles under one personality, USB Host needs to support composite device including IAD (Interface Association Descriptor). IAD is re- quired to support USB function which requires multiple interfaces, and without IAD, it is not possible to associate multiple interfaces with single USB functionality. Requirement: USB host (TM client) must support composite device including IAD. Recommendation: It is recommended for USB device (TM server) to provide MTP and ACM as part of the Terminal Mode personality. 3.1.2 Wireless Local Area Networks (WLAN) Support for Wireless LAN is optional. 3.2 Networking and Transport Layer Networking mechanisms are used to abstract the different physical transport mechanisms. The Internet Protocol is a well-established and known networking solution. IP protocol packets are encapsulated into Ethernet packages. Requirement: The networking layer must support IPv4. Support for IPv6 is optional. This specification does anticipate only USB and WLAN networking. Other wired or wireless links are supported optionally, as long as they allow carrying IP packets, as shown in Figure 2. 2 According to USB 2.0 specification, a USB device, which does not support a message, must re- turn a STALL PID (Request Error). As the endpoint is control endpoint, there is no further action required. USB host can consider device returning STALL as non-terminal mode device and can proceed with non-terminal mode behavior. 3 A user will be able to manually switch between USB device personalities at any time. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  15. 15. - 15 - DHCP UDP TCP IPv4 WLAN USB 2.0 <Other Link Layer> Figure 2: Terminal Mode Networking and Transport Stack 3.2.1 USB Networking Support for USB Networking is mandatory. Three competing Communication Device Class sub- class drivers are available. In all cases, the USB connection basically looks like an Ethernet 802.3 networking card. • RNDIS: Remote Network Device Interface Specification is a Microsoft proprietary specifi- cation. RNDIS is partly Operating System dependent. • CDC/ECM [3]: Communications Device Class/Ethernet Networking Control Model al- lows one Ethernet package inside a single USB transfer. ECM has been developed for USB full-speed. • CDC/NCM [4]: Communications Device Class/Network Control Model allows multiple Ethernet packages inside a single USB transfer. NCM can therefore offer a much higher performance. NCM has been particular developed for high-bandwidth. Requirement: The USB networking must follow CDC/NCM device class, revision 1.0 speci- fication.4 Recommendation: The host and client should support a Maximum Transmission Unit (MTU) size bigger than 1,500 Bytes. It is recommended to support MTU sizes up to 9000 Byte. Requirement: The USB host must follow the maximum Ethernet frame size supported from the USB device as discovered from the value wMaxSegmentSize in the Ether- net Networking Functional Descriptor (For details refer to [3]) and supported from the host (Least common denominator). The Dynamic Host Configuration Protocol (DHCP) is used by the terminal mode client (DHCP client) to obtain configuration information for operation in an IP network from the mobile device (DHCP server). This protocol reduces system administration workload, allowing devices to be added to the network with little or no manual intervention. DHCP is using UDP for the negotia- tion. • Packets sent from the client have source port 68 and destination port 67. • Packets sent from the server have source port 67 and destination port 68. Requirement: A DHCP Server must be available on the mobile device, connected to the USB interface. The DCHP client must use the standard DHCP port. 4 According to USB CDC/NCM specification, the device and host MUST support 16-bit NTB structures (NTB-16) and MAY also support 32-bit NTB structures (NTB-32). Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  16. 16. - 16 - Requirement: To minimize IP address conflicts on the terminal mode client, the DHCP Server must provide an IP address within the 192.168.x.y with x in the range of 2 to 127 and y in the range of 0 to 254. The netmask must be 255.255.255.0. Requirement: The DHCP Server may provide a default gateway address for the DHCP client. Provisioning of the default gateway address must not be interpreted as if the Terminal Mode server provides Internet connectivity. The Terminal Mode specification does not intend to specify the setup of IP routing functio- nality on the Terminal Mode server. Recommendation: Use ARP to resolve potential IP conflicts on client side. 3.2.2 WLAN Networking Support for Wireless Local Area Network (WLAN) is optional. IP packets are carried over WLAN connections, using the Ethernet LLC/SNAP framing. On other network types than Ethernet, LLC and SNAP headers are required to multiplex different protocols on the link layer. Requirement: In case WLAN is supported, the mobile device must support ad-hoc and in- frastructure networks. In latter case, the access point must be implemented in the terminal mode client. 3.2.3 Transport Layer The IP protocol enables two transport mechanisms, • User Datagram Protocol (UDP) to provide connectionless communication, used for ser- vice advertising, multi-casting, and most real-time streaming protocols • Transmission Control Protocol (TCP) to provide connection-oriented communication Requirement: The transport layer must support UDP and TCP transport protocols on top of IP. 3.3 Session & Application Layer The Terminal Mode application layer consists of three basic session layer components using ei- ther UDP or TCP sockets to interact as shown in the figure below. • Audio, responsible for providing and exchanging audio content, using UDP sockets. • VNC, responsible for exchanging display and control information, using TCP sockets. • UPnP, responsible for service negotiation and remote application control, using UDP broadcasting and TCP sockets. Audio UPnP VNC SOAP RTCP RTP SSDP HTTP 1.1 VNC UDP TCP Figure 3: Session Layer The application layer is specified in the following chapters. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  17. 17. - 17 - 4 AUTHENTICATION AND SECURITY Giving the terminal mode client control over the server requires addressing security and authen- tication concerns. Potential threats in the Terminal Mode setup include 1. Attacker can read and/or modify communication between terminal mode client and server. 2. Terminal mode server (e.g. a mobile device) does not connect to the intended terminal mode client (e.g. a vehicle head-unit) or vice versa. 3. Terminal mode client connects to a non-compliant server. Threats 1 is addressed via confidentiality and integrity mechanisms, where as threats 2 is ad- dressed via host authentication. Threat 3 is addressed with device attestation. The term “security mechanisms” is used to refer to confidentiality, integrity and authentication mechanisms. Those mechanisms are described in the following. 4.1 Host Authentication Mechanisms Host authentication in this context refers to the terminal mode client authenticating itself towards the terminal mode server, to mitigate the threat that the server connects to unintended client (or vice versa). 4.1.1 USB Networking Additional authentication and integrity mechanisms in wired USB networking are not required. 4.1.2 WLAN Networking Authentication and integrity mechanisms in WLAN networks are available on Link Layer. Requirement: Link-Layer authentication mechanisms, like WPA, must be used, if mandated from the terminal mode server or from the terminal mode client. 4.2 Confidentiality and Integrity Mechanisms Confidentiality and integrity mechanisms mitigate the threat that an external attacker can read, change or inject any data. 4.2.1 USB Networking Additional confidentiality and integrity mechanisms in wired USB networking are not required. 4.2.2 WLAN Networking Confidentiality and integrity mechanisms in WLAN networks are available on Link Layer. Requirement: Link-Layer confidentiality mechanisms, like WPA, must be used if mandated from the terminal mode server or from the terminal mode client. 4.3 Device Identification Mechanism Device identification in this context refers to the mobile device identifying itself to the terminal mode client. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  18. 18. - 18 - Device identification (proving the identity of the device and or the device manufacturer are avail- able from different sources during the connection setup. 1. USB standard device descriptor entries idVendor and idProduct 2. UPnP XML device description: <manufacturer> and <modelName> Requirement: The Device must set the USB vendor and product ID as well as UPnP device manufacturer and model name accordingly. Requirement: The device must prevent 3rd party software to change USB vendor ID and UPnP device manufacturer according to the state-of-the-art of the particular device platform. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  19. 19. - 19 - 5 DISPLAY OUTPUT AND CONTROL INPUT The contents of the Terminal Mode server device’s screen are transferred to the Terminal Mode client device. The control inputs are transferred from the Terminal Mode client to the Terminal Mode server. Screen copy methods can be used to copy the content of the Terminal Mode server's display (frame buffer copy) to the Terminal Mode client’s display. The copy operation may in- clude rotation or color conversion. The frame buffer is used as an abstraction layer, which avoid any changes to the applications and services running on the mobile device can be avoided. For this purpose the Virtual Networking Computing (VNC) protocol is used. The Virtual Networking Computing (VNC) uses the Remote Framebuffer Protocol (RFB) as a simple protocol for remote access to any sort of framebuffer based user interfaces. The remote endpoint is called the VNC Client, whereas the endpoint driving the framebuffer is called the VNC Server. In the Terminal Mode context, the VNC Client resides in the car head-unit (Terminal Mode client) and the VNC Server is in the mobile device (Terminal Mode server). The VNC client will show the remote display either on the entire local display or on a subset of it, as shown in Figure 4. HU CE User VNC VNC CE Display Display Input Client Server Display RFB Protocol Display Control Consumer Automotive Electronics Head Unit Device Figure 4: Terminal Mode VNC Setup The command and control input is handled as part of the VNC protocol by key and pointer events. A key or pointer event on terminal mode client will be signaled to the terminal mode server via a specific key symbol value, which uniquely identifies the event. The mobile device and/or its application will not necessarily support all possible keys defined. Some applications may even have a dynamic behavior on the selection of key inputs they expect. The RFB protocol is originating from the desktop computing world and has been designed as a thin client protocol, i.e. it assumes a client with only a few requirements, and a server having access to more processing capabilities. The protocol allows the client to be as simple as possible. In the Terminal Mode context this assumption needs to be reconsidered, as mobile devices are experiencing performance limitations as well. Requirement: The terminal mode client must implement the VNC client functionality. Requirement: The terminal mode server must implement the VNC server functionality Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  20. 20. - 20 - 5.1 Connection Setup The connection setup is facilitated via UPnP. Once the VNC server is up and running, its service is announced using UPnP protocol mechanisms. The VNC Server must listen for the VNC client to connect. Connection is done via establishment of a TCP/IP socket. The Server IP address and VNC port number are provided as part of the UPnP protocol (see respective chapter). The established connection is facilitating the execution of the VNC protocol. Once the TCP/IP connection is disconnected, the VNC server will wait for the VNC client to re- connect. The VNC protocol does not provide specific messaging to shut down the connection. Be- fore reconnection, the VNC client has to verify, whether the VNC server is still available (using UPnP mechanisms). The connection setup is high-lighted in the following picture. The red dotted lines indicate trigger points between the Server and Client operation. Start VNC Client Start VNC Server Wait for VNC Server to VNC Server available become available Connect Wait for VNC Client to Connect to VNC Server (Re-)Connect VNC VNC Operation VNC Operation Protocol No Still Yes Close Connection connected ? Disconnect Figure 5: VNC Connection Setup The Server IP address and VNC port number are provided as part of the UPnP protocol (see re- spective chapter). Requirement: The VNC Server must listen for the VNC client to connect. If the VNC client disconnects, the VNC Server must listen for the client to reconnect. Requirement: The VNC Server must listen on a TCP/IP socket. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  21. 21. - 21 - 5.2 Traditional VNC Protocol Phases After the connection between the VNC server and client has been established, the VNC protocol processing will start according to the VNC specification. The VNC protocol basically consists of three main steps (1) Exchange of handshaking messages. In this step, the VNC connection between both end- points is set up. After the handshaking phase, the VNC connection parameters are nego- tiated and the connection is established. (2) Exchange of initialization messages. After this phase, both ends have agreed on all needed parameter for the following operational phase. (3) Client to server and server to client messages are used to reflect changes of the framebuf- fer content on the local endpoint and user interaction on the remote endpoint. These three VNC protocol phases are specified in more detail in the following. 5.2.1 Handshaking Phase The handshaking phase defines a couple of messages, which are being exchanges between the VNC Client and the VNC Server, as shown in the following figure. In general the VNC Server presents its capabilities and the VNC Client selects the best option with regard to its own capabili- ties. VNC Server VNC Client Server Protocol Version Client Protocol Version Security Type Support Security Type Security_ Selection type = 0 Security Failure Reason Security Result Security Failure Reason (3.8 only) Figure 6: VNC Handshaking Phase The following parameters have to be supported from the VNC Client and the Server. Message Origin Parameter Mandatory Values Protocol Version Server Max. protocol version At least 3.7 Protocol Version Client Max. protocol version At least 3.7 # of security types (as specified in RFB) Security Type Support Server Security types 1 (None) Security Type Selection Client Security type (as specified in RFB) Reason length Security Failure Reason Client (as specified in RFB) Reason string Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  22. 22. - 22 - Message Origin Parameter Mandatory Values Security Result Server Security status (as specified in RFB) Security Failure Reason Reason length Server (as specified in RFB) (RFB version 3.8 only) Reason string Table 2: Requirements for Handshaking Phase Messages Authentication and security is handled outside the VNC protocol on link-layer and transport- layer. The VNC Client cannot expect the VNC Server to offer additional security or authentication features. 5.2.2 Initialization Phase The initialization phase defines a couple of messages, which are being exchanged between the VNC Client and the VNC Server, as shown in the following figure. In general the VNC Server presents its capabilities and the VNC Client selects the best option with regard to its own capabili- ties. VNC Server VNC Client Client Init Server Init Set Encodings Set Pixel Format Figure 7: Initialization Phase The following parameters have to be supported from the VNC Client and the Server. Message Origin Parameter Mandatory Values Client Init Client Shared-flag (as specified in RFB) Framebuffer-width Framebuffer-height Name-length Name-string Server Init Bits-per-pixel Server (as specified in RFB) (using native framebuf- Depth fer configuration) Big-endian-flag True-colour-flag Red-/Green-/Blue max Red-/Green-/Blue shift Number of encodings (as specified in RFB) Set Encodings Client -223 (Desktop Size Pseudo Encoding-type Encoding – optional for client) Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  23. 23. - 23 - Message Origin Parameter Mandatory Values -523 (Terminal Mode Pseu- do Encoding) Bits-per-pixel 32, 16 Depth 24, 16 Big-endian-flag (as specified in RFB) Set Pixel Format Client True-colour-flag 1 (true color) Red-/Green-/Blue max RGB888, RGB565 Red-/Green-/Blue shift Table 3: Requirements for Initialization Phase Messages The Terminal Mode limits the RFB protocol, as shown in the above table with regard to supported color formats, to allow for efficient implementations. Some more specific recommendations and requirements are given below. Requirement: The VNC Client must not select any other pixel format, unless the server has indicated support, using the ServerDisplayConfiguration VNC extension message. Requirement: The VNC Client must send a Set Pixel Format message, prior to any Frame- buffer Update Request message. Recommendation: It is recommended that the VNC Client selects the native color format of the VNC server. On device color conversion will lead to a significant reduction of achievable frame rate. 5.2.3 Framebuffer Update and Event Phase The update and event phase defines a couple of messages, which are being exchanged between the VNC Client and the VNC Server. The VNC Server only responds to framebuffer update re- quests, as shown in Figure 8. No response message is sent to any of the other messages. VNC Server VNC Client Framebuffer Update Request Framebuffer Update Figure 8: Framebuffer Update Phase The following parameters have to be supported from the VNC Client and the Server. Message Origin Parameter Mandatory Values Incremental x-position Framebuffer Update Client y-position (as specified in RFB) Request Width Height Number-of-rectangles Framebuffer Update Server (as specified in RFB) x-position Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  24. 24. - 24 - Message Origin Parameter Mandatory Values y-position Width Height encoding-type 0 (Raw) first color number-of-colours (Message not implemented Set Colour Map Entries Server Red at Server) Green Blue Down-flag Key Event Client (as specified in RFB) Key Button-mask Pointer Event Client x-position (as specified in RFB) y-position Length Client Cut Text Client (as specified in RFB) Text Bell Server No parameter (as specified in RFB) Length Server Cut Text Server (as specified in RFB) Text Table 4: Requirements for Framebuffer Update and Event Phase Messages The VNC Client can request two types of framebuffer updates, using the incremental flag in the FramebufferUpdateRequest message. • A ‘0’ indicates the VNC server to provide a non-incremental update, i.e. the VNC server must provide the requested area or a superset of it, independent of whether it has changed or not. • A ‘1’ indicates the VNC server to provide an incremental update, i.e. the VNC server must provide only the area(s) containing all framebuffer changes. Requirement: The VNC Client must support Zero framebuffer update messages, i.e. Fra- mebuffer Update messages where the width and height equals zero.5 Requirement: The VNC Client must support framebuffer update messages of a bigger fra- mebuffer area, than the original requested one.6 Requirement: The VNC Client must support that the VNC Server may ignore framebuffer update request messages, if multiple are sent at a time. Multiple non- incremental framebuffer update request messages may be combined into one 5 Zero Framebuffer updates are not forbidden from the VNC specification specifically. Though some existing VNC clients, display warnings. 6 This occurs, if the VNC Client requests a non-incremental framebuffer update for a specific area and other areas have changed in the mean time as well. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  25. 25. - 25 - framebuffer update response. It is recommended that the VNC Client sends only one Framebuffer Update Request message at a time. Requirement: The VNC Client must support that the VNC Server will not respond imme- diately to an incremental framebuffer update request. The Server may wait for the next response until the framebuffer has changed on the Server side. Requirement: The VNC Server must respond immediately to a non-incremental framebuf- fer update request. Recommendation: It is recommended that the VNC Client has a copy of the server side frame- buffer locally available. Therefore it is recommended that the VNC Client re- quests incremental framebuffer updates. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  26. 26. - 26 - 5.3 VNC Terminal Mode Extension Messages The existing RFB protocol specification is not sufficient to cover the mobile device – remote car display configuration space. Therefore extensions to the current protocol are specified in this chapter. Extensions are made in compliance with the RFB protocol, introducing a new Terminal Mode message type (128). Under the Terminal Mode message type, a couple of additional messages are introduced to the VNC protocol. These can be distinguished via unique extension-types. All extension messages must use the Terminal Mode message type and must follow the following fundamental design principle. # bytes Type Value Description 1 U8 128 Terminal Mode Message-type 1 U8 Extension-type 2 U16 N Payload length N U8 array Message specific payload data Table 5: VNC Extension Type Message Structure Requirement: The VNC Server and Client must handle Terminal Mode extension messages with unknown extension types, by reading the whole message (body and payload) and ignoring it. The Terminal Mode Message type defines the following set of new VNC messages given in Table 6. All extension messages are mandatory server-side, but the execution of some messages can be disabled within the Server or Client Event Configuration message. Exten- Disable sion- Message Message Name Origin Support Execu- Type tion 1 Display ServerDisplayConfiguration Server Mandatory No Configura- 2 tion ClientDisplayConfiguration Client Mandatory No 3 Event Con- ServerEventConfiguration Server Mandatory No 4 figuration ClientEventConfiguration Client Mandatory No 5 Event Map- EventMapping Server Mandatory No 6 ping EventMappingRequest Client Optional No 7 Key Event KeyEventListing Server Mandatory Yes 8 Listing KeyEventListingRequest Client Optional Yes 9 Virtual Key- VirtualKeyboardTrigger Server Mandatory Yes board Trig- 10 ger VirtualKeyboardTriggerRequest Client Optional Yes 11 Device Sta- DeviceStatus Server Mandatory No 12 tus DeviceStatusRequest Client Optional No 13 Content At- AttestationResponse Server Mandatory No 14 testation AttestationRequest Client Optional No Framebuffer 16 FramebufferBlockingNotification Client Optional No Blocking Table 6: New Extension Types for Terminal Mode Messages Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  27. 27. - 27 - Requirement: The Client must disable the key event listing and the virtual keyboard trigger support in the Client Event Configuration message, if it has not implemented the respective request messages. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  28. 28. - 28 - 5.3.1 Display Configuration Messages The Server and Client Configuration message pair exchanges additional display information be- tween the client and the server. Based on the received information the client may change the pixel format, sending a Set Pixel Format message. The server will use this information to optionally provide higher resolution virtual framebuffer copies. The message flow is shown in Figure 9. VNC Server VNC Client Set Encodings -523 (Terminal Mode) Server Display Configuration Client Display Configuration Set Pixel Format Figure 9: Server and Client Display Configuration Requirement: The Server Display Configuration Message must be sent immediately in re- sponse to the Set Encoding message, indicating Terminal Mode (-523) sup- port, prior any other message. Requirement: The Client Display Configuration Message must be sent immediately in re- sponse to the Server Display Configuration Message, prior any other mes- sage. The Server Display Configuration message is given in Table 7. It will be responded from the Client with a Client Display Configuration message. # bytes Type Value Description 1 U8 128 Message-type 1 U8 1 Extension-type 2 U16 12 Payload length 1 U8 1 Terminal Mode Server Major Version 1 U8 0 Terminal Mode Server Minor Version Bit Framebuffer configuration (1 = yes, 0 = no) Server-side framebuffer orientation switch available 1. The VNC server must start in default orientation, which 0 is given in the Server Init message (values width and height). 2 U16 Server-side framebuffer rotation available 1 • The VNC server must start with no rotation. Server-side framebuffer scaling available • The VNC server must be able to scale the framebuffer to 2 the client resolution (as provided from the VNC client in the Client Display Configuration message) 2 U16 Relative pixel width (set to zero, if relative width not known) 2 U16 Relative pixel height (set to zero, if relative height not known) Bit RGB Color conversion support (1 = yes, 0 = no) 4 U32 0 32 bit ARGB 888 (mandatory support for VNC server) Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  29. 29. - 29 - # bytes Type Value Description 7 All other 32 bit formats 8 16 bit RGB 565 (mandatory support for VNC server) 9 16 bit RGB 555 15 All other 16 bit formats 16 bit single color (grayscale) 25 • Client must use red_shift and red_mask to set gray range 8 bit single color (grayscale) 26 • Client must use red_shift and red_mask to set gray range Table 7: Server Display Configuration Message Recommendation: The relative pixel width and height should be used to compensate for differ- ent pixel aspect rotation on client and server side. Requirement: The Server Display Configuration message must be sent only once. The Client Display Configuration message is shown in the following table. # bytes Type Value Description 1 U8 128 Message-type 1 U8 2 Extension-type 2 U16 14 Payload length 1 U8 1 Terminal Mode Client Major Version 1 U8 0 Terminal Mode Client Minor Version Bit Framebuffer Configuration (1 = yes, 0 = no) Server-side framebuffer orientation switch used 0 • If enabled, the VNC client must use the Device Status Request message (section 5.3.6) 2 U16 Server-side framebuffer rotation used 1 • If enabled, the VNC client must use the Device Status Request message (section 5.3.6) Client-side framebuffer scaling available • If enabled, the VNC client must be able to scale the 2 server framebuffer (as provided in the Server Display 7 Configuration message) to the client resolution 2 U16 Client display width [pixel] (unknown value must be 0) 2 U16 Client display height [pixel] (unknown value must be 0) 2 U16 Client display width [mm] (unknown value must be 0) 2 U16 Client display height [mm] (unknown value must be 0) Distance driver – client display [mm] (unknown value must be 2 U16 0) Table 8: Client Display Configuration Message 7 According to the RFB specification, the client must support any server framebuffer resolution. A client must not expect the server to support scaling. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  30. 30. - 30 - Requirement: The Client Display Configuration message must be sent only once. Requirement: The VNC client must support Desktop Size Pseudo Encoding, if it enables bit 0 or 1 in the framebuffer configuration entry. Requirement: If the VNC client does not support scaling (disabled bit 2 in the framebuffer configuration entry), the VNC server must provide the framebuffer content in the requested client display resolution in one of the following options shown in Figure 10. Scaling Framing Clipping Figure 10: VNC Server Options for non-scalable Clients Scaling, if the VNC server supports scaling (maintaining the server as- pect ratio – with (0, 0) offset; other client area remains unspecified), or Framing, if the VNC server does not support scaling and the server reso- lution is smaller than the client one (with (0, 0) offset; other client area remains unspecified), or Clipping, if the VNC server does not support scaling and the server reso- lution is bigger than the client one (framebuffer content aligned to the upper left corner). Requirement: No pixel data must be transmitted for unspecified client area (shown in red in Figure 10) Requirement: The VNC client must not provide a Terminal Mode minor and major version, which is higher than the VNC server supported version. Requirement: VNC client and server must be backward compatible with regard to different Terminal Mode versions. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  31. 31. - 31 - 5.3.2 Event Configuration Messages The Server and Client Event Configuration message pair provides additional information about the event handling, i.e. which key and pointer events are natively supported on the server and client. This information helps the server to map specific incoming client events to server events and helps the client to directly use specific server events. The message flow is shown in Figure 11. VNC Server VNC Client Server Event configuration Client Event configuration Figure 11: Server and Client Event Configuration Requirement: The Server Event Configuration Message must be sent immediately in re- sponse to the Set Encoding message, indicating Terminal Mode (-523) sup- port. The message may be delayed only until reception of the Client Display Configuration message. Requirement: The Client Event Configuration Message must be sent immediately in re- sponse to the Server Event Configuration Message, prior any other message. The Server Event Configuration message is given Table 9. # bytes Type Value Description 1 U8 128 Message-type 1 U8 3 Extension-type 2 U16 28 Payload length 2 U16 Keyboard layout – Language code (according ISO 639-1) Keyboard layout – Country code (according ISO 3166-1 al- 2 U16 pha-2) 2 U16 UI Language – Language code (according ISO 639-1) UI Language – Country code (according ISO 3166-1 alpha- 2 U16 2) Knob shift & rotate configuration (Bitmask according Table 37) 4 U32 Bit l • ‘1’: Server supports knob key events8 • ‘0’: Server does not support knob key events Device keys (Bitmask according Table 38) • Bitmask defined in Table 38 4 U32 Bit m • ‘1’: Server supports device key events • ‘0’: Server does not support the key events Multimedia keys (Bitmask according Table 38) 4 U32 Bit n • Bitmask defined in Table 38 • ‘1’: Server supports multimedia key events 8 The Server can claim e.g. the device cursor keys plus a Device_Ok key to represent a simple knob, i.e. supporting shift up, down, left right and push knob events. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  32. 32. - 32 - # bytes Type Value Description • ‘0’: Server does not support the multimedia key events Bit Key related (1 = support, 0 = no support) 0 ITU keypad (T9) events (‘0’, … ,’9’, ‘#’, ‘*”) 1 Virtual keyboard trigger support 4 U32 2 Key event listing support # additional soft and hard keys, the server requires 8 – 15 • Key events defined in Table 38 • Key events start with TM_Key 0, no subsequent gaps Bit Pointer related (1 = support, 0 = no support) 0 Single-touch events 4 U32 1 Multi-touch events 8 – 15 Pointer event button mask (according RFB spec) Table 9: Server Event Configuration Message Requirement: Server event configuration must be sent only once. The payload structure of the Client Event Configuration message is fully symmetrical with the Server Event Configuration message, as shown in Table 10. # bytes Type Value Description 1 U8 128 Message-type 1 U8 4 Extension-type 2 U16 28 Payload length 2 U16 Keyboard layout – Language code (according ISO 639-1) Keyboard layout – Country code (according ISO 3166-1 al- 2 U16 pha-2) 2 U16 UI Language – Language code (according ISO 639-1) UI Language – Country code (according ISO 3166-1 alpha- 2 U16 2) Knob shift & rotate configuration (Bitmask according Table 37) 4 U32 Bit l • ‘1’: Client supports knob key events • ‘0’: Client does not support knob key events Device keys (Bitmask according Table 38) 4 U32 Bit m • ‘1’: Client supports device key events • ‘0’: Client does not support device key events Multimedia keys (Bitmask according Table 38) 4 U32 Bit n • ‘1’: Client supports multimedia key events • ‘0’: Client does not support multimedia key events Bit Key related (1 = support, 0 = no support) 0 ITU keypad (T9) events (‘0’, … ,’9’, ‘#’, ‘*”) 1 Virtual keyboard trigger support (ignored at server) 4 U32 2 Key event listing support (ignored at server) # additional soft and hard keys, the client supports 8 – 15 • Key events defined in Table 38 • Key events start with TM_Key 0, no subsequent gaps 4 U32 Bit Pointer related (1 = support, 0 = no support) Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  33. 33. - 33 - # bytes Type Value Description 0 Single-touch events 1 Multi-touch events 8 – 15 Pointer event button mask (according RFB spec) Table 10: Client Event Configuration Message Requirement: Client Event Configuration must be sent only once. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  34. 34. - 34 - 5.3.3 Event Mapping Messages The Event Mapping and Event Mapping Request message pair provides the client with informa- tion about the server mapping of high key symbol values. The Client can send an Event Mapping Request message at any time, either requesting or setting a specific mapping within the server. The server must always send an Event Mapping message in response of an Event Mapping Re- quest message. If the server changes any event mapping locally, the server must inform the client via an Event Mapping message, if the client has requested the mapping at least once. The message flow follows the following steps, as shown in Figure 12. VNC Server VNC Client Event Mapping Request Event Mapping Server changes Event Mapping Event Mapping Figure 12: Example Event Mapping Message Flow Requirement: The Server must respond to any Event Mapping Request message imme- diately. The Event Mapping message is given in Table 11. # bytes Type Value Description 1 U8 128 Message-type 1 U8 5 Extension-type 2 U16 8 Payload length 4 U32 Client Key Symbol Value Server Key Symbol Value 4 U32 (0 = client key value not support from server) Table 11: Event Mapping Message The Default Mapping Request message is given in Table 12. # bytes Type Value Description 1 U8 128 Message-type 1 U8 6 Extension-type 2 U16 8 Payload length 4 U32 Client Key Symbol Value Server Key Symbol Value 4 U32 (0 = request value from Server) Table 12: Event Mapping Request Message Requirement: If the server locally changes any event mapping, it must send an Event Map- ping message with appropriate Client and Server Key Symbol Values. The Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  35. 35. - 35 - server must send the Event Mapping message only, if the Client has re- quested the Client Key Symbol Value previously using an Event Mapping Request message. Requirement: If the server does not support a new mapping request according to the Event Mapping Request message, it must send an Event Mapping message, con- taining the existing mapping. The server key symbol value is set to zero if the server does not allow any mapping for the client symbol value. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  36. 36. - 36 - 5.3.4 Key Event Listing Message The Key Event Listing and Key Event Listing Request message pair provides the client with in- formation about the next valid characters, while entering text information. The VNC Server sup- port is announced in the Server Event Configuration message. The Client may start the key event listing at any time. In case a text entry is required, the server will provide the initial list of valid keys, which is getting updated after each key event, either as incremental or full update. An ex- ample message flow is shown in Figure 13. VNC Server VNC Client Key Event Listing Request Start Information Flow Key Event Listing Initial List – Counter n Key Event Key Event Listing List Update – Counter n+1 Key Event Listing Request Stop Information Flow Figure 13: Key Event Listing Messages Requirement: The VNC Server must send Key Event Listing messages only, if the VNC Client has enabled or requested them. Requirement: The VNC Client must not assume, that the VNC server will send Key Event Listing messages even if it has indicated support (the current application may not be able to support this feature). The Key Event Listing message is given in Table 13. # bytes Type Value Description 1 U8 128 Message-type 1 U8 7 Extension-type 2 U16 4 + 4*n Payload length Bit Configuration 0 Update flag (0 = non-incremental, 1 = incremental) 1 Listing flag (0 = black list, 1 = white list) Default event list flag (0 = normal list, 1 = default list ) • To reference to the default list, set this flag and set # 1 U8 2 key events in list to zero. • To set the default event list, set this flag and add key events to the KeySymValue list Other key event listing follows (0 = no, 1 = yes) 3 • Next key event listing must follow immediately • Black lists and white lists can be mixed 1 U8 n # key events in list 2 U16 Key event counter Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  37. 37. - 37 - # bytes Type Value Description 4*n U32 array KeySymValue list used to define the next valid character Table 13: Key Event Listing Message The flow using incremental updates is shown in Figure 14. Here, a default layout must be defined once per VNC session. This list can be used as a reference point prior the incremental update process. During the incremental update, the white list contains all key symbols to be added to the key event list, where as black lists contains all key symbols to be removed from the key event list. Default layout = 1 # key events in list != 0 Wait for Text Event Default layout = 1 # key events in list = 0 Wait for Key Event Last Yes Char? No Update flag = 1 Yes No Update flag = 1 White Listing flag = 1 Listing flag = 0 listing? Default layout flag = 0 Default layout flag = 0 No Other Yes Yes Other No List? List? Figure 14: Key Event Listing – Incremental Updates In case of non-incremental (i.e. full) updates, the key event listing looks like in Figure 15. Incre- mental and non-incremental updates can be mixed at any time. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.
  38. 38. - 38 - Wait for Text Event Default layout = 0 Update flag = 0 # key events in list != 0 Other Yes List? No Wait for Key Event No Last Yes Char? Figure 15: Key Event Listing – Non-Incremental Updates Requirement: The valid key symbol list must not differentiate between upper and lower case characters The Key Event Listing Request message is shown in Table 14. # bytes Type Value Description 1 U8 128 Message-type 1 U8 8 Extension-type 2 U16 4 Payload length Bit Configuration (0 = Disable, 1 = Enable) 0 Server key event listing 4 U32 1 Incremental updates 2 Reset key event counter Table 14: Key Event Listing Request Message The key event counter can be used to synchronize the server key event listing message to key events. The counter value must represent all received key events (key press events only). The counter must be set to zero on the reception of the Client Event Configuration message and if the specific bit is set in the Key Event Listing Request message. The counter must roll over to zero, once the maximum number is reached. The VNC Client must not assume that the VNC Server will send a key event list update before the client sends the next key press event. This specifically can happen if the key events are entered faster than the Server can provide key event list updates. In such case, the client should use the default key event list instead. Copyright © 2010 Nokia Corporation, Audi AG, BMW AG, Daimler AG, Porsche AG, Volkswagen AG. All rights reserved.

×