A 5-part course on the two most popular application layer protocols for the IoT namely MQTT and CoAP.

1. Application Layer Protocols
for the IoT
Damien Magoni
University of Bordeaux
2017/10/18
Version 1

2. Attribution
• The material contained inside is intended for teaching.
• This document is licensed under the CC BY-NC-SA license.
• Relevant sources are listed on the following References slide.
• All figures and text borrowed from these sources retain the rights of
their respective owners.
2

3. References
• https://www.oasis-open.org/standards#mqttv3.1.1
• http://mqtt.org/
• https://github.com/mqtt/mqtt.github.io/wiki
• http://www.cs.wustl.edu/~jain/cse570-15/m_14mqt.htm
• http://mqtt.org/new/wp-content/uploads/2009/06/MQTT-
SN_spec_v1.2.pdf
• https://www.slideshare.net/zdshelby/coap-tutorial
• https://tools.ietf.org/html/rfc7252
• https://www.slideshare.net/zdshelby/oma-lightweightm2-mtutorial
3

4. Acknowledgments
• Special thanks to Zach Shelby and Raj Jain for their high-quality
teaching material that I have extensively reproduced here.
4

5. Table of Contents
1. IoT and Web of Things networking
2. Message Queuing Telemetry Transport
3. Constrained Application Protocol
4. Implementations on Raspberry Pi
5. Programming with MQTT/CoAP APIs on Raspbian
5

6. IoT Networking
Part 1

7. Networking on the Edge
• Low power, low speed, lossy wireless networks
• Huge number of low capability end devices
• Automatically generated (big) data
• Terse, purposeful, uncritical
7

8. Current Human-oriented Protocol Stack
• Communications are mostly one-to-one/unicast and sender-oriented
• M2M comms do not need protocols for human comms
• High overhead features: smooth streaming, character echoing, presentation
format, etc.
• Low-end devices can not store/use a full network protocol stack
• Functionnality needs hardware which costs money
8

9. IoT Paradigms
• Wireless networks to connect them all
• Cheap and adaptive infrastructures compared to wired networks
• Many-to-few/few-to-many/many-to-many communications
• Group-oriented/multicast communications
• Can cope with huge number of devices
• Reliability through redundancy
• Losses and errors are frequent and unavoidable
• Large number of unreliable, uncritical messages
• Simple and autonomous devices
9

10. Zones of Interest
• In nature, zones of interest are formed by affinities
• Common medium used by all
• Individuals act upon specific signals among countless other signals
• Interested observing individuals can select a zone of interest
• Analyze/send data only to/from this zone
• Receiver-oriented
• Wireless channels can broadcast all messages to neighbors
• Neighbors process messages if they wish
10

11. Publish/Subscribe Paradigm
• Devices publish information tagged with category (i.e. zone)
• They broadcast it to everyone
• Devices subscribe to some categories
• They act only on what they are interest in
• No tight coupling between publishers and subscribers
• Unmanaged loose coupling
• Signals/messages are simple and small
• This communication paradigm scales
11

12. Publish/Subscribe Topologies
• With intermediary
• Publishers post messages to an intermediary message broker
• Subscribers register subscriptions with that broker
• The broker performs message filtering and “store and forward” routing from
publishers to subscribers
• The broker may prioritize messages in a queue before routing
• Without intermediary
• Each publisher and subscriber shares meta-data about each other via
multicast
• Publishers and subscribers cache this information locally and route messages
based on the discovery of each other interests
12

13. Functional Levels (IoT Hierarchy Revisited)
• End devices: mostly publishers
• Meshed to one another and connected to propagators
• Generate data (sensors) or act on command (actuators)
• Propagator nodes: brokers
• Connected to the Internet
• Process and aggregate data from end devices to integrators
• Integrator nodes: mostly subscribers
• Connected to the Internet
• Analyse data, control the system, and interface with humans
13

14. Web of Things
14

15. Web Architecture
15

16. Web Naming
16

17. URL Resolution
17

18. HTTP Request
18

19. Web Paradigms
19

20. REST Request
20

21. Message Queue Telemetry
Transport
Part 2

22. What is MQTT
• ISO standard publish-subscribe-based lightweight messaging protocol
• Telemetry = tele-metering = remote measurements
• However, no message queuing in MQTT
• Lightweight = low network bandwidth and small code footprint
• For use on top of the TCP/IP protocol stack
• The publish-subscribe messaging pattern requires a message broker
• The broker is responsible for distributing messages to interested
clients based on the topic of a message
• Data agnostic, adapt to any content formats
22

23. Publish / Subscribe
23
• Clients can subscribe to a topic
or a set of related topics
• Clients can also publish to one or
more topics
• Publish/Subscribe via message
exchanges
• Topics organized as trees using /
character
• /# matches all sublevels
• /+ matches only one sublevel

24. Quality of Service Levels
• Three levels
• 0 = At most once (best effort, no Ack)
• 1 = At least once (Acked, retransmitted if Ack not received)
• 2 = Exactly once
• Control Messages Sequence: Publish → Pubrec → Pubrel → Pubcomp
• Server keeps/retains messages even after sending it to all subscribers
• New subscribers get the retained messages
24

25. Control Messages
• Indicate the desired action to be
performed on the identified
resource/server
• Publish messages can be to/from
Client/Server
• Often, the resource corresponds
to a file or the output of an
executable residing on the
server
• CONNECT/CONNACK
• PUBLISH/PUBACK (QoS 1)
• PUBREC/PUBREL/PUBCOMP (QoS 2)
• SUBSCRIBE/SUBACK
• UNSUBSCRIBE/UNSUBACK
• PINGREQ/PINGRESP
• DISCONNECT
25

26. Sessions and Connections
• Clean or continuous sessions with durable connections
• At connection set up, if Clean Session flag → all subscriptions are removed on
disconnect
• Otherwise subscriptions remain in effect after disconnection → subsequent
messages with high QoS are stored for delivery after reconnection
• Wills
• at connection set up, a client can set a Will flag to inform that it has will
message that should be published if unexpected disconnection → alarm if the
client looses connection
• Periodic keep-alive messages → if a client is still alive
26

27. Example
• Sensors publish
• Brokers can be bridged
27

28. Comparison with HTTP
28

29. MQTT for Sensor Networks (MQTT-SN)
• Variation of the main protocol aimed at embedded devices on non-
TCP/IP networks such as ZigBee
• Zigbee is a low-power, low data rate, close proximity wireless ad hoc PAN
• Optimized for implementation on low-cost, battery-operated devices
with limited processing and storage resources
• Enforce short messages (<<128B)
• Support for sleeping clients, CONNECT message split in 3, topic names
replaced by 2B topic ids, predefined topic ids with no registration,
server/gateway network @ discovery procedure
29

30. MQTT-SN Architecture
30
The forwarder encapsulates/ decapsulates the
MQTT-SN frames it receives from the
client/the GW and forwards them unchanged

31. Client Connection
31

32. Client State Transition Diagram
32

33. Alternative Protocols
• Constrained Application Protocol (CoAP)
• Advanced Message Queuing Protocol (AMQP)
• Streaming Text Oriented Messaging Protocol (STOMP)
• Extensible Messaging and Presence Protocol (XMPP)
• Web Application Messaging Protocol (WAMP)
• Object linking and embedding for Process Control - Unified
Architecture (OPC UA)
• M2M communication protocol for industrial automation
33

34. Constrained Application Protocol
Part 3

35. CoAP Design Requirements
35

36. CoAP Architecture
36

37. What is CoAP
• CoAP is
• A very efficient RESTful protocol
• Ideal for constrained devices and networks
• Specialized for M2M applications
• Easy to proxy to/from HTTP
• CoAP is not
• A general replacement for HTTP
• HTTP compression
• Restricted to isolated “automation” networks
37

38. Features
• Embedded web transfer protocol (coap://)
• Asynchronous transaction model
• UDP binding with reliability and multicast support
• GET, POST, PUT, DELETE methods
• URI support
• Small, simple 4 byte header
• DTLS based PSK, RPK and Certificate security
• Subset of MIME types and HTTP response codes
• Built-in discovery
• Optional observation and block transfer
38

39. Transaction Model
• Transport
• CoAP currently defines UDP binding with DTLS security
• CoAP over SMS or TCP possible
• Base Messaging
• Simple message exchange between endpoints
• Confirmable or Non-Confirmable Message
• Acknowledgement or Reset Message
• REST Semantics
• REST Request/Response piggybacked on CoAP Messages
• Method, Response Code and Options (URI, content-type, etc.)
39

40. Message Header
40

41. Option Format
41

42. Specification Options
42

43. Request
43

44. Dealing with Packet Loss
44

45. Separate Response
45

46. Bits and Bytes
46

47. Caching
• CoAP includes a simple caching model
• Cacheability determined by response code
• An option number mask determines if it is a cache key
• Freshness model
• Max-Age option indicates cache lifetime
• Validation model
• Validity checked using the Etag Option
• A proxy often supports caching
• Usually on behalf of a constrained node,
• a sleeping node,
• or to reduce network load
47

48. Proxying and Caching
48

49. Observation
49

50. Block Transfer
50

51. Web Linking
• Web Linking formalizes links with defined relations, typed links
• HTML and Atom have allow links
• RFC5988 defines a framework for Web Linking
• Combines and expands the Atom and HTML relation types
• Defines a unified typed link concept
• A link can be serialized in any number of formats
• RFC5988 revives the HTTP Link Header and defines its format
• Atom and HTML are equivalent serializations
51

52. Typed Link
• A type link consists of
• Context URI – What the link is from
• Relation Type – Indicates the semantics of the link
• Target URI – What the link is to
• Attributes – Key value pairs describing the link or its target
• Relations include e.g. copyright, author, chapter, service, etc.
• Attributes include e.g. language, media type, title, etc.
• Example in HTTP Link Header format
Link: <http://example.com/TheBook/chapter2>; rel="previous"; title="previous chapter"
52

53. Resource Discovery
• Service Discovery
• Find the IP address, port and protocol of the service
• Usually performed by DNS-SD when DNS is available
• Resource Discovery
• Finding URIs
• Performed using Web Linking or some REST interface
• CoRE Link Format is designed to enable resource discovery
53

54. CoRE Link Format
• RFC6690 is aimed at Resource Discovery for M2M
• Defines a link serialization suitable for M2M
• Defines a well-known resource where links are stored
• Enables query string parameters for filtered GETs
• Can be used with unicast or multicast (CoAP)
• Resource Discovery with RFC6690
• Discovering the links hosted by CoAP (or HTTP) servers
• GET /.well-known/core?optional_query_string
• Returns a link-header style format
• URL, relation, type, interface, content-type etc.
54

55. CoRE Resource Discovery
55

56. Resource Directory
• CoRE Link Format only defines
• The link format
• Peer-to-peer discovery
• A directory approach is also useful
• Supports sleeping nodes
• No multicast traffic, longer battery life
• Remote lookup, hierarchical and federated distribution
• The CoRE Link Format can be used to build Resource Directories
• Nodes POST (register) their link-format to an RD
• Nodes PUT (refresh) to the RD periodically
• Nodes may DELETE (remove) their RD entry
• Nodes may GET (lookup) the RD or resource of other nodes
56

57. CoAP M2M Interface
57

58. Using CoRE in Real Applications
• Resources need meaningful naming (rt=)
• A resource needs an interface (if=)
• Use WADL for this
• A payload needs a format (EXI, JSON, etc.)
• Deployment or industry specific today
• oBIX, SensorML, EEML, sMAP, etc.
• SenML is a promising format [draft-jennings-senml]
• What can we make universal? What should be market specific? How
do we enable innovation?
58

59. CoRE Link Format Semantics
• RFC6690 = Simple semantics for machines
• IANA registry for rt= and if= parameters
• Resource Type (rt=)
• What is this resource and what is it for?
• e.g. Device Model could be rt=“ipso.dev.mdl”
• Interface Description (if=)
• How do I access this resource?
• e.g. Sensor resource accessible with GET if=“core.s”
• Content Type (ct=)
• What is the data format of the resource payloads?
• e.g. text/plain (0)
59

60. CoRE Interfaces
60
A paradigm for REST profiles made up of function sets

61. Open Mobile Alliance (OMA)
• Standards body which develops open standards for the mobile phone
industry
• Only standardizes applicative protocols
• Specifications are meant to work with any cellular network
technologies
• Example specifications
• MMS multimedia messaging
• Instant Messaging and Presence Service
• Lightweight M2M
61

62. Lightweight M2M
• Provide device management functionality over sensor or cellular
networks
• 6LoWPAN, WiFi, ZigBee, any IP based constrained devices/networks
• Transfer service data from the network to devices
• Can be extended to meet the requirements of any application
• Supported in OneM2M and integrated with ETSI M2M
62

63. LWM2M 1.0 Features
• Simple object based resource model
• Global registry and public lookup of all objects
• Standard device management objects already defined by OMA
• Resource operations of creation/retrieval/update/deletion/configuration of attribute
• Resource observation/notification
• TLV/JSON/Plain Text/Opaque data format support
• UDP and SMS transport layer support
• DTLS based security
• Pre-shared and Public Key modes, Provisioning and Bootstrapping
• Queue mode for NAT/Firewall environment
• Multiple server support
• Basic M2M functionalities
• Access Control, Device, Connectivity, Firmware Update, Location, Connectivity Statistics
63

64. LWM2M Architecture
64

65. IoT Standardization
• IETF
• 6LoWPAN Working Group (IPv6 anywhere)
• ROLL (Routing Over Low-power Lossy Networks) WG
• CoRE WG (REST for IoT, CoAP, Resource Directory etc.)
• TLS WG (DTLS)
• OMA
• Lightweight M2M Enabler Standard (CoAP/DTLS based)
• Device Management 2.0 Enabler Standard (HTTP/TLS based)
• ETSI / OneM2M
• Ongoing work on M2M system standardization (CoAP, HTTP binding)
• W3C
• Efficient XML Interchange (EXI) standardization
• ZigBee IP
• An open-standard 6LoWPAN stack for e.g. Smart Energy 2.0
65

66. Implementations of Protocols on
the Raspberry Pi
Part 4

67. Eclipse for IoT
• https://iot.eclipse.org/
• Plenty of open-source protocol implementations
• https://iot.eclipse.org/standards/
67

68. MQTT
• Eclipse Mosquitto server (in C)
• https://projects.eclipse.org/projects/technology.mosquitto
• https://mosquitto.org/
• https://github.com/eclipse/mosquitto
• Eclipse Paho client
• Open-source client implementations of MQTT/MQTT-SN protocols
• http://www.eclipse.org/paho/
• MQTT Paho client library (in Python)
• https://pypi.python.org/pypi/paho-mqtt
• pip install paho-mqtt
68

69. CoAP
• Implementations
• http://coap.technology/impls.html
• libcoap (client and server in C)
• https://libcoap.net/
• OpenWSN CoAP Python library
• https://github.com/openwsn-berkeley/coap
• txThings (in Python)
• https://github.com/mwasilak/txThings
• CoAPthon
• https://github.com/Tanganelli/CoAPthon
• Copper (Cu) CoAP user-agent for Firefox
• https://addons.mozilla.org/en-US/firefox/addon/copper-270430/
69

70. Programming with MQTT and
CoAP APIs on Raspbian
Part 5

71. MQTT Client Subscription Example
import paho.mqtt.client as mqtt
# The callback for when the client receives a
CONNACK response from the server.
def on_connect(client, userdata, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose
the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("$SYS/#")
# The callback for when a PUBLISH message is
received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("iot.eclipse.org", 1883, 60)
# Blocking call that processes network traffic,
..dispatches callbacks and handles reconnecting.
# Other loop*() functions are available that give
.. a threaded interface and a manual interface.
client.loop_forever()
71

72. Other MQTT Client Examples
• https://github.com/eclipse/paho.mqtt.python/tree/master/examples
72

73. CoAP GET Request Client Example
• https://github.com/mwasilak/txThings/blob/master/examples/clientGET.py
73