Lightweight Messaging for a Connected Planet


Published on

Video online at

The Internet of Things is one of the buzz phrases of the moment - the idea that there are a of a lot of devices around, from tiny sensors and RFID tags, through smartphones, GPS location-aware devices, laptops, and embedded systems. They are nearly all natively "online" to some extent, and most will have an Internet address of their own (even if the connections are not always super-high bandwidth, always-on, or reliable). So, they are becoming interconnected, either directly to one another across local networks, or indirectly via clouds. These devices typically have enough computing power to at least gather and transmit data, and some of them have enough to respond to requests to modify their behaviour. Many of these devices run Linux!

So, with the lightweight nature of the systems involved, what about a lightweight but reliable messaging protocol that can help to connect them? Well, there's one available, it's called MQTT (MQ Telemetry Transport, see, and it's increasingly being adopted for some very cool projects like, for example, mosquitto ( This presentation will look at why lightweight reliable messaging is useful; what MQTT is and how to use it (including examples in a number of languages); and how it can be used in a variety of systems from home automation solutions and Arduino-based gadgets all the way through to city-wide transport systems, and even used to bridge up to messaging systems used by large enterprises.

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Hello. Thanks for coming all the way to L block I'm Andy Been at IBM nearly 10 years working in integration software Been a Linux user and hacker for longer than that. Really wanted to come here to talk about some of the cool stuff we've been doing
  • Getting straight into this Fits really nicely with some of the things we've been talking about this week around mobile, and around things like Arduino and automation
  • If you've seen any IBM adverts recently you may have seen these icons or heard the words “Smarter Planet”. What does that mean? As the world becomes instrumented, interconnected and intelligent, we have the opportunity to think and act in new ways—economically, socially and technically.
  • Corporate stuff isn't always popular at a conference like this but of course it is worth understanding
  • Let's talk about one of those protocols at the edge of the network. About 10 years ago one of our Distinguished Engineers, Dr Andy Stanford-Clark, was working with oil companies and industrial automation companies and saw the opportunity for a lightweight protocol for connecting devices
  • I don't want to ignite any kind of flamewar here, I just want to highlight a couple of the key differences which may make us think about how we get data between devices
  • For clients requiring better remote operations, buffering, concentrators then there are more topology options. The WMQT Daemon for Devices was formerly Really Small Message Broker available on alphaWorks MQTT is already available in Microbroker (a component of Lotus Expeditor)
  • Phone by bed monitors pacemaker Dial-up -> MQTT to WMB -> analytics engine Avoid regular clinic visits
  • Energy co can send small changes and commands to homes over mobile network, minor temp adjustments -> significant savings
  • Lightweight Messaging for a Connected Planet

    1. 1. Lightweight Messagingfor a Connected Planet Andy Piper WebSphere Messaging Community Lead, IBM
    2. 2. Whats this all about?
    3. 3. The Internet of ThingsTrillions of smartdevices instrumentour world today Interconnecting these smart devices creates a Central Nervous System
    4. 4. Building a Smarter Planet + + =
    5. 5. The Networked Enterprise Connectivity for Applications Head Offices, Transport WebSphere MQ Data centres REST/HTTP Intelligent Cloud Computing WS* Mission-Critical Connectivity & Intelligence Messages WebSphere MQ Regional Offices Stores, Outlets Files WebSphere MQ File Transfer Edition Interconnected Transform, WebSphere Message Broker Enrich, WebSphere ESB Mediate DataPower WebSphere Sensor Events Catalog WebSphere Service Registry & Repository Intelligence WebSphere Business Events Instrumented Remote Systems and Devices Cognos. ILOG, SPSS InfoSphere StreamsConnectivity for Smart Devices Other InfoSphere and Tivoli productsTransport MQ Telemetry HTTP Multicast Actuators Embedded Controllers Sensors Power meters, weather data Tag printers, status lights, LoadMillions of Filtering of duplicate read events, Store- based HVAC & lighting controls, SCADA sensors, pressure, volume, generation, HVAC and lighting, RFID readers, Motion detectors… Valves, switches and pumps…devices Industrial Network Gateways (SCADA)
    6. 6. MQTT =MQ Telemetry Transport
    7. 7. Design principles■ Publish/subscribe messaging paradigm (useful for the majority of sensor applications)■ Minimise the on-the-wire footprint.■ Expect and cater for frequent network disruption – built for low bandwidth, high latency, unreliable, high cost networks■ Expect that client applications may have very limited processing resources available.■ Provide traditional messaging qualities of service where the environment allows.■ Publish the protocol royalty-free, for ease of adoption by device vendors and third- party software developers.
    8. 8. Key facts■ Low complexity and footprint■ Simple publish/subscribe messaging semantics  Asynchronous (“push”) delivery of messages to applications  Simple verbs: connect, publish, (un)subscribe, disconnect Minimised on-the-wire format  Plain byte array message payload  No application message headers  Protocol compressed into bit-wise headers and variable length fields  Smallest possible packet size is 2 bytes■ In-built constructs to support loss of contact between client and server  “Last will and testament” to publish a message if the client goes offline  Stateful “roll-forward” semantics and “durable” subscriptions
    9. 9. What aboutthat HTTP thing?
    10. 10. Good point.Heres a (very) quick comparison.
    11. 11. Data-centricityMQTT is agnostic of data content and transferssimple byte arrays, making drip-feeds ofupdating information trivial.HTTP is (basically) document-centric.
    12. 12. SimplicityMQTT has few methods(publish/subscribe/unsubscribe) and is quick tolearn.HTTP can be complex (although it is often well-understood) - there are a multitude of returncodes and methods. REST is a great principle but not always the bestfor simple data applications(POST/PUT/GET/DELETE? er what?)
    13. 13. Light on the networkThe smallest possible packet size for an MQTTmessage is 2 bytes. The protocol was optimised from the start forunreliable, low-bandwidth, expensive, high-latency networks.HTTP is relatively verbose - lots of "chatter" in aPOST
    14. 14. Easy distribution of dataMQTT distributes 1-to-none, 1-to-1 or 1-to-nvia the publish/subscribe mechanism→ very efficientHTTP is point-to-point (can bemediated/clustered but no distributionmechanism). To distribute to multiple receivers alarge number of POSTs may be required.
    15. 15. Lightweight Stack (CPU/Mem)MQTT has been trivially implemented on tiny tolarger platforms in very small libraries[IBM ref implementation = ~80Kb for full broker]HTTP (often with associated XML or JSONlibraries for SOAP and REST etc) can be relativelylarge on top of OS network librariesPlus... even if the client is small, considerwhether it is really necessary to run an HTTPserver on every device
    16. 16. Variable Quality-of-ServiceMQTT supports fire-and-forget or fire-and-confirm (aka QoS 0/1/2)HTTP has no retry / confirmation / attempt atonce-only delivery. It is basically brittle, i.e. retryneeds to be written in at the application level.Applications must also handle timeouts.
    17. 17. Sounds like theres a place for both! Where is MQTT in use?
    18. 18. • Simple• Lightweight (CPU,Mem,**Net)• Data-centric• Distributes data (pub/sub)• Range of QoS → strong developer community
    19. 19. “strong developer community” huh...
    20. 20. Ok, to be fair, I haveno knowledge of theirphysical strength, butthey are all awesome...
    21. 21. Home automation
    22. 22. Gardening “It all started with the seemingly simple question – “How can I water the garden without leaving my laptop/phone/sofa using tech?”” - Dan Fish
    23. 23. TV, Android, burglar “control”
    24. 24. Mind-controlled Taxis b “Kevin already had the headset hooked up to MQTT, so it would be trivial to use my Arduino MQTT library to get them all talking.” - Nick OLeary
    25. 25. Flashing Arduino-controlled ducks “Now, you may wonder why I would want 20 rubber ducks to flash when my phone goes off.... There is no scientific or technical reason in itself. I just had a Mini Cooper’s worth of rubber ducks sitting around, unemployed.” - Chris Phillips
    26. 26. This sounds moderatelyinteresting (and fun) Lemme at it!
    27. 27. Seriously.Weve been waiting for 25 slides already. CODE, DAMMIT
    28. 28. The IBM way•• Download• Unzip• Run nohup ./broker >> /dev/null &• Play with C client utils, code, and be merry!• Available for Linux IA32, IA64 kernel 2.6.8+; Linux on IBM System z; Linux for ARM XScale, kernel 2.0.0+ (Crossbow Stargate or Eurotech Viper); Windows XP; Mac OS X Leopard; Unslung (Linksys NSLU2) – Binary only, request other platforms from IBM
    29. 29. Alternatively...•• On e.g. Ubuntu: sudo add-apt-repository ppa:mosquitto-dev/mosquitto- ppa && sudo apt-get update && sudo apt-get install mosquitto (optional: mosquitto-pub, mosquitto-sub, python- mosquitto)• Runs as a daemon; IPv4/IPv6-capable• C++ and Python examples available• Packaged for Ubuntu, Fedora, RHEL, OpenSuSE, CentOS, Debian, Mandriva; Windows - binary; OS X – binary (homebrew compile via github package); source tarball; dev version in bitbucket
    30. 30. Client APIs• C• C++• Java (your route to Android and Kindle goodness)• .NET (yes, Mono too...)• Delphi • Perl• Python • PHP • Ruby • Erlang • Arduino • mbed
    31. 31. Show us the code!public void sendAMessage() throws MqttException { MqttProperties mqttProps = new MqttProperties(); Create a connection using the mqttProps.setCleanStart( true ); connection factory, this time MqttClient client = MqttClientFactory. INSTANCE. for a clean starting client createMqttClient("testClient", “tcp://localhost:1883”, mqttProps); Register the class as a listener and client.registerCallback(this); connect to the broker client.connect(); client.publish(“abc/123”, new MqttPayload((“Hello World!”).getBytes(),0), (byte) 2, false); client.disconnect(); Publish a message to the} given topic and disconnectpublic void publishArrived (String topicName, MqttPayload payload, byte qos, boolean retained, int msgId) { System.out.println(“Got it!”);} On receipt of a publication, simply print out a message on the console to say we received it
    32. 32. Moar code plz#!/usr/bin/pythonimport pynotifyimport mosquitto Setup a callback – post a# define what happens after connection desktop notification with message topic and payloaddef on_connect(rc): when a publication arrives print "Connected"# On receipt of a message create a pynotification and show itdef on_message(msg): n = pynotify.Notification (msg.topic, msg.payload) ()# create a client connectionmqttc = mosquitto.Mosquitto("python_sub")# define the callbacksmqttc.on_message = on_messagemqttc.on_connect = on_connect Set a client id (“python_sub”), connect to localhost, and# connect subscribe on topic “test”mqttc.connect("localhost", 1883, 60, True)# subscribe to topic testmqttc.subscribe("test", 2)# keep connected to brokerwhile mqttc.loop() == 0: pass
    33. 33. Community• (including wiki)• rsmb forum at IBM alphaWorks• #mqtt on freenode• mosquitto project on launchpad•• many bloggers, developers, etc...
    34. 34. More random-but-cool schtuffs• File sync over MQTT?• Desktop notifications and• Web thermometers• Digital-to-analogue readouts• CEIT @ UQ research projects• LEGO microscope control
    35. 35. Elsewhere in theworld, exciting things were occurring...
    36. 36. Integrating to the EnterpriseWebSphere MQ Telemetry Sensors DIRECT MQTT CLIENT WebSphere Mobile, smart meters MQ 7.0.1 MQXR Applications BRIDGERich clients MQTT BRIDGErequiring buffering, WebSphere MQ Telemetryremote management Daemon for Devices DIRECTcapabilities, or MQTT CLIENT Microbrokeradvanced data handling (Lotus Expeditor, WebSphere Sensor Events)
    37. 37. Smarter HealthcareMedical organization created a remote pace-maker monitoringsolution to provide better patient care Client Pains  Physicians needed better monitoring of cardiac patients  Improve efficiency of checkups  Meet healthcare data capture standards Enables higher level of patient care and peace of mind Improves administrative efficiency and maintenance Helps conform to standards and ease integration of data
    38. 38. Smarter EnergyUtility company developing an Intelligent Utility Network offering foroptimizing load on electricity grids Business Partner  Needs robust middleware technology to connect to remote smart meters  Needs to be able to rapidly scale solution nationally Able to offer daily energy savings of 15-20% Enables utilities to reduce peaks and avoid punitive charges Helps save electricity through better peak load management
    39. 39. KTHXBAI! Andy Piper @andypiper
    40. 40. Thanks!!• Roger Light @ralight (mosquitto awesomeness)• Nick OLeary @knolleary (Arduino/MQTT awesomeness – images from Flickr – and general uber-awesomeness)• Chris Yeoh @ckbyeoh (home hacking awesomeness)• Benjamin Hardill @hardillb (TV hacking awesomeness)• Chris Phillips @cminion (Rubber Duck awesomeness)• Oliver Smith @chemicaloliver (lots of webby awesomeness)• Dan Fish @ossmedicine (garden awesomeness)• Donna Benjamin @kattekrab (Inkscape awesomeness)