• Save
Lightweight Messaging (Apache Retreat Hursley 2010)
Upcoming SlideShare
Loading in...5
×
 

Lightweight Messaging (Apache Retreat Hursley 2010)

on

  • 4,049 views

A discussion of MQTT presented at the Apache Retreat at IBM Hursley in 2010

A discussion of MQTT presented at the Apache Retreat at IBM Hursley in 2010

Statistics

Views

Total Views
4,049
Slideshare-icon Views on SlideShare
3,674
Embed Views
375

Actions

Likes
4
Downloads
0
Comments
3

6 Embeds 375

http://andypiper.co.uk 358
http://lanyrd.com 13
http://www.theinfoboom.com 1
http://sotech.org.uk 1
url_unknown 1
http://andypiper.wordpress.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

13 of 3 Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • The image credit for slide 18 goes to Nick O'Leary borrowed from his Arduino library for MQTT page http://knolleary.net/arduino-client-for-mqtt/
    Are you sure you want to
    Your message goes here
    Processing…
  • Thanks Ben. This was a cut-down version of a longer session I'd normally do, to fit the audience and Barcamp style - some background around Smarter Planet and the Internet of Things, and how MQTT extends the MQ ecosystem. I expect one of the longer versions will get posted before too long :-)
    Are you sure you want to
    Your message goes here
    Processing…
  • a very good presentation, Andy!
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Who am I I want to talk about a very exciting new area and way in which IBM is extending its messaging middleware technology
  • http://mccabism.blogspot.com/2008/08/transistors-and-grains-of-rice.html 10^11 stars in the Milky Way galaxy, 10^14 cells in the human body, 10^15 grains of rice produced annually worldwide, and 10^17 ants on planet Earth, it seems that the annual world production of transistors exceeds all these figures, at 10 quintillion, 10^19. Not only that, but the cost of producing each transistor is about one-hundreth of the cost of producing each grain of rice!
  • As the world becomes instrumented, interconnected and intelligent, we have the opportunity to think and act in new ways—economically, socially and technically. INSTRUMENTED Support for event capture and filtering Sensor solutions Instrumentation for regulatory compliance (auditing and governance) INTERCONNECTED Flexible any-any connectivity with a range of “qualities of service” Extended reach ( Sensors, Remote locations; Business to Business) Unified operational Model for a globally connected infrastructure Dynamic Provisioning and Management Security and Governance Scale, performance Programming Model for a globally connected infrastructure Open Standards, Ease of solution construction INTELLIGENT Event Processing capabilities Business rules Integration with data and stream analytics Systems that automatically adjust to your business
  • MQTT has actually been around for a while already, but often I find that customers have not heard of it! It has had a few different names and has hidden away in different corners of the product family - known as SCADA (node in Message Broker), Mqisdp, pervasive Stable and widely used, 10 years old NOT MQLLM!
  • MQTT is really simple, lightweight, and easy to learn It has gained a lot of popularity in development communities
  • Separate purchase on top of QM as not all users need Telemetry Initial availability Windows and UNIX but should run on any platform supporting JVM – service may require recreation of issues on a formally supported platform No MQSC / PCF support initially Let's see how this looks in MQ Explorer
  • The MQ Telemetry support is fully integrated into MQ Explorer Connections to a queue manager are set up via a simple wizard and new Telemetry channels (default port 1883) MQ and MQTT publish and subscribe is interoperable – so MQ applications can publish to MQTT subscribers, and vice versa The test client is built-in. Similar to the GUI Utility shipped previously in the IA92 SupportPac (which will still work as the protocol is the same, but this is integrated into the Eclipse environment)
  • * performance report to be issued
  • For clients requiring better remote operations, buffering, concentrators then there are more topology options. The WMQT Daemon for Devices was formerly Really Small Message Broker available on alphaWorks MQTT is already available in Microbroker (a component of Lotus Expeditor)
  • Why do you think it might be important to secure TT clients? the devices are likely to be portable, and used in places that cannot be carefully controlled. WebSphere MQ security (OAM) also applies
  • For clients requiring better remote operations, buffering, concentrators then there are more topology options. The WMQT Daemon for Devices was formerly Really Small Message Broker available on alphaWorks MQTT is already available in Microbroker (a component of Lotus Expeditor)
  • VIEW/NOTES PAGE Mode – PRESS F5 or DOUBLE CLICK ON SLIDE TO VIEW SLIDE IN FULL SCREEN A national railway infrastructure company Synopsis: A country’s railway infrastructure utility improves information flow, reduces costs and increases flexibility with a sophisticated middleware solution designed and implemented by IBM Global Business Services Customer Background : The company has responsibility for the the country’s railways infrastructure , which involves running, maintaining and developing the country's railway tracks, signaling system, rail bridges, tunnels, level crossings, viaducts and key stations. Business Need: The company had difficulty integrating and sharing information throughout its organization. Key data was often needed in near real-time by the business but could not be provided quickly enough. Systems were joined on a point-to-point basis and this resulted in a plethora of nonstandard solutions that were expensive to implement and maintain. The company is implementing a new Information Management strategy, to improve information sharing throughout the organization. Both new and legacy systems need to communicate effectively with each other and the company wanted to achieve this by connecting systems in a consistent way, promoting corporate standards for integration and messaging as well as removing costly "point-to-point" integration. By implementing the correct middleware infrastructure, the company knew it would have the opportunity to introduce a service oriented architecture (SOA) that will provide an integrated information services environment and offer users the tools they need to improve their efficiency and effectiveness. The company needed a skilled and experienced partner to help it to achieve this goal. Solution Implementation: IBM Global Business Services was engaged to create generic enterprise middleware for the company that is capable of providing a core messaging backbone throughout the organization. Based on a "hub and spoke" model, it allows data to be sent from a single system to wherever it is needed within the enterprise. A set of core services have been built within WebSphere Business Integration Message Broker (WBIMB) enabling the middleware to be used in any client environment. The main messaging hub is in place and IBM is continuing to add legacy and new systems to it, enabling an increasing number of systems to exchange information. One of the first integration projects involves taking information from trackside assets - such as power supplies or devices that measure technical information like the state of the wheels on a train as it passes a certain point - and driving it through the enterprise infrastructure so that the company's operational centers can see what is happening on the track more easily and quickly. Any problems, alerts or warning messages can be picked up and responded to appropriately, enabling resources to be allocated more efficiently and increasing train reliability by improving management of the network. The messaging hub also handles the thousands of train information messages that the company receives from all over the UK every day. These messages relate to train times and delays from all the train operating companies that run services on the railway infrastructure and are used by the company to report on how each of the train operating companies is performing on a daily basis. IBM is currently working with the company on an external gateway that will enable appropriate information to be fed to external stakeholders such as the train operating companies and freight companies. This will help all stakeholders to work in a more integrated and efficient way, improving the performance of the rail system even further. Benefits: - Timely flow of information throughout the organization, providing governance and control. - IT environment has become far more flexible, allowing applications to be plugged into the messaging hub quickly and easily, reducing the cost involved in creating individual "point-to-point" data interfaces and facilitating reuse - Maintenance and operating costs reduced with the use of standardized code - Improved reliability of rail services and ultimately increased satisfaction for passengers and freight customers. Usage level : Win/Ongoing project/Completed project: Industry : External Complete Travel & Transportation The UK's railway infrastructure utility improves information flow, reduces costs and increases flexibility with a sophisticated middleware solution designed and implemented by IBM Global Business Services Back to top Client information Client name:Location: Network RailUnited Kingdom Industry: Travel & Transportation Focus area: Openness, Service Oriented Architecture Integrated/Aligned AccountGeography/IOT/GMU: Northeast Europe IOT Region/IMT/GMT: United Kingdom and Ireland IMT Client background: In October 2002, Network Rail took responsibility for the infrastructure on which the UK's railways depend. Its mandate involves running, maintaining and developing the country's railway tracks, signalling system, rail bridges, tunnels, level crossings, viaducts and 17 key stations. Britain's railway network carries 2.75 million passengers every day along with 79 percent of the UK's coal and coal products and 34 percent of the UK's metals. URL: http://www.networkrail.co.uk Project start: 04/2004 Project end: 03/2006 Back to top Special handling instructions The client has agreed to be a reference for sales situations. The status of any installation or implementation can change, so you should always contact the Primary Contact or Additional Contact named in the reference prior to discussing it with your client. Any public use, such as in marketing materials, on WWW sites, in press articles, etc., requires specific approval from the client. It is the responsibility of the person or any organization planning to use this reference to make sure that this is done. The IBM representative will, as appropriate, contact the client for review. You should not contact the client directly. Business need Network Rail had difficulty integrating and sharing information throughout its organization. Key data was often needed in near real-time by the business but could not be provided quickly enough. Systems were joined on a point-to-point basis and this resulted in a plethora of nonstandard solutions that were expensive to implement and maintain. Network Rail is implementing organizational changes that include the introduction of a new Information Management strategy. The overall aim of this strategy is to improve information sharing throughout the organization in order to enhance the way that Britain’s railways are run. To achieve this aim, both new and legacy systems need to communicate effectively with each other and Network Rail wanted to achieve this by connecting systems in a consistent way, promoting corporate standards for integration and messaging as well as removing costly "point-to-point" integration. By implementing the correct middleware infrastructure, Network Rail knew it would have the opportunity to introduce a service oriented architecture (SOA) that will provide an integrated information services environment and offer users the tools they need to improve their efficiency and effectiveness. Network Rail needed a skilled and experienced partner to help it to achieve this goal. Back to top Solution implementation IBM Global Business Services was engaged to create generic enterprise middleware for Network Rail that is capable of providing a core messaging backbone throughout the organization. Based on a "hub and spoke" model, it allows data to be sent from a single system to wherever it is needed within the enterprise. A set of core services have been built within WebSphere Business Integration Message Broker (WBIMB) enabling the middleware to be used in any client environment. IBM is involved from end to end on the project, helping Network Rail to develop its integration strategy, providing project management, designing and developing the middleware, and implementing a thorough architectural control and testing regime that has resulted in an almost fault free solution. The IBM team meets weekly with Network Rail to make sure the work stays on track and, due to the close working relationship that has developed, IBM is now seen as a trusted advisor. The main messaging hub is in place and IBM is continuing to add legacy and new systems to it, enabling an increasing number of systems to exchange information. One of the first integration projects involves taking information from trackside assets - such as power supplies or devices that measure technical information like the state of the wheels on a train as it passes a certain point - and driving it through the enterprise infrastructure so that Network Rail's operational centers can see what is happening on the track more easily and quickly. Any problems, alerts or warning messages can be picked up and responded to appropriately, enabling resources to be allocated more efficiently and increasing train reliability by improving management of the network. Two ruggedized hardware devices from IBM business partners Husky and Symbol were trialled as a central point for collecting status messages from multiple trackside devices and then routing them through to the messaging hub using WebSphere MQ Telemetry Transport (MQTT). The messaging hub is also used to handle the thousands of train information messages that Network Rail receives from all over the UK every day. These messages relate to train times and delays from all the train operating companies that run services on the railway infrastructure and are used by Network Rail to report on how each of the train operating companies is performing on a daily basis. Handling information consistently within Network Rail and basing it on open standards offers a wide variety of possibilities for improvement both now and into the future. For example, IBM is currently working with Network Rail on an external gateway that will enable appropriate information to be fed to external stakeholders such as the train operating companies and freight companies. This will help all stakeholders to work in a more integrated and efficient way, improving the performance of the rail system even further. Business Partner information: Husky United States The Second Business Partner information: Symbol United States Back to top Benefit of the solution The enterprise middleware system implemented by IBM supports Network Rail's Information Management strategy by improving the timely flow of information throughout the organization. Information can be transferred into common views for use in portal or other systems and there is now a standard way for integration to occur within the organization, providing governance and control. Any type of information can be processed whether asset information, passenger information or train information, supporting and providing value for the entire Network Rail organization. Network Rail's overall IT environment has become far more flexible since the introduction of enterprise middleware. Applications can be plugged into the messaging hub quickly and easily, reducing the cost involved in creating individual "point-to-point" data interfaces and facilitating reuse in multiple business processes across the enterprise. Maintenance and operating costs are also reduced with the use of standardized code. Applications are less dependent upon each other as they can use their own data formats and let the messaging hub deal with any integration issues. Once an application is ready to be retired it is simply unplugged and removed without affecting remaining systems. The enterprise middleware will continue to provide benefits for Network Rail into the future. It has the potential to provide exactly the right information, in the correct format, wherever it is needed, whether for Network Rail's use or its key stakeholders. Timely provision of the correct information will enable Network Rail to allocate resources more effectively in order to better manage the railway network. This improves the reliability of rail services and ultimately leads to increased satisfaction for train passengers and freight customers. Solutions/Offerings Services: IBM Global Services - Application Services (GBS-AIS and/or AMS): GBS Application Innovation Services: Enterprise Architecture and Technology Back to top Contact information Primary contact information: W (James) Chaloner Associate Partner (Travel & Transport) Global Business Services IBM Div 70 United Kingdom Telephone: 44-1252-558563 Fax: 44-1252-558001 James Chaloner/UK/IBM@IBMGB [email_address] Additional contact information: Mark Tikerpae Managing Consultant IBM Div 70 United Kingdom Telephone: 44-207-021-9015 Mark Tikerpae/UK/IBM@IBMGB mark.tikerpae@uk.ibm.com
  • Phone by bed monitors pacemaker Dial-up -> MQTT to WMB -> analytics engine Avoid regular clinic visits
  • Energy co can send small changes and commands to homes over mobile network, minor temp adjustments -> significant savings
  • HVAC stands for heating, ventilation, and air conditioning. MQTT very popular with home hackers – home automation, weather stations, power monitoring at home

Lightweight Messaging (Apache Retreat Hursley 2010) Lightweight Messaging (Apache Retreat Hursley 2010) Presentation Transcript

    • Lightweight Messaging
      Andy Piper (@andypiper)
    • The Internet of Things
      Trillions of smart devices instrument our world today
      Interconnecting these smart devices creates a Central Nervous System
    • Our World is Filling with Devices
      Grains grown each year
      Pop Quiz
      Crawling the earth
      Manufactured each year
      1,000,000,000,000,000
      10,000,000,000,000,000,000
      100,000,000,000,000,000
      (1 quadrillion)
      (100 quadrillion)
      (10 quintillion)
      1 billion transistors for every one of us
    • +
      +
      =
      An opportunity to think and act in new ways— economically, socially and technically.
        Building a Smarter Planet
      • Connectivity for Smart Devices
        Transport
        MQ Telemetry HTTP Multicast
        Connectivity for Applications
        Transport
        WebSphere MQ REST/HTTP WS*
        Sensors Power meters, weather data SCADA sensors, pressure, volume, RFID readers, Motion detectors…
        Actuators Tag printers, status lights, Load generation, HVAC and lighting, Valves, switches and pumps…
        Embedded Controllers Filtering of duplicate read events, Store-based HVAC & lighting controls, Industrial Network Gateways (SCADA)
        Regional Offices Stores, Outlets
        Head Offices, Data centres Cloud Computing
        Remote Systems and Devices
        Instrumented
        Interconnected
        Intelligent
        Blueprint for Edge Connectivity
      • Technology
      • Introducing MQTT
      • The challenges of industrial control systems (supervisory control and data acquisition, or SCADA) well-suited to a messaging solution
        • Loose coupling, multi-protocol, separation of concerns...
      • IBM developed a protocol for the MQSeries Integrator product designed for the constraints of the SCADA world.
        • MQ Integrator SCADA Device Protocol (Mqisdp)
        • Renamed MQ Telemetry Trnsport (MQTT) with broader telemetry role
      • Support has been available via SCADA nodes in WebSphere Message Broker from version 2.0 through version 6.1
      • Design principles of MQTT
      • Publish/subscribe messaging paradigm as required by the majority of SCADA and sensor applications.
      • Minimise the on-the-wire footprint.
      • Expect and cater for frequent network disruption – built for low bandwidth , high latency , unreliable , high cost networks
      • Expect that client applications may have very limited processing resources available.
      • Provide traditional messaging qualities of service where the environment allows.
      • Publish the protocol for ease of adoption by device vendors and third-party client software.
      • Key facts about MQTT
      • Reduced complexity and footprint
      • Simple, minimal pub/sub messaging semantics
        • Asynchronous (“push”) delivery of messages to applications
        • Simple set of verbs: connect, publish, subscribe, disconnect
      • Minimised on-the-wire format
        • Plain byte array message payload
        • No application message headers
        • Protocol compressed into bit-wise headers and variable length fields
        • Smallest possible packet size is 2 bytes
      • In-built constructs to support loss of contact between client and server
        • “ Last will and testament” to publish a message if the client goes offline
        • Stateful “roll-forward” semantics and “durable” subscriptions
      • Qualities of Service
        QoS 0: At most once delivery (non-persistent) – No retry semantics are defined in the protocol. – The message arrives either once or not at all. QoS 1: At least once delivery (persistent, dups possible) – Client sends message with Message ID in the message header – Server acknowledges with a PUBACK control message – Message resent with a DUP bit set If the PUBACK message is not seen QoS 2: Exactly once delivery (persistent) – Uses additional flows to ensure that message is not duplicated – Server acknowledges with a PUBREC control message – Client releases message with a PUBREL control message – Server acknowledges completion with a PUBCOMP control message
        Three qualities of service for both publishing and subscribing :
      • Example: connect and send an MQTT message
        public void sendAMessage() throws MqttException { MqttProperties mqttProps = new MqttProperties(); mqttProps.setCleanStart( true ); MqttClient client = MqttClientFactory. INSTANCE . createMqttClient("testClient", “ tcp://localhost:1883”, mqttProps); client.registerCallback(this); client.connect(); client.publish(“abc/123”, new MqttPayload((“Hello World!”).getBytes(),0), (byte) 2, false); client.disconnect(); } public void publishArrived( String topicName, MqttPayload payload, byte qos, boolean retained, int msgId) { System.out.println(“Got it!”); }
        Create a connection using the connection factory, this time for a clean starting client
        Register the class as a listener and connect to the broker
        Publish a message to the given topic and disconnect
        On receipt of a publication, simply print out a message on the console to say we received it
      • WMQT Implementation
      • New MQ service – MQXR ('eXtended Reach')
        • Separate purchasable extension
        • Available on Windows and 64-bit Linux
        • eGA 27 August 2010, GA 3 September 2010
      • Fully integrated / interoperable with MQ
        • Publishers and subscribers can exchange messages
      • Telemetry channels enable MQTT connections to Qmgr
      • Supports MQTTv3 protocol (most common in use)
      • Ships with reference Java (for MIDP upwards) and C clients
        • other APIs and implementations available via 3 rd parties e.g. Python, .NET, Delphi etc.
      • MQ Explorer integration
      • Topology example: “simple” clients
        WebSphere MQ 7.0.1
        MQXR
        Sensors
        Mobile
        Applications
        Scaling to tens of thousands of MQTT connections to a single queue manager*
        WebSphere MQ Telemetry
        * performance evaluation covers tests up to 100,000 concurrent connections on Linux, but message rate/restart considerations apply
      • Topology example: “advanced” clients
        WebSphere MQ 7.0.1
        MQXR
        Sensors
        Mobile, smart meters
        Applications
        Microbroker (Lotus Expeditor, WebSphere Sensor Events)
        WebSphere MQ Telemetry
        WebSphere MQ Telemetry Daemon for Devices
        Rich clients requiring buffering, remote management capabilities, or advanced data handling
      • Security options
        Securing mobile / remote clients can be vital! WMQ Telemetry supports two key technologies:
        • SSL – encryption and authentication
        • JAAS – authorization
        WebSphere MQ
        MQXR
        JAAS
        -> userid/pw ← “ OK / no way!”
        Telemetry channel may also use SSL
        Configurable:
      • clientId
      • userid/pw on API
      • Static username on channel
      • Topology example: enterprise gateway
        WebSphere MQ 7.0.1
        MQXR
        WebSphere MQ Telemetry
        WebSphere Message Broker
        DB
        CICS, IMS
        Web Services
        HTTP
        Files
        Multiplex devices at edge of enterprise Integrate “any-to-any” via WMB
    •  
      • Case studies
      • Improved reliability and timeliness of train services Able to allocate railway resources more efficiently Instant access to real-time data across organization
        National railway infrastructure company created solution for real-time rail monitoring
        • Client Pains
        • D ifficulty integrating and sharing information
        • Lack of real-time data
        • Home-grown solution expensive to build and maintain
        Transforming Railway Operations
      • Medical organization created a remote pace-maker monitoring solution to provide better patient care
        • Client Pains
        • Physicians needed better monitoring of cardiac patients
        • Improve efficiency of checkups
        • Meet healthcare data capture standards
        Smarter Healthcare
        Enables higher level of patient care and peace of mind Improves administrative efficiency and maintenance Helps conform to standards and ease integration of data
      • Utility company developing an Intelligent Utility Network offering for optimizing load on electricity grids
        • Business Partner
        • Needs robust middleware technology to connect to remote smart meters
        • Needs to be able to rapidly scale solution nationally
        Improving Energy Usage
        Able to offer daily energy savings of 15-20% Enables utilities to reduce peaks and avoid punitive charges Helps save electricity through better peak load management
      • Pipeline Monitoring and Control
        Flood Defence Warning
        Vehicle Telematics
        • Cars / Military – Diagnostics and Prognostics
        • Pay As You Drive Insurance
        Field Force Automation
        • Sales Force Automation
        • Field Service Engineers
        • Service Delivery
        Asset Management And Monitoring
        Home Automation
        Chemical Detection
        RFID
        HVAC Control
        Kiosks
        Fire Sensors
        POS
        Stock Checks
        Trickle Feed
        Parking Tickets
        Some areas that MQTT has been used…
        Kiosks
        Energy Monitoring
      • Useful links
        Announcement Letter
        http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=877&letternum=ENUSZP10-0255
        MQTT – The Smarter Planet Protocol (blog post)
        http://andypiper.co.uk/2010/08/05/mqtt-the-smarter-planet-protocol/
        MQTT 3.1 specification – IBM developerWorks
        http://www.ibm.com/developerworks/webservices/library/ws-mqtt/index.html
        WebSphere MQ Telemetry Performance Evaluation – SupportPac MP0A
        http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg24027711&loc=en_US&cs=utf-8&lang=en
        MQTT Community
        http://mqtt.org
      • Thank you!
        Contact: Andy Piper [email_address]
        http:// twitter.com/andypiper | http:// andypiper.co.uk