Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Introducing IBM Message Hub: Cloud-scale messaging based on Apache Kafka


Published on

IBM Message Hub is a new Bluemix service for messaging in the cloud. It's ideal for linking together microservices to build a scalable, flexible application in the cloud. It's great for feeding data at speed into other services such as analytics. You can also use it to bridge securely from your enterprise MQ systems into the cloud.

Published in: Technology
  • Be the first to comment

Introducing IBM Message Hub: Cloud-scale messaging based on Apache Kafka

  1. 1. 3499: Introducing IBM Message Hub Cloud-scale messaging based on Apache Kafka Andrew Schofield Chief Architect, Hybrid Cloud Messaging
  2. 2. Organizations must restlessly re-invent themselves • Outpace existing – and emerging – competitors • Deliver an increasingly engaging customer experience • Continuously learn how to improve and anticipate shifts 1 Business success increasingly depends on applying new and innovative business models and implementing IT solutions more rapidly than competitors…
  3. 3. Digital IT Enterprise IT Message Hub (Based on Apache Kakfa) MQ Light API MQ Light API 2 Enterprise Messaging & Integration Hybrid Messaging MQ ExplorerMQ ClientMQ Light API Kafka API REST API IBM MQ Appliance Bluemix Public Bluemix Dedicated (Announced, GA March) Bluemix Local (Coming soon) IBM MQ Cloud On-Prem Cloud On-Prem IBM Message Connect
  4. 4. What is “Digital IT” anyway? 3 Speed and agility to drive innovation and growth •Explore, adopt, adapt •Rapid, iterative prototypes Valuable business assets Exposed as APIs Consumed by app developers To develop innovative apps Delightful experience to customers Feedback loop enabling rapid evolution Or delivered as events
  5. 5. IBM Bluemix 4 Bluemix is an open-standards, cloud-based platform for building, running, and managing applications Build your apps, your way Use the most prominent compute technologies to power your app: Cloud Foundry, Docker, OpenStack. Extend apps with services A catalog of IBM, third party, and open source services allow the developer to stitch an application together quickly. Scale more than just instances Development, monitoring, deployment, and logging tools allow the developer to run and manage the entire application. Layered security IBM secures the platform and infrastructure and provides you with the tools to secure your apps. Deploy and manage hybrid apps seamlessly Get a seamless dev and management experience across a number of hybridoptions. Flexible pricing Try compute options and services for free and, when you’re ready, pay only for what you use. Pay as you go and subscription models offer choice and flexibility.
  6. 6. Message Hub is….. • A scalable, distributed, high-throughput message bus based on Apache Kafka • Wide compatibilityvia 3 APIs : REST, Kafka and MQ Light API (coming soon) • Tightly integrated with the Bluemix platform MQ (plus other on-prem data sources) MQ Light API AMQP RESTful API HTTP Apache Kafka TCP Streams … Bluemix Public Bluemix Dedicated (Announced, GA March) Bluemix Local (Coming soon) Available for … 5 IBM Message Connect
  7. 7. What is Apache Kafka? • Apache Kafka is publish-subscribe messaging re-thought as a distributed commit log. The following attributes are claimed of Kafka: • Fast - a single broker on commodity hardware can handle hundreds of megabytes of reads and writes per second from thousands of clients. • Scalable - a single cluster can serve as the central data backbone for a large organization. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine. • Durable - messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact. • LinkedIn developed Kafka to solve its internal data integration problems • Felt available commercial solutions didn’t address their scalability requirements • Open sourced early 2011 • Apache incubator graduation fall 2012 6
  8. 8. Why Apache Kafka? 7 Fast: Responds to your needs Scalable: Scales according to demands Durable: Prevents data loss through replication Distributed: Fault-tolerant Used by companies including:  LinkedIn  Yahoo  Twitter  Netflix  Spotify  Pinterest  Paypal  Tumblr
  9. 9. Anatomy of a topic  Kafka topics are made up of multiple partitions – Partitions are distributed across Kafka nodes  Partitions are ordered streams of messages – Replicated copies distributed across Kafka nodes T0P0 T0P0 T0P1 T0P1 T1P1 T1P1 T1P0 T1P0 Kafka Node 1 Kafka Node 2 Kafka Node 3 Kafka Node 4 M0 M1 M2 M3 8
  10. 10. What does Message Hub enable? 9 Hub for asynchronously connecting services inside Bluemix or beyond Applications connected to events happening in other Bluemix services, or from beyond the cloud Insights from the data you already have Data needs to be streamed from anywhere to one or many analytics engines 1 2 3 Microservices allow applications to evolve rapidly Open protocols support polyglot runtimes, application controlled behavior and reactive scale MQ plus other on-prem data sources 4
  11. 11. What does Message Hub enable? 10 Hub for asynchronously connecting services inside Bluemix or beyond Applications connected to events happening in other Bluemix services, or from beyond the cloud MQ plus other on-prem data sources Speeds production: Messy wiring between services taken care of for you Offers a range of solutions: Choose the services that you need, and use Message Hub for the messaging Connects with enterprise on- premise: All the benefits of working in a hybrid environment 1
  12. 12. Getting messages from one service to another 11 Think of Message Hub as the conduit between your cloud services…. …taking data, in the form of messages, from one service to another
  13. 13. What does Message Hub enable? 12 2 Microservices allow applications to evolve rapidly Open protocols support polyglot runtimes, application controlled behavior and reactive scale Innovate and adapt quickly: Deploy and scale microservices independently Flexibility: Work in a range of languages that suit you Reliability: Remove the interdependency between microservices
  14. 14. One day in a world without microservices… 13 (Except he has to re-build the entire app, because of the way that it has been built, so it’s not so quick…) Chris, your developer, decides to make a quick change to the app His change introduces an error, and the whole thing crashes A supermarket runs an app that allows customers to take advantage of special offers when they are in specific areas of the store
  15. 15. And with microservices? 14 Working in a microservices framework means that applications are broken into smaller parts And because they are independent So that changes to individual parts can be quickly made One change does not affect the other parts
  16. 16. The same scenario with microservices 15 IT’S QUICK because he doesn’t have to rebuild the entire app OK, his code breaks, but the rest of app is unaffected QUICK TO FIX because he doesn’t have to rebuild the entire app Chris, your developer, decides to make a quick change to the app A supermarket runs an app that allows customers to take advantage of special offers when they are in specific areas of the store
  17. 17. MQ Light API – Messaging for app developers • Messaging that application developers will love to use, helping them make responsive applications that scale easily • 3 ways to get it: – MQ Light software download for developers – IBM MQ and later – Message Hub in Bluemix • Open APIs crafted to feel natural in a growing range of popular languages • Tooling that makes modular app development easy 16
  18. 18. What does Message Hub enable? 17 Insights from the data you already have Data needs to be streamed from anywhere to one or many analytics engines 3 Adapt to change: React to changing trends as they happen Consume data in your way: Run real time and batch analytics on the same data Protection of valuable data: Acts as a buffer between your data and the analytics engine
  19. 19. Imagine the possibilities… 18 Fraud and risk detection Real-time traffic flow optimization Accurate and timely threat detection Understand and act on customer sentiment Predict and act on intent to purchase
  20. 20. What does Message Hub enable? 19 Hub for asynchronously connecting services inside Bluemix or beyond Applications connected to events happening in other Bluemix services, or from beyond the cloud Insights from the data you already have Data needs to be streamed from anywhere to one or many analytics engines 1 2 3 Microservices allow applications to evolve rapidly Open protocols support polyglot runtimes, application controlled behavior and reactive scale MQ plus other on-prem data sources 4
  21. 21. Message Hub delivery strategy 20 Message HubMessage Hub Incubator Features & capabilities Promoted when ready • MQ Light API Future: • … • [continuously delivered new features]
  22. 22. Introducing Message Hub Incubator • A separate, experimental service whose content changes over time – Find it in the Bluemix Labs catalog: • Early access to forthcoming Message Hub features – Under active development so we need freedom to make significant changes – We reserve the right to discard data, so we don’t need to handle migration as we develop • The first delivery of MQ Light API support – Full MQ Light API support for Java and Node.js (but not Ruby) clients – Limited infrastructure behind the scenes • Come and try it out in Lab 3508: IBM MQ Light – Cloud, Developer and Enterprise – MGM Grand, Room 319 – Wednesday – 1:15PM to 2:15PM 21
  23. 23. AMQP Adapter Message Hub architecture Apache Kafka Cluster HTTP Load Balancer MQ Light API AMQP RESTful API HTTP Apache Kafka TCP Topicmapper ZooKeeper AMQP Adapter (stateless) AMQP Adapter AMQP Adapter AMQP Adapter Kafka REST Interface AMQP State Store Service Lookup 22
  24. 24. Performance • Throughput of the cluster is about 300,000 msg/s – 100-byte messages – Secure connection from public network – Disk encryption of Kafka log – Secure connections for all user data, even within DMZ • Latency ranges from 20ms to 100ms, average 50ms 23
  25. 25. Continuous delivery and microservices • Message Hub is developed using a continuous delivery process and a microservice architecture – It’s built out of small pieces that we can enhance rapidly • Multiple deploymentenvironments with promotion from dev/test towards production – We can deploy changes into production as often as required, even several times as day if we had to • When there’s a critical patch for Kafka, we… – Build the patch as deploy to dev/test – Test it thoroughly – Evaluate suitability for promotion to staging… – And repeat until the patch reaches production 24
  26. 26. Message Hub continuous delivery pipeline Build Unit Test Docker Image Promotion process Dev-test Staging Bluemix (internal) Bluemix (public) Marathon / Mesos / Docker Jenkins CI Pipeline Integration testing “Chaos Monkey” Load testing 25
  27. 27. Involvement in the community • Contributed several patches, particularly as release of 0.9 approached – Primarily related to SSL/TLS and SASL • Proposed a Kafka Improvement Proposal to augment SASL capabilities – KIP-43 • Started discussion on exactly-once delivery to improve inter-protocol bridging • Have an accepted session for the Kafka Summit in April 2016 26
  28. 28. Why Apache Kafka as a service? 27 Option 1 • Deploy your own Kafka cluster • Deploy Zookeeper cluster to support it • Put systems in place to operate the service 24x7 • Keep up-to-date with the open source community • Handleversion-to-version migration and updates Option 2 • Use Message Hub See our 3-minute video showing how to get started!
  29. 29. IBM Message Hub is live! 28 A scalable, distributed, high throughput message bus for your cloud-native applications Connects services inside Bluemix with your systems beyond Connects microservices, using open-standard protocols Streams data to analytics for real-time insights Find us in the Bluemix catalog and sign up at Built on Apache
  30. 30. Where do I get more information? IBM Messaging developerWorks IBM Messaging Youtube LinkedIn Twitter @IBMMessaging IBM MQ Facebook
  31. 31. Monday 10:30-11:30 3592 New MQ features 3452 Managing applications 12:00-13:00 2835 MQ on z/OS and Distributed 15:00-16:00 3470 Latest MQ z/OSfeatures 2833 Where is my message? 3544 MQ Light in an MQ infrastructure 16:30-17:30 3573 Hybrid cloud messaging 2941 MQ Advanced Tuesday 08:30-09:30 3540 The MQ Light API 12:00-13:00 3456 The IBM MQ Appliance 13:15-14:15 3499 Introducing Message Hub 3458 MQ Appliance administration 14:30-15:30 6432 MQ updates and futures (InnerCircle) 2849 Messaging feedback roundtable 16:00-17:00 3544 MQ Light in an MQ infrastructure 3513 MQ hands on lab Wednesday 08:30-09:30 3602 Effectively Managing Hybrid Messaging 12:00-13:00 3613 Designing MQ self service 6408 Hybrid messaging roadmap (InnerCircle) 13:15-14:00 3416 HA and DR with MQ 3433 Why secure your messaging? 15:45-16:30 3429 Securing MQ 2847 Meet the messaging experts 16:00-17:00 3508 MQ Light hands on lab 16:45-17:30 2275 Migrating to the IBM MQ Appliance Thursday 08:30-09:15 3420 MQ Clustering 2931 Business agility with self service MQ 09:30-10:15 3479 MQ z/OS clusters and shared queue 3450 Optimising MQ applications 2849 Messaging feedback roundtable 10:30-11:15 3465 MQ Appliance high availability 3481 MQ z/OS messaging connectivity 11:30-12:15 3474 Active-active messaging 3537 Monitoring and managing MQ 3425 MQ publish/subscribe Find us at the EXPO: Hybrid Integration peds 65-68 Check out the Hybrid Messaging sub topic under the Hybrid Integration topic for further customer and business partner sessions Hybrid Messaging from the IBM experts at InterConnect 2016 Sunday 14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)
  32. 32. • Hybrid Integration Strategy • Cloud Integration • Accelerating Digital Business • Integration Bus • IBM MQ • API Management • BPM / ODM • DataPower • CICS • WAS Spend time with IBM experts, at the home of many of IBM's software products. This summit is by invitation only - a limited seating engagement for executives and architects who would like to learn how to harness IBM connectivity and application integration solutions to deliver access to data, applications and information regardless of platform, device or data formats - across both on-premises and cloud environments. Learn more about how we are transforming our technologies using Hybrid Cloud to enable you to harness your existing assets to achieve greater capacity, efficiency and integration across platforms, whilst retaining the security, capability and resiliency you would expect from IBM. • Discover and influence IBM's strategy for key messaging and integration technologies, including, IBM MQ, IBM Integration Bus and IBM API Management • Engage in technical sessions and one-on-one interactions with top IBM Hursley Lab architects and senior executives to refine your 2016 strategic plans • Expand your network with industry-leading peers from other companies • Plus learn about other IBM technology, such as IBM intelligent business process management solutions (BPM & ODM), DataPower gateways, CICS and WebSphere Application Server on-premise and cloud This event is conducted under a Non-Disclosure agreement, so we will be able to share product directions with you. Hursley: a visit to talk about The IBM Hursley Lab is the largest softw are development facility in Europe; situated in a beautiful 100 acre park w ith a historic setting. Attendees stay in the local city of Winchester w hich is a vibrant heritage destination w ith many attractions and classical architecture including a magnificent cathedral. Enjoy the aw ard-winning pubs and restaurants and a tempting array of independent shops. Talk to your IBM rep to find out more Be part of the conversation Keep up to date with the latest information, join the conversations and help to shape the event to meet your interests. Use #IBMhursum in your Tweets to keep in touch. #IBMhursum European & North American Hursley Summit 2016 Integration across applications, data and processes for mobile and cloud May 10 – 12 & May 16 - 19 | IBM Hursley Lab, UK #IBMhursum
  33. 33. Notices and Disclaimers 32 Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government UsersRestricted Rights- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISINGFROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. Any statementsregarding IBM'sfuture direction, intent or product plansare subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intendedto, nor shall constitutelegal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as tothe identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
  34. 34. Notices and Disclaimers Con’t. 33 Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperatewith IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo,, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at:
  35. 35. Thank You Your Feedback is Important! Access the InterConnect 2016 Conference Attendee Portal to complete your session surveys from your smartphone, laptop or conference kiosk.