Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IBM Message Hub service in Bluemix - Apache Kafka in a public cloud


Published on

This talk was presented at the Kafka Meetup London meeting on 20 January 2016. You can find more information about Message Hub here:

Published in: Technology

IBM Message Hub service in Bluemix - Apache Kafka in a public cloud

  1. 1. 1 © 2016 IBM Corporation IBM Message Hub service in Bluemix Apache Kafka in a public cloud Andrew Schofield IBM Hursley Park
  2. 2. 2 © 2016 IBM Corporation A scalable, distributed, high throughput message bus for your cloud-native applications Connects services inside Bluemix with your systems beyond Connects microservices, using open standard protocols Streams data to analytics services for real-time insights Built on Apache What is Message Hub?
  3. 3. 3 © 2016 IBM Corporation Bluemix is an open-standards, cloud-based platform for building, running, and managing applications Build your apps, your way Use the most prominent compute technologies to power your app: Cloud Foundry, Docker, OpenStack. Extend apps with services A catalog of IBM, third party, and open source services allow the developer to stitch an application together quickly. Scale more than just instances Development, monitoring, deployment, and logging tools allow the developer to run and manage the entire application. Layered security IBM secures the platform and infrastructure and provides you with the tools to secure your apps. Deploy and manage hybrid apps seamlessly Get a seamless dev and management experience across a number of hybrid implementations options. Flexible pricing Try compute options and services for free and, when you’re ready, pay only for what you use. Pay as you go and subscription models offer choice and flexibility. What is IBM Bluemix?
  4. 4. 4 © 2016 IBM Corporation Start with a boilerplate
  5. 5. 5 © 2016 IBM Corporation Or a runtime for building your app from scratch
  6. 6. 6 © 2016 IBM Corporation Choose from a wide range of services to use in your app
  7. 7. 7 © 2016 IBM Corporation Including Message Hub
  8. 8. 8 © 2016 IBM Corporation Message Hub is Kafka in Bluemix
  9. 9. 9 © 2016 IBM Corporation Bind to the service to get your credentials
  10. 10. 10 © 2016 IBM Corporation Why are we using Kafka? Kafka has many valuable characteristics • Fast • High throughput to handle whatever you throw at it • Scalable • Easily add more brokers to increase throughput • Durable • Consumers can be stopped for maintenance and catch up without impacting the service • Distributed • Replication and failure characteristics make managing the service easy
  11. 11. 11 © 2016 IBM Corporation Kafka in Bluemix • Some services in Bluemix integrate well with Kafka’s streaming data-flow model • For example, analytics such as Spark • Exposing, not hiding, the Kafka interface gives an event- driven programming interface for application code • Some internal services also use Kafka as plumbing • Some compound services make use of Kafka for internal communication • Think SaaS built using a PaaS • Message Hub is intended to support all of these scenarios
  12. 12. 12 © 2016 IBM Corporation Why might you use Kafka on the cloud? • The benefits of Kafka are applicable to services in the cloud • For example, feeding data into real-time analytics • Serious use of any software takes investment • Either grow your own skills • Or pay someone else to support you • Pay-as-you-go pricing makes it a cost-effective platform for experimentation • For example, try some analytics on telemetry data from your mobile app
  13. 13. 13 © 2016 IBM Corporation Kafka use cases which fit with Message Hub nicely • Messaging • Replacement for a traditional message broker • Website activity tracking • High-volume feeds of site activity such as views, searches • Metrics • Aggregating operational monitoring data • Log aggregation • Copying physical log files from servers to central point • Stream processing • Data flowing on the topics between processing stages
  14. 14. 14 © 2016 IBM Corporation Kafka use cases which don’t fit so well • Event sourcing • State changes are logged as time-ordered sequence, rather than storing the current state • Commit log • Assists with resynchronisation of failed nodes • Change data capture • Take a feed of database changes and store them as a compact change history in Kafka, and feed data into multiple consumers from Kafka The difference is that log data needs to be retained and perhaps compacted
  15. 15. 15 © 2016 IBM Corporation In the public cloud, you’re sharing resources • Two instances of Message Hub available now • Dallas and London • Each is a 5-node cluster, shared among the tenants • Cluster-wide configuration is common • Replication factor: 3 • Clean-up policy: delete • Log retention: 24 hours • Maximum message size: 1 MB • If you need more performance or isolation • Bluemix Dedicated • Dedicated hardware in IBM datacenters, IBM ops team • Bluemix Local • Customer hardware, IBM ops team
  16. 16. 16 © 2016 IBM Corporation Pricing
  17. 17. 17 © 2016 IBM Corporation Pricing example • A workload of 10 msg/second published to a topic with 1 partition • 26,784,000 messages in a month • About $25
  18. 18. 18 © 2016 IBM Corporation Modifications to Kafka for use in the cloud • Client connections must provide credentials • Username or API key tell us which tenant is connecting • Client connections are always encrypted • Connections are over the public internet • Real topic names have a tenant-specific prefix • Kafka topics are not hierarchical • Number of partitions is capped per tenant • Per-tenant metering data used to generate bills
  19. 19. 19 © 2016 IBM Corporation Topic management • Message Hub topics do not auto-create • Auto-create can only give a default topic config anyway • Kafka users can use • You can’t access zookeeper or a shell • Two options for Message Hub: • Admin REST API for create/delete/list topics • Bluemix UI
  20. 20. 20 © 2016 IBM Corporation Topic management in the UI
  21. 21. 21 © 2016 IBM Corporation Topic configuration Property Can set in MH? Default Description cleanup.policy No “delete” “compact” would enable log compaction use cases, but would consume more storage No 24 hours Applies to log compaction retention.bytes No 1 GB Maximum log size Not yet 24 hours How long a log segment is retained partitions Yes 1 replicas No 3 Defaults are taken from the cluster config, and that’s common for all tenants
  22. 22. 22 © 2016 IBM Corporation Message Hub deployment architecture Red zone (internet) Yellow zone (DMZ) Green zone (internal) FIREWALL FIREWALL App using Kafka client HA proxy HA proxy Kafka ZK Bare-metal Bluemix services: Logging, billing, authentication Metering Alerting Cloudant DB TLS TLS Bare-metal
  23. 23. 23 © 2016 IBM Corporation Continuous delivery and microservices • Message Hub is developed using a continuous delivery process and a microservice architecture • It’s built out of small pieces that we can enhance rapidly • Multiple deployment environments with promotion from dev/test towards production • We can deploy changes into production as often as required, even several times as day if we had to • When there’s a critical patch for Kafka, we… • Build the patch as deploy to dev/test • Test it thoroughly • Evaluate suitability for promotion to staging… • And repeat until the patch reaches production
  24. 24. 24 © 2016 IBM Corporation Example – multiple partitions • The first delivery of Message Hub actually only supports one partition per topic • Deliver quickly and iterate often • We are about to add multiple partitions • Poised in our staging environment awaiting promotion • Includes: • Admin API • UI • Metering
  25. 25. 25 © 2016 IBM Corporation Performance • Throughput of the cluster is about 300,000 msg/s • 100-byte messages • Secure connection from public network • Disk encryption of Kafka log • Secure connections for all user data, even within DMZ • Latency ranges from 20ms to 100ms, average 50ms
  26. 26. 26 © 2016 IBM Corporation Involvement in the community • Contributed several patches, particularly as release of 0.9 approached • Primarily related to SSL/TLS and SASL • Planning to propose Kafka Improvement Proposals shortly to augment SASL capabilities • Have submitted sessions for the Kafka Summit in April 2016
  27. 27. 27 © 2016 IBM Corporation A scalable, distributed, high throughput message bus for your cloud-native applications Connects services inside Bluemix with your systems beyond Connects microservices, using open standard protocols Streams data to analytics services for real-time insights Built on Apache What is Message Hub?
  28. 28. 28 © 2016 IBM Corporation Questions? Twitter: @IBMmessaging DeveloperWorks: Bluemix blogs: LinkedIn: Message Hub service: