Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Microservices in a Streaming World


Published on

Presentation on Apache Kafka and Microservices. Presented in Melbourne and Perth Australia meetup events.

Published in: Data & Analytics
  • Be the first to comment

Microservices in a Streaming World

  1. 1. Microservices in a Streaming World Modern distributed applications and infrastructure
  2. 2. About me Hans Jespersen U of Waterloo – Punch Cards, COBOL, Assembler, RJE, Screen Scraping AT&T – Unix, Client/Server, Tuxedo & Transactions, Pre-Web Internet Sun – Solaris, ONC RPC, CORBA TIBCO – Yahoo!Quotes, Pub/Sub, Multicast, JMS, ESB, SOA, WS-* Solace – Messaging HW, MQTT ,AMQP, REST Confluent – Kafka, Event Stream Processing Personally - active in IoT, open source, MQTT
  3. 3. Not going to talk about... Meaningless, vendor marketing, definition of Micro Services
  4. 4. Let’s parse this out together…
  5. 5. Love Martin Fowler’s Work
  6. 6. – or Google “Kafka Summit Martin Fowler”
  7. 7. I am going to talk about... Patterns in Modern Distributed Architecture and Application Infrastructure
  8. 8. You’ve probably heard of Microservices.
  9. 9. But what are they really?
  10. 10. Are they SOA all over again? SOA Microservices
  11. 11. Or are they an evolution? SOA
  12. 12. We used to build monolithic applications.
  13. 13. These applications came in a variety of different forms.
  14. 14. Companies built lots of them.
  15. 15. But the teams in these companies were grouped into business areas. Silos.
  16. 16. So the applications would also form into silos along their business lines.
  17. 17. This led to a lot of duplication across different applications, in different silos.
  18. 18. The duplication made the IT function expensive to run.
  19. 19. Then someone had an idea.
  20. 20. What if they pulled out the common pieces into reusable “services”? Service
  21. 21. Less code in each silo seemed like a good idea. Service
  22. 22. Then, they realized they could build whole companies based on these reusable services.
  23. 23. This seemed like a very good idea!
  24. 24. The trick was to rearrange the people to match the services.
  25. 25. So instead of sitting in silos…
  26. 26. The teams would match the architecture that they wanted to end up with.
  27. 27. SoA This pattern was termed a ‘Service Oriented Architecture.’
  28. 28. But - there was a problem.
  29. 29. Refactoring companies turned out to be slow and difficult. Services
  30. 30. So people took the same pattern designed for big companies…
  31. 31. …and applied it to break up monolithic applications...
  32. 32. …knowing that with this architecture, the companies would be able to grow…
  33. 33. …and grow…
  34. 34. …and grow.
  35. 35. This was termed Microservices: Change through small, well-defined services that are easy to reuse.
  36. 36. So silos were broken down…
  37. 37. …and monoliths were broken up.
  38. 38. Evolving incrementally towards contemporary service estates.
  39. 39. But things were not entirely rosy yet.
  40. 40. While silos seemed expensive and wasteful, they had one big advantage...
  41. 41. Each application was free to handle change independently of the applications around it.
  42. 42. Every application was, in some sense, an island.
  43. 43. Microservices are built with HTTP, REST, or some other protocol made from requests and replies. Request Reply
  44. 44. This works well when ecosystems are small… Buying Widgets
  45. 45. But gets harder as they grow more complex and more interconnected.
  46. 46. Services are tightly coupled. No islands here!
  47. 47. So if one service fails…
  48. 48. …or even just runs slowly… Buy Widgets
  49. 49. The fall-out could be much larger.
  50. 50. Others end up feeling that pain. Buy Widgets
  51. 51. Yet, at company scale, the majority of processes run in the background anyway. They are asynchronous to one another. Online Billing Inventory Fulfillment Fraud Offline
  52. 52. So it makes sense to DECOUPLE services from one another. Billing Inventory Fulfillment Fraud Offline Online Decouple
  53. 53. Apache Kafka™ helps with this as it provides a data backbone for your services. Billing Inventory Fulfillment Finance Fraud HTTP etc Offline Online
  54. 54. This connects services together.
  55. 55. It also connects their DATA together. All Your Data
  56. 56. The three tenets of messaging… embedded into a layer of permanence. Decoupling Notification Data Transfer Permanence
  57. 57. Messaging that remembers.
  58. 58. So every service gets the data it needs. All Your Data
  59. 59. Unlike typical service frameworks, it also DECOUPLES services from one another, so they can evolve independently.
  60. 60. This makes it easier to move away from legacy architectures, to evolve away from the past… All Your Data LEGACY
  61. 61. …towards a better-factored future, whatever that may look like. All Your Data LEGACYNew Services Analytics
  62. 62. So wherever your business ends up… Cloud Another Device Another Geography
  63. 63. Apache Kafka provides the Service Backbone built to handle today’s data-centric world... Big Data Ready
  64. 64. In a way that can adapt to your company's future, wherever you might take it.
  65. 65. Services built on the POWER and IMMEDIACY of an Event Streaming Platform. Event Streaming Platform
  66. 66. So what is this Kafka thing you speak of so highly?
  67. 67. How do you see Kafka?
  68. 68. Traditional Messaging Functionality Decoupling of Producers and Consumers
  69. 69. Message Exchange Patterns (MEP) Topic = Publish/Subscribe - N of N delivery Queues = Point-to-Point - 1 of N delivery
  70. 70. Message Exchange Patterns (MEP) Request/Reply Content-Based Router
  71. 71. Message Delivery Semantics At-Most-Once “Best effort” “Reliable” QoS 0 At-Least-Once “Guaranteed” “Certified” QoS 1 Exactly Once “Once-and-Only-Once” “Transactional” QoS 2
  72. 72. Wide Spectrum of Messaging Offerings Ultra- low Latency (often no broker in the middle) High Volume (Persistent or Non-Persistent) Highly Available (Clustered and Fault Tolerant) Embedded Messaging (inside apps) Cross Datacenter / Organizational / B2B Enterprise Message Bus Messaging-as-a-Service Web / IoT Messaging Instant Messaging
  73. 73. “publish-subscribe messaging rethought as a distributed commit log”
  74. 74. Kafka is a Mashup Mashup of some well proven concepts into something even greater and easier to use: EAI + ETL Messaging Middleware + Big Data Batch + Real-time Data Movement + Data Processing Log Data Streams + Structured Database Tables
  75. 75. + Distributed clustered storage Kafka is a blend of messaging, stream processing, ETL and modern database designs built around a distributed log + Streaming platform Pub/Sub Messaging ETL Connectors Spark Flink Beam IBM MQ TIBCO RabbitMQ Mulesoft Talend Informatica Kafka is much more than messaging + Exactly Once + Designed for the Cloud + Inter DC replication + Schema evolution Stream Processing Confluent Confidential
  76. 76. What’s different about Kafka? Topics are also Queues Consumers can share one copy of the data • Independent consumers share the same log • Inter-dependent consumers share the same log • No need for Topic/Queue bridging or multiple copies of the data Message processing is greatly simplified - There is no “head’ of the queue - Writes are sequential, distributed, and parallel
  77. 77. What’s different about Kafka? Messages are not deleted when consumed Messages in the commit logs are persistent and immutable Slow Consumers are (very) decoupled from Fast Producers Batch and real-time are unified Message Replay, Replication, and Auditing are built-in (for free) All production messaging deployment need some form of these Message Retention is not a waste of disk space You need to size for offline/disconnected consumers anyway Distributed State can always be recreated from a common commit log Makes distributed HA apps much easier to build
  78. 78. What’s different about Kafka? Topic Partitions and Keyed Messages - Topics/Queues are not the smallest unit of scalability - Topics partitions are distributed across brokers for parallel in-order consumption - This is very different from a cluster of traditional message brokers - [graphic of topic partitions with parallel Producers, Brokers, and Consumers] - Sometime you can just use more keys instead of more topics - Eg. don’t create a new topic for every user, or IoT device, create unique keys - This is proven to scale to many millions of connected users, cars and IoT devices - [graphic to show Keyed messages get distributed across topic partitions]
  79. 79. From an event stream / transaction log we can derive all of the following database centric features: - Replication - Secondary Indexing - Caching - Materialized Views What’s different about Kafka? Duality of streams and databases Duality of a message streams and database tables is a key design point =
  80. 80. (Good) Microservices avoid shared mutable state Shared, mutable state
  81. 81. Old World: REST Based Microservices Interconnect GUI UI Service Order s Returns Pay Fulfilment Stock Confluent Confidential Each Microservice has to maintain their own stateful nature by using their own databases 1. Difficult to Enforce Same REST API standards across many languages and micro-services. 2. Rest APIs Inherently Slow: Limited to Thousands calls/sec. 3. Inter Service Dependencies are Messy. 4. Each Service Needs to Maintain State. 5. Difficult to enforce consistent security standards. 6. Logging is distributed between services. 7. Version compatibility between services is difficult.
  82. 82. Streaming Microservices with Kafka GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Confluent Confidential Database Sources Now Centralized on the Kafka Bus for all microservices 1. Service inter-communication standard enforce by Kafka Schema Registry. 2. Millions of messages per second on cheap hardware. 3. No Inter-Service Dependency: just depend on Kafka. 4. Each service can be stateless: Kafka maintains state. 5. Security can be enforced by ACLs from Kafka. 6. Logs can be aggregated into Kafka. 7. Version compatibility can be enforced by Scheme Registry. 8. Kafka is inherently HA, horizontal scalable: still no central point of failure.
  83. 83. What’s different about Kafka? Ecosystem and Adoption The Kafka ecosystem is flourishing and developer adoption continues to grow • Confluent Platform additions (REST Proxy, Schema Registry, KSQL etc.) • Third Party Connectors ( Confluent Hub) • Open Source contributions from individuals, corporations, vendors, consulting organizations • Inside and outside of Big Data/Stream Processing
  84. 84. Adoption of Event Streaming 60%Fortune 100 Companies Using Apache Kafka
  85. 85. Event Streaming at the Heart of the Enterprise
  86. 86. Embrace a Distributed World