Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Devoxx Morocco 2016 - Microservices with Kafka

685 views

Published on

Scalability, fault tolerance, distributed log…these are terms which we hear more and more these days. Make them happen is quite a challenge sometimes especially if our business need to be data intensive, agile and fast to market.

One way to answer to this challenge is microservices. These are small services that communicate to each other to deliver business value. The key word here is _communication_. Without communication all the power of microservices falls apart. And communication is not a trivial fact when involves systems with multiple data systems that are talking to one another over many channels. Each of the channel requiring their own protocol and communication methods. This is where communication can become a bottleneck if not handled properly.

One answer to this problem is Kafka, a distributed messaging system providing fast, highly scalable and redundant message exchange using a publish-subscribe model. And when we talk about fast we talk about one of the fastest messaging systems out there.

This presentation will show you an alternative way of doing microservices with event-driven architecture through Kafka.

Presenters:
Laszlo-Robert Albert (albertlaszlorobert [at] gmail [dot] com)
Dan Balescu (dfbalescu [at] gmail [dot] com)

Published in: Software
  • Be the first to comment

Devoxx Morocco 2016 - Microservices with Kafka

  1. 1. Microservices with Kafka László-Róbert ALBERT Dan BALESCU Casablanca • 02 November 2016
  2. 2. Róbert Dan
  3. 3. ING Software Development Center Bucharest 3
  4. 4. ING Software Development Center Bucharest 4 IT 5,7% of GDP with 100k ppl Agriculture 4,4% of GDP with 1.8 million ppl
  5. 5. 5 FAULT TOLERANCE PERFORMANCE RELIABILITY SECURITY AVAILABILITY HIGH SCALABILITY REPLICATION CHANGE SELF HEALING
  6. 6. 6
  7. 7. 7
  8. 8. 8 Account Manager
  9. 9. 9 Account Manager Core Banking
  10. 10. 10 Account Manager Core Banking Transfer
  11. 11. 11 Account Manager Core Banking Transfer
  12. 12. 12 Source: http://www.slideshare.net/dberkholz/ how-microservices-are-redefining- modern-application-architecture
  13. 13. 13 Source: http://www.slideshare.net/dberkholz/ how-microservices-are-redefining- modern-application-architecture
  14. 14. 14
  15. 15. 500 million messages / day 15
  16. 16. 500 million messages / day 700 billion messages / day 16
  17. 17. 500 million messages / day 700 billion messages / day 1.4 trillion messages / day 17
  18. 18. 500 million messages / day 700 billion messages / day 1.4 trillion messages / day 18
  19. 19. Messaging Brokers Features 19 Kafka ActiveMQ RabbitMQ
  20. 20. Messaging Brokers Features 20 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ
  21. 21. Messaging Brokers Features 21 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ Java Messaging System
  22. 22. Messaging Brokers Features 22 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ Java Messaging System Java Authentication and Authorization Service Simple Authentication and Security Layer
  23. 23. Messaging Brokers Features 23 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ Java Messaging System Java Authentication and Authorization Service Simple Authentication and Security Layer Transactions
  24. 24. Messaging Brokers Features 24 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ Java Messaging System Java Authentication and Authorization Service Simple Authentication and Security Layer Transactions Filtering
  25. 25. Messaging Brokers Features 25 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ Java Messaging System Java Authentication and Authorization Service Simple Authentication and Security Layer Transactions Filtering Persistence
  26. 26. 26 Performance Features
  27. 27. 27 Messaging Brokers Performance Message producing Message consumption LinkedIn – Kafka: a Distributed Messaging System for Log Processing
  28. 28. 28 Performance Features
  29. 29. 29 Usecases of Kafka Consumer Microservice A Microservice B Transport API Producer
  30. 30. 30 Usecases of Kafka Consumer Microservice A Microservice B Transport API Producer Consumer Producer
  31. 31. 31 Usecases of Kafka Microservice A Microservice B Transport API Producer Consumer Microservice C Consumer
  32. 32. 32 System Availability & Performance
  33. 33. 33 System Availability & Performance
  34. 34. 34 Scalability
  35. 35. 35 Scalability
  36. 36. 36 Usecases of Kafka Microservice A ELK stackKafka ConsumerKafka Appender K A F K A
  37. 37. © ING 2016 37 Producer // build a 'default' factory based on your-implicit-kafka-producer.properties // - the properties file contains the list of brokers to connect to ProducerFactory factory = ProducerFactory.builder().build(); // create a message producer with a Payload that sends the messages to topic ‘test’ Producer<Message<Payload>> producer = factory.createMessageProducer("test", Payload.class); // send a Message { header: Map<String, String>; payload: Payload } producer.send( Message.<Payload> builder() .header("applicationId", "sample-producer") .header("timestamp", now()) .payload(new Payload("/some/file/path")) .build() );
  38. 38. 38 Producer Native Kafka Producer API your-implicit-producer.properties your-security.properties
  39. 39. 39 Consumer // build a 'default' factory – your-implicit-kafka-consumer.properties // - the properties contains the list of brokers from which it will receive messages ConsumerFactory factory = ConsumerFactory.builder().build(); // obtain a consumer instance that can process Message<Payload> messages Consumer<Message<Payload>> consumer = factory.createMessageConsumer(Payload.class); // prepare the message (Message { header: Map<String, String>; payload: Payload } ) handler ReceiverHandler<Message<Payload>> messageReceiver = new ReceiverHandler<Message<Payload>>() { @Override public void onReceive(Message<Payload> message) { System.out.printf("message received: %s%n", message); } }; // register the handler to receive messages from the topic named “test” consumer.registerHandler("test", messageReceiver); © ING 2016
  40. 40. 40 Consumer Native Kafka Consumer API your-implicit-consumer.properties your-security.properties
  41. 41. 41 KafkaAppender ELK KAFKA Microservice A Logback File Appender Kafka Appender Kafka Producer Logstash Fordwarder
  42. 42. 43 Mutual TLS - your-security.properties # The location of the key store file. kafka.tls.keystore.location=/full/path/to/a/keystore.jks # The store password for the key store file. kafka.tls.keystore.password=some-good-password # The file format of the key store file. kafka.tls.keystore.type=JKS # The password of the private key in the key store file kafka.tls.key.password=the-private-keys-password # The location of the trust store file. kafka.tls.truststore.location=/path/to/truststore.p12 # The password for the trust store file. kafka.tls.truststore.password=some-other-good-password # The file format of the trust store file kafka.tls.truststore.type=PKCS12 © ING 2016
  43. 43. 44 Serialize Producer 1 Metadata N Metadata 1 N Transformer Serializer MetadataMetadata KAFKA1 2 3
  44. 44. 45 Filter Microservice A Kafka Producer Microservice B Kafka Consumer KAFKA Topic A Kafka Consumer Kafka Producer Metadata Topic B Filter
  45. 45. Transformer 46 Transformer Microservice A Kafka Producer Microservice B Kafka Consumer KAFKA Topic A Topic B 1 Kafka Consumer Kafka Producer Metadata 1 2 1 2 2
  46. 46. 47 Forward & Dispatch Microservice A Kafka Producer Microservice B Kafka Consumer KAFKA Topic A Topic C Kafka Consumer Metadata Topic B Topic N Kafka Producer 1 1 1 1 Microservice D Kafka Consumer Microservice Z Kafka Consumer Forward & Dispatch
  47. 47. 48 Replay 0 1 2 3 i NN-1 beginning i end Microservice A
  48. 48. 49 Performance Features
  49. 49. 50 Performance Features FILTERINGDISPATCH
  50. 50. 51 Performance FeaturesFILTERINGDISPATCH
  51. 51. Use microservices with Kafka … … code ahead of the curve László-Róbert ALBERT Dan BALESCU Casablanca • 02 November 2016 @danbalescu https://www.linkedin.com/in/albertlaszlorobert

×