Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1Confidential
Apache Kafka + Apache Mesos
Highly Scalable Streaming Microservices with Kafka Streams
Kai Waehner
Technolog...
2Confidential
Abstract
Microservices establish	many	benefits	like	agile,	flexible	development	and	deployment	of	business	l...
3Highly Scalable Microservices with Apache Kafka + Mesos
Agenda
1) Scalable Microservices
2) Apache Kafka and Confluent Pl...
4Highly Scalable Microservices with Apache Kafka + Mesos
Agenda
1) Scalable Microservices
2) Apache Kafka and Confluent Pl...
5Highly Scalable Microservices with Apache Kafka + Mesos
Microservices
Orders
Service
Basket
Service
Payment
Service
Fulfi...
6Highly Scalable Microservices with Apache Kafka + Mesos
Independently Deployable
Orders
Service
Basket
Service
Payment
Se...
7Highly Scalable Microservices with Apache Kafka + Mesos
Allows us to scale
8Highly Scalable Microservices with Apache Kafka + Mesos
Scale in people terms
9Highly Scalable Microservices with Apache Kafka + Mesos
Scale in infrastructure terms
Service
Service
Service
Service A i...
10Highly Scalable Microservices with Apache Kafka + Mesos
Scalable Microservices
How do we get there?
• Loose Coupling
• D...
11Highly Scalable Microservices with Apache Kafka + Mesos
Agenda
1) Scalable Microservices
2) Apache Kafka and Confluent P...
12Highly Scalable Microservices with Apache Kafka + Mesos
Apache Kafka – A Distributed, Scalable , Fault-Tolerant Commit L...
13Highly Scalable Microservices with Apache Kafka + Mesos
14Highly Scalable Microservices with Apache Kafka + Mesos
15Highly Scalable Microservices with Apache Kafka + Mesos
16Highly Scalable Microservices with Apache Kafka + Mesos
17Highly Scalable Microservices with Apache Kafka + Mesos
Apache Kafka – A Streaming Platform
18Highly Scalable Microservices with Apache Kafka + Mesos
Single, Shared Source of Truth
Orders Customers
Payments
Stock
19Highly Scalable Microservices with Apache Kafka + Mesos
Agenda
1) Scalable Microservices
2) Apache Kafka and Confluent P...
20Highly Scalable Microservices with Apache Kafka + Mesos
Stream Processing
Data at Rest Data in Motion
21Highly Scalable Microservices with Apache Kafka + Mesos
Stream Processing Pipeline
APIs
Adapters /
Channels
Integration
...
22Highly Scalable Microservices with Apache Kafka + Mesos
When to use Kafka Streams for Stream Processing?
23Highly Scalable Microservices with Apache Kafka + Mesos
When to use Kafka Streams for Stream Processing?
24Highly Scalable Microservices with Apache Kafka + Mesos
25Highly Scalable Microservices with Apache Kafka + Mesos
26Highly Scalable Microservices with Apache Kafka + Mesos
27Highly Scalable Microservices with Apache Kafka + Mesos
Kafka Streams (shipped with Apache Kafka)
Map, filter,
aggregate...
28Highly Scalable Microservices with Apache Kafka + Mesos
A complete streaming microservice, ready for production at large...
29Highly Scalable Microservices with Apache Kafka + Mesos
KSQL – A Streaming SQL Engine for Apache Kafka
SELECT STREAM
CEI...
30Highly Scalable Microservices with Apache Kafka + Mesos
Agenda
1) Scalable Microservices
2) Apache Kafka and Confluent P...
31Highly Scalable Microservices with Apache Kafka + Mesos
Executor
(Task)
Mesos Architecture
Leader
Standby Standby
Execut...
32Highly Scalable Microservices with Apache Kafka + Mesos
Components of a Kafka Cluster
33Highly Scalable Microservices with Apache Kafka + Mesos
Executor
(Kafka Broker)
Mesos Architecture
Leader
Standby Standb...
34Highly Scalable Microservices with Apache Kafka + Mesos
Why DC/OS for the Kafka Ecosystem?
• Automated provisioning and ...
35Highly Scalable Microservices with Apache Kafka + Mesos
Agenda
1) Scalable Microservices
2) Apache Kafka and Confluent P...
36Highly Scalable Microservices with Apache Kafka + Mesos
Live Demo
Use Case:
Airline Flight Delay Prediction
Machine Lear...
37Highly Scalable Microservices with Apache Kafka + Mesos
Executor
(Kafka Broker 1 of 3)
Architecture – Live Demo
Leader
S...
38Highly Scalable Microservices with Apache Kafka + Mesos
H2O.ai Model + Kafka Streams
Filter
Map
2) Configure Kafka Strea...
39Highly Scalable Microservices with Apache Kafka + Mesos
Live Demo
40Highly Scalable Microservices with Apache Kafka + Mesos
DC/OS on AWS
https://downloads.dcos.io/dcos/stable/aws.html
41Highly Scalable Microservices with Apache Kafka + Mesos
Kafka Brokers on DC/OS
42Highly Scalable Microservices with Apache Kafka + Mesos
Kafka Client (compatible with Kafka Streams) on DC/OS
https://hu...
43Highly Scalable Microservices with Apache Kafka + Mesos
Kafka Streams Microservice
dcos kafka --name confluent-kafka top...
44Highly Scalable Microservices with Apache Kafka + Mesos
Kafka Streams Microservice on DC/OS
45Highly Scalable Microservices with Apache Kafka + Mesos
Video Recording of the Live Demo
https://www.youtube.com/watch?v...
46Highly Scalable Microservices with Apache Kafka + Mesos
KSQL on DC/OS – Why not? J
SELECT STREAM
CEIL(timestamp TO HOUR)...
47Highly Scalable Microservices with Apache Kafka + Mesos
Key Take-Aways
Ø Apache Kafka Ecosystem on DC/OS for Highly Scal...
48Highly Scalable Microservices with Apache Kafka + Mesos
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWae...
Upcoming SlideShare
Loading in …5
×

Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Microservices

1,981 views

Published on

Microservices establish many benefits like agile, flexible development and deployment of business logic. However, a Microservice architecture also creates many new challenges like increased communication between distributed instances, the need for orchestration, new fail-over requirements, and resiliency design patterns.
 
This session discusses how to build a highly scalable, performant, mission-critical microservice infrastructure with Apache Kafka and Apache Mesos. Apache Kafka brokers are used as powerful, scalable, distributed message backbone. Kafka’s Streams API allows to embed stream processing directly into any external microservice or business application; without the need for a dedicated streaming cluster. Apache Mesos can be used as scalable infrastructure for both, the Apache Kafka brokers and external applications using the Kafka Streams API, to leverage the benefits of a cloud native platforms like service discovery, health checks, or fail-over management.
 
A live demo shows how to develop real time applications for your core business with Kafka messaging brokers and Kafka Streams API and how to deploy / manage / scale them on a Mesos cluster using different deployment options.

Key takeaways for the audience
- Successful Microservice architectures require a highly scalable messaging infrastructure combined with a cloud-native platform which manages distributed microservices
- Apache Kafka offers a highly scalable, mission critical infrastructure for distributed messaging and integration
- Kafka’s Streams API allows to embed stream processing into any external application or microservice
- Mesos allows management of both, Kafka brokers and external applications using Kafka Streams API, to leverage many built-in benefits like health checks, service discovery or fail-over control of microservices
- See a live demo which combines the Apache Kafka streaming platform and Apache Mesos

https://www.youtube.com/watch?v=OTCuWK8PA1g

Published in: Technology
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • You might also like this slide 'Apache Kafka vs MapR-ES: Fit for purpose/Decision tree': https://www.slideshare.net/sbaltagi/apache-kafka-vs-mapres-fit-for-purposedecision-tree
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Microservices

  1. 1. 1Confidential Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai Waehner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
  2. 2. 2Confidential Abstract Microservices establish many benefits like agile, flexible development and deployment of business logic. However, a Microservice architecture also creates many new challenges like increased communication between distributed instances, the need for orchestration, new fail-over requirements, and resiliency design patterns. This session discusses how to build a highly scalable, performant, mission-critical microservice infrastructure with Apache Kafka and Apache Mesos. Apache Kafka brokers are used as powerful, scalable, distributed message backbone. Kafka’s Streams API allows to embed stream processing directly into any external microservice or business application; without the need for a dedicated streaming cluster. Apache Mesos can be used as scalable infrastructure for both, the Apache Kafka brokers and external applications using the Kafka Streams API, to leverage the benefits of a cloud native platforms like service discovery, health checks, or fail-over management. A live demo shows how to develop real time applications for your core business with Kafka messaging brokers and Kafka Streams API and how to deploy / manage / scale them on a Mesos cluster using different deployment options. Key takeaways for the audience - Successful Microservice architectures require a highly scalable messaging infrastructure combined with a cloud-native platform which manages distributed microservices - Apache Kafka offers a highly scalable, mission critical infrastructure for distributed messaging and integration - Kafka’s Streams API allows to embed stream processing into any external application or microservice - Mesos allows management of both, Kafka brokers and external applications using Kafka Streams API, to leverage many built-in benefits like health checks, service discovery or fail-over control of microservices - See a live demo which combines the Apache Kafka streaming platform and Apache Mesos
  3. 3. 3Highly Scalable Microservices with Apache Kafka + Mesos Agenda 1) Scalable Microservices 2) Apache Kafka and Confluent Platform 3) Kafka Streams 4) Scalable Microservices with Kafka and DC/OS 5) Use Case - Scalable Flight Prediction
  4. 4. 4Highly Scalable Microservices with Apache Kafka + Mesos Agenda 1) Scalable Microservices 2) Apache Kafka and Confluent Platform 3) Kafka Streams 4) Scalable Microservices with Kafka and DC/OS 5) Use Case - Scalable Flight Prediction
  5. 5. 5Highly Scalable Microservices with Apache Kafka + Mesos Microservices Orders Service Basket Service Payment Service Fulfillment Service Stock Service
  6. 6. 6Highly Scalable Microservices with Apache Kafka + Mesos Independently Deployable Orders Service Basket Service Payment Service Fulfillment Service Stock Service
  7. 7. 7Highly Scalable Microservices with Apache Kafka + Mesos Allows us to scale
  8. 8. 8Highly Scalable Microservices with Apache Kafka + Mesos Scale in people terms
  9. 9. 9Highly Scalable Microservices with Apache Kafka + Mesos Scale in infrastructure terms Service Service Service Service A instance 1 Service A instance 2 Service B instance 1 Service B instance 2 Cluster (many machines)
  10. 10. 10Highly Scalable Microservices with Apache Kafka + Mesos Scalable Microservices How do we get there? • Loose Coupling • Data Enabled • Event Driven • Operational Transparency
  11. 11. 11Highly Scalable Microservices with Apache Kafka + Mesos Agenda 1) Scalable Microservices 2) Apache Kafka and Confluent Platform 3) Kafka Streams 4) Scalable Microservices with Kafka and DC/OS 5) Use Case - Scalable Flight Prediction
  12. 12. 12Highly Scalable Microservices with Apache Kafka + Mesos Apache Kafka – A Distributed, Scalable , Fault-Tolerant Commit Log
  13. 13. 13Highly Scalable Microservices with Apache Kafka + Mesos
  14. 14. 14Highly Scalable Microservices with Apache Kafka + Mesos
  15. 15. 15Highly Scalable Microservices with Apache Kafka + Mesos
  16. 16. 16Highly Scalable Microservices with Apache Kafka + Mesos
  17. 17. 17Highly Scalable Microservices with Apache Kafka + Mesos Apache Kafka – A Streaming Platform
  18. 18. 18Highly Scalable Microservices with Apache Kafka + Mesos Single, Shared Source of Truth Orders Customers Payments Stock
  19. 19. 19Highly Scalable Microservices with Apache Kafka + Mesos Agenda 1) Scalable Microservices 2) Apache Kafka and Confluent Platform 3) Kafka Streams 4) Scalable Microservices with Kafka and DC/OS 5) Use Case - Scalable Flight Prediction
  20. 20. 20Highly Scalable Microservices with Apache Kafka + Mesos Stream Processing Data at Rest Data in Motion
  21. 21. 21Highly Scalable Microservices with Apache Kafka + Mesos Stream Processing Pipeline APIs Adapters / Channels Integration Messaging Stream Ingest Transformation Aggregation Enrichment Filtering Stream Preprocessing Process Management Analytics (Real Time) Applications & APIs Analytics / DW Reporting Stream Outcomes • Contextual Rules • Windowing • Patterns • Analytics • Machine Learning • … Stream Analytics Index / SearchNormalization
  22. 22. 22Highly Scalable Microservices with Apache Kafka + Mesos When to use Kafka Streams for Stream Processing?
  23. 23. 23Highly Scalable Microservices with Apache Kafka + Mesos When to use Kafka Streams for Stream Processing?
  24. 24. 24Highly Scalable Microservices with Apache Kafka + Mesos
  25. 25. 25Highly Scalable Microservices with Apache Kafka + Mesos
  26. 26. 26Highly Scalable Microservices with Apache Kafka + Mesos
  27. 27. 27Highly Scalable Microservices with Apache Kafka + Mesos Kafka Streams (shipped with Apache Kafka) Map, filter, aggregate, apply analytic model, „any business logic“ Input Stream (Kafka Topic) Kafka Cluster Output Stream (Kafka Topic) Kafka Cluster Stream Processing Microservice (Kafka Streams) Deployed Anywhere Java App, Docker, Kubernetes, Mesos, “you-name-it”
  28. 28. 28Highly Scalable Microservices with Apache Kafka + Mesos A complete streaming microservice, ready for production at large-scale Word Count App configuration Define processing (here: WordCount) Start processing
  29. 29. 29Highly Scalable Microservices with Apache Kafka + Mesos KSQL – A Streaming SQL Engine for Apache Kafka SELECT STREAM CEIL(timestamp TO HOUR) AS timeWindow, productId, COUNT(*) AS hourlyOrders, SUM(units) AS units FROM Orders GROUP BY CEIL(timestamp TO HOUR), productId; timeWindow | productId | hourlyOrders | units ------------+-----------+--------------+------- 08:00:00 | 10 | 2 | 5 08:00:00 | 20 | 1 | 8 09:00:00 | 10 | 4 | 22 09:00:00 | 40 | 1 | 45 ... | ... | ... | ...
  30. 30. 30Highly Scalable Microservices with Apache Kafka + Mesos Agenda 1) Scalable Microservices 2) Apache Kafka and Confluent Platform 3) Kafka Streams 4) Scalable Microservices with Kafka and DC/OS 5) Use Case - Scalable Flight Prediction
  31. 31. 31Highly Scalable Microservices with Apache Kafka + Mesos Executor (Task) Mesos Architecture Leader Standby Standby Executor (Task) Scheduler ZK ZK ZK Scheduler Framework A Framework B Framework A Framework B Mesos Master Quorum Slave 1 Slave N Offer Offer Offer Offer Master 2 Master 3 Master 1
  32. 32. 32Highly Scalable Microservices with Apache Kafka + Mesos Components of a Kafka Cluster
  33. 33. 33Highly Scalable Microservices with Apache Kafka + Mesos Executor (Kafka Broker) Mesos Architecture Leader Standby Standby Executor (Kafka Streams) Scheduler Scheduler Marathon Kubernetes Marathon Kubernetes Mesos Master Quorum Slave 1 Slave N Offer Offer Offer Offer Master 2 Master 3 Master 1 ZK ZK ZK
  34. 34. 34Highly Scalable Microservices with Apache Kafka + Mesos Why DC/OS for the Kafka Ecosystem? • Automated provisioning and upgrading of Kafka components • Broker, REST Proxy, Schema Registry, Connect … • Kafka applications (Java / Go / .NET / Python Clients, Kafka Streams, KSQL) • Monitoring (Confluent Control Center, etc.) • Unified management and monitoring • Easy interactive installation • Multiple Kafka Cluster on one infrastructure + multi-tenancy • Combination with other Big Data components (Spark, Cassandra, etc.) on one infrastructure • Integration with syslog-compatible logging services for diagnostics and troubleshooting • Elastic scaling, fault tolerance and self-healing • Stateful and stateless services • Service discovery and routing (using the corresponding Mesos framework, i.e. Marathon or Kubernetes) • Kafka VIP Connection (one “static” bootstrap server url) • Storage volumes for enhanced data durability, known as Mesos Dynamic Reservations and Persistent Volumes
  35. 35. 35Highly Scalable Microservices with Apache Kafka + Mesos Agenda 1) Scalable Microservices 2) Apache Kafka and Confluent Platform 3) Kafka Streams 4) Scalable Microservices with Kafka and DC/OS 5) Use Case - Scalable Flight Prediction
  36. 36. 36Highly Scalable Microservices with Apache Kafka + Mesos Live Demo Use Case: Airline Flight Delay Prediction Machine Learning Algorithm: Gradient Boosting (GBM) using Decision Trees Technologies: DC/OS Kafka Broker Kafka Streams H2O.ai
  37. 37. 37Highly Scalable Microservices with Apache Kafka + Mesos Executor (Kafka Broker 1 of 3) Architecture – Live Demo Leader Standby Standby Executor (Kafka Streams 5 of 5) Scheduler Marathon Marathon Marathon Mesos Master Quorum Slave 1 Slave N Offer Offer Offer Master 2 Master 3 Master 1 ZK ZK ZK
  38. 38. 38Highly Scalable Microservices with Apache Kafka + Mesos H2O.ai Model + Kafka Streams Filter Map 2) Configure Kafka Streams Application 3) Apply H2O ML Model to Streaming Data 4) Start Kafka Streams App 1) Create H2O ML Model
  39. 39. 39Highly Scalable Microservices with Apache Kafka + Mesos Live Demo
  40. 40. 40Highly Scalable Microservices with Apache Kafka + Mesos DC/OS on AWS https://downloads.dcos.io/dcos/stable/aws.html
  41. 41. 41Highly Scalable Microservices with Apache Kafka + Mesos Kafka Brokers on DC/OS
  42. 42. 42Highly Scalable Microservices with Apache Kafka + Mesos Kafka Client (compatible with Kafka Streams) on DC/OS https://hub.docker.com/r/mesosphere/kafka-client/ https://hub.docker.com/r/megachucky/mesos-kafka-client/ dcos node ssh --master-proxy –leader docker run -it megachucky/mesos-kafka-client From Apache Kafka 0.9 to 0.11 (Kafka Streams messages require timestamps) Dockerfile: FROM java:openjdk-8-jreMAINTAINER Kai Waehner curl http://apache.mirrors.spacedump.net/kafka/0.11.0.1/kafka_2.11-0.11.0.1.tgz | tar xvz --strip-components=1 WORKDIR /bin
  43. 43. 43Highly Scalable Microservices with Apache Kafka + Mesos Kafka Streams Microservice dcos kafka --name confluent-kafka topic create AirlineInputTopic --partitions 10 --replication 3 dcos kafka --name confluent-kafka topic create AirlineOutputTopic --partitions 10 --replication 3 https://hub.docker.com/r/megachucky/kafka-streams-machine-learning-docker-microservice/ https://github.com/kaiwaehner/kafka-streams-machine-learning-docker-microservice Dockerfile: FROM java:8 ADD /opt/kafka-streams-h2o-docker-microservice-1.0-SNAPSHOT-jar-with-dependencies.jar /opt/ ENTRYPOINT ["java", "-jar", "/opt/kafka-streams-h2o-docker-microservice-1.0-SNAPSHOT-jar-with-dependencies.jar"]
  44. 44. 44Highly Scalable Microservices with Apache Kafka + Mesos Kafka Streams Microservice on DC/OS
  45. 45. 45Highly Scalable Microservices with Apache Kafka + Mesos Video Recording of the Live Demo https://www.youtube.com/watch?v=OTCuWK8PA1g
  46. 46. 46Highly Scalable Microservices with Apache Kafka + Mesos KSQL on DC/OS – Why not? J SELECT STREAM CEIL(timestamp TO HOUR) AS timeWindow, productId, COUNT(*) AS hourlyOrders, SUM(units) AS units FROM Orders GROUP BY CEIL(timestamp TO HOUR), productId; timeWindow | productId | hourlyOrders | units ------------+-----------+--------------+------- 08:00:00 | 10 | 2 | 5 08:00:00 | 20 | 1 | 8 09:00:00 | 10 | 4 | 22 09:00:00 | 40 | 1 | 45 ... | ... | ... | ...
  47. 47. 47Highly Scalable Microservices with Apache Kafka + Mesos Key Take-Aways Ø Apache Kafka Ecosystem on DC/OS for Highly Scalable, Fault-Tolerant Microservices Ø DC/OS offers many Kafka Features out-of-the-box (one-click-provisioning, VIP connection, …) Ø Kafka Streams Microservices run and scale on DC/OS via Marathon or Kubernetes
  48. 48. 48Highly Scalable Microservices with Apache Kafka + Mesos Kai Waehner Technology Evangelist kontakt@kai-waehner.de @KaiWaehner www.confluent.io www.kai-waehner.de LinkedIn Questions? Feedback? Please contact me!

×