Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Kafka on Mesos
https://github.com/mesos/kafka
Joe Stein
CEO of Elodina, Inc. Elodina http://www.elodina.net/ is a startup focusing
on the support & maintenance of third...
◉ Quick intro to Mesos
◉ Quick intro to Kafka
◉ Making Kafka elastic with
Mesos
Overview
Quick intro to Mesos
Static partitioning
Static partitioning
Static partitioning
Static partitioning
Better option
Data Center Operating System
Mesos
Quick intro to Kafka
Not Streaming Data
Kafka decouples data pipelines
Making Kafka elastic with Mesos
Goals we set out with
● smart broker.id assignment.
● preservation of broker placement (through constraints
and/or new fea...
mesos/kafka
https://github.com/mesos/kafka
Scheduler
● Provides the operational automation for a Kafka Cluster.
● Manages the changes to the broker's configuration.
...
CLI & REST API
● scheduler - starts the scheduler.
● add - adds one more more brokers to the cluster.
● update - changes r...
Launch 20 brokers in seconds
./kafka-mesos.sh add 1000..1019 --cpus 0.01 --heap 128 --mem 256 --options num.io.threads=1
....
Kafka is available on DCOS
https://mesosphere.com/product/
dcos install kafka
dcos kafka help
Mesosphere DCOS
Questions?
Joe Stein
http://www.elodina.net
Making Apache Kafka Elastic with Apache Mesos
Upcoming SlideShare
Loading in …5
×

Making Apache Kafka Elastic with Apache Mesos

7,558 views

Published on

Running Apache Kafka as a framework on Apache Mesos

Making Apache Kafka Elastic with Apache Mesos

  1. 1. Kafka on Mesos https://github.com/mesos/kafka
  2. 2. Joe Stein CEO of Elodina, Inc. Elodina http://www.elodina.net/ is a startup focusing on the support & maintenance of third party open source software (like Mesos frameworks) as well as its own open source products & SaaS solutions. Joe Stein is also the Founder and Principal Consultant of Big Data Open Source Security http://stealth.ly and has been working for the last couple of years on implementing and assisting organizations with their Kafka, Mesos, Hadoop, Cassandra, Accumulo, Storm, Spark, etc, Big Data systems. Prior to this, Joe Stein was responsible for building out a platform that ingested and processed the analytics for 6 billion unique mobile devices.
  3. 3. ◉ Quick intro to Mesos ◉ Quick intro to Kafka ◉ Making Kafka elastic with Mesos Overview
  4. 4. Quick intro to Mesos
  5. 5. Static partitioning
  6. 6. Static partitioning
  7. 7. Static partitioning
  8. 8. Static partitioning
  9. 9. Better option
  10. 10. Data Center Operating System
  11. 11. Mesos
  12. 12. Quick intro to Kafka
  13. 13. Not Streaming Data
  14. 14. Kafka decouples data pipelines
  15. 15. Making Kafka elastic with Mesos
  16. 16. Goals we set out with ● smart broker.id assignment. ● preservation of broker placement (through constraints and/or new features). ● ability to-do configuration changes. ● rolling restarts (for things like configuration changes). ● scaling the cluster up and down with automatic, programmatic and manual options. ● smart partition assignment via constraints visa vi roles, resources and attributes.
  17. 17. mesos/kafka https://github.com/mesos/kafka
  18. 18. Scheduler ● Provides the operational automation for a Kafka Cluster. ● Manages the changes to the broker's configuration. ● Exposes a REST API for the CLI to use or any other client. ● Runs on Marathon for high availability. Executor ● The executor interacts with the kafka broker as an intermediary to the scheduler Scheduler & Executor
  19. 19. CLI & REST API ● scheduler - starts the scheduler. ● add - adds one more more brokers to the cluster. ● update - changes resources, constraints or broker properties one or more brokers. ● remove - take a broker out of the cluster. ● start - starts a broker up. ● stop - this can either a graceful shutdown or will force kill it (./kafka-mesos.sh help stop) ● rebalance - allows you to rebalance a cluster either by selecting the brokers or topics to rebalance. Manual assignment is still possible using the Apache Kafka project tools. Rebalance can also change the replication factor on a topic. ● help - ./kafka-mesos.sh help || ./kafka-mesos.sh help {command}
  20. 20. Launch 20 brokers in seconds ./kafka-mesos.sh add 1000..1019 --cpus 0.01 --heap 128 --mem 256 --options num.io.threads=1 ./kafka-mesos.sh start 1000..1019
  21. 21. Kafka is available on DCOS https://mesosphere.com/product/ dcos install kafka dcos kafka help Mesosphere DCOS
  22. 22. Questions? Joe Stein http://www.elodina.net

×