SlideShare a Scribd company logo

Containerized Data Persistence on Mesos

Joe Stein
Joe Stein

Elodina is a startup focused on supporting open source software like Mesos frameworks. They have experience implementing and assisting with Kafka, Mesos, Hadoop, Cassandra, and other big data systems. The document discusses containerizing data persistence on Mesos with various data systems like Kafka, MySQL, Cassandra and HDFS. It provides overviews of file systems, databases, and Mesos concepts like roles and resources. It also summarizes efforts to run these data systems as Mesos frameworks.

1 of 38
Download to read offline
Containerized Data Persistence
on Mesos with Kafka, MySQL,
Cassandra, HDFS and More!
CEO of Elodina, Inc. Elodina http://www.elodina.net/ is a startup focusing
on the support & maintenance of third party open source software (like
Mesos frameworks) and offering SaaS based solutions for those
systems. Elodina started as Big Data Open Source Security
http://stealth.ly and has been working for the last couple of years on
implementing and assisting organizations with their Kafka, Mesos,
Hadoop, Cassandra, Accumulo, Storm, Spark, etc, Big Data systems.
Twitter: https://twitter.com/allthingshadoop
LinkedIn: https://www.linkedin.com/in/charmalloc
Joe Stein
◉ File systems, databases, object
stores, storage solutions, etc.
◉ Apache Mesos and the Datacenter
Operating System.
◉ Kafka, Cassandra, MySQL, HDFS.
Overview
File Systems
- manages space
- directories
- file names
- meta-data
- permissions
- compression
- quotas
Distributed File System
- Remote block management
- Replication
- Streaming Data Access
- Large Data Sets
Replicated Log
- Immutable Appends
- Replicated Partitions
- Messaging Features
- Log Retention
Transactional and/or Relational Databases ~ ACID
Dynamo
Storage Solutions
- Storage Area Networks
- Network Attached Storage
- W.O.R.M.
- Cold Storage
Quick intro to Mesos
Static partitioning
Static partitioning
Static partitioning
Static partitioning
Better option
Data Center Operating System
Containerized Data Persistence on Mesos
Mesos
Containerized Data Persistence on Mesos
Resources & Attributes
The Mesos system has two basic methods to describe the
slaves that comprise a cluster. One of these is managed
by the Mesos master, the other is simply passed onwards
to the frameworks using the cluster.
--attributes='disks:sata;raid:jbod;dc:1;rack:3'
Roles
Total consumable resources per slave, in the form 'name(role):value;name(role):value...'. This value can be set
to limit resources per role, or to overstate the number of resources that are available to the slave.
--resources="cpus(*):8; mem(*):15360; disk(*):710534; ports(*):[31000-32000]"
--resources="cpus(prod):8; cpus(stage):2 mem(*):15360; disk(*):710534; ports(*):[31000-32000]"
All * roles will be detected, so you can specify only the resources that are not all roles (*). --
resources="cpus(prod):8; cpus(stage)"
Frameworks bind a specific roles or any. A default roll (instead of *) can also be configured.
Roles can be used to isolate and segregate frameworks.
In coming release(s) to make things even better!
MESOS-2018 Dynamic Reservations
MESOS-1554 Persistent resources support for storage-like
services
MESOS-1279 Add resize task primitive
Apache Kafka with Apache Mesos
mesos/kafka
https://github.com/mesos/kafka
Goals we set out with
● smart broker.id assignment.
● preservation of broker placement (through constraints
and/or new features).
● ability to-do configuration changes.
● rolling restarts (for things like configuration changes).
● scaling the cluster up and down with automatic,
programmatic and manual options.
● smart partition assignment via constraints visa vi
roles, resources and attributes.
Scheduler
● Provides the operational automation for a Kafka Cluster.
● Manages the changes to the broker's configuration.
● Exposes a REST API for the CLI to use or any other
client.
● Runs on Marathon for high availability.
Executor
● The executor interacts with the kafka broker as an
intermediary to the scheduler
Scheduler & Executor
CLI & REST API
● scheduler - starts the scheduler.
● add - adds one more more brokers to the cluster.
● update - changes resources, constraints or broker properties one or more
brokers.
● remove - take a broker out of the cluster.
● start - starts a broker up.
● stop - this can either a graceful shutdown or will force kill it (./kafka-mesos.sh
help stop)
● rebalance - allows you to rebalance a cluster either by selecting the brokers
or topics to rebalance. Manual assignment is still possible using the Apache
Kafka project tools. Rebalance can also change the replication factor on a
topic.
● help - ./kafka-mesos.sh help || ./kafka-mesos.sh help {command}
Launch 20 brokers in seconds
./kafka-mesos.sh add 1000..1019 --cpus 0.01 --heap 128 --mem 256 --options num.io.threads=1
./kafka-mesos.sh start 1000..1019
Kafka is available on DCOS
https://mesosphere.com/product/
dcos install kafka
dcos kafka help
Mesosphere DCOS
Apache Cassandra with Apache
Mesos
Cassandra on Mesos
https://github.com/mesosphere/cassandra-mesos
The Mesos scheduler is the component with the most
high-level intelligence in the framework. It will need to
possess the ability to bootstrap a ring and distribute
the correct configuration to all subsequently started
nodes. The Scheduler will also be responsible for
orchestrating all tasks with regard to restarting nodes
and triggering and monitoring periodic administrative
tasks required by a node.
Cassandra Scheduler
◉ Bootstrapping a ring
◉ Adding nodes to a ring
◉ Restarting a node that has crashed
◉ Providing configuration to nodes
o Seed nodes, Snitch Class, JVM
OPTS
◉ Scheduling and running administrative
utilities
o nodetool repair
o nodetool cleanup
◉ Registers with a failover timeout
◉ Supports framework authentication
◉ Declines offers to resources it
doesn't need
◉ Only use necessary fraction of
offers
◉ Deal with lost tasks
◉ Does not rely on in-memory state
◉ Verifies supported Mesos Version
◉ Supports roles
◉ Able to provide set of ports to be
used by Nodes
◉ Initial implementation will be for a
static set of ports with a potential
for longer term dynamic port usage.
Cassandra Executor
◉ Monitor health of running node
◉ Use JMX Mbeans for interfacing
with Cassandra Server Process
◉ Communicate results of
administrative actions via
StatusUpdates to scheduler
when necessary
◉ Does not rely on file system
state outside sandbox
◉ Pure libprocess
communication with Scheduler
leveraging StatusUpdate
◉ Does not rely on running on a
particular slave node
◉ Data directories will be
created and managed by
Mesos leveraging the features
provided in MESOS-1554
Apache HDFS with Apache Mesos
HDFS on Mesos
https://github.com/mesosphere/hdfs
◉ 3 journal nodes
◉ 2 name nodes (active/standby)
◉ data nodes, lots of them!
◉ Fault tolerance more
than just what Hadoop
gives.
◉ Ease of configuration and
distributing nodes.
◉ Elastic DFS
◉ Run multiple frameworks
at a time for new
solutions
MySQL with Apache Mesos
MySQL on Mesos (Apache Incubating)
◉ Open sourced by Twitter https://github.com/twitter/mysos
◉ Moving to Apache https://twitter.com/ApacheMysos
◉ Dramatically simplifies the management of a MySQL cluster:
o Efficient hardware utilization through multi-tenancy (in performance-
isolated containers)
o High reliability through preserving the MySQL state during failure and
automatic backing up to/restoring from HDFS
o An automated self-service option for bringing up new MySQL clusters
o High availability through automatic MySQL master failover
o An elastic solution that allows users to easily scale up and down a MySQL
cluster by changing the number of slave instances
Questions?
Joe Stein
http://www.elodina.net

Recommended

Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...Joe Stein
 
Making Apache Kafka Elastic with Apache Mesos
Making Apache Kafka Elastic with Apache MesosMaking Apache Kafka Elastic with Apache Mesos
Making Apache Kafka Elastic with Apache MesosJoe Stein
 
Apache Kafka, HDFS, Accumulo and more on Mesos
Apache Kafka, HDFS, Accumulo and more on MesosApache Kafka, HDFS, Accumulo and more on Mesos
Apache Kafka, HDFS, Accumulo and more on MesosJoe Stein
 
Developing Frameworks for Apache Mesos
Developing Frameworks  for Apache MesosDeveloping Frameworks  for Apache Mesos
Developing Frameworks for Apache MesosJoe Stein
 
Introduction To Apache Mesos
Introduction To Apache MesosIntroduction To Apache Mesos
Introduction To Apache MesosJoe Stein
 
Ceph-Mesos framework
Ceph-Mesos frameworkCeph-Mesos framework
Ceph-Mesos frameworkZhongyue Luo
 
Get started with Developing Frameworks in Go on Apache Mesos
Get started with Developing Frameworks in Go on Apache MesosGet started with Developing Frameworks in Go on Apache Mesos
Get started with Developing Frameworks in Go on Apache MesosJoe Stein
 
Building and Deploying Application to Apache Mesos
Building and Deploying Application to Apache MesosBuilding and Deploying Application to Apache Mesos
Building and Deploying Application to Apache MesosJoe Stein
 

More Related Content

What's hot

Introduction to Mesos
Introduction to MesosIntroduction to Mesos
Introduction to Mesoskoboltmarky
 
Introduction of mesos persistent storage
Introduction of mesos persistent storageIntroduction of mesos persistent storage
Introduction of mesos persistent storageZhou Weitao
 
Streaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogStreaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogJoe Stein
 
Mesos and containers
Mesos and containersMesos and containers
Mesos and containersJiang Yan Xu
 
Elastic HBase on Mesos - HBaseCon 2015
Elastic HBase on Mesos - HBaseCon 2015Elastic HBase on Mesos - HBaseCon 2015
Elastic HBase on Mesos - HBaseCon 2015Cosmin Lehene
 
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloReal-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein
 
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax
 
8a. How To Setup HBase with Docker
8a. How To Setup HBase with Docker8a. How To Setup HBase with Docker
8a. How To Setup HBase with DockerFabio Fumarola
 
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...C4Media
 
Cassandra Introduction & Features
Cassandra Introduction & FeaturesCassandra Introduction & Features
Cassandra Introduction & FeaturesPhil Peace
 
Introduction to mesos bay
Introduction to mesos bayIntroduction to mesos bay
Introduction to mesos bayhongbin034
 
Introduction to Apache Mesos
Introduction to Apache MesosIntroduction to Apache Mesos
Introduction to Apache Mesostomasbart
 
Introduction to Apache Mesos
Introduction to Apache MesosIntroduction to Apache Mesos
Introduction to Apache MesosKnoldus Inc.
 
Scaling Big Data with Hadoop and Mesos
Scaling Big Data with Hadoop and MesosScaling Big Data with Hadoop and Mesos
Scaling Big Data with Hadoop and MesosDiscover Pinterest
 
HBaseConEast2016: HBase on Docker with Clusterdock
HBaseConEast2016: HBase on Docker with ClusterdockHBaseConEast2016: HBase on Docker with Clusterdock
HBaseConEast2016: HBase on Docker with ClusterdockMichael Stack
 
Cassandra and Rails at LA NoSQL Meetup
Cassandra and Rails at LA NoSQL MeetupCassandra and Rails at LA NoSQL Meetup
Cassandra and Rails at LA NoSQL MeetupMichael Wynholds
 
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google CloudDrupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google CloudDropsolid
 
Friends of Solr - Nutch & HDFS
Friends of Solr - Nutch & HDFSFriends of Solr - Nutch & HDFS
Friends of Solr - Nutch & HDFSSaumitra Srivastav
 
Develop with linux containers and docker
Develop with linux containers and dockerDevelop with linux containers and docker
Develop with linux containers and dockerFabio Fumarola
 

What's hot (20)

Introduction to Mesos
Introduction to MesosIntroduction to Mesos
Introduction to Mesos
 
Introduction of mesos persistent storage
Introduction of mesos persistent storageIntroduction of mesos persistent storage
Introduction of mesos persistent storage
 
Streaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogStreaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit Log
 
Apache Mesos
Apache MesosApache Mesos
Apache Mesos
 
Mesos and containers
Mesos and containersMesos and containers
Mesos and containers
 
Elastic HBase on Mesos - HBaseCon 2015
Elastic HBase on Mesos - HBaseCon 2015Elastic HBase on Mesos - HBaseCon 2015
Elastic HBase on Mesos - HBaseCon 2015
 
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloReal-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
 
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
 
8a. How To Setup HBase with Docker
8a. How To Setup HBase with Docker8a. How To Setup HBase with Docker
8a. How To Setup HBase with Docker
 
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
 
Cassandra Introduction & Features
Cassandra Introduction & FeaturesCassandra Introduction & Features
Cassandra Introduction & Features
 
Introduction to mesos bay
Introduction to mesos bayIntroduction to mesos bay
Introduction to mesos bay
 
Introduction to Apache Mesos
Introduction to Apache MesosIntroduction to Apache Mesos
Introduction to Apache Mesos
 
Introduction to Apache Mesos
Introduction to Apache MesosIntroduction to Apache Mesos
Introduction to Apache Mesos
 
Scaling Big Data with Hadoop and Mesos
Scaling Big Data with Hadoop and MesosScaling Big Data with Hadoop and Mesos
Scaling Big Data with Hadoop and Mesos
 
HBaseConEast2016: HBase on Docker with Clusterdock
HBaseConEast2016: HBase on Docker with ClusterdockHBaseConEast2016: HBase on Docker with Clusterdock
HBaseConEast2016: HBase on Docker with Clusterdock
 
Cassandra and Rails at LA NoSQL Meetup
Cassandra and Rails at LA NoSQL MeetupCassandra and Rails at LA NoSQL Meetup
Cassandra and Rails at LA NoSQL Meetup
 
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google CloudDrupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
 
Friends of Solr - Nutch & HDFS
Friends of Solr - Nutch & HDFSFriends of Solr - Nutch & HDFS
Friends of Solr - Nutch & HDFS
 
Develop with linux containers and docker
Develop with linux containers and dockerDevelop with linux containers and docker
Develop with linux containers and docker
 

Viewers also liked

Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and CassandraReal-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and CassandraJoe Stein
 
[SICP] 4.4 Logic Programming : 논리로 프로그램 짜기
[SICP] 4.4 Logic Programming : 논리로 프로그램 짜기[SICP] 4.4 Logic Programming : 논리로 프로그램 짜기
[SICP] 4.4 Logic Programming : 논리로 프로그램 짜기종빈 오
 
하스켈학교 세미나 - Haxl
하스켈학교 세미나 - Haxl하스켈학교 세미나 - Haxl
하스켈학교 세미나 - HaxlJooyung Han
 
jstein.cassandra.nyc.2011
jstein.cassandra.nyc.2011jstein.cassandra.nyc.2011
jstein.cassandra.nyc.2011Joe Stein
 
Storing Time Series Metrics With Cassandra and Composite Columns
Storing Time Series Metrics With Cassandra and Composite ColumnsStoring Time Series Metrics With Cassandra and Composite Columns
Storing Time Series Metrics With Cassandra and Composite ColumnsJoe Stein
 
Developing Realtime Data Pipelines With Apache Kafka
Developing Realtime Data Pipelines With Apache KafkaDeveloping Realtime Data Pipelines With Apache Kafka
Developing Realtime Data Pipelines With Apache KafkaJoe Stein
 
Apache Cassandra 2.0
Apache Cassandra 2.0Apache Cassandra 2.0
Apache Cassandra 2.0Joe Stein
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaJoe Stein
 
Introduction Apache Kafka
Introduction Apache KafkaIntroduction Apache Kafka
Introduction Apache KafkaJoe Stein
 
Developing with the Go client for Apache Kafka
Developing with the Go client for Apache KafkaDeveloping with the Go client for Apache Kafka
Developing with the Go client for Apache KafkaJoe Stein
 
Current and Future of Apache Kafka
Current and Future of Apache KafkaCurrent and Future of Apache Kafka
Current and Future of Apache KafkaJoe Stein
 
SMACK Stack 1.1
SMACK Stack 1.1SMACK Stack 1.1
SMACK Stack 1.1Joe Stein
 
Hadoop Streaming Tutorial With Python
Hadoop Streaming Tutorial With PythonHadoop Streaming Tutorial With Python
Hadoop Streaming Tutorial With PythonJoe Stein
 
cloud scheduling
cloud schedulingcloud scheduling
cloud schedulingMudit Verma
 
core.logic (Clojure)
core.logic (Clojure)core.logic (Clojure)
core.logic (Clojure)Seonho Kim
 
Apache Kafka
Apache KafkaApache Kafka
Apache KafkaJoe Stein
 

Viewers also liked (17)

Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and CassandraReal-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
 
[SICP] 4.4 Logic Programming : 논리로 프로그램 짜기
[SICP] 4.4 Logic Programming : 논리로 프로그램 짜기[SICP] 4.4 Logic Programming : 논리로 프로그램 짜기
[SICP] 4.4 Logic Programming : 논리로 프로그램 짜기
 
하스켈학교 세미나 - Haxl
하스켈학교 세미나 - Haxl하스켈학교 세미나 - Haxl
하스켈학교 세미나 - Haxl
 
jstein.cassandra.nyc.2011
jstein.cassandra.nyc.2011jstein.cassandra.nyc.2011
jstein.cassandra.nyc.2011
 
Data Pipeline with Kafka
Data Pipeline with KafkaData Pipeline with Kafka
Data Pipeline with Kafka
 
Storing Time Series Metrics With Cassandra and Composite Columns
Storing Time Series Metrics With Cassandra and Composite ColumnsStoring Time Series Metrics With Cassandra and Composite Columns
Storing Time Series Metrics With Cassandra and Composite Columns
 
Developing Realtime Data Pipelines With Apache Kafka
Developing Realtime Data Pipelines With Apache KafkaDeveloping Realtime Data Pipelines With Apache Kafka
Developing Realtime Data Pipelines With Apache Kafka
 
Apache Cassandra 2.0
Apache Cassandra 2.0Apache Cassandra 2.0
Apache Cassandra 2.0
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache Kafka
 
Introduction Apache Kafka
Introduction Apache KafkaIntroduction Apache Kafka
Introduction Apache Kafka
 
Developing with the Go client for Apache Kafka
Developing with the Go client for Apache KafkaDeveloping with the Go client for Apache Kafka
Developing with the Go client for Apache Kafka
 
Current and Future of Apache Kafka
Current and Future of Apache KafkaCurrent and Future of Apache Kafka
Current and Future of Apache Kafka
 
SMACK Stack 1.1
SMACK Stack 1.1SMACK Stack 1.1
SMACK Stack 1.1
 
Hadoop Streaming Tutorial With Python
Hadoop Streaming Tutorial With PythonHadoop Streaming Tutorial With Python
Hadoop Streaming Tutorial With Python
 
cloud scheduling
cloud schedulingcloud scheduling
cloud scheduling
 
core.logic (Clojure)
core.logic (Clojure)core.logic (Clojure)
core.logic (Clojure)
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
 

Similar to Containerized Data Persistence on Mesos

Fully fault tolerant real time data pipeline with docker and mesos
Fully fault tolerant real time data pipeline with docker and mesos Fully fault tolerant real time data pipeline with docker and mesos
Fully fault tolerant real time data pipeline with docker and mesos Rahul Kumar
 
Fully Fault tolerant Streaming Workflows at Scale using Apache Mesos & Spark ...
Fully Fault tolerant Streaming Workflows at Scale using Apache Mesos & Spark ...Fully Fault tolerant Streaming Workflows at Scale using Apache Mesos & Spark ...
Fully Fault tolerant Streaming Workflows at Scale using Apache Mesos & Spark ...Akhil Das
 
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating SystemOSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating SystemNETWAYS
 
DataStax | Building a Spark Streaming App with DSE File System (Rocco Varela)...
DataStax | Building a Spark Streaming App with DSE File System (Rocco Varela)...DataStax | Building a Spark Streaming App with DSE File System (Rocco Varela)...
DataStax | Building a Spark Streaming App with DSE File System (Rocco Varela)...DataStax
 
Realtime Data Pipeline with Spark Streaming and Cassandra with Mesos (Rahul K...
Realtime Data Pipeline with Spark Streaming and Cassandra with Mesos (Rahul K...Realtime Data Pipeline with Spark Streaming and Cassandra with Mesos (Rahul K...
Realtime Data Pipeline with Spark Streaming and Cassandra with Mesos (Rahul K...DataStax
 
Real time data pipeline with spark streaming and cassandra with mesos
Real time data pipeline with spark streaming and cassandra with mesosReal time data pipeline with spark streaming and cassandra with mesos
Real time data pipeline with spark streaming and cassandra with mesosRahul Kumar
 
Getting Started Running Apache Spark on Apache Mesos
Getting Started Running Apache Spark on Apache MesosGetting Started Running Apache Spark on Apache Mesos
Getting Started Running Apache Spark on Apache MesosPaco Nathan
 
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...DataStax Academy
 
xPatterns on Spark, Tachyon and Mesos - Bucharest meetup
xPatterns on Spark, Tachyon and Mesos - Bucharest meetupxPatterns on Spark, Tachyon and Mesos - Bucharest meetup
xPatterns on Spark, Tachyon and Mesos - Bucharest meetupRadu Chilom
 
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesReusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesYevgeniy Brikman
 
DockerCon14 Cluster Management and Containerization
DockerCon14 Cluster Management and ContainerizationDockerCon14 Cluster Management and Containerization
DockerCon14 Cluster Management and ContainerizationDocker, Inc.
 
Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)
Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)
Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)Spark Summit
 
Abhishek Kumar - CloudStack Locking Service
Abhishek Kumar - CloudStack Locking ServiceAbhishek Kumar - CloudStack Locking Service
Abhishek Kumar - CloudStack Locking ServiceShapeBlue
 
In Memory Analytics with Apache Spark
In Memory Analytics with Apache SparkIn Memory Analytics with Apache Spark
In Memory Analytics with Apache SparkVenkata Naga Ravi
 
Cassandra - A decentralized storage system
Cassandra - A decentralized storage systemCassandra - A decentralized storage system
Cassandra - A decentralized storage systemArunit Gupta
 
HPC and cloud distributed computing, as a journey
HPC and cloud distributed computing, as a journeyHPC and cloud distributed computing, as a journey
HPC and cloud distributed computing, as a journeyPeter Clapham
 
Big data processing using hadoop poster presentation
Big data processing using hadoop poster presentationBig data processing using hadoop poster presentation
Big data processing using hadoop poster presentationAmrut Patil
 
Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...
Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...
Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...Accumulo Summit
 
Hadoop cluster configuration
Hadoop cluster configurationHadoop cluster configuration
Hadoop cluster configurationprabakaranbrick
 

Similar to Containerized Data Persistence on Mesos (20)

Fully fault tolerant real time data pipeline with docker and mesos
Fully fault tolerant real time data pipeline with docker and mesos Fully fault tolerant real time data pipeline with docker and mesos
Fully fault tolerant real time data pipeline with docker and mesos
 
Fully Fault tolerant Streaming Workflows at Scale using Apache Mesos & Spark ...
Fully Fault tolerant Streaming Workflows at Scale using Apache Mesos & Spark ...Fully Fault tolerant Streaming Workflows at Scale using Apache Mesos & Spark ...
Fully Fault tolerant Streaming Workflows at Scale using Apache Mesos & Spark ...
 
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating SystemOSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
 
DataStax | Building a Spark Streaming App with DSE File System (Rocco Varela)...
DataStax | Building a Spark Streaming App with DSE File System (Rocco Varela)...DataStax | Building a Spark Streaming App with DSE File System (Rocco Varela)...
DataStax | Building a Spark Streaming App with DSE File System (Rocco Varela)...
 
Realtime Data Pipeline with Spark Streaming and Cassandra with Mesos (Rahul K...
Realtime Data Pipeline with Spark Streaming and Cassandra with Mesos (Rahul K...Realtime Data Pipeline with Spark Streaming and Cassandra with Mesos (Rahul K...
Realtime Data Pipeline with Spark Streaming and Cassandra with Mesos (Rahul K...
 
Real time data pipeline with spark streaming and cassandra with mesos
Real time data pipeline with spark streaming and cassandra with mesosReal time data pipeline with spark streaming and cassandra with mesos
Real time data pipeline with spark streaming and cassandra with mesos
 
Getting Started Running Apache Spark on Apache Mesos
Getting Started Running Apache Spark on Apache MesosGetting Started Running Apache Spark on Apache Mesos
Getting Started Running Apache Spark on Apache Mesos
 
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...
 
xPatterns on Spark, Tachyon and Mesos - Bucharest meetup
xPatterns on Spark, Tachyon and Mesos - Bucharest meetupxPatterns on Spark, Tachyon and Mesos - Bucharest meetup
xPatterns on Spark, Tachyon and Mesos - Bucharest meetup
 
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesReusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modules
 
DockerCon14 Cluster Management and Containerization
DockerCon14 Cluster Management and ContainerizationDockerCon14 Cluster Management and Containerization
DockerCon14 Cluster Management and Containerization
 
Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)
Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)
Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)
 
Sparkstreaming
SparkstreamingSparkstreaming
Sparkstreaming
 
Abhishek Kumar - CloudStack Locking Service
Abhishek Kumar - CloudStack Locking ServiceAbhishek Kumar - CloudStack Locking Service
Abhishek Kumar - CloudStack Locking Service
 
In Memory Analytics with Apache Spark
In Memory Analytics with Apache SparkIn Memory Analytics with Apache Spark
In Memory Analytics with Apache Spark
 
Cassandra - A decentralized storage system
Cassandra - A decentralized storage systemCassandra - A decentralized storage system
Cassandra - A decentralized storage system
 
HPC and cloud distributed computing, as a journey
HPC and cloud distributed computing, as a journeyHPC and cloud distributed computing, as a journey
HPC and cloud distributed computing, as a journey
 
Big data processing using hadoop poster presentation
Big data processing using hadoop poster presentationBig data processing using hadoop poster presentation
Big data processing using hadoop poster presentation
 
Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...
Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...
Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...
 
Hadoop cluster configuration
Hadoop cluster configurationHadoop cluster configuration
Hadoop cluster configuration
 

Recently uploaded

Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Establishing data sharing standards to promote global industry development
Establishing data sharing standards to promote global industry developmentEstablishing data sharing standards to promote global industry development
Establishing data sharing standards to promote global industry developmentThorsten Huelsmann
 
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...2toLead Limited
 
Achieving Excellence IESVE for HVAC Simulation.pdf
Achieving Excellence IESVE for HVAC Simulation.pdfAchieving Excellence IESVE for HVAC Simulation.pdf
Achieving Excellence IESVE for HVAC Simulation.pdfIES VE
 
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...ShapeBlue
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueShapeBlue
 
Python For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ emPython For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ emNho Vĩnh
 
Java Optional (Kitworks Team Study 김성호 발표)
Java Optional (Kitworks Team Study 김성호 발표)Java Optional (Kitworks Team Study 김성호 발표)
Java Optional (Kitworks Team Study 김성호 발표)Wonjun Hwang
 
Trading Software Development_ Trends to Watch in 2024.pdf
Trading Software Development_ Trends to Watch in 2024.pdfTrading Software Development_ Trends to Watch in 2024.pdf
Trading Software Development_ Trends to Watch in 2024.pdfLucas Lagone
 
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUG
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUGBoosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUG
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUGRick Ossendrijver
 
Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfMostafa Higazy
 
Large Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareLarge Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareAsma Ben Abacha
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueVM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueShapeBlue
 
AGFM - Toyota Coaster 1HZ Install Guide.pdf
AGFM - Toyota Coaster 1HZ Install Guide.pdfAGFM - Toyota Coaster 1HZ Install Guide.pdf
AGFM - Toyota Coaster 1HZ Install Guide.pdfRodneyThomas28
 
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024BookNet Canada
 
Mastering Play Store App Listing and Optimization
Mastering Play Store App Listing and OptimizationMastering Play Store App Listing and Optimization
Mastering Play Store App Listing and OptimizationAppsthentic Technology
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueCloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueShapeBlue
 
Centralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-ManagerCentralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-ManagerSaiLinnThu2
 
MuleSoft Online Meetup Group - B2B Crash Course: PM Insider Lecture
MuleSoft Online Meetup Group - B2B Crash Course: PM Insider LectureMuleSoft Online Meetup Group - B2B Crash Course: PM Insider Lecture
MuleSoft Online Meetup Group - B2B Crash Course: PM Insider LectureManik S Magar
 

Recently uploaded (20)

Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Establishing data sharing standards to promote global industry development
Establishing data sharing standards to promote global industry developmentEstablishing data sharing standards to promote global industry development
Establishing data sharing standards to promote global industry development
 
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...
 
Achieving Excellence IESVE for HVAC Simulation.pdf
Achieving Excellence IESVE for HVAC Simulation.pdfAchieving Excellence IESVE for HVAC Simulation.pdf
Achieving Excellence IESVE for HVAC Simulation.pdf
 
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
 
Python For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ emPython For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ em
 
Java Optional (Kitworks Team Study 김성호 발표)
Java Optional (Kitworks Team Study 김성호 발표)Java Optional (Kitworks Team Study 김성호 발표)
Java Optional (Kitworks Team Study 김성호 발표)
 
Trading Software Development_ Trends to Watch in 2024.pdf
Trading Software Development_ Trends to Watch in 2024.pdfTrading Software Development_ Trends to Watch in 2024.pdf
Trading Software Development_ Trends to Watch in 2024.pdf
 
Sue Loth: Job Search Strategies using personal connections
Sue Loth: Job Search Strategies using personal connectionsSue Loth: Job Search Strategies using personal connections
Sue Loth: Job Search Strategies using personal connections
 
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUG
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUGBoosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUG
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUG
 
Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdf
 
Large Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareLarge Language Models and Applications in Healthcare
Large Language Models and Applications in Healthcare
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueVM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
 
AGFM - Toyota Coaster 1HZ Install Guide.pdf
AGFM - Toyota Coaster 1HZ Install Guide.pdfAGFM - Toyota Coaster 1HZ Install Guide.pdf
AGFM - Toyota Coaster 1HZ Install Guide.pdf
 
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
 
Mastering Play Store App Listing and Optimization
Mastering Play Store App Listing and OptimizationMastering Play Store App Listing and Optimization
Mastering Play Store App Listing and Optimization
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueCloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
 
Centralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-ManagerCentralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-Manager
 
MuleSoft Online Meetup Group - B2B Crash Course: PM Insider Lecture
MuleSoft Online Meetup Group - B2B Crash Course: PM Insider LectureMuleSoft Online Meetup Group - B2B Crash Course: PM Insider Lecture
MuleSoft Online Meetup Group - B2B Crash Course: PM Insider Lecture
 

Containerized Data Persistence on Mesos

  • 1. Containerized Data Persistence on Mesos with Kafka, MySQL, Cassandra, HDFS and More!
  • 2. CEO of Elodina, Inc. Elodina http://www.elodina.net/ is a startup focusing on the support & maintenance of third party open source software (like Mesos frameworks) and offering SaaS based solutions for those systems. Elodina started as Big Data Open Source Security http://stealth.ly and has been working for the last couple of years on implementing and assisting organizations with their Kafka, Mesos, Hadoop, Cassandra, Accumulo, Storm, Spark, etc, Big Data systems. Twitter: https://twitter.com/allthingshadoop LinkedIn: https://www.linkedin.com/in/charmalloc Joe Stein
  • 3. ◉ File systems, databases, object stores, storage solutions, etc. ◉ Apache Mesos and the Datacenter Operating System. ◉ Kafka, Cassandra, MySQL, HDFS. Overview
  • 4. File Systems - manages space - directories - file names - meta-data - permissions - compression - quotas
  • 5. Distributed File System - Remote block management - Replication - Streaming Data Access - Large Data Sets
  • 6. Replicated Log - Immutable Appends - Replicated Partitions - Messaging Features - Log Retention
  • 9. Storage Solutions - Storage Area Networks - Network Attached Storage - W.O.R.M. - Cold Storage
  • 10. Quick intro to Mesos
  • 18. Mesos
  • 20. Resources & Attributes The Mesos system has two basic methods to describe the slaves that comprise a cluster. One of these is managed by the Mesos master, the other is simply passed onwards to the frameworks using the cluster. --attributes='disks:sata;raid:jbod;dc:1;rack:3'
  • 21. Roles Total consumable resources per slave, in the form 'name(role):value;name(role):value...'. This value can be set to limit resources per role, or to overstate the number of resources that are available to the slave. --resources="cpus(*):8; mem(*):15360; disk(*):710534; ports(*):[31000-32000]" --resources="cpus(prod):8; cpus(stage):2 mem(*):15360; disk(*):710534; ports(*):[31000-32000]" All * roles will be detected, so you can specify only the resources that are not all roles (*). -- resources="cpus(prod):8; cpus(stage)" Frameworks bind a specific roles or any. A default roll (instead of *) can also be configured. Roles can be used to isolate and segregate frameworks.
  • 22. In coming release(s) to make things even better! MESOS-2018 Dynamic Reservations MESOS-1554 Persistent resources support for storage-like services MESOS-1279 Add resize task primitive
  • 23. Apache Kafka with Apache Mesos
  • 25. Goals we set out with ● smart broker.id assignment. ● preservation of broker placement (through constraints and/or new features). ● ability to-do configuration changes. ● rolling restarts (for things like configuration changes). ● scaling the cluster up and down with automatic, programmatic and manual options. ● smart partition assignment via constraints visa vi roles, resources and attributes.
  • 26. Scheduler ● Provides the operational automation for a Kafka Cluster. ● Manages the changes to the broker's configuration. ● Exposes a REST API for the CLI to use or any other client. ● Runs on Marathon for high availability. Executor ● The executor interacts with the kafka broker as an intermediary to the scheduler Scheduler & Executor
  • 27. CLI & REST API ● scheduler - starts the scheduler. ● add - adds one more more brokers to the cluster. ● update - changes resources, constraints or broker properties one or more brokers. ● remove - take a broker out of the cluster. ● start - starts a broker up. ● stop - this can either a graceful shutdown or will force kill it (./kafka-mesos.sh help stop) ● rebalance - allows you to rebalance a cluster either by selecting the brokers or topics to rebalance. Manual assignment is still possible using the Apache Kafka project tools. Rebalance can also change the replication factor on a topic. ● help - ./kafka-mesos.sh help || ./kafka-mesos.sh help {command}
  • 28. Launch 20 brokers in seconds ./kafka-mesos.sh add 1000..1019 --cpus 0.01 --heap 128 --mem 256 --options num.io.threads=1 ./kafka-mesos.sh start 1000..1019
  • 29. Kafka is available on DCOS https://mesosphere.com/product/ dcos install kafka dcos kafka help Mesosphere DCOS
  • 30. Apache Cassandra with Apache Mesos
  • 31. Cassandra on Mesos https://github.com/mesosphere/cassandra-mesos The Mesos scheduler is the component with the most high-level intelligence in the framework. It will need to possess the ability to bootstrap a ring and distribute the correct configuration to all subsequently started nodes. The Scheduler will also be responsible for orchestrating all tasks with regard to restarting nodes and triggering and monitoring periodic administrative tasks required by a node.
  • 32. Cassandra Scheduler ◉ Bootstrapping a ring ◉ Adding nodes to a ring ◉ Restarting a node that has crashed ◉ Providing configuration to nodes o Seed nodes, Snitch Class, JVM OPTS ◉ Scheduling and running administrative utilities o nodetool repair o nodetool cleanup ◉ Registers with a failover timeout ◉ Supports framework authentication ◉ Declines offers to resources it doesn't need ◉ Only use necessary fraction of offers ◉ Deal with lost tasks ◉ Does not rely on in-memory state ◉ Verifies supported Mesos Version ◉ Supports roles ◉ Able to provide set of ports to be used by Nodes ◉ Initial implementation will be for a static set of ports with a potential for longer term dynamic port usage.
  • 33. Cassandra Executor ◉ Monitor health of running node ◉ Use JMX Mbeans for interfacing with Cassandra Server Process ◉ Communicate results of administrative actions via StatusUpdates to scheduler when necessary ◉ Does not rely on file system state outside sandbox ◉ Pure libprocess communication with Scheduler leveraging StatusUpdate ◉ Does not rely on running on a particular slave node ◉ Data directories will be created and managed by Mesos leveraging the features provided in MESOS-1554
  • 34. Apache HDFS with Apache Mesos
  • 35. HDFS on Mesos https://github.com/mesosphere/hdfs ◉ 3 journal nodes ◉ 2 name nodes (active/standby) ◉ data nodes, lots of them! ◉ Fault tolerance more than just what Hadoop gives. ◉ Ease of configuration and distributing nodes. ◉ Elastic DFS ◉ Run multiple frameworks at a time for new solutions
  • 37. MySQL on Mesos (Apache Incubating) ◉ Open sourced by Twitter https://github.com/twitter/mysos ◉ Moving to Apache https://twitter.com/ApacheMysos ◉ Dramatically simplifies the management of a MySQL cluster: o Efficient hardware utilization through multi-tenancy (in performance- isolated containers) o High reliability through preserving the MySQL state during failure and automatic backing up to/restoring from HDFS o An automated self-service option for bringing up new MySQL clusters o High availability through automatic MySQL master failover o An elastic solution that allows users to easily scale up and down a MySQL cluster by changing the number of slave instances