This document discusses the ceph-mesos framework, which implements a Mesos scheduler and executor for Ceph. The goal is to provide RADOS services like the RADOS gateway in a Mesos cluster. The scheduler has callback modules that interact with the Mesos master and provide a REST API and static file server. The executor launches Ceph Docker containers as tasks. The framework is still in early development and future work includes improving support for host hardware selection and networking configurations to optimize Ceph performance. A video demo of ceph-mesos is available online.

1. Big Data Technologies
Ceph-Mesos Framework
Zhongyue Luo zhongyue.nah@intel.com
2nd Beijing MUG Meetup – 2015/06/28

2. SSG / STO / BDT
Content
Apache Mesos PoC
The ceph-mesos framework
Demo video

3. SSG / STO / BDT
Apache Mesos PoC
A PoC to enable Big Data PaaS based on Apache Mesos
• Myriad
• HDFS-mesos
• ceph-mesos
• A framework to scale a Ceph cluster
• Project open sourced
• CDAP (Cask Data Analysis Platform)
• PaaS layer
• Marathon
• Mesos-DNS

4. SSG / STO / BDT
Motivation for ceph-mesos
We have a team working on RGWFS
• HCFS
Ceph as a storage backend for Big Data analytics based on Mesos
• Node Managers are managed by Myriad
• Data locality is important
• Need for a framework to manage Ceph
Apache Mesos is becoming an application deployment platform
• It is not only for running Spark now

5. SSG / STO / BDT
Goal & Scope
Implement a Mesos framework for Ceph
Provide RADOS service in a Mesos cluster
• Our main target is to support RADOS gateway
• RADOS block device and Ceph file system will be supported in future releases
The scheduler will have three modules
• The callback interaction with the Master
• Static file server (libmicrohttpd)
• REST API service (libzmq IPC communication with the callbacks)
The executor will launch Ceph Docker containers as tasks
• Configuration files will be downloaded from the static file server

6. SSG / STO / BDT
Architecture
Mesos Master
ceph-mesos-scheduler
REST APIFile server
Call backs (Mesos SDK)
Mesos
Slave
ceph-mesos-executor

7. SSG / STO / BDT
“flexup” API
Mesos Master
Mesos
Slave
ceph-mesos-scheduler
REST APIFile server
Call backs (Mesos SDK)
ZMQ
ZMQ
“flexup”
ceph-mesos-executor
1
2
3
4

8. SSG / STO / BDT
Challanges
Ceph-Mesos is still v0.0.1
• HDD
• OSD’s need to run on multiple disks on host
• Journal should write on SSD
 We will implement host HDD selection
• NIC
• All data traffic must pass through 10Gb (or greater) network
 We will implement OSD NIC selection
• “flexdown” REST API, reconciliation, MACVLAN, etc.

9. SSG / STO / BDT
Check out our code on Github
https://github.com/Intel-bigdata/ceph-mesos

10. SSG / STO / BDT
Video Demo
http://v.youku.com/v_show/id_XMTI3MjMxNTU5Ng==.html
(Audio in Chinese)