Ceph and Storage
Management with openATTIC
Ceph Tech Talks
2016-06-23
Lenz Grimmer <lenz@openattic.org>
openATTIC – Our Vision
─ Develop an open source alternative to proprietary storage
management systems
─ „Traditional” unified storage (NAS/SAN)
─ Support Ceph for scale-out scenarios
─ Backed with commercial support and services
openATTIC – Notable Recent Changes
─ Removed Enterprise/Community Edition split
─ Now fully under the GPLv2
─ Removed requirement for CLA
─ DCA (Signed-off-by) is all that's needed
─ Public Jira Bug Tracker
─ Public Pull requests / code reviews on BitBucket
─ Development/Release Branch
─ Entire code base (Backend/WebUI/Test/Docs) in one branch
openATTIC – What Sets us Apart?
─ Focus on data center storage management
─ Support both SAN and NAS functionality without limitations
─ Fully Open Source (GPLv2)
─ No arbitrary functional restrictions
─ Low entrance barrier for adoption
─ Based on Linux / OSS tools
─ Multiple Linux distributions (Debian/Ubuntu/Red Hat/SUSE)
─ Well-established technology stack (e.g. drivers, hardware support)
─ Broad user base
─ Modern Web UI
─ RESTful API (Software-
Defined Storage)
─ Unified Storage
─ NAS (NFS, CIFS, HTTP)
─ SAN (iSCSI, Fibre Channel)
─ LVM, XFS, ZFS, Btrfs,
ext3/4
─ Volume mirroring (DRBD®)
─ Multi-node support
─ Monitoring (Nagios/Icinga)
built-in
─ Ceph management and
monitoring (WIP)
─ Development sponsored by
it-novum
openATTIC – Open Source Storage Management
─ Backend
─ Python (Django)
─ Django REST Framework
(RESTful API)
─ Linux tools for storage
management, e.g. LVM, LIO,
filesystem utilities, DRBD,
etc.
─ Nagios/Icinga & PNP4Nagios
(Monitoring and Graphing)
─ Web Frontend
─ AngularJS (JS framework)
─ Bootstrap (HTML, CSS, and
JS framework)
─ Uses REST API exclusively
─ Automated Test Suites
─ Python Unit Tests
─ REST Backend Tests
(Gatling)
─ WebUI Tests
(Protractor/Jasmine)
openATTIC – Components
openATTIC – High Level Architecture
openATTIC – Ceph Management Challenges
─ Managing and monitoring Ceph is complex
─ Many tools exist (e.g. Calamari, Intel VSM, ceph-dash)
─ Limited functionality, unclear roadmaps
─ Finding the best approach for managing Ceph
openATTIC – Ceph Management Goals
─ Create a management & monitoring GUI tool
─ A tool that administrators actually want to use
─ That scales without becoming overwhelming
─ Still should allow changes to be made elsewhere, without
becoming inconsistent
openATTIC – Ceph Management Implementation
─ Which Ceph Management API?
─ How to manage a distributed system?
─ How to monitor the cluster‘s health/performance?
─ How to perform remote management tasks?
─ How to monitor cluster nodes in a scalable way?
Django REST
Framework
Django Models
Client (GUI)
REST API
DB
(PostgreSQL)
nodb
Python librados / librbddjango.db.models
openATTIC – “NoDB” Django models
openATTIC – Current Ceph Development Status
─ “NoDB” backend architecture / framework in place
─ Create and map RBDs as block devices (volumes)
─ Pool Management Web UI (table view)
─ OSD Management Web UI (table view)
─ RBD Management Web UI (table view)
─ Monitor a cluster health and performance
─ CRUSH Map Editor
─ Support for managing multiple Ceph clusters
openATTIC – Ceph REST API Overview
openATTIC – Ceph Development Roadmap
─ Ceph Cluster Status Dashboard incl. Performance Graphs
─ Extend Pool Management
─ OSD Monitoring/Management
─ RBD Management/Monitoring
─ CephFS Management
─ RGW Management (users, buckets keys)
─ Deployment, remote configuration of Ceph nodes (via Salt)
─ Public Roadmap on the openATTIC Wiki to solicit
community feedback: http://bit.ly/28PCTWf
openATTIC – Storage Management Dashboard
openATTIC – Ceph Pool List
openATTIC – Ceph OSD List
openATTIC – Ceph CRUSH map editing
openATTIC – Volume Management
openATTIC – API Recorder
openATTIC – Attending SUSE Hackweek
─ June 24
th
– July 1st
─ https://hackweek.suse.com/
─ Pick open Jira issues
─ Join #openattic on Freenode
─ Submit your project
─ Have fun, learn new stuff
─ openATTIC Team will be in the SUSE HQ in Nuremberg
─ www.openattic.org
─ demo.openattic.org
─ blog.openattic.org
─ docs.openattic.org
─ bitbucket.org/openattic
─ tracker.openattic.org
─ @openATTIC
─ openATTIC
openATTIC – Resources
Questions / Discussion
Thank you!

openATTIC Ceph Management @ Ceph Tech Talks - 2016-06-23

  • 1.
    Ceph and Storage Managementwith openATTIC Ceph Tech Talks 2016-06-23 Lenz Grimmer <lenz@openattic.org>
  • 2.
    openATTIC – OurVision ─ Develop an open source alternative to proprietary storage management systems ─ „Traditional” unified storage (NAS/SAN) ─ Support Ceph for scale-out scenarios ─ Backed with commercial support and services
  • 3.
    openATTIC – NotableRecent Changes ─ Removed Enterprise/Community Edition split ─ Now fully under the GPLv2 ─ Removed requirement for CLA ─ DCA (Signed-off-by) is all that's needed ─ Public Jira Bug Tracker ─ Public Pull requests / code reviews on BitBucket ─ Development/Release Branch ─ Entire code base (Backend/WebUI/Test/Docs) in one branch
  • 4.
    openATTIC – WhatSets us Apart? ─ Focus on data center storage management ─ Support both SAN and NAS functionality without limitations ─ Fully Open Source (GPLv2) ─ No arbitrary functional restrictions ─ Low entrance barrier for adoption ─ Based on Linux / OSS tools ─ Multiple Linux distributions (Debian/Ubuntu/Red Hat/SUSE) ─ Well-established technology stack (e.g. drivers, hardware support) ─ Broad user base
  • 5.
    ─ Modern WebUI ─ RESTful API (Software- Defined Storage) ─ Unified Storage ─ NAS (NFS, CIFS, HTTP) ─ SAN (iSCSI, Fibre Channel) ─ LVM, XFS, ZFS, Btrfs, ext3/4 ─ Volume mirroring (DRBD®) ─ Multi-node support ─ Monitoring (Nagios/Icinga) built-in ─ Ceph management and monitoring (WIP) ─ Development sponsored by it-novum openATTIC – Open Source Storage Management
  • 6.
    ─ Backend ─ Python(Django) ─ Django REST Framework (RESTful API) ─ Linux tools for storage management, e.g. LVM, LIO, filesystem utilities, DRBD, etc. ─ Nagios/Icinga & PNP4Nagios (Monitoring and Graphing) ─ Web Frontend ─ AngularJS (JS framework) ─ Bootstrap (HTML, CSS, and JS framework) ─ Uses REST API exclusively ─ Automated Test Suites ─ Python Unit Tests ─ REST Backend Tests (Gatling) ─ WebUI Tests (Protractor/Jasmine) openATTIC – Components
  • 7.
    openATTIC – HighLevel Architecture
  • 8.
    openATTIC – CephManagement Challenges ─ Managing and monitoring Ceph is complex ─ Many tools exist (e.g. Calamari, Intel VSM, ceph-dash) ─ Limited functionality, unclear roadmaps ─ Finding the best approach for managing Ceph
  • 9.
    openATTIC – CephManagement Goals ─ Create a management & monitoring GUI tool ─ A tool that administrators actually want to use ─ That scales without becoming overwhelming ─ Still should allow changes to be made elsewhere, without becoming inconsistent
  • 10.
    openATTIC – CephManagement Implementation ─ Which Ceph Management API? ─ How to manage a distributed system? ─ How to monitor the cluster‘s health/performance? ─ How to perform remote management tasks? ─ How to monitor cluster nodes in a scalable way?
  • 11.
    Django REST Framework Django Models Client(GUI) REST API DB (PostgreSQL) nodb Python librados / librbddjango.db.models openATTIC – “NoDB” Django models
  • 12.
    openATTIC – CurrentCeph Development Status ─ “NoDB” backend architecture / framework in place ─ Create and map RBDs as block devices (volumes) ─ Pool Management Web UI (table view) ─ OSD Management Web UI (table view) ─ RBD Management Web UI (table view) ─ Monitor a cluster health and performance ─ CRUSH Map Editor ─ Support for managing multiple Ceph clusters
  • 13.
    openATTIC – CephREST API Overview
  • 14.
    openATTIC – CephDevelopment Roadmap ─ Ceph Cluster Status Dashboard incl. Performance Graphs ─ Extend Pool Management ─ OSD Monitoring/Management ─ RBD Management/Monitoring ─ CephFS Management ─ RGW Management (users, buckets keys) ─ Deployment, remote configuration of Ceph nodes (via Salt) ─ Public Roadmap on the openATTIC Wiki to solicit community feedback: http://bit.ly/28PCTWf
  • 15.
    openATTIC – StorageManagement Dashboard
  • 16.
  • 17.
  • 18.
    openATTIC – CephCRUSH map editing
  • 19.
  • 20.
  • 21.
    openATTIC – AttendingSUSE Hackweek ─ June 24 th – July 1st ─ https://hackweek.suse.com/ ─ Pick open Jira issues ─ Join #openattic on Freenode ─ Submit your project ─ Have fun, learn new stuff ─ openATTIC Team will be in the SUSE HQ in Nuremberg
  • 22.
    ─ www.openattic.org ─ demo.openattic.org ─blog.openattic.org ─ docs.openattic.org ─ bitbucket.org/openattic ─ tracker.openattic.org ─ @openATTIC ─ openATTIC openATTIC – Resources
  • 23.
  • 24.