Ceph and Storage Management
with openATTIC
SUSE Monthly Open Source Talks
2016-07-06
Lenz Grimmer <lenz@openattic.org>
2
openATTIC – Our Vision
─Develop an open source alternative to proprietary
storage management systems
─„Traditional” unified storage (NAS/SAN)
─Support Ceph for scale-out scenarios
─Backed with commercial support and services
3
openATTIC – Notable Recent Changes
─Removed Enterprise/Community Edition split
─Now fully under the GPLv2
─Removed requirement for CLA
─DCA (Signed-off-by) is all that's needed
─Public Jira Bug Tracker
─Public Pull requests / code reviews on BitBucket
─Development/Release Branch
─Entire code base (Backend/WebUI/Test/Docs) in one branch
4
openATTIC – What Sets us Apart?
─ Focus on data center storage management
─ Support both SAN and NAS functionality without limitations
─ Fully Open Source (GPLv2)
─ No arbitrary functional restrictions
─ Low entrance barrier for adoption
─ Based on Linux / OSS tools
─ Multiple Linux distributions (Debian/Ubuntu/Red Hat/SUSE)
─ Well-established technology stack (e.g. drivers, hardware support)
─ Broad user base
5
─Modern Web UI
─RESTful API (Software-
Defined Storage)
─Unified Storage
─NAS (NFS, CIFS, HTTP)
─SAN (iSCSI, Fibre Channel)
─LVM, XFS, ZFS, Btrfs,
ext3/4
─Volume mirroring (DRBD®)
─Multi-node support
─Monitoring (Nagios/Icinga)
built-in
─Ceph management and
monitoring (WIP)
─Development sponsored
by it-novum
openATTIC – Open Source Storage Management
6
─Backend
─Python (Django)
─Django REST Framework
(RESTful API)
─Linux tools for storage
management, e.g. LVM,
LIO, filesystem utilities,
DRBD, etc.
─Nagios/Icinga &
PNP4Nagios (Monitoring
and Graphing)
─Web Frontend
─AngularJS (JS framework)
─Bootstrap (HTML, CSS, and
JS framework)
─Uses REST API exclusively
─Automated Test Suites
─Python Unit Tests
─REST Backend Tests
(Gatling)
─WebUI Tests
(Protractor/Jasmine)
openATTIC – Components
7
openATTIC – High Level Architecture
8
openATTIC – Installation on SUSE Linux
─OBS Project filesystems:openATTIC
─Packages available for openSUSE Leap 42.1 & SLES12
─Thanks to Eric Jackson (swiftgist) for the support!
─Feedback is welcome!
9
openATTIC – Storage Development Roadmap
─ Add Disk and Storage Pool Management to the API & WebUI
─ Creating/Modifying LVM Volume Groups / MD RAID setups
─ Creating/Modifying Btrfs/ZFS Pools (incl. RAID setups)
─ Automatic discovery of disks/pools (via udev)
─ Monitoring Disk health (SMART)
─ Manage HW RAID controllers
─ Add volume mirroring support to the WebUI
─ Extend SAN functionality (more iSCSI/FC features)
─ Public Roadmap on the openATTIC Jira/Wiki to solicit
community feedback
10
openATTIC – Ceph Management Challenges
─Managing and monitoring Ceph is complex
─Many tools exist (e.g. Calamari, Intel VSM, ceph-dash)
─Limited functionality, unclear roadmaps
─Finding the best approach for managing Ceph
11
openATTIC – Ceph Management Goals
─Create a management & monitoring GUI tool
─A tool that administrators actually want to use
─That scales without becoming overwhelming
─Still should allow changes to be made elsewhere, without
becoming inconsistent
12
openATTIC – Ceph Management Implementation
─Which Ceph Management API?
─How to manage a distributed system?
─How to monitor the cluster‘s health/performance?
─How to perform remote management tasks?
─How to monitor cluster nodes in a scalable way?
13
Django REST Framework
Django Models
Client (GUI)
REST API
DB
(PostgreSQL)
nodb
Python librados / librbddjango.db.models
openATTIC – “NoDB” Django models
14
openATTIC – Current Ceph Development Status
─“NoDB” backend architecture / framework in place
─Create and map RBDs as block devices (volumes)
─Pool Management Web UI (table view)
─OSD Management Web UI (table view)
─RBD Management Web UI (table view)
─Monitor a cluster health and performance
─CRUSH Map Editor
─Support for managing multiple Ceph clusters
15
openATTIC – Attending SUSE Hackweek
─June 24th – July 1st
─https://hackweek.suse.com/
─openATTIC Team visited SUSE HQ in Nuremberg
─Implementation design decisions
─ Salt integration
─New features implemented
─ Ceph Monitoring Dashboard
─ Ceph RBD Management
16
openATTIC – Ceph Management REST API
https://wiki.openattic.org/display/OP/openATTIC+Ceph+REST+API+overview
17
openATTIC – Ceph Development Roadmap
─ Ceph Cluster Status Dashboard incl. Performance Graphs
─ Extend Pool Management
─ OSD Monitoring/Management
─ RBD Management/Monitoring
─ CephFS Management
─ RGW Management (users, buckets keys)
─ Deployment, remote configuration of Ceph nodes (via Salt)
─ Public Roadmap on the openATTIC Wiki to solicit community
feedback: http://bit.ly/28PCTWf
18
openATTIC – Storage Management Dashboard
19
openATTIC – Volume Management
20
openATTIC – API Recorder
21
openATTIC – Ceph Pool List
22
openATTIC – Ceph OSD List
23
openATTIC – Ceph RBD list
24
openATTIC – Create Ceph RBD
25
openATTIC – Ceph CRUSH map editing
26
─ www.openattic.org
─ demo.openattic.org
─ blog.openattic.org
─ docs.openattic.org
─bitbucket.org/openattic
─tracker.openattic.org
─@openATTIC
─openATTIC
openATTIC – Resources
SUSE CERTIFIED LINUX PROFESSIONAL ACADEMY
Technical Training for SUSE Linux Enterprise Server 12
No Cost Flexible 12-Week Online Training
This popular training model is for technical sales/engineering employees of
SUSE partners interested in learning to install and manage SUSE Linux
Enterprise Server 12 (SLES 12). Participation in our SUSE Academy also
offers no-cost exam vouchers for participants seeking SUSE certification.
Learn through self-study of the SLES 12 curriculum and weekly webinar
reviews with our field-experienced instructor.
Each Academy participant will need 5-7 hours per week to self-study the
SLES 12 course assignments, plus 1 hour each week to attend a webinar
review of the key concepts studied. Certification is not required to participate.
Contact your company’s Partner Executive to ask to be sponsored in our
next Academy.
28
Questions / Discussion
29
Thank you!

openATTIC & Ceph Management @ Suse Monthly Open Source Talks - 2016-06-07

  • 1.
    Ceph and StorageManagement with openATTIC SUSE Monthly Open Source Talks 2016-07-06 Lenz Grimmer <lenz@openattic.org>
  • 2.
    2 openATTIC – OurVision ─Develop an open source alternative to proprietary storage management systems ─„Traditional” unified storage (NAS/SAN) ─Support Ceph for scale-out scenarios ─Backed with commercial support and services
  • 3.
    3 openATTIC – NotableRecent Changes ─Removed Enterprise/Community Edition split ─Now fully under the GPLv2 ─Removed requirement for CLA ─DCA (Signed-off-by) is all that's needed ─Public Jira Bug Tracker ─Public Pull requests / code reviews on BitBucket ─Development/Release Branch ─Entire code base (Backend/WebUI/Test/Docs) in one branch
  • 4.
    4 openATTIC – WhatSets us Apart? ─ Focus on data center storage management ─ Support both SAN and NAS functionality without limitations ─ Fully Open Source (GPLv2) ─ No arbitrary functional restrictions ─ Low entrance barrier for adoption ─ Based on Linux / OSS tools ─ Multiple Linux distributions (Debian/Ubuntu/Red Hat/SUSE) ─ Well-established technology stack (e.g. drivers, hardware support) ─ Broad user base
  • 5.
    5 ─Modern Web UI ─RESTfulAPI (Software- Defined Storage) ─Unified Storage ─NAS (NFS, CIFS, HTTP) ─SAN (iSCSI, Fibre Channel) ─LVM, XFS, ZFS, Btrfs, ext3/4 ─Volume mirroring (DRBD®) ─Multi-node support ─Monitoring (Nagios/Icinga) built-in ─Ceph management and monitoring (WIP) ─Development sponsored by it-novum openATTIC – Open Source Storage Management
  • 6.
    6 ─Backend ─Python (Django) ─Django RESTFramework (RESTful API) ─Linux tools for storage management, e.g. LVM, LIO, filesystem utilities, DRBD, etc. ─Nagios/Icinga & PNP4Nagios (Monitoring and Graphing) ─Web Frontend ─AngularJS (JS framework) ─Bootstrap (HTML, CSS, and JS framework) ─Uses REST API exclusively ─Automated Test Suites ─Python Unit Tests ─REST Backend Tests (Gatling) ─WebUI Tests (Protractor/Jasmine) openATTIC – Components
  • 7.
    7 openATTIC – HighLevel Architecture
  • 8.
    8 openATTIC – Installationon SUSE Linux ─OBS Project filesystems:openATTIC ─Packages available for openSUSE Leap 42.1 & SLES12 ─Thanks to Eric Jackson (swiftgist) for the support! ─Feedback is welcome!
  • 9.
    9 openATTIC – StorageDevelopment Roadmap ─ Add Disk and Storage Pool Management to the API & WebUI ─ Creating/Modifying LVM Volume Groups / MD RAID setups ─ Creating/Modifying Btrfs/ZFS Pools (incl. RAID setups) ─ Automatic discovery of disks/pools (via udev) ─ Monitoring Disk health (SMART) ─ Manage HW RAID controllers ─ Add volume mirroring support to the WebUI ─ Extend SAN functionality (more iSCSI/FC features) ─ Public Roadmap on the openATTIC Jira/Wiki to solicit community feedback
  • 10.
    10 openATTIC – CephManagement Challenges ─Managing and monitoring Ceph is complex ─Many tools exist (e.g. Calamari, Intel VSM, ceph-dash) ─Limited functionality, unclear roadmaps ─Finding the best approach for managing Ceph
  • 11.
    11 openATTIC – CephManagement Goals ─Create a management & monitoring GUI tool ─A tool that administrators actually want to use ─That scales without becoming overwhelming ─Still should allow changes to be made elsewhere, without becoming inconsistent
  • 12.
    12 openATTIC – CephManagement Implementation ─Which Ceph Management API? ─How to manage a distributed system? ─How to monitor the cluster‘s health/performance? ─How to perform remote management tasks? ─How to monitor cluster nodes in a scalable way?
  • 13.
    13 Django REST Framework DjangoModels Client (GUI) REST API DB (PostgreSQL) nodb Python librados / librbddjango.db.models openATTIC – “NoDB” Django models
  • 14.
    14 openATTIC – CurrentCeph Development Status ─“NoDB” backend architecture / framework in place ─Create and map RBDs as block devices (volumes) ─Pool Management Web UI (table view) ─OSD Management Web UI (table view) ─RBD Management Web UI (table view) ─Monitor a cluster health and performance ─CRUSH Map Editor ─Support for managing multiple Ceph clusters
  • 15.
    15 openATTIC – AttendingSUSE Hackweek ─June 24th – July 1st ─https://hackweek.suse.com/ ─openATTIC Team visited SUSE HQ in Nuremberg ─Implementation design decisions ─ Salt integration ─New features implemented ─ Ceph Monitoring Dashboard ─ Ceph RBD Management
  • 16.
    16 openATTIC – CephManagement REST API https://wiki.openattic.org/display/OP/openATTIC+Ceph+REST+API+overview
  • 17.
    17 openATTIC – CephDevelopment Roadmap ─ Ceph Cluster Status Dashboard incl. Performance Graphs ─ Extend Pool Management ─ OSD Monitoring/Management ─ RBD Management/Monitoring ─ CephFS Management ─ RGW Management (users, buckets keys) ─ Deployment, remote configuration of Ceph nodes (via Salt) ─ Public Roadmap on the openATTIC Wiki to solicit community feedback: http://bit.ly/28PCTWf
  • 18.
    18 openATTIC – StorageManagement Dashboard
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
    25 openATTIC – CephCRUSH map editing
  • 26.
    26 ─ www.openattic.org ─ demo.openattic.org ─blog.openattic.org ─ docs.openattic.org ─bitbucket.org/openattic ─tracker.openattic.org ─@openATTIC ─openATTIC openATTIC – Resources
  • 27.
    SUSE CERTIFIED LINUXPROFESSIONAL ACADEMY Technical Training for SUSE Linux Enterprise Server 12 No Cost Flexible 12-Week Online Training This popular training model is for technical sales/engineering employees of SUSE partners interested in learning to install and manage SUSE Linux Enterprise Server 12 (SLES 12). Participation in our SUSE Academy also offers no-cost exam vouchers for participants seeking SUSE certification. Learn through self-study of the SLES 12 curriculum and weekly webinar reviews with our field-experienced instructor. Each Academy participant will need 5-7 hours per week to self-study the SLES 12 course assignments, plus 1 hour each week to attend a webinar review of the key concepts studied. Certification is not required to participate. Contact your company’s Partner Executive to ask to be sponsored in our next Academy.
  • 28.
  • 29.