The document discusses implementing a high availability NAS gateway using CephFS. A 6 node Ceph cluster with 80TB of usable storage was configured with 2 gateway servers running Samba and Ganesha NFS to provide SMB and NFS shares of the CephFS filesystem. CTDB was used to enable failover between the gateways and floating IP addresses allow seamless failover. Some issues were encountered around snapshots, quotas and performance, but the overall solution provides high availability NAS capabilities using standard protocols and CephFS.

1. HA NAS with CephFSWyllys Ingersoll – Keeper Technology, LLC

2. Data and Storage Management Experts
Focus on IC & Commercial Customers for 12 Years
• Mul?-PB Enterprise Systems
• Imagery, Computer Forensics, Big Data
• High Volume/High Velocity Data Analysis
• Full Solu?on Provider
• Keeper Products + Partner Products
Introductions
Keeper Technology

3. • How we implemented HA NAS gateways using
cephfs
• Cluster configura?on
• SoPware used
• Issues encountered
• Performance Sta?s?cs
Overview
Overview
3/17/17 - 3

4. • Ceph Jewel 10.2.5 on Ubuntu-based Linux
• 6 Storage servers, ~ 80 TB Usable (3-copy)
• ~15 OSD per server
• 2, 3, and 4TB 7200RPM spinning drives (no SSD)
• 2 Gateways
• HP DL380 G9 Servers w/48GB RAM
• 3 Monitors + 3 MDS Servers
• HP DL360 G6 w/48GB RAM
Configuration
Cluster Configuration
3/17/17 - 4

5. • Provide NFS and/or SMB filesystem shares with
redundancy
• Failover if a gateway goes down
• Clients should not lose data
• Minimal interrup?on of client workflow
• “Seamless” for NFSv3 – others are WiP.
• Minimum 2 gateways required
Goals
HA NAS Goals
3/17/17 - 5

6. • SAMBA 4.5.5 w/CTDB support
• Built with “—with-cluster-support” flag
• CTDB is key to HA func?onality
• CTDB = Clustered Trivial Database
• Node monitor, failover, IP takeover
• Define mul?ple floa?ng IP Addresses in DNS
• CTDB configured with virtual shared IPs and real IP of
each GW.
• CTDB nodes communicate on private network
• Insecure protocol
Software
SMB
3/17/17 - 6

7. • Ganesha NFS 2.4.3
• User space NFS service, replaces kernel NFS
• Building code from github repo code
• Store ganesha config on shared FS
• Ex: /cephfs/nfs/ganesha.conf
• HA gateways must have common NFS Export IDs
• Use “VFS” FSAL (not “CEPH”) for Ganesha exports
Software
NFS
3/17/17 - 7

8. • Single FS per cluster (for now) - /cephfs
• Disable snapshots or restrict to top level
• Hard-linking bug prevents reliable snapshots on subdirs
• Prefer kernel mounted over fuse for performance
• Kernel 4.8.10
• Each export is a subdirectory
• /cephfs/exports/foobar
Software
NAS with Cephfs
3/17/17 - 8

9. Data Flow
3/17/17 - 9
10GB
Ceph
GW-1
GW-2
GW-3
Virtual IP
Client
1
Client
2
Client
3
SMB
CTDB
CTDB
CTDB
10GB Network

10. • SAMBA locks stored on shared FS
• Ex: /cephfs/ctdb
• CTDB monitors SAMBA and Ganesha services
• Starts and stops as necessary via “callout” scripts
• CTDB assigns virtual IP addrs as needed
Software
NAS With Cephfs
3/17/17 - 10

11. • Kernel support for cephfs varies
• Using “bleeding edge” kernels for best results
• Cannot set quotas on subdirectories
• kernel cephfs limita?on
• Cannot limit size available for a single export
• Each share has max size = en?re cephfs data pool
• Snapshots only at top level
• Cannot snapshot each exported subdirectory
Issues
Issues and Problems
3/17/17 - 11

12. • mds_cache_size = 8,000,000 (default was 100k)
• Uses more RAM, but we have 48GB
• Avoid “failing to respond to cache pressure” errors
• Use “default” crush tunables (not “jewel”).
• Works beqer with older kernels
Issues
Adjustments
3/17/17 - 12

13. • FIO parameters
• Vary block sizes (4K, 64K, 1m, 4m)
• Vary # of jobs (1, 16, 32, 64)
• Iodepth = 1
• Read/write + randread/randwrite
• Ioengine = sync
• direct=1
• Simultaneous 4 dis?nct clients on 10GB link
Performance
Test Methodology
3/17/17 - 13

14. Performance
Performance Configuration
3/17/17 - 14
NAS Gateway
Client
1
Client
2
Client
3
Client
4
/cephfs
Shared FS (/cephfs)
mounted with NFS or SMB
/cephfs
/cephfs
/cephfs
/cephfs
FIO directIO used on each client
to read & write data to the share.
Ceph Cluster

15. Performance
NFS Read (kernel mount)
3/17/17 - 15

16. Performance
NFS Writes (kernel mount)
3/17/17 - 16

17. Performance
NFS Read (Fuse mount)
3/17/17 - 17

18. Performance
NFS Write (Fuse mount)
3/17/17 - 18

19. Performance
SMB Read (kernel mount)
3/17/17 - 19

20. Performance
SMB Write (kernel mount)
3/17/17 - 20

21. Performance
SMB Read (fuse mount)
3/17/17 - 21

22. Performance
SMB Write (fuse mount)
3/17/17 - 22

23. • High Availability NAS is possible with cephfs
• Some issues remain
• Deep snapshots
• Quotas/limits on subdirs
• Performance is “OK”
• Standard NAS protocol limita?ons (NFS, SMB)
Summary
Summary
3/17/17 - 23

24. Thank You
| 21740 Beaumeade Circle | Suite 150 | Ashburn, VA 20147 | P [571] 333 2725 | F [703] 738 7231 | solu?ons@keepertech.com | www.keepertech.com
Thank You