This document discusses using OpenStack Manila to provide NFS shares backed by CephFS storage. It describes deploying this solution using TripleO to enable production-quality CephFS with NFS. Key steps include customizing the TripleO deployment to add an isolated StorageNFS network, deploying NFS Ganesha under Pacemaker for high availability, and post-deployment tasks for the cloud administrator such as creating share types and networks. The workflow for cloud users to create VMs, attach them to the StorageNFS network, and mount the NFS shares is also outlined.

1. Practical CephFS with NFS today
using OpenStack Manila
Tom Barron
Ceph Day Berlin
12 November 2018

2. About me
● PTL for upstream manila project since Rocky
● Work downstream for Red Hat on OpenStack Storage
○ Co-ordinated with Ceph, Ganesha, Openstack HA, and OpenStack TripleO teams on the CephFS
with NFS solution presented here
● Feel free to follow up:
○ Email: tbarron@redhat.com
○ irc: tbarron
■ #openstack-manila

3. Agenda
● The target: CephFS with NFS using OpenStack Manila
○ What?
○ Why?
● Development/Experimental vs Production Deployment
○ Isolated data center networks
○ Fault Tolerance
● Deploying with TripleO
● Post-deployment tasks for the Cloud administrator
● Cloud user workflow to use manila shares with NFS protocol backed by CephFS
● Next ...

4. Target: Use TripleO to deploy CephFS back
end for OpenStack Manila NFS shares

5. Use work from these open source projects to enable real-world,
production quality deployment of OpenStack Manila with CephFS back
end storage exposed as NFS shares
Manila

6. CephFS: Use Ceph for File Shares as a Service
in OpenStack

7. CephFS
RGW
S3 and Swift compatible object
storage with object versioning,
multi-site federation, and
replication
LIBRADOS
A library allowing apps to direct access RADOS (C, C++, Java, Python, Ruby, PHP)
RADOS
A software-based, reliable, autonomic, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes (OSDs) and lightweight monitors (Mons)
RBD
A virtual block device with
snapshots, copy-on-write clones,
and multi-site replication
CEPHFS
A distributed POSIX file system
with coherent caches and
snapshots on any directory
OBJECT BLOCK FILE

8. Manila: OpenStack tenant-aware abstraction
layer for file share management

9. Tenant B
OpenStack Storage Model
■ Tenant (keystone project, user) aware self-service storage
■ Abstracts the actual physical storage back ends
■ Multiple protocols available: NFS, CIFS, CephFS, GlusterFS, HDFS, ...
■ Standard REST API for managing life-cycle of and access-control for
shares
Tenant A

10. TripleO: deploy real cloud infrastructure!

11. development/prototyping environment vs production cloud
● Ansible playbook to build devstack using vagrant with libvirt/KVM, deploying
manila with CephFS native or CephFS with NFS back end
○ https://github.com/tombarron/vagrant-libvirt-devstack
● But what if you want:
○ Fault tolerant service processes
○ Survive failure of any one hardware node
○ Scale out compute
○ Scale out storage
● TripleO!
○ There are other deployment tools out there of course
○ We can help you if you want to make those work for manila with CephFS via NFS

12. Ganesha: expose CephFS shares via NFS.

13. NFS Ganesha
■ User-space NFSv2, NFSv3, NFSv4, NFSv4.1 and pNFS server
■ Modular architecture: Pluggable FSAL allow for various storage backend (e.g. vfs,
xfs, glusterfs, cephfs)
■ Dynamic export/unexport/update with DBUS
■ Can manage huge metadata caches
■ Simple access for other user-space services (e.g. KRB5, NIS, LDAP)
■ Open source

14. https://www.openstack.org/user-survey/survey-2017
14
Most Openstack users are
also running a Ceph cluster
already
Open source storage
solution
CephFS metadata
scalability is ideally suited
to cloud environments.
There’s a perfectly good native CephFS
solution for Manila

15. CephFS native driver deployment with TripleO
Public OpenStack Service API (External) network
Storage (Ceph public) network
External
Provider
Network
Storage
Provider
Network
Router Router
Tenant VMs with 2 nics
Manila
Share
service
Ceph MON
Ceph MDS
Ceph OSD Ceph OSD
Ceph OSD
Controller
Nodes
Storage nodes
Tenant A Tenant B
Compute Nodes
Manila API
service

16. Why NFS Ganesha?
■ If you want NFS backed by an open source storage technology
■ If you want to leverage an existing Ceph deployment while keeping
your NFS shares
■ Ubiquitous, well-understood client software
■ Familiar IP based access control
■ Allows clear separation between trusted cloud administrators and
untrusted guests

17. CephFS NFS driver deployment with TripleO
Public OpenStack Service API (External) network
Storage (Ceph public) network
External
Provider
Network
Storage NFS
Network
Router Router
Tenant VMs with 2 nics
Manila
Share
service
Ceph MON
Ceph MDS
Ceph OSD Ceph OSD
Ceph OSD
Controller
Nodes
Storage nodes
Tenant A Tenant B
Compute Nodes
Manila API
service

18. NFS Ganesha deployment challenges
■ Ceph MON, MDS, MGR, OSDs manage their own HA
■ But not NFS-Ganesha
■ Only one NFS-Ganesha instance can run at a time!
■ But we cannot have a SPOF in the data path
■ So we need to run NFS-ganesha under control of
pacemaker-corosync
○ Expose exports via a VIP
○ Migrate the service to a new node as required

19. TripleO Deployment

20. Standard TripleO topology
■ Undercloud node running its own specialized OpenStack
■ Three Controller Nodes
■ M x Compute Nodes
■ N X Storage Nodes

21. Queens +
■ No undercloud deployment changes
■ New containers
■ Custom controller role
■ Custom isolated network
■ New environment files
■ Uses Ceph Luminous
■ NFS ganesha 2.5 latest

22. Install ceph-ansible on undercloud
■ Deploy undercloud as normal
■ Install ceph-ansible on undercloud
■ Required step for all TripleO ceph deployments
[stack@undercloud ~]$ sudo yum install -y ceph-ansible
...
Total download size: 196 k
Installed size: 996 k
Downloading packages:
Ceph-ansible-3.1.9-1.el7.noarch.rpm
...
Installed:
ceph-ansible.noarch 0:3.1.9-1.el7
Complete!

23. Containerized Deployment
■ Queens deploys all storage service daemons in docker containers
■ So on the undercloud, be sure to include the relevant ceph and
manila environment files when preparing containers for the
overcloud.
■ Ganesha will run its own container but it uses the standard ceph
container image
[stack@undercloud ~]$ openstack overcloud container image prepare
...
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/manila.yaml
...

24. Generating the Custom Roles File
● The ControllerStorageNFS custom role is used to set up the
isolated StorageNFS network.
● This role is similar to the default Controller.yaml role file with the
addition of the StorageNFS network and the CephNfs (aka
nfs-ganesha) service.
[stack@undercloud ~]$ openstack overcloud roles generate --roles-path
/usr/share/openstack-tripleo-templates/roles -o /home/stack/roles_data.yaml
ControllerStorageNfs Compute CephStorage

25. Custom network-data-ganesha file
● By default overcloud deploy uses network definitions from
network-data.yaml in /usr/share/openstack-tripleo-heat-templates
● We instead use network-data-ganesha.yaml* which adds definitions for the
StorageNFS network
name: StorageNFS
enabled: true
vip: true
ame_lower: storage_nfs
vlan: 70
ip_subnet: '172.16.4.0/24'
allocation_pools: [{'start': '172.16.4.4', 'end': '172.16.4.250'}]
ipv6_subnet: 'fd00:fd00:fd00:7000::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:7000::10', 'end':
'fd00:fd00:fd00:7000:ffff:ffff:ffff:fffe'}]
*If you have customized network-data.yaml, make corresponding adjustments to the ganesha
equivalent.

26. Deploy the overcloud
● Use custom network (1) and roles (2) files.
● Use environment files for ceph-ansible (3), ceph-mds (4) , and
manila with nfs-ganesha (5)
[stack@undercloud ~]$ openstack overcloud deploy
--templates /usr/share/openstack-tripleo-heat-templates
-n /usr/share/openstack-tripleo-heat-templates/network_data_ganesha.yaml (1)
-r /home/stack/roles_data.yaml (2)
-e /home/stack/containers-default-parameters.yaml
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml
-e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml
-e /home/stack/network-environment.yaml
-e/usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml (3)
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-mds.yaml (4)
-e /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml (5)

27. CephFS NFS driver deployment with TripleO
Public OpenStack Service API (External) network
Storage (Ceph public) network
External
Provider
Network
Storage NFS
Network
Router Router
Tenant VMs with 2 nics
Manila
Share
service
Ceph MON
Ceph MDS
Ceph OSD Ceph OSD
Ceph OSD
Controller
Nodes
Storage nodes
Tenant A Tenant B
Compute Nodes
Manila API
service

28. Post-Deployment tasks for the
Cloud Administrator

29. Create Overcloud Neutron Storage-NFS
network
● And map it to the isolated Storage NFS network in the data center
● NFS clients (e.g. Nova VMs) will connect to ganesha over this
software defined network
(overcloud) [stack@undercloud-0 ~]$ openstack network create StorageNFS --share --provider-network-type vlan
--provider-physical-network datacentre --provider-segment 70
Create StorageNFS: by convention we use ‘StorageNFS’ as the name of the neutron SDN that maps to the
data centre isolated StorageNFS network
--share: This provider network can be shared by multiple tenants
--provider type vlan: We define the isolated data center StorageNFS network as vlan 70
--provider-physical-network datacentre: Name of the physical network on which the isolated StorageNFS
network was defined

30. Create Overcloud Neutron Storage-NFS
sub-network
● Create subnet on the StorageNFS neutron network
● Give it a DHCP server, using an allocation pool compatible with
that defined for the undercloud’s StorageNFS allocation pool
● No gateway/default route is needed since this network will only be
used for NFS mounts
● NFS clients (e.g. Nova VMs) will connect to ganesha over this
software defined network
(overcloud) [stack@undercloud-0 ~]$ openstack subnet create --allocation-pool start=172.16.4.150,end=172.16.4.250
--dhcp --network StorageNFS --subnet-range 172.16.4.0/24 --gateway none StorageNFSSubnet

31. Create default share type
● Manila needs a default-share type, which is used when creating
shares if an explicit share-type argument is not supplied
● TripleO deploys manila configured to expect a default share type
named ‘default’ but it does not itself create the share type
● So the cloud administrator needs to create it:
(overcloud) [stack@undercloud-0 ~]$ manila type-create default False
● The manila type-create command requires that the implicit DHSS
field be set
● “DHSS” means “driver handles share servers.” In our case the
share server is implemented by ganesha and its life cycle is
controlled by TripleO rather than by the manila driver, so we set it
to False

32. Cloud-user workflow

33. Create security group
● One-time task that isolates a tenant’s VMs from others attaching to the
Storage NFS network
● Content of this security group is the same as the original content of the
‘default’ security group but the latter has often been changed so we make a
new security group just to be safe:
(user) [stack@undercloud-0 ~]$ openstack security group create no-ingress
● As suggested by its name, this group allows egress packets but no ingress
packets from unestablished connections
● Cloud-administrator can do this specifying ‘--project’ for each tenant to make
cloud-user workflow simpler.

34. Create port on StorageNFS network
● Per VM (nova instance) task
● Use the no-ingress security group just created when creating the port
(user) [stack@undercloud-0 ~]$ openstack port create nfs-port0 --network StorageNFS
--security-group no-ingress
● Neutron will assign an IP address to this new port from the allocation-range
set up for the StorageNFS network, set up DHCP to serve that address when
an interface is bound to this port, and ensure that an interface can only use
this address if it is bound to this port.

35. Add the port to Nova VM
(user) [stack@undercloud-0 ~]$ openstack server add port instance0 nfs-port0
(user) [stack@undercloud-0 ~]$ openstack server list -f yaml -
Flavor: m1.micro
ID: 0b878c11-e791-434b-ab63-274ecfc957
Image: manila-test
Name: demo-instance
Networks: demo-network=172.20.0.4, 10.0.0.53; StorageNFS=172.17.5.160
Status: ACTIVE
● Before the port was added, server instance0 had a private address 172.20.0.4 and a
floating public IP 10.0.0.53.
● After the port is added, the server also has assigned to it an address on the
StorageNFS network, 172.17.5.160

36. Activate the StorageNFS address on the
Nova VM
● Assigning a port to an OpenStack compute instance reserves an IP for it and
sets up DHCP for it but does not perform any actions on the compute instance
itself to activate a new interface with the IP.
● Procedures to do this last part are compute-instance image specific.
● If the second interface does not already exist in the image it must be created
and configured.
● Then the interface must be toggled or networking restarted (or the VM
rebooted) so that the interface actually comes up with the Storage NFS IP.

37. Allow access to manila shares from the
Nova VM at its StorageNFS address
● share-01 is created using the default share-type, NFS protocol, and is 2
gigabytes
● Access is allowed to share-01 from IP address 172.17.5.169
● This is the StorageNFS network IP assigned to compute instance instance0
two slides back
(user) [stack@undercloud-0 ~]$ manila create --name share-01 nfs 2
(user) [stack@undercloud-0 ~]$ manila access-allow share-01 ip 172.17.5.160

38. List share’s export locations
● 172.17.5.13 is the IP at which the ganesha server is listening for mount requests
● The string at the right of the colon is the export path to share with uuid
e840b4ae-6a04-49ee-9d6e-67d4999fbc01
(user) [stack@undercloud-0 ~]$ manila share-export-location-list share-01
172.17.5.13:/volumes/_nogroup/e840b4ae-6a04-49ee-9d6e-67d4999fbc01
● The manila share-export-location-list command reveals export locations to be used in mount
commands to mount a share.

39. Mount share on compute instance
● Login to the compute instance:
(user) [stack@undercloud-0 ~]$ openstack server ssh demo-instance0 --login root
# hostname
demo-instance-o
● Mount the share using the export location infromation from the previous slide:
# mount.nfs -v 172.17.5.13:/volumes/_nogroup/e840b4ae-6a04-49ee-9d6e-67d4999fbc01 /mnt
mount.nfs: timeout set for Wed Sep 19 09:14:46 2018
mount.nfs: trying text-based options 'vers=4.2,addr=172.17.5.13,clientaddr=172.17.5.160'
172.17.5.13:/volumes/_nogroup/e840b4ae-6a04-49ee-9d6e-67d4999fbc01 on /mnt type nfs
# mount | grep mnt
172.17.5.13:/volumes/_nogroup/e840b4ae-6a04-49ee-9d6e-67d4999fbc01 on /mnt type nfs4
(rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=
600,retrans=2,sec=sys,clientaddr=172.17.5.160,local_lock=none,addr=172.17.5.13)

40. Futures

41. Current CephFS NFS Driver
Pros
● Security: isolates user VMs from ceph public network and its daemons.
● Familiar NFS semantics, access control, and end user operations.
● Large base of clients who can now use Ceph storage for file shares without doing
anything different.
○ NFS supported out of the box, doesn’t need any specific drivers
● Path separation in the backend storage and network policy (enforced by neutron
security rules on a dedicated StorageNFS network) provide multi-tenancy support.

42. Current CephFS NFS Driver
Cons
● Ganesha is a “man in the middle” in the data path and a potential performance
bottleneck.
● HA using the controller node pacemaker cluster impacts our ability to scale
● As does the (current) inability to run ganesha active-active, and
● We’d like to be able to spawn ganesha services on demand, per-tenant, as required
rather than statically launching them at cloud deployment time.

43. ● High Availability
○ Kubernetes managed Ganesha container
■ Container life-cycle and resurrection not managed by Ceph.
■ ceph-mgr creates shares and launches containers through Kubernetes
● Scale-Out (avoid Single Point of Failure)
○ ceph-mgr creates multiple Ganesha containers for a share.
○ (Potentially) Kubernetes load balancer allows for automatic multiplexing between
Ganesha containers via a single service IP.
HA and Scale-Out

44. Ganesha per Tenant running under k8s control
Public OpenStack Service API (External) network
Ceph public network
External
Provider
Network
Router Router
Tenant VMs
Manila
Share
service
Ceph MON
Ceph MDS Ceph OSD Ceph OSD
Ceph OSD
Controller
Nodes Tenant A Tenant B
Compute Nodes
Manila API
service
Ceph MGR kubernetes

45. Q&A

46. Thank you!