Presentation on upcoming feature for Ceph Pacific release -- CephFS Snapshot Mirroring.

1. CephFS Snapshot Mirroring

2. Goal
 asynchronous
 cephfs-mirror daemon(s)
 unidirectional
 user (de)selects directories to mirror
 HA, scale out

3. Overview, components
 manager module
 user interface
 mirror daemon(s)
 per cluster

4. Overview, interface
 mgr/mirroring
 ceph fs snapshot mirror …
 enable/disable
 peer_add/peer_remove
 add/remove
 dirmap/show distribution
 <remote>: <client>@<cluster>

5. Overview, mirror daemon
 Create user in primary cluster
 ceph auth get-or-create client.mirror mon 'allow r' mds 'allow r' osd
'allow rw tag cephfs metadata=*, allow r tag cephfs data=*' mgr 'allow r'
 create user in secondary cluster
 ceph fs authorize <fs_name> client.mirror_remote / rwps
 start mirror daemon
 systemctl enable cephfs-mirror@mirror
 systemctl start cephfs-mirror@mirror

6. Design, Tracking Peers
 peers tracked in fsmap
 ClusterInfo : client_name, cluster_name, fs_name
 Peer : UUID, ClusterInfo
 MirrorInfo : Peers
 Filesystem : MirrorInfo

7. Design, Index Objects
 global index object in metadata pool
 cephfs_mirror
 created when mirroring is enabled for a filesystem
 mirror daemon discovery (via RADOS watch/notify)
 per mirror daemon instance object in metadata pool
 cephfs_mirror.<instance_id>
 assign/release a directory (via RADOS watch/notify)

8. Design, mgr/mirroring
 assign directories to mirror instances
 M/N policy (M: number of directories, N: number of mirror
instances)
 persist mapping in omap (cephfs_mirror object)
 acquire/release notify to mirror instances
 shuffle mapping when required

9. Design, mirror daemon
 subscribe to fsmap
 get notified on peer changes
 establish watch on global object
 respond to notify call
 establish watch on instance object
 handle acquire/release calls

10. Design, mirror daemon (cotd..)
 snapshot synchronization
 bulk data copy to remote filesystem
 buggy rstats: https://github.com/ceph/ceph/pull/32126 (“mds: track global rstat
progagation”)
 snapshot via ceph_mksnap()
 /**
 * Create a snapshot
 * …...
 * @param snap_metadata array of snap metadata structs
 * @param nr_snap_metadata number of snap metadata struct entries
 */
 int ceph_mksnap(struct ceph_mount_info *cmount, const char *path, const char *name,
 mode_t mode, struct snap_metadata *snap_metadata, size_t nr_snap_metadata);

11. Design, mirror daemon (cotd..)
 snapshot synchronization (cotd..)
 snap-id (directory snap in primary filesystem) persisted in snapshot
metadata in target
 allows identifying snap renames and recreates

12. Pacific
 support single mirror daemon instance
 fix memory leaks, cleanups
 https://github.com/ceph/ceph/pull/39138 (“cephfs-mirror: run mirror daemon
with valgrind”)
 register with service map
 ancestor/subtree mirroring? – https://tracker.ceph.com/issues/48953 (“cephfs-
mirror: support snapshot mirror of subdirectories and/or ancestors of a mirrored
directory”)
 Future
 https://tracker.ceph.com/issues/47034 (“mds: readdir for snapshot diff”)

13. Q & A