Email storage at Deutsche Telekom is migrating from NFS to Ceph object storage using a new open source Ceph plugin for Dovecot called librmb. Emails and metadata will be stored directly in RADOS namespaces and accessed via the librados library. A proof-of-concept is being tested using 9 SSD nodes for CephFS metadata and 12 HDD nodes for email storage, laid out across 3 fire compartments for redundancy. The goal is to replace the NFS solution and store over 1.3 petabytes of email for 39 million customer accounts on a scale-out Ceph cluster using erasure coding for reliability and reduced costs.

1. Email Storage
with
Ceph
Deutsche Telekom AG
Danny Al-Gaaf

2. TelekomMail platform

3. TelekomMail
DT's mail platform for customers
dovecot
Network-Attached Storage (NAS)
NFS (sharded)
~1.3 petabyte net storage
~39 million accounts

4. NFS Operations

5. NFS Traffic

6. NFS Statistics
~42% usable raw space
NFS IOPS
max: ~835,000
avg: ~390,000
relevant IO:
WRITE: 107,700 / 50,000
READ: 65,700 / 30,900

7. Email Statistics
6.7 billion emails
1.2 petabyte net
compression
1.2 billion index/cache/metadata files
avg: 24 kiB
max: ~600 MiB

8. Email Distribution

9. How are emails stored?
Emails are written once, read many (WORM)
Usage depends on:
protocol (IMAP vs POP3)
user frontend (mailer vs webmailer)
usually separated metadata, caches and indexes
lost of metadata/indexes is critical
without attachments easy to compress

10. Where to store emails?
Filesystem
maildir
mailbox
Database
SQL
Objectstore
S3
Swift

11. Ceph

12. RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self managing, intelligent storage nodes
LIBRADOS
A library allowing
apps to directly
access RADOS, with
support for C, C++,
Java, Python, Ruby
and PHP
RADOSGW
A bucket-based REST
Gateway, compatible with
S3 and Swift
RBD
A reliable and fully-
distributed block device,
with a Linux kernel client
and a QEMU/KVM driver
CEPH FS
A POSIX-compliant
distributed file system,
with a Linux kernel client
and support for FUSE
CLIENTHOST/VMAPPAPP

13. Motivation
Scale-out vs Scale-up
Fast self healing
Commodity hardware
Prevent vendor lock-in
Open Source where feasible
Reduce Total Cost of Ownership (TCO)

14. Ceph Options
Filesystem
CephFS
NFS Gateway via RGW
Any filesystem on RBD
Objectstore
S3/Swift via RGW
RADOS

15. Where to store in Ceph?
CephFS
same issues as NFS
mail storage on POSIX layer adds complexity
no option for emails
usable for metadata/caches/indexes
M
M
RADOS CLUSTER
LINUX HOST
KERNEL MODULE
01
10 metadatadata

16. Where to store in Ceph?
RBD
needs sharding and large RBDs
needs account migration
needs RBD/fs extend scenarios
no sharing between clients
impracticable
M
M
RADOS CLUSTER
HYPERVISOR
LIBRBD
VM

17. Where to store in Ceph?
RadosGW
can store emails as objects
extra network hops
potential bottleneck
very likely not fast enough
M
M
RADOS CLUSTER
APPLICATION
RADOSGW
LIBRADOS
REST
socket

18. Where to store in Ceph?
Librados
direct access to RADOS
parallel I/O
not optimized for emails
how to handle metadata/caches/indexes?
M
M
RADOS CLUSTER
APP
LIBRADOS
APP
LIBRADOS

19. Dovecot and Ceph

20. Dovecot
Open source project (LGPL 2.1, MIT)
72% market share (openemailsurvey.org, 02/2017)
Objectstore plugin available (obox)
supports only REST APIs like S3/Swift
not open source
requires Dovecot Pro
large impact on TCO

21. Dovecot Pro obox Plugin
IMAP4 / POP3 / LMTP process
Storage API
dovecot obox backend
metacacheRFC 5322 mails
fs API
fscache backend
object store backend
RFC 5322 objects
object store
index & cache bundles
sync local
index & cache with
object store
write index & cache
to local store
local storage
mail cache
local storage

22. DT's approach
no open source solution on the market
closed source is no option
develop / sponsor a solution
open source it
partner with:
Wido den Hollander (42on.com) for consulting
Tallence AG for development
SUSE for Ceph

23. Ceph plugin for Dovecot
First Step: hybrid approach
Emails
Store in RADOS Cluster
Metadata and indexes
Store in CephFS
Be as generic as possible
Split out code into libraries
Integrate into corresponding upstream projects
Mail User Agent
RADOS CLUSTER
Ceph Client
Dovecot
rbox storage plugin
librmb CephFS
librados Linux Kernel
IMAP/POP

24. Librados mailbox (librmb)
Generic email abstraction on top of librados
Out of scope:
User data and credential storage
target are huge installations where usually are already solutions in place
Full text indexes
There are solutions already available and working outside email storage

25. Librados mailbox (librmb)
M
M
RADOS CLUSTER
M
M
RADOS CLUSTER
RFC5322
CACHE
LOCAL STORAGE
IMAP4 / POP3 / LMTP process
Storage API
dovecot rbox backend
librmb
librados
dovecot lib-index
CephFS
Linux kernel
RFC 5322 mails
RFC 5322 objects index & metadata

26. librmb - Mail Object Format
Mails are immutable regarding the RFC-5322 content
RFC-5322 content stored in RADOS directly
Immutable attributes used by Dovecot stored in RADOS xattr
rbox format version
GUID
Received and save date
POP3 UIDL and POP3 order
Mailbox GUID
Physical and virtual size
Mail UID
writable attributes are stored in Dovecot index files

27. Dump email details from RADOS
$> rmb -p mail_storage -N t1 ls M=ad54230e65b49a59381100009c60b9f7
mailbox_count: 1
MAILBOX: M(mailbox_guid)=ad54230e65b49a59381100009c60b9f7
mail_total=2, mails_displayed=2
mailbox_size=5539 bytes
MAIL: U(uid)=4
oid = a2d69f2868b49a596a1d00009c60b9f7
R(receive_time)=Tue Jan 14 00:18:11 2003
S(save_time)=Mon Aug 21 12:22:32 2017
Z(phy_size)=2919 V(v_size) = 2919 stat_size=2919
M(mailbox_guid)=ad54230e65b49a59381100009c60b9f7
G(mail_guid)=a3d69f2868b49a596a1d00009c60b9f7
I(rbox_version): 0.1
[..]

28. RADOS Dictionary Plugin
make use of Ceph omap key/value store
RADOS namespaces
shared/<key>
priv/<key>
used by Dovecot to store metadata, quota, ...

29. It's open source!
License: LGPLv2.1
Language: C++
Location:
Supported Dovecot versions:
2.2 >= 2.2.21
2.3
github.com/ceph-dovecot/

30. Ceph Requirements
Performance
Write performance for emails is critical
metadata/index read/write performance
Cost
Erasure Coding (EC) for emails
Replication for CephFS
Reliability
MUST survice failure of disk, server, rack and even fire compartments

31. Which Ceph Release?
Required Features:
Bluestore
should be at least 2x faster than filestore
CephFS
Stable release
Multi-MDS
Erasure coding

32. SUSE Products to use
SLES 12-SP3 and SES 5

33. Hardware

34. Hardware
Commodity x86_64 server
HPE ProLiant DL380 Gen9
Dual socket
Intel® Xeon® E5 V4
2x Intel® X710-DA2 Dual-port 10G
2x boot SSDs, SATA, HBA, no separate RAID controller
CephFS, Rados, MDS and MON nodes

35. Storage Nodes
CephFS SSD Nodes
CPU: E5-2643v4 @ 3.4 GHz, 6 Cores, turbo 3.7GHz
RAM: 256 GByte, DDR4, ECC
SSD: 8x 1.6 TB SSD, 3 DWPD, SAS, RR/RW 125k/92k iops
Rados HDD Nodes
CPU: E5-2640v4 @ 2.4 GHz, 10 Cores, turbo 3.4GHz
RAM: 128 GByte, DDR4, ECC
SSD: 2x 400 GByte, 3 DWPD, SAS, RR/RW 108k/49k iops
for BlueStore database etc.
HDD: 10x 4 TByte, 7.2K, 128 MB cache, SAS

36. Compute Nodes
MDS
CPU: E5-2643v4 @ 3.4 GHz, 6 Cores, turbo 3.7GHz
RAM: 256 GByte, DDR4, ECC
MON / SUSE admin
CPU: E5-2640v4 @ 2.4 GHz, 10 Cores, turbo 3.4GHz
RAM: 64 GByte, DDR4, ECC

37. Why this specific HW?
Community recommendations?
OSD: 1x 64-bit AMD-64, 1GB RAM/1TB of storage, 2x 1GBit NICs
MDS: 1x 64-bit AMD-64 quad-core, 1 GB RAM minimum per MDS, 2x 1GBit NICs
NUMA, high clocked CPUs and large RAM overkill?
Vendor did not offer single CPU nodes for number of drives
MDS performance is mostly CPU clock bound and partly single threaded
High clocked CPUs for fast single threaded performance
Large RAM: better caching!

38. Placement

39. Issues
Datacenter
usually two independent fire compartments (FCs)
may additional virtual FCs
Requirements
Lost of customer data MUST be prevented
Any server, switch or rack can fail
One FC can fail
Data replication at least 3 times (or equivalent)

40. Issues
Questions
How to place 3 copies in two FCs?
How independent and reliable are the virtual FCs?
Network architecture?
Network bandwidth?

41. Fire Compartment A
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
Switches
MON
MDS
HDDnode
HDDnode
HDDnode
HDDnode
HDDnode
HDDnode
HDDnode
SSDnode
SSDnode
SSDnode
Fire Compartment B
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
Switches
MON
MDS
HDDnode
HDDnode
HDDnode
HDDnode
HDDnode
HDDnode
HDDnode
SSDnode
SSDnode
SSDnode
SUSE
Admin
Fire Compartment C
(third room)
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
Switches
MON
MDS
SSDnode
SSDnode
SSDnode

42. Network
10G network
2 NICs / 4 ports per node
SFP+ DAC
Multi-chassis Link Aggregation (MC-LAG/M-LAG)
For aggregation and fail-over
Spine-Leaf architecture
Interconnect must not reflect theoretical rack/FC bandwidth
L2: terminated in rack
L3: TOR <-> spine / spine <-> spine
Border Gateway Protocol (BGP)

43. 2x 40G QSFP
L3 crosslink
40G QSFP
LAG, L3, BGP
N*10G SFP+
MCLAG
rack , L2 terminated
Spine switch
DC-R
FC1 FC2 vFC

44. Status and Next Steps

45. Status
Dovecot Ceph Plugin
Open sourced
github
still under development
still includes librmb
planned: move to Ceph project

46. Testing
Functional testing
Setup small 5-node cluster
SLES12-SP3 GMC
SES5 Beta
Run Dovecot functional tests against Ceph

47. Proof-of-Concept
Hardware
9 SSD nodes for CephFS
12 HDD nodes
3 MDS / 3 MON
2 FCs + 1 vFC
Testing
run load tests
run failure scenarios against Ceph
improve and tune Ceph setup
verify and optimize hardware

48. Further Development
Goal: Pure RADOS backend, store metadata/index in Ceph omap
M
M
RADOS CLUSTER
RFC5322
CACHE
LOCAL STORAGE
IMAP4 / POP3 / LMTP process
Storage API
dovecot rbox backend
librmb
librados
RFC 5322 mails
RFC 5322 objects and index

49. Next Steps
Production
verify if all requirements are fulfilled
integrate in production
migrate users step-by-step
extend to final size
128 HDD nodes, 1200 OSDs, 4,7 PiB
15 SSD nodes, 120 OSDs, 175 TiB

50. Conclusion

51. Summary and conclusions
Ceph can replace NFS
mails in RADOS
metadata/indexes in CephFS
BlueStore, EC
librmb and dovecot rbox
Open Source, LGPLv2.1, no license costs
librmb can be used in non-dovecot systems
still under development
PoC with dovecot in progress
Performance optimization

52. Be invited to: Participate!
Try it, test it, feedback and report bugs!
Contribute!
Thank you.
github.com/ceph-dovecot/

53. Questions?

55. https://dalgaaf.github.io/CephMeetUpBerlin20170918-librmb