Gerd Prüßmann & Ievgen Nelen, Deutsche Telekom

1. CEPH@DeutscheTelekom
A 2+ Years ProductionLiaison
IevgenNelen, GerdPruessmann - Deutsche Telekom AG, DBU Cloud Services,P&I

2. 07.05.2015 2
Speakers
Ievgen Nelen & Gerd Prüßmann
• Cloud Operations Engineer
• Ceph cuttlefish
• Openstack diablo
• @eugene_nelen
• i.nelen@telekom.de
• Head of PlatformEngineering
• CEPH argonaut
• Openstack cactus
• @2digitsLeft
• g.pruessmann@telekom.de

3. Overview
the business case

4. 07.05.2015 4
Overview
Business Marketplace
• https://portal.telekomcloud.com/
• SaaS Applications from Software Partners (ISVs) and
DT offered to SME customers
• i.e.Saperion, Sage,PadCloud, Teamlike, Fastbill, Imeet,
Weclapp, SilverERP, Teamdisk ...
• Complements othercloud offerings from Deutsche Telekom (Enterprise cloud from T-Systems, Cisco
Intercloud, Mediencenter etc.)
• IaaS platform based only on Open Source technologies like OpenStack, CEPH and Linux
• Project started in 2012 with OS Essex, CEPH in production since 3/2013 (bobtail)

5. 07.05.2015–Strictlyconfidential,Confidential,Internal– Author/Presentationtitle 5
Overview
why opensource?Why ceph?
• no vendorlock in!
• easier to change and adapt newtechnologies / concepts - more independent from vendor priorities
• low cost of ownership and operation, utilizing commodity hardware and Open Source
• no license fees - but professional support
• modular and horizontally scalable platform
• automation and flexibility allow for faster deployment cycles, than in traditional hosting
• control overopen sourcecode - faster bug fixing and feature delivery

6. DETAILS
BASICS

7. 07.05.2015 7
DETAILS
ceph basics
• Bobtail > Cuttlefish > Dumpling > Firefly (0.80.9)
• Multiple CEPH clusters
• overall raw capacity 4.8 PB
• OneS3and cluster(~810TB raw capacity - 15 storage nodes - 3 MONs)
• multiple smaller RBD clusters for REF, LIFE and DEV
• S3 storage for cloud native apps (Teamdisk, Teamlike) and for backups(i.eRBD)
• RBD for persistent volumes / data via Openstack Cinder(i.e.DB volumes)

8. 07.05.2015 8
Details
ceph basics

9. DETAILS
Hardware

10. 07.05.2015 10
DETAILS
hardware

11. • Supermicro
2x Intel Xeon E5-2640 v2
@ 2.00GHz
64GB RAM
7x SSDs
18x HDDs
• Seagate Terascale
ST4000NC000
4TB HDDs
• LSI MegaRAID SAS 9271-8i
• 18 OSDs per node: RAID1 with 2
SSD for /, 3 RAID0 with 1 SSD for
journals, 18 raid0 with 1 hdd for OSD
• 2x10Gb network adapters
07.05.2015 11
DETAILS
hardware
• Supermicro
1x Intel Xeon E5-2650L
@ 1.80GHz
64GB RAM
36x HDDs
• Seagate Barracuda
ST3000DM001
3TB HDDs
• LSI MegaRAID
SAS 9271-8i
• 10 OSDs per node: RAID1 for /, 10
RAID0 with 1 hdd for journals, 10
raid0 with 2 hdd for OSD
• 2x10Gb network adapters

12. Details
Configuration& deployment

13. 07.05.2015 13
details
configuration& deployment
• Razor
• Puppet
• https://github.com/TelekomCloud/pup
pet-ceph
• dm-crypt disk encryption
• osd location
• XFS
• 3 replica
• OMD/Check_mk http://omdistro.org/
• ceph-dash https://github.com/TelekomCloud/ceph-dash for dashboard
and API
• check_mkplugins (Cluster health, OSDs,S3)

14. Details
performancetuning

15. 07.05.2015 15
details
performance tuning
• Problem - Low IOPS,IOPSdrops
• fio
• Enable RAID0 Writeback cache
• Useseparate disks for cephjournals (better use SSDs –scale out project)
• Problem - Recovery/Backfilling consumes a lot of cpu, decreaseof performance
• osd_recovery_max_active 1 numberofactiverecoveryrequestsperOSDatonetime
• osd_max_backfills 1 maximumnumberofbackfills allowedtoorfromasingleOSD

16. 07.05.2015 16
details
performance Tests – current hardware / IO

17. 07.05.2015 17
details
performance Tests – curr.Hardware/Bandwidth

18. lessonslearned

19. 07.05.2015 19
lessonslearned
operational experience
• Chose your hardware well !!
• I,e. RAID and hard disks -> enterprise gradedisks (desktop HDs aremissing important features like TLER/ERC)
• CPU/RAM planning: calculate 1GHz CPU powerand 2GB RAM persingle OSD
• pick nodes with lowstorage capacity density for smaller clusters
• At least 5 nodes for a 3 replica cluster (i.e.for PoC, testing and development purposes)
• Cluster configuration “adjustments”:
• increasing PG num> impact on cluster becauseofmassive data migration
• Rolling software updates / upgrades workedperfectly
• CEPH: has a character– buthighly reliable - neverlost data

20. 07.05.2015 20
lessonslearned
operational experience
• Failed / ”Slow”disks
• Inconsistent PGs
• Incomplete PGs
• RBD pool configured with min_size=2
• Blocks IO operations to the pool / cluster
• fixed in Hammer (allows PG replication while replica level below min_size pool/OSD)

21. /var/log/syslog.log
Apr 12 04:59:47 cephosd5 kernel: [12473860.669262] sd 6:2:10:0: [sdk]
Unhandled error code
root@cephosd5:/var/log# mount | grep sdk /dev/mapper/cephosd5-journal-sdk on
/var/lib/ceph/osd/journal-disk9
root@cephosd5:/var/log# grep journal-disk9 /etc/ceph/ceph.conf osd journal =
/var/lib/ceph/osd/journal-disk9/osd.151-journal
/var/log/ceph/ceph-osd.151.log.1.gz
2015-04-12 04:59:47.891284 7f8a10c76700 -1 journal FileJournal::do_ write:
pwrite(fd=25, hbp.length=4096) failed :(5) Input/output error
07.05.2015 21
lessonslearned
operational experience

22. 5/7/2015 22
lessonslearned
incompletePGs- what happened?
OSDnode
OSD
Journal
pg pg
OSD
Journal
OSDnode
OSD
Journal
pg pg
OSD
Journal
OSDnode
OSD
Journal
pg pg
OSD
Journal
pg

23. glimpseof the future

24. 07.05.2015 24
Overview
SCALE OUT Project
+40%
Current overall capacity:
 ~60 storage nodes
 5,4 PB Storage Gross
 ~0,5 PB S3 Storage Net
Planned Capacity for 2015:
 ~90 storage nodes
 7,5 PB Storage Gross
 ~1,5 PB S3 Storage Net

25. 07.05.2015 25
Future setup
scale out project
• 2 physically separated rooms
• Data distributed accordingthe rule
• not more than 2 replicas in - oneroom not more than 1
replica in onerack

26. 07.05.2015 26
Future setup
New crushmap rules
rule myrule {
ruleset 3
type replicated
min_size 1
max_size 10
step take default
step choose firstn 2 type room
step chooseleaf firstn 2 type rack
step emit
}
crushtool -i real7 --test --show-
statistics --rule 3 --min-x 1 --
max-x 1024 --num-rep 3 --show-
mappings
CRUSH rule 3 x 1 [12,19,15]
CRUSH rule 3 x 2 [14,16,13]
CRUSH rule 3 x 3 [3,0,7]
…
Listing 1: crushmap rule Listing 2: Simulate 1024Objects

27. 07.05.2015 27
Future setup
dreams
• cachetiering
• make use of shiny newSSDs in a hot zone / cachepool
• SSD pools
• Openstack live migration for VMs(boot from rbd volume)

28. Q & a

29. 07.05.2015 29
QUESTION & ANSWERS
• Ievgen Nelen
• @eugene_nelen
• i.nelen@telekom.de
• Gerd Prüßmann
• @2digitsLeft
• g.pruessmann@telekom.de