Ceph on 64-bit ARM with X-Gene

1© AppliedMicro Proprietary & Confidential
CEPH on 64-bit ARM with X-Gene®
Kumar Sankaran
Associate Vice President,
Software and Platform Engineering
CEPH Day Mar 30th 2016

Applied Micro: A Global Presence
San Diego, CA
Ottawa, Canada
Andover, MA
Austin, TX
Raleigh, NC
Copenhagen
SANTA CLARA, CA
India
Pune
Bangalore
Vietnam
Shenzhen
TokyoShanghai
BeijingParis
Manchester
Munich
China
Founded in 1978
Headquarters: Santa Clara, CA, USA
NASDAQ: Ticker AMCC
~500 Employees Worldwide

X-Gene®
Industry First 64-bit ARM Server-on-Chip
CPU Performance
 Strong Single Thread Performance
Power
 Low Power Envelope
 High Perf/Watt
IO Integration
 Integrated 10G NICs
 PCIe Gen 3
 SATA Gen 3
 Programmable Offloads
DRAM Memory
 Low latency
 Multiple channels
 Large Memory
Addressability

© AppliedMicro Proprietary & Confidential
NOVEMBER 2015
X-Gene 3 Announced
X-Tend Announced
OCTOBER 2014
HP’s m400
product launch
MARCH 2013
X-Gene silicon
samples
to customers
APRIL 2012
X-Gene FPGA release
enables ecosystem
partners
MARCH 2010
World’s 1st ARMv8
Server Architecture
Licensee
2012 2014
2013 2015
OCTOBER 2009
Dedicated processor
team begins 64b
development
2009
2010 2014
2014
SPRING 2014
X-Gene 2 samples
to customers
OCTOBER 2014
X-Gene 2 wins ARM
TechCon “Best in
Show”
OEM
Hyperscale
ODM
HPC
Networking Storage
Memcache Enterprise
Search
Mainstream
Web Tier
X-Gene Driving ARM in Data Centers
Target Customer Types Target Deployments

X-Gene® Market Segments
Web Serving/Proxy  Apache, NGINX, HAProxy
Web Tier
Cold Storage/Big
Data/Data Analytics
High Performance
Computing
Cold Storage  CEPH, GlusterFS, OpenStack Swift/Cinder
CPU + GPU combination for HPC workloads
Web Apps/Hosting  Dhrupal, WordPress, Rails
Web Caching  Memcached, Redis
Databases  MySQL, MongoDB, Cassandra, PostgreSQL
Big Data  Hadoop, MapReduce
Data Analytics  Lucene, ElasticSearch, SPARK

X-Gene® Features
Memory subsystem
 Multiple DDR3/DDR4 Channels
X-Gene®  Server on Chip
 Multi-core 64-bit ARMv8-A CPUs
 Three Level Cache Hierarchy
 L1 /L2/L3 caches
 Out-of-Order Execution
 HW virtualization
 Coherent Network
Boot/Power Management
 UEFI with SBBR compliance
 ACPI compliant
Connectivity
PCIe 3.0, SATA 3.0, USB 3.0
Integrated 1GbE/10GbE
 UART, I2C, SPI, MDIO, GPIO
X-Gene® Evaluation Kits Available Today
APM X-C1 (Mustang)
Platform APM X-C2 (Merlin)
Platform

© AppliedMicro Proprietary & Confidential 7
X-Gene® Production Platforms
Multiple SKUs from Leading OEM and ODM Partners

CEPH Storage Server Based on X-Gene 1
 Microserver featuring the Applied Micro X-Gene 1 “Storm” ARM 64bit
2.4GHz, 8-Core SoC
– Memory
– DDR3 UDIMMs 1600MHz ECC
– Integrated I/O
– 10GbE
– PCIe Gen3
– SATA Gen3
– Programmable Offloads
– Sub 40W TDP
 Two customer-defined workloads
– Cold-archival (one motherboard) or Hadoop storage (two motherboards)
– Unique 1U form factor in a cost-effective
-- Monolithic chassis to maximize density
 14 internal SATA storage devices: 12 x 3.5” hot-plug, Up to 2 x 2.5” boot
devices
 Optional JBOD building block capability for increasingly dense storage needs
 Redundant hot-plug power supplies
 Flexible I/O capability, including integrated 10GbE SFP+ support

© 2014 Applied Micro Circuits Corporation 9
Wistron X5 OCP CEPH Server
QS Compute Tray 8 APM X-Gene 1 ARM64 Compute Nodes
Each Compute Node has 8 DIMMs, 1 128GB M.2 SSD
2 QSFP+ connectors from OCP MEZZ card support 40GbE
DS Storage Tray 1 APM X-Gene 1 Compute Node
Marvell Storage Controller, Aspeed AST1250 BMC
10 3TB 3.5” SATA 6Gb HDD
1 SFP+ connector support 10GbE network
Software Open Source Linux, Ubuntu 14.04 LTS
Power Supply 3 +1 Redundant Power Supply design, max power 4800W
Sampling Now

• Red Hat is working in partnership with Applied Micro
• World’s First Full Ceph reference design on 64-bit ARM using
Applied Micro CEPH ARMv8 Server
• Complete the first Performance and Configuration Guide
Applied Micro RedHat Partnership for CEPH

CEPH Network Hardware & Software
• X-Gene 1 based Cold Storage Solution
– 4 node cluster with CEPH servers
– 12 SATA HDD drives per node
– ~40TB data storage
– Marvell 4-port PCIE SATA expander
• Platform Software
– Operating System: RHELSA 7.2, CentOS 7.2
– BIOS: Tianocore UEFI
• Cluster Software
– Apache v2.4.7, Python v2.7.6
– Ceph “Jewel” early access release
– Ceph Cluster Monitors, Client Load generators
– Ceph Benchmark Toolkit (Open Source tool for Ceph testing/benchmarks)
– FIO, RADOS Bench, other standardized tools for common test results

CEPH Hardware Topology
10GbE/1GbE Switch
Servers Under Test
10GbE/1GbE 10GbE/1GbE
10GbE/1GbE
1GbE
HP Moonshot
CEPH cluster monitor/load generator
10GbE/1GbE

Cluster Network Topology
• Ceph Cluster Network - Backend network – Data replication by OSD-daemons
• Ceph Public network - Frontend network – ceph-client to cluster control path network
• External Network - Management interface, Software Installation, Gateway
Node1 Node2 Node3 Node4
Ceph Public Network – 192.168.3.x
Ceph Cluster Network – 192.168.4.x
ExternalNetwork-10.58.12.x

CEPH FEATURE SUMMARY
CEPH
TRAINING
OBJECT
STORAGE
Unlimited Object Size
Integrates with
Openstack Keystone
Multi-Tenant
S3/SWIFT API
Usage Statistics
BLOCK
STORAGE
Thin Provisioning
Copy on Write
Integrates with
KVM/QEMU/Libvirt
Linux Kernel
Snapshots/Cloning
FILE SYSTEM
Mainline Linux
Kernel Support
Auto-balancing
Metadata Servers
POSIX Compliant
Openstack Cinder

CEPH Software Stack
S3/SWIFT
HOST/HYPERVI
SOR
iSCSI CIFS/NFS SDK
INTERFACESSTORAGECLUSTERS
MONITORS OBJECT STORAGE DAEMONS (OSD)
BLOCK
STORAGE
FILE SYSTEM
OBJECT
STORAGE

CEPH Software Architecture
Applied Micro CEPH Storage Server using X-Gene 1
Tianocore UEFI
EXT4
Ceph
Config Ceph
OSD
RADOS
NTP SSH

CEPH Components
 Ceph OSD
 Daemon to manage and store
cluster data
 Per disk
 Total #48 in cluster
 EXT4 File system on OSD
 Ceph MON (M)
 Monitor to check health of
cluster and OSD status
Image Courtsey: InkTank

CEPH Cluster Access
– librados interface
• ceph-admin
• rados client
– RADOS gateway
• AmazonWS S3 compatible
Image Courtsey: InkTank

Librados Library
RESTful
client
1. LIBRADOS
a client library
for direct access
of data in a
RADOS cluster
2. Ceph
Object
Gateway
host/VM
client
POSIX fs
client
3. Ceph
Block Device
4. Ceph
File System
native
client
RADOS
a reliable, autonomous, distributed object store

CEPH Performance Benchmarking – Seq Read/Write

CEPH Performance Benchmarking – Random Read/Write

• There is enough Community / Upstream and Customer interest
to begin the path to a supported build of Ceph for ARM
• Step 1: RHELSA – RHEL Server for ARM is released as a
Developers Tech Preview in RHEL 7.2 currently
• Step 2: the first Ceph on ARM builds will be available in the
Jewel Release in May 2016
• Ceph on ARM becomes available upstream!
• Step 3: Community, Developers, and customers to start making
use of Ceph on ARM
CEPH Upstream Status

• Goal of Ceph Performance and Configuration Guide is to offer guidance to
users wishing to utilize APM ARM Servers
• Guide will offer guidance on what performance can be expected for the
various tested workloads with guidance on configurations that were tested
for repeatable results in the field.
• Block Use Cases for Rados Block Device (RBD) in various block sizes
w/results for each (ie. 4K Random, etc)
• RBD Linux Kernel, Virtual Machines & Container persistent back end
storage, KVM, QEMU, Libvirt
• Object Use Cases for Swift/S3 compatible Ceph Objects (RADOS
Benchmark) for sequential read/write Object workloads
• Digital Media, Content Delivery Networks, Archive storage, Cloud
Object Storage Services
Conclusion

Ceph on 64-bit ARM with X-Gene

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Viewers also liked

Viewers also liked (20)

Similar to Ceph on 64-bit ARM with X-Gene

Similar to Ceph on 64-bit ARM with X-Gene (20)

Recently uploaded

Recently uploaded (20)

Ceph on 64-bit ARM with X-Gene