제3회난공불락 오픈소스 인프라세미나 - lustre

1. Intel High Performance Storage Solution
WanHee, Kim | kimwh@isbc.co.kr | ISBC Inc 2016
1
2016-07-22

2. Intel® Scalable System Framework
A design foundation enabling a wide range of highly workload-optimized solutions
Compute Memory/Storage
Fabric Software
Small Clusters Through Supercomputers
Compute and Data-Centric Computing
Standards-Based Programmability
On-Premise and Cloud-Based
Intel Silicon
Photonics
Intel® Xeon® Processors
Intel® Xeon Phi™ Processors
Intel® Xeon Phi™ Coprocessors
Intel® Server Boards and Platforms
Intel® Solutions for Lustre*
Intel® Optane™ Technology
3D XPoint™ Technology
Intel® SSDs
Intel® Omni-Path Architecture
Intel® True Scale Fabric
Intel® Ethernet
Intel® Silicon Photonics
HPC System Software Stack Intel®
Software Tools
Intel® Cluster Ready Program
Intel Software Defined Visualization
2

3. ISBC Korea3
High Performance File System - Lustre

4. ISBC Korea4
Intel Enterprise Edition for Lustre

5. • Lustre는 오브젝트 기반, 오픈소스, 분산, 병렬, 클러스터 파일 시스템
• Runs externally from compute cluster
• Accessed by clients over network (Ethernet, InfiniBand)
• Up to 512 PB file system size, 32 PB per file
• Production file systems have exceeded 2TB/sec
• 스토리지 및 네트워크의 자원을 최대한 활용 할 수 있도 설계
• POSIX 표준 호환
• 공유 가능한 단일 네임 스페이스
• 모든 클라이언트는 볼륨에 있는 모든 데이터 액세스 가능
• 비용 대비 최대한의 자원을 활용 가능
ISBC Korea5
Lutre File System

6. ISBC Korea6
IEEL Sample Architecture

7. ISBC Korea7
IML - Intel Enterprise Edition for Lustre

8. ISBC Korea8
IML - Monitoring a Lustre File System

9. ISBC Korea9
IML - Monitoring a Lustre File System

10. ISBC Korea10
IML - Monitoring a Lustre File System

11. ISBC Korea11
IML - Lustre File System Performance (8 OSSs)

12. ISBC Korea12
IML - Monitoring a Lustre File System

13. ISBC Korea13
IML – Add Servers

14. ISBC Korea14
IML – Cluster Status

15. ISBC Korea15
IML – LUN Configuration

16. ISBC Korea16
IML – Create File System

17. ISBC Korea17
IML – File System up and Running

18. Lustre today uses ldiskfs(ext4) or, more recently, ZFS as backend file
system
► Ldiskfs(ext4)
•ldiskfs(ext4) is used widely and exhibits well-tuned performance and
proven reliability
•But, its technical limitations are increasingly apparent, it is missing
important novel features, lacks scalability, while offline fsck is extremely
time-consuming
► ZFS
• ZFS is an extremely feature-rich file system
• but, for use as Lustre OSD further performance optimization is
required
• Licensing and potential legal issues have prevented a broader
distribution
► Btrfs
• Btrfs shares many features and goals with ZFS
• Btrfs is becoming sufficiently stable for production usage
• Btrfs is available through the major mainstream Linux distributions
• Pachless server support. Simple Lustre setup and easy maintenance
ISBC Korea18
Lustre Backend File System

19. ISBC Korea19
ZFS – L2ARC Benefit
L2ARC device can improve by 8x n. IOPS of a single OST
Test 1 with L2ARC completes in 157 sec
Test 2 without L2ARC completes in 581 sec
Reading 3.84M files each 64K from 16 clients in parallel. Lustre configured using ZFS on 4 OSS. On each OSS, Intel configured 1 OST using 16
HDD and 1 Intel DCP 3700 SSD as L2ARC device. In the chart: operations/sec during both experiments from a single OST

20. Lustre depends on the “ZFS on Linux”
implementation of ZFS
•Lustre targets run on a local file system on Lustre
servers. Object Storage Device (OSD) layer supported
are:
• ldiskfs (EXT4) is the commonly used driver
• ZFS is the 2nd use of the OSD layer based on OpenZFS
implementation
•Targets can be different types (for example LDISKFS
as MDTs and ZFS as OSTs )
•Lustre Clients are unaffected by the choice of OSD
file system
•ZFS as backend is functional since Lustre 2.4
•ZFS fully supported by Intel
ISBC Korea20
How do Lustre and ZFS interact?

21. ISBC Korea21
Maximum data protection with ZFS
• ZFS as backend file system for Lustre, increases the relia
bility of data on the next generation of high capacity H
DD and protects from silent data corruption (SDC) issue
s
• ZFS can enable cheaper storage solution for Lustre enab
ling JBODs
• ZFS is using advanced algorithms for data protection in
single, double, triple parity
• Benefits of using Enterprise Edition for Lustre with ZFS
– Hybrid Storage Pool
– L2ARC – Intel SSD devices to speed up random and
small files read I/O
– Copy on Write (COW) – protect Lustre to unaligne
d write I/O
– Checksum on data block – in conjunction with the
LNET checksum provides an end-to-end data protec
tion for Lustre
– Always consistent on disk – waiting for a very lon
g file system check is no more needed.
– Resilvering – very fast rebuilding time based on the
space used on the disk
– Scrubbing – automatic and online data correction f
or SDC
– Manageability – simple to manage, simple to troub
leshoot
– Compression – compression can be enabled to ma
ximize the ROI
– Snapshots
Device Hard Read Erro
r Rate
Equivalent in TB’s
SATA Consumer 10E14 10
SATA/SAS Nearline Enter
prise
10E15 110
Enterprise SAS/FC 10E16 1110
LTO and some Enterprise
SAS SSD’s
10E17 11100

22. ISBC Korea22
Zpool history

23. ISBC Korea23
Zpool status

24. ISBC Korea24
Zpool iostat

25. • zpool list
• zpool get all
• zpool status
• zpool iostat
• zpool import
• zfs list
• zfs get all
• zfs get / set
ISBC Korea25
Zfs command

26. ISBC Korea26
Community Version Release Roadmap

27. ISBC Korea27
Commuity Lustre Version and Download
https://downloads.hpdd.intel.com/public/lustre/latest-release/
2012 20152013
2.3 2.5 2.7
2016
2.8

28. ISBC Korea28
Intel Enterprise Lustre Version
Intel OS Lustre
V3.0.0.0 RHEL/CentOS7.2 V2.7.15.3
V2.4.2.0 RHEL/CentOS6.8 V2.5.42.12
V2.4.1.3 RHEL/CentOS6.7 V2.5.42.6
V2.2.0.2 RHEL/CentOS6.6 V2.5.34.1
Enterprise
HPC, commercial technical
computing, and analytics
Lustre 2.7.x
Enterprise Edition 3.1
Expanded OpenZFS support in IML
IML graphical UI enhancements
Bulk IO Performance (16MB IO)
OpenZFS metadata performance
Subdirectory mounts
Patchless server support
Lustre 2.7.x
Enterprise Edition 3.0
Client metadata RPC scaling
OpenZFS performance
OpenZFS Snapshots
SELinux on client + Kerberos
Intel Omni-Path (OPA) support
RHEL 7 server/client, SLES 12 SP1 client
OpenZFS 0.6.5.3
Q2 ’16 Q3 ’16 Q4 ’16 Q1 ’17 Q2 ’17

29. ISBC Korea29
Convert NFS Backup to Lustre File System
2.2PB Size
Single Name Space
– Lustre File System

30. ISBC Korea30
Key Markets using intel lustre today
Government research
and defense
Large-scale manufacturing
Mechanical computer-aided desig
n & computer-aided engineering s
ystems
Energy
Seismic processing, reservoir modeli
ng / characterization, sensor data an
alysis
Finance
Fraud detection, Monte Carlo simu
lations, risk management analysis
Weather and climate
Highly complex CGI rendering
Bioscience
Genomic data analysis, modeling
and simulations

31. ISBC Korea31