In this deck from the DDN User Group at SC19, Gael Delbray from CEA presents: Optimizing Flash at Scale. CEA, a major player in research and innovation, has been recognized as an expert in HPC through the momentum of the "Simulation Programme" supported by its Direction des Applications Militaires (CEA / DAM) and implemented by the Department of Simulation Sciences and Information (DSSI). "The major challenges that the HPC will face in the coming years are manifold, such as the development of hardware and software architectures able to deliver very high computing power, modelling methods combining different scales and physical models and the management of huge volumes of numerical data. High performance computing for numerical simulation has become an essential tool in scientific and technological research, as well as for industrial applications. Simulation can replace experiments that are too dangerous (accidental situations), beyond reach in terms of time or scale (climate, astrophysics) or banned (nuclear tests). Simulation is also time-saving and leverages productivity in many situations." Watch the video: https://wp.me/p3RLHQ-li0 Learn more: https://www.ddn.com/company/events/user-group-sc/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
FROM RESEARCH TO INDUSTRY
Commissariat à l’énergie atomique et aux énergies alternatives - www.cea.fr
CEA & SFA18KE STORY
November 18th, 2019
Gaël DELBARY

2. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
TOPICS
2
Site update
T1KF project
Building methodology
One story
Numbers
18KE key features
Thanks

3. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
Compute centers at CEA/DAM
3
2 production compute centers:
- TERA: Defense application
- TGCC: European research
 Hosting France Génomique (storage of DNA sequencing data)
 Hosting CCRT (for industrial companies)
 Hosting Human Brain project (soon)
1 lab compute center:
- R&D compute nodes
- R&D storage clusters
Compute power:
- TERA1K: 28 Pflops
- TGCC: 25 Pflops
- LAB: 1 Pflops (heterogeneous nodes)
2 production compute centers with a similar design:
- Nearly the same architecture, technologies, tools and system software

4. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
Global storage: data centric approach
4
User interface: Lustre/HSM (Lustre 2.10)
- Lustre 2.12.3 upgrade planned for Q1 2020
Seamless integration of HPSS as a Lustre backend

5. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
CEA HPC storage numbers
5
Production systems TGCC
(Usable Capacity/Write throughput)
TERA
(Usable Capacity/Write throughput)
Scratch FS (Cray) 15 PB (390 GB/s) 10 PB (530 GB/s)
Store FS (DDN) 16 PB (200 GB/s) 23 PB (200 GB/s)
HPSS HDD Tier 5 PB (30 GB/s) 4 PB (30 GB/s)
Total 36 PB (620 GB/s) 37 PB (760 GB/s)

6. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
CEA HPC DDN numbers
6
Production systems TGCC TERA
Store FS (DDN)
6x SFA14KXE (w 10x 8462
enclosures)
5x SFA14KXE (w 10x 8462
enclosures)
HPSS HDD Tier
1x SFA14KXE (w 10x 8462
enclosures)
1x SFA14KXE (w 10x 8462
enclosures)
Total 7 6

7. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
T1KF Project – Key numbers
7
T1KF: Flash Parallel Filesystem (Q4 2019) for the next exascale
supercomputers
Target:
- First store storage level (same Lustre FS as STORE) with Lustre Pool
- 1 TB/s in write sequential (5% of time)
- Mounted by all supercomputers
- Data migration triggered by RobinHood (via lfs migrate)
- Hide complexity to end users
Usage:
- Checkpoint restart files (mostly sequential writes)
- Final files (mostly sequential writes)
- Data post-processing (sequential writes + random read)

8. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
T1KF Project – Flash required?
8
Reasons:
- Low footprint (limit to 5 racks)
- Random read
- High write throughput (focus here)
External things to monitor:
- Weight (density is not light)
- Power (SSDs consume energy)
- Cooling (linked to power )
Goal: Find a storage solution which answers to these requirements

9. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
Building methodology
9
Layers by layers approach (not new):
SSD choice
Drive protocol validation
Network interface choice
PCIe validation
Raw devices IO
Network performance
Distributed filesystem IO

10. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE story: history
10
A long time ago in a small country, CEA decided to build a Flash parallel
filesystem: July 2016
- Embedded Targeted platform 14KE (X ?)
- SSDs in head enclosure (Sandisk Lightning Ascend Gen II 800 GB)
- Legacy RAID Mode
- Defect on SS14K (slots issue)
- Defect on SAS load balancing algorithm
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
pool93 pool94 pool95
GB/s
Parallel write - perf per pool
3.0.1.5
3.1.0.1
3.1.2.1a

11. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE story
11
July 2017:
- Switch to 14KXE
- DCR Beta
- New drives: 20x Hitachi SS200 (3840 GB)
- Push unmap feature
Summary:
- DCR stable
- 14KXE boost performance (~45GB/s in IOR write with more drives!)
- PCIe ICL limitation (4,5 GB/s)
- First words about 18K
- Dual port drives feature

12. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE story
12
January 2018:
- Decision to choose 18KE for production configuration
- Start of drives deep dive
- 20x Hitachi SS300 (1,6 TB) (costly)
- Start of raw drive benchs
July 2018:
- Drives winner list:
 Hitachi SS530 3,2 TB (DWPD 3)
 Samsung PM1643 3,2 TB (DWPD 3)
- Multiple VD feature supported
- Unmap design progress

13. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE story
13
November 2018:
- 18KE sample
- First runs with SSD drives
- Issues with loopback cables (copper)
- Not too stable (pretty stable for a sample )
- Push hardware remarks for GA version
- Power button hole size
- Plastic cpu cover
- Metal canister cover
January 2019:
- SSD Drives fight (be patient, next slides are coming…)
- Troubleshooting instability
 Failed heatsinks (fixed for GA canisters)
- SAS bandwidth analysis

14. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE story
14
April 2019:
- SAS bandwidth analysis finished
 +10% SAS bandwidth (SS530 or SS300 drives)
 Landed in 11.5
- SSD chosen: SS530
- Lustre write sequential benchs starts on lab
July 2019:
- Unmap feature in SFAOS 11.5
- 4x 18KE (56x drives inside) received
- Tuning locked for IOR write sequential workload
- IO500 discovery (some stuff…)
- Lnetself_test benchs warmup

15. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
Pause in story: How to choose a SSD?
15
XSR methodology
- Goal: monitor SSD capability to regain « nominal » performance
3DWPD Hitachi SS530
1DWPD Samsung PM1643
3DWPD Samsung PM1643 Low power
2 hours 128KB sequential write 2 hours 128KB random write 2 hours 128KB sequential write
Recovery time, shorter is better
1h40
47 minutes
1h10

16. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE story
16
September 2019:
- +5x 18KE received
- Slot available (1,5 day) to run benchs at scale
- Stuck on Infiniband networks (bad surprise)
- Full configuration received (mid-september)
October 2019:
- Full systems integration
- Infiniband troubleshooting
- 4x SS530 failures
- 4x power interposer failures
- 1x riser card failure

17. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE story
17
November 2019:
- Upgrade to SFAOS 11.6.1
- New slot available
- Infiniband optimization
- Lnetselftest validation (1,5 TB/s)
- One canister failure
- 1x SS530 failure
- 1x SS530 slow
- First full benchmark run
Could run IO500 and acceptance benchmark 

18. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE: numbers
18
IO500 (valid result) Lustre 2.12.3 LTS:
[RESULT] BW phase 1 ior_easy_write 836.649 GB/s : time 367.18 seconds
[RESULT] IOPS phase 1 mdtest_easy_write 1317.850 kiops : time 372.54 seconds
[RESULT] BW phase 2 ior_hard_write 4.494 GB/s : time 797.99 seconds
[RESULT] IOPS phase 2 mdtest_hard_write 465.515 kiops : time 312.24 seconds
[RESULT] IOPS phase 3 find 795.810 kiops : time 801.51 seconds
[RESULT] BW phase 3 ior_easy_read 789.340 GB/s : time 389.19 seconds
[RESULT] IOPS phase 4 mdtest_easy_stat 384.570 kiops : time 1276.63 seconds
[RESULT] BW phase 4 ior_hard_read 14.509 GB/s : time 247.18 seconds
[RESULT] IOPS phase 5 mdtest_hard_stat 730.733 kiops : time 198.92 seconds
[RESULT] IOPS phase 6 mdtest_easy_delete 676.117 kiops : time 726.13 seconds
[RESULT] IOPS phase 7 mdtest_hard_read 348.595 kiops : time 416.97 seconds
[RESULT] IOPS phase 8 mdtest_hard_delete 243.326 kiops : time 607.60 seconds
[SCORE] Bandwidth 81.0068 GB/s : IOPS 545.738 kiops : TOTAL 210.258
During IO500 benchmark:
- 3x slow drives (latency > 64ms in write)
- SFAOS failed one at the end of the bench

19. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE: numbers
19
Acceptance passed : 1,09 TB/s
- 1 drive missing
- IOR parameters weren’t optimal:
- One shot (no more time slot)
- Buffered IO (not perfect for this workload)
- Throughput sustained during 10 minutes
- Bandwidth space on drives (15%)
- Reachable target: 1,2 TB/s

20. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE: Key features for performance
20
Loopback cables:
- +40% on raw devices
- GA with fibers only (hard stuff to stabilize)
SAS optimization:
- +10% on raw devices
- No effect on hard drives (to check)
FIO parameters:
- Write sequential
- Block size=3M (strange or not?)
- Numjobs=8
- Latency hunt (iosched=none)
0
10
20
30
40
50
60
70
80
90
w loopback w/o loopback
GB/s
FIO write seq (3M)
SFAOS 11.4
SFAOS 11.5

21. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
SFA18KE loves Lustre
21
Lustre 2.12.3 LTS:
- One 18KE (56 drives)
- 24 clients
- EDR infiniband network
- Lustre checksums disable
- max_pages_per_rpc=2M
- rpc_in_flight=2
Lustre overhead is pretty light
Lustre loves #clients
- If not, play with “large rpc”
77.6
77.8
78
78.2
78.4
78.6
78.8
79
raw ext4 ior
GB/s
SFAOS 11.5 (Lustre overhead)
-0,7%
-1%

22. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
Lustre improvements
22
Lustre 2.12.3 LTS doesn’t like non aligned I/O
- ior_hard run kills Lustre performance
Lustre master brings major improvements
- Big boost in ior_hard_write (from 4,49 GB/s to 19,72 GB/s)
 More improvements could be done
- mdtest_easy_write (from 1.317.850 kiops to 3.184.910 kiops)
 Reason: bump to 128 MDTs (stable with IO500 benchmarks)
 Scalibilty is nearly linear
- Results not submitted
IO500 pushes Lustre to be more and more powerful

23. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
Thanks
23
IO500 and acceptance:
- L3 support guy: Sebastien PIECHURSKI (ATOS)
- Network guy: Damien GROS (CEA)
- Best qualified guy: Dominique MARTINET (CEA)
- Gael DELBARY (CEA)
 Contribution to the flash project:
- DDN: Paul, Bret, Ryan, Laurent, Lee, Cedric, Rich, Kevin, Vasu, Scott,
Thomas, William, Martin, Stefan, Richard, James, Mark, Reggie, Neil, Bill
- CEA: Jean-Marc Ducos (Cooling), Stephane Caillat (Infrastructure), Matthieu
Hautreux (brainstorming on IB topology), Damien GROS (Network), Gael
Delbary (DDN challenging )
Many people involved

24. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
Summary
24
SFA18KE story: exciting
Many things learned
Powerful platform
Strong relationship between CEA and DDN engineering
Todo:
- Unmap improvements
- Insert 16x18KE in Store FS
- CentOS 8 benchmarks
- Performance with 9012 enclosures
- Next Lustre improvements
- Next IO500?

25. November 18th, 2019Commissariat à l’énergie atomique et aux énergies alternatives Gaël DELBARY
Commissariat à l’énergie atomique et aux énergies alternatives - www.cea.fr
Questions?