In this deck from the 2017 DDN User Group meeting at ISC, James Coomer from DDN presents: IME Update - I/O Profiling of applications. "DDN’s Infinite Memory Engine (IME) is a scale-out, flash-native, software-defined, storage cache that streamlines the data path for application IO. IME interfaces directly to applications and secures IO via a data path that eliminates file system bottlenecks."

1. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
11
11
IO Management with IME1.1

2. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
12 What are the IO challenges?
Application
Developer
approach
IO Characterisation IO Challenge
Multi-physics Non-trivial mix of IO behaviours Filesystem must be optimised for all
IO types, not just a small subset
Workflows and
Ensembles
Coordination of files/objects across many jobs. Different
IO access patterns from different elements of the
workflow
Requires a global namespace with
strong consistency
Adaptive mesh
refinement
Very difficult to manage and maintain logical application
“block” sizes with underlying filesystem
Non-optimal access to files, incurs
RMW, fs lock contention
Metadata-
orientation
Small, random IOs, Shared file formats Limited by filesystem locking
Machine learning High read activity Tough for HDD: disk thrashing
Higher Concurrency
and heterogeneous
CPUs
Large thread counts make sequential IO look like
random IO
avoids all assistance from Filesystem
and Storage caches

3. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
13
DDN Confidential
How do I make my application go faster?
► Optimal/Natural behaviour already exist within the algorithms
► Difficult to optimise for multiple aspects simultaneously (GPU, multi-
threading, network changes, caches)
► Difficult to optimise our application for new environments
► Difficult to maintain efficient porting even between different node counts

4. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
14
DDN Confidential
How do I make my application go faster?

5. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
15 DDN | IME DataFlow
Fast Data
NVM & SSD
Persistent
Data (Disk)
Diverse, high
concurrency
applications
Compute
Application issues IO
to IME client.
Erasure Coding
applied
IME client sends
fragments to IME
servers
IME servers write buffers
to NVM and manage
internal metadata
IME servers write
aligned sequential
I/O to SFA backend
Parallel File system
operates at
maximum efficiency

6. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
16
DDN Confidential
Applying Flash-Cache Ecomonics
► Real World IO sub-system activity
• 99% of the time <30%
• 70% of the time <5%
► Build a cache for the peaks, PFS for the capacity
► 10x Performance/Watt improvement for flash-cache over PFS
Argonne: P. Carns, K. Harms et al., Understanding and Improving Computational Science Storage Access through Continuous
Characterization
Parallel File System
Scale-Out Flash Cache

7. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
17
problem > client cache
problem(s) with Life Science IO
ftp ftp-mouse.sanger.ac.uk: anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
230 Anonymous access granted
ftp> cd REL-1210-BAM
ftp> ls –lh
-rw-rw-r-- 1 proftpd proftpd 138.5G Dec 7 2009 129P2_OlaHsd.bam
-rw-rw-r-- 1 proftpd proftpd 182.3G Oct 9 2012 129S1_SvImJ.bam
-rw-rw-r-- 1 proftpd proftpd 60.0G Dec 10 2009 129S5SvEvBrd.bam
-rw-rw-r-- 1 proftpd proftpd 107.0G Oct 9 2012 A_J.bam
-rw-rw-r-- 1 proftpd proftpd 112.8G Oct 9 2012 AKR_J.bam

8. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
18 IME1.1 | Protected, Fast, Flexible IO
IME
Server0
IME
Server1
IME
Server2
IME
Server3
streaming writes
streaming reads
random reads
random writes
shared file
high concurrency
Single Shared
Namespace to
all clients
Tackle Tough IO dial-in erasure coding
blisteringly fast IOPs
wirespeed throughput
adaptive IO
insane rebuild speeds
FILE CACHE

9. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
19 IME240 Performance (EDR)
► 100% of peak SSD performance to clients
► Consistently high performance across IO sizes and SSF/FPP

10. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
20
DDN Confidential
IME1.1 | Blistering Sequential Performance
File Per Process
1M Seq IO4k Seq IO
► IME240 Sequential performance
to file per process over 23GB/s
per server (POSIX IO)
► >21GB/s with 4k IOs (MPIIO)
► consistent read and write
performance across IO sizes
► POSIX IO performance hits max
values at ~32K IO sizes

11. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
21
DDN Confidential
IME1.1 | Blistering Sequential Performance
Single Shared File
1M Seq IO4k Seq IO
► IME240 Sequential
performance for Single
Shared File over 22GB/s per
server (POSIX IO)
► 15GB/s with 4k IOs (MPIIO)
► consistent read and write
performance across IO
sizes
► POSIX IO performance hits
max values at ~32K IO sizes

12. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
22
DDN Confidential
IME | WRF with I/O quilting
one I/O process per node
Measurement Speed Up
IO Wallclock 17.2x
Total Execution Wallclock 3.5x
► Application wall time
reduced by 3.5 using IME
► I/O time reduces by x17.2
► Move from I/O bound to
compute bound

13. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
23 IME1.1 | True Dial-In Erasure Coding
IME
Server0
IME
Server1
IME
Server2
IME
ServerN
FILE CACHE
8+38+1
6+0 8+2
6+1
4+1
4+14+1
8+0
1+1
► IME1.1 supports multiple resilience
levels through flexible, adaptive
erasure coding
► System Wide Default up to 15+3
► Applications can overide defaults
and select a specific Erasure
Coding Scheme
DEFAULT: 8+3

14. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
24 IME1.1 Node Failure Management
► IME1.1 allows continued IO Service through IME server failure(s)
► Following server failure, IME commences background rebuild of all
missing extents for each Parity Group Instance (PGI) : 3+1 across
the 3 remaining servers
• 1 server will be overloaded with 2 blocks for each PGI
• Following a server failure but prior to rebuild completion, immediate read requests may be
satisfied through on-the-fly reconstruction of missing file extents
► Service interruption will occur with a second IME server failure in
this case (but not with NVM device failures that occur after rebuild)
► New writes still maintain 3+1 (with overloading)
► IME restart is needed to re-introduce failed node and restore
original redundancy. (requires full sync to the BFS)
4 IME Servers, Default Parity Schema= 3+1
Continued Production

15. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
25 IME1.1 Node Failure Management
► With higher node counts (>4) and using an erasure coding geometry of
higher parity count (>1), IME 1.1 can sustain >1 node failures
► In the below case after one node fails, reads which rely upon missing
extents are reconstructed from parity. Writes can continue to be written with
6+2 with full redundancy since we still have 8 nodes
► If another server fails, service continues, with writes at 6+2, but one server
will contain 2 objects in each PGI
9 IME Servers, Default Parity Schema= 6+2
Continued Production

16. ddn.com© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.
26
Apply Scale-Out FLASH-CACHE Economics with
Best Performance Density 500GB/s per rack
Adaptive IO route around IO bottlenecks
Insane Rebuild Speeds minutes not hours
Flexible Erasure Coding SW-defined, adaptive N+M
Flash-Native minimise write amplification. maximise SSD lifetime
IME240 | IME1.1SCALE-OUT | OPTIMISED DATAPATH | FLASH-CACHE | BURST-BUFFER
EXTREME IOPS | MASSIVE THROUGHPUT | SOFTWARE DEFINED