The document discusses the future challenges of data management and I/O performance as data scales to exabytes. It outlines growing issues with metadata, random access performance, multi-tenancy, and data retrieval. New workloads are more heterogeneous and driven by latency. Solutions proposed include improved file systems, analytics-driven administration, and technologies from DDN's acquisition of Tintri to address these challenges at extreme scales.

1. DDN Storage | ©2018 DataDirect Networks, Inc.
A Glimpse into the Future of I/O
DDN Usergroup Meeting SC18
November, 2018 Sven Oehme – Chief Research Officer DDN

2. DDN Storage | ©2018 DataDirect Networks, Inc.
A Glimpse into the Future of I/O

3. DDN Storage | ©2018 DataDirect Networks, Inc.
Data At Scale
HPC AI
Analytics
BigData
Cloud Integration
Data Management
Distributed Object
Advanced Data Management
I/O Performance and Behavior
Latency Latency-Tail Throughput
Byte Addressable
GPUs
More Heterogeneous
CPU
Economic Model
Cloud Integration
Workloads
SuperComputing
Themes
Meta-Analytics Simplicity
Insight
Scale
Security
Multi Tenancy
Data Path Optimisation
Tag&Search
HW
Accelerators
Fat Nodes
Orchestrators
Containers
Open Monitoring
Openstack/
Kubernetes
NVMe/SCM
KV
HDFS
Trending Technologies
Audit
Data Views

4. DDN Storage | ©2018 DataDirect Networks, Inc.
Challenges in Broader Multi-Tenanted Security and Flexibly
Managing Data Collections
► Multi-Tenancy == Tenanted Isolation by
• Data Collections ß Flexible allocation of collections of data to users, groups, organizations, allowing
collaborative access, publication, etc
• Performance ß The Noisy Neighbor problem - particularly for latency critical applications being impacted by
throughput hungry workloads
• Resilience ß failures in domain A should not impact domain B
► Virtualization and Containerization - Data should be accessible from within containers
► Multi tenancy needs to be a logical allocation, not a HW or physical separation unless desired
► Auditable - All events – both system and tenant in the system should be auditable
► One data platform needs to run on your own HW, the cloud and highly optimized integrated systems
► Need a more native approach to resolving all these challenges

5. DDN Storage | ©2018 DataDirect Networks, Inc.
Challenges in Security and deployment – Starting down the path
► Lustre takes strong steps towards better multi-tenancy support
► This addresses some of the requirements
► dynamic ‘views’ on data based on policy and access control rather than namespace segregation

6. DDN Storage | ©2018 DataDirect Networks, Inc.
Challenges in I/O Performance and Behavior
► Newer applications need to operate on byte addressable data
► Significant shift from sequential to random I/O
► Multifold increase of metadata to data ratio
► Average data sizes are less homogeneous and are now fractions or multiples of previous
workloads. gap between small and large data seems to wide (bytes on one end , GB’s on the
other end of the spectrum)
► Interactive, outcome and event driven analytics are driven by latency rather than bandwidth
Big Data NoSQL
Analytics
Supercomputin
g
Multi-
Physics
Machine Learning Workflow
s
Adaptive Mesh
Refinement
Checkpointing
Trend for ever more mixed, complex I/O Workloads

7. DDN Storage | ©2018 DataDirect Networks, Inc.
I/O Performance several techniques to address them
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
Oakforest-PACS
at JCAHPC (IME)
Shaheen at
Kaust
(Datawarp)
Mistral at DKRZ
(Lustre)
EMSL Cascade at
PNNL (Lustre)
Ratio of Easy:Hard
IO500 Results
Ratio of Easy:Hard (systems with 100 clients or more)
Write Ratio Read Ratio
► Overlapping i/o and conflicting access to
same resources are well addressed with IME
► Log structured filesystems mitigate i/o
amplification and significant slowdown of
random i/o patterns
► Leverage HDD for capacity tiers and IME for
performance
► HAWC / LROC in Scale , HDFS flash
accelerators are other examples

8. DDN Storage | ©2018 DataDirect Networks, Inc.
Challenges in Data Management
► Metadata growth is substantial, people try to solve this with external data repositories with mixed
success, but at current growth rate its not sustainable
► At Exabyte scale data reliability and availability will be substantially challenged by traditional
recovery models
► We see requirements to support Quadrillions (millions of billions) of objects in storage systems and
we are still at the beginning of this
► Data management at scale is complicated , snapshots, tiering (to disk, tape, cloud) and breaks in
larger systems
► People can’t find their data, no easy way to tag and ‘search’ for it

9. DDN Storage | ©2018 DataDirect Networks, Inc.
Challenges in Data Management – need for extensive analytics
► The acquisition of Tintri provides us technology to run analytics for systems at scale
► More insights into system to make decisions based on data and trends rather manual crafted
administrative policies
► data placement based on historic data

10. DDN Storage | ©2018 DataDirect Networks, Inc.
conclusions
► Already many strong inroads into these data challenges for 2019+
• IME
• Lustre
• SFAOS
► Heavy engineering investment underway to resolve this challenges in more native ways
► Come and talk to us and ask for the ‘secret’ presentation room ;-)

11. ddn.com©2018 DataDirect Networks, Inc. *Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change.
Thank You!
Keep in touch with us.
9351 Deering Avenue
Chatsworth, CA 91311
1.800.837.2298
1.818.700.4000
company/datadirect-networks
@ddn_limitless
sales@ddn.com