Emerging Storage Trends Discussed at Scalar Presentation

Emerging Trends & Technologies
Friday, February 28th, 2014
Neil Bunn, P.Eng.
neil.bunn@scalar.ca
© 2013 Scalar Decisions Inc. Not for distribution outside of intended audience

Ø About Scalar
Ø NAND/Flash Memory Technologies
§ Flash Arrays
§ Hybrid Storage
§ Flash-In-The-System
Ø Object Storage
§ What is Object Storage
§ Leaders in Object Storage
§ Open Object Storage (Ceph/Openstack)
Ø Software Defined Storage
§ What is Software Defined Storage
§ What Types of Implementations Are There?
§ Implications of Software Defined Storage

} Traditional Disk Arrays
– Largely commoditized
– Limited future performance/scale
improvements without change
} Tape
– Great Medium, but limited innovation
} Disks themselves
– 4TB/5TB currently, with clear roadmap to
6/8TB, but limited performance
improvements (5yrs plus!) (HAMR / BPR)
} Parallel Filesystems
– Lustre & GPFS are cool. But far from
practical in the enterprise for broad
application

} Broad innovations causing product
churn
} Many new suppliers, many of whom
may not exist in their present form in
5yrs (acquisitions, death, etc.)
} Trends that are shaping the design of
next-generation application platforms
(ie: ‘Cloud’)
} The concepts are the key, the benefits
vary from client to client

INTRODUCTION

} Agnostic Solutions Vendor
– ~120 Employees
– Cross Country Coverage
– Offices in Toronto, London & Ottawa
– ~100M+ In Annual Revenue
– Technically focused organization
﹘ ~40+ Technical Personnel
– Own & Operate clusters of our own
(over 1,200 servers and growing)

} Some Of Our Partners

Source: Seagate Technology

1980
2013
2018+
Source: Multiple, IBM, HP, EMC, US Gov.

} The sky is falling!
} So, we know we have multiple problems:
– Performance issue (storage is always a bottleneck)
– Capacity issue (requirements continue to outstrip
traditional options)
– Management issue (divergent needs / classes)
– Locality issue (electrons take time…)
– Security issue (data protection costs a lot)
– Networking issue (back to electrons taking time…)

} There are coherent roadmaps to address almost all of the
recognized issues.
} Triggers change in the enterprise.
} Change is bad for some, but a terrific opportunity for
others.
} Storage focus:
– Disruptive Innovation based on:
﹘ Simple Management (Web / REST / Pervasive)
﹘ Higher Abstraction (Object / Software Defined)
﹘ Flash and future memory technologies (Flash / PCM / SCM)
﹘ Intensive services (Compression, Deduplication)

NAND / Flash

} Originally slow, consumable, persistent storage
} Rapid advancements in fabrication technologies
leading to tremendous density improvements
} Similar improvements in process leading to
significant longevity improvements
} Available in a variety of forms:
– Traditional ‘drive’ formats (SSDs)
– PCIe adapters (leveraging higher speed data
bus)
– Integrated DIMM packaging (Diablo controller)
– Chip Embedded Storage (next-gen)
} Broadly these trends apply to traditional “NAND” an
also to other non-volatile high speed storage for the
future such as PCM / DWM, etc.

} Commoditization of x86 technology, and
robustness of new platforms with Linux and/or
BSD derivatives results in low cost of entry for
new storage vendors and the resulting
proliferation of them.
} Three main approaches:
– Accelerators
– Primary Storage
– Hybrid / Tiered Storage Systems

} Simple, limited function
} Typically Block devices
} Connectivity options often:
– Fibre Channel
– Infiniband (SRP)*
– 10/40G* Ethernet (iSCSI)
} Often limited total scaling, for both cost and
complexity reasons
} Key Players:
– Fusion IO (ION Accelerator)
– IBM (FlashSystem)
– Violin Memory
} Note: Non-Flash Accelerators as well! ie: Kove

} Differentiation predominantly based on:
– Density
– Latency (large battle ground), much debate
about consistency
– Scalability
} Largely a space relegated to niche problems at
the moment with high costs ($500k+ for even
moderate storage solutions)
} Early movers differentiated on controller
architecture, SSD vs. dedicated cards, etc.
} Expect rapidly falling prices due to rapid
commoditization, and loss of many players

} Designed based on commodity platforms to
combine flash (typically SSD) with low-cost bulk
storage (ie: SATA drives)
} Predicated on use-cases following an 80/20 or
90/10 rule (80% of work on 20% of data) or on
low-rate-of-change data
} Really the value is the caching infrastructure
} Ideally suited to replicated uses such as VDI
(linked clones) or other scale-out topologies
where force-multipliers can be put in place
against the SSD usage

} Lower function devices
} Typical file (NAS) devices or iSCSI type for
supporting virtualization platforms
} Heavily focused on virtualization environments
– 1/10/40G* Ethernet
} Scale-out design, sometimes with aggregated
administrative interface, but often based on VAAI
for almost complete management from
Virtualization interface.
} Key Players:
– Nimble
– Tintri
– All traditional players in some “form”

Datastore performance stats
Performance fuel gauge

VM
Name
Latency
Real
/me
Graphs

} High function to compete (compression, deduplication, etc.)
} Typically Block devices
– Fibre Channel
– Infiniband (SRP)*
– 10/40G* Ethernet (iSCSI)
} High scaling via clustering approaches, but typically require
very-high performance internode interconnect (Infiniband)
} Destroy the “tiering” approach of traditional vendors. There
is data you need to access, and data you don’t. No in-between.
} Key Players:
– Pure Storage
– Solidfire
– Whiptail (Cisco)
– Xtreme IO (EMC)

} Significant focus on:
– Distributed data parity (similar to XIV in the disk
world)
– Potential for n+x protection in the future as
distributed data parity systems often using the
basis for erasure coding under the covers
– Block level deduplication
﹘ Differentiation on inline dedupe vs. post-process dedupe
– Compression
﹘ This adds challenges in doing array to array replication,
depending on the compression method used
– Thin provisioning
﹘ SSD/Flash math currently requires all of thin provisioning +
dedupe + compression against traditional disk array $/GB
without those features

} At the system level Flash / NAND or other non-volatile
storage is appearing in four areas:
– Basic Disk Subsystem (SSD’s)
– PCIe / Card based accelerators (FusionIO)
– Memory-Coupled Systems (ULLtraDIMM)
– On-CPU NVRAM (futures, not necessarily
Flash based)
} SSD’s are well understood, leveraging standard
SAS/SATA interfaces and characterized by a
current “Moore’s Law” associated with capacity/
density improvements

} Controller tech largely developed by Ottawa
based “Diablo Technologies”
} Commercialized by Smart Storage (SanDisk)
} Shortly to be broadly available from IBM
(Lenovo?) as eXFlash DIMMs
– With Intel E7-8xxx v2 processors enables 12+
TB memory systems
} Likely the first of many implementations of
another “mid-layer” of storage, closer to RAM
than even PCIe storage

} Opens up tremendous possibilities for all
databases being mostly in-memory databases,
and at least all indexes in memory
} Realistic state-snapshotting of in-execution
environments (ie: virtual machines) beyond just
disk-state snapshotting
} ULLtraDIMM represents the first phase (no pun
intended) in transitioning to “Storage Class
Memory”

Object Storage

} The fringe of scale is driving a new look
at storage for the future.
} Facebook / Google / Flickr, etc.
} Argonne, Lawrence Livermore, etc.
} Different problem domains, similar
storage issue – most POSIX filesystems
don’t scale well.
} Exceptions:
– ZFS, BTRFS
} Others fake it… (Lustre, GPFS, etc.)

Object storage
simplifies the back-end
requirements on how
data is housed,
however also implies
that the user or
application-space
technologies capture,
handle and manage
metadata associated
with the objects.

} Fully separate metadata and BLOB
functions from each other.
} Both in terms of performance, medium,
and reliability characteristics.

Applications use an API for access.
Simple Object
Metadata stored and protected
by customer/application
Gateway
Metadata stored and protected
OpenStack Swift
Swift compatible API
SITE 1 SITE 2 SITE 3
Named Object
(S3 Compatible)
Custom Application
Embedded Accesser
Java Libraries

Dramatic increases
in R&D funding and
investment over the
most recent decade
have largely taken
object stores from
the corner cases of
extreme-scale
providers only to the
more common use-case.
Growth has also
been driven by the
rapid uptake of “Big
Data” type
repositories.

• Leaders mostly driven by a
focus on heavy use of
commoditized hardware with
open API access
• Initially supportable only by
development organizations
capable of writing internally to
leverage object stores
• Standardization driven by
Amazon S3 adoption rates and
similar API (OpenStack Swift)
has enabled a rapidly maturing
base of supported applications
Capabilities
Strategies

} Simple
– Object storage solutions handle the vast
majority of failure scenarios in software
– Operationally, common repairs should
be “KISS” based
} Inexpensive
– Success with Ceph/Swift tends to rest
on metrics of $/TB with reasonable
access patterns
} Standard
– Basic, industry standard components,
adaptable and interchangeable with
commonly available hardware

} Replicate industry success
– Organizations running large object
storage solutions focus on highly
commoditized hardware
– Custom / White-box is pervasive in
Facebook, Google, eBay, Apple, etc.
because of unique form factors &
low price
– Density rules
– Performance & Reliability required
but also predominantly provided by
scale

} Not focused on single object performance, designed for
high-scale, large touchpoints. (ex. Flickr)
} Typically use the concept of “eventually consistent” or
“optionally consistent” – it takes data a while to
replicate.
} Limited update capability, just emerging, for byte-range
updating of containers/objects, but still “eventually
consistent”.
} Makes for an excellent backup / archive target.
} Now supported by many applications (Avere, Commvault,
etc.)

} Erasure Coding
– Mostly via Cleversafe / Scality
– Limited inclusion in open-source
products thus far, but on roadmaps
} Filesystem / POSIX interface
– Being built into Ceph
} API capabilities
– Vast differentiator, some (Swift /
S3) provide limited specific API,
others broadly allow different
approaches (Cleversafe / Ceph)

Erasure Coding… (Cleversafe / Scality)
10 11 12
SITE 3
DATA
Accessers®
1 2 3 4 5 8
6 7 9
SITE 1 SITE 2
[Total slice1s = 1
WIDTH]
Slicestors®
[Subset to read = THRESHOLD]
Data is virtualized,
encrypted, and
sliced using
Information Dispersal
Algorithms.
1
2 Slices are
dispersed to
separate disks,
storage nodes and
geographic
locations.
3
Even with
individual servers
or entire sites
down, real time
bit perfect data is
retrieved from a
subset of slices.
2 3 4 5 6 7 8
Accessers®
DATA
SCALABLE RELIABLE SECURE

Software Defined Storage

Next Generation Datacentres
Prineville, Oregon Lenoir, North Carolina Chicago, Illinois
What’s in common?
• Uniformity
• x86 hardware
• Local Disks
• Software defined networks
• Flat Layer 2 Designs
• (Minimal overhead lighting… PUE
driven!)
What’s missing?
• Fibre Channel Networks
• Disk shelves
• Physical SAN/NAS controllers
• Large UPS Systems
• Multitudes of Cabling

In general characteristics of software-defined storage could include any or
all of the following features
} Pooling and abstraction of the logical storage services and capabilities
from the underlying physical storage systems. This theme was also
reflected in the technical term storage virtualization.
} Automation with policy-driven storage provisioning with
service-level agreements replacing technology details. This requires
management interfaces that span traditional storage array products, as
a particular definition of separating "control plane" from "data plane"
} Virtual volumes or additional abstraction layers (objects, buckets,
containers, volumes, etc.) supporting different characteristics
} Commodity hardware with storage logic abstracted into a software
layer. This is also described as a clustered file system for
converged storage.
} Scale-out storage architecture (typical).

} Largely “Software Defined Storage” really just means
storage virtualization.
} But there is more to it than that, as truly defining storage
only via software allows architects to break away from
historical physical concepts.
} Will be a long time before things change. Witness
historical terms in server virtualization (virtual machine,
virtual disk, virtual network) rather than “instance”,
“datastore / container”, “data transport”, etc.

} EMC technology meeting the definition of both an Object Store, and
Software Defined Storage
} Emerging capabilities to support the realities of heterogeneous
storage demands in the enterprise
} Primary focus is on automation & intelligent provisioning
VNX Isilon
3rd Party
ViPR
Data Services
ViPR
Controller
VMAX
Commodity
EMC ViPR Platform
Provisioning Self-Service Reporting Automation

What is Virtual SAN? Key Features
§ Converged compute + storage solution
§ Utilizes DAS and server attached disks
§ Auto-tiering between SSD and HDD
§ Intelligent policy driven data placement
across the cluster
§ vCenter-integrated, instant storage
provisioning
Customer Benefits
§ High performance storage at up to
50% lower cost
§ Radically Simple Storage – configure
and manage without complex workflows
§ Designed for dynamic scalability
vCenter Server
vSphere
Virtual SAN
SSD Hard
Clusters internal server disks to and high resiliency
provide scalable shared storage with
cloud agility and efficiency.
Hard
disks
SSD
…………….
Hard
disks
disks
SSD
Distributed Storage
Aggregated Datastore

} As data moves between VM2 and the LUN
(shown with gold arrows), the FVP layer acts
as a read and write-back cache while also
mirroring the write-back data to another host
node (shown with a white arrow).
} This is done over the vMotion network by
default, but you can change this to use a
different network, or make a dedicated
network, if desired.
} If the virtualization administrator or DRS
migrates VM2 from host ESXi A to B with a
vMotion, the flash will already contain the
hot write-back data and performance should
remain relatively high.
} In the case of write-through data, FVP will
allow the host to read from a peer host’s
flash tier to avoid going back to the SAN until
the local flash has warmed up.

Virtual Machine/Virtual Disk
Virtual Storage Control Virtual Storage Control
Flash HDD
Fastest Performance
Localization, tiering,
deduplication
Hypervisor Agnostic
vSphere, KVM,
Hyper-V
Enterprise Storage
Clones, snapshots,
replication,
compression, thin
provisioning

NDFS Cluster
Infinite Scalability
• Controller VMs create NDFS cluster
• Storage Pool created from all Disks (RF:2)
• NFS datastore presented to hypervisor (SMB for Hyper-V)
• I/O reads local, I/O writes synchronous

The great majority of Read/Write I/O is serviced locally
• vMotion or HA event triggers RF review and data locality
• Only accessed blocks are brought local (RF maintained)
• Background process (no flooding network)

• Replication Factor (RF) : 2
• Synchronous replication of blocks between node OpLogs
• Fault Tolerance for Node failure
• All nodes participate in OpLog replication (no “hot nodes”)
• Asynchronous data drained to extent store (HDD)
© 2013 Scalar Decisions Inc. Not for distribution outside of intended audience 10

Implications?

QUESTIONS?

THANK YOU.

Emerging Storage Trends Discussed at Scalar Presentation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Emerging Storage Trends Discussed at Scalar Presentation

Similar to Emerging Storage Trends Discussed at Scalar Presentation (20)

More from patmisasi

More from patmisasi (20)

Recently uploaded

Recently uploaded (20)

Emerging Storage Trends Discussed at Scalar Presentation