SlideShare a Scribd company logo

 your choice
 reduce
File Based StorageBlock StorageSAN AlternativeHyper-Converged Cloud Fabric
SMB3
Hyper-V cluster with local storage
Hyper-V Cluster
Scale-Out File Server Cluster
Hyper-V Cluster(s)
SMB3 Storage Network Fabric
 Software-Defined Storage Design Calculator
 Software-Defined Storage Design Considerations
Guide





•
•
•
• When compute + storage needs grow together
• Great for small clusters
•
•
•
•
•
•
•
•
3-node
Windows Server
scale-out file server
Shared SAS storage enclosures
SAS cables
Node
w/disks
Node
w/disks
Node
w/disks
Node
w/disks
SMB Network w/RDMA
Servers Enclosures Disk/Enclosure Resiliency Raw/Net Capacity (TB) $/GB
2 1 4x 800GB SSD, 8x 6TB HDD 2-way mirror 50 / 23 1.41
2 2 12x 1.6TB SSD, 48x 8TB HDD 3-way mirror 770 / 230 1.14
4 4 12x 1.6TB SSD, 48x 8TB HDD 3-way w/ EA 1500 / 450 1.18
Servers Disks (local) Resiliency Raw/Net Capacity (TB) $/GB
2
4x 800GB SSD, 12x 4TB HDD
2-way mirror 96 / 42 1.02
3 3-way mirror 144 / 42 1.53
4
3-way mirror/dual-parity
192 / 90 0.95
8 384 / 217 0.79
12 576 / 368 0.70
16 768 / 505 0.68
Storage
Options
Flash support SAS SSD
Flash caches 1 GB WBC per virtual disk
Performance tier Mirror
Capacity tier Mirror
Tier optimization Nightly (default)
Flash sizing 5-6% of raw capacity
SAS SSD SAS or SATA SSD, NVMe
1 GB WBC per virtual disk
ReFS read cache
(40% of SSD capacity)
Read/Write cache
Mirror Replaced by flash cache
Parity
Mirror (~10%)
Parity (~90)
Real-time w/ReFS
5-6% of raw capacity
Windows Server 2016
Disaggregated | Hyper-converged
Shared SAS w/ReFS* Storage Space Direct*
OS Windows Server 2012 R2
Architecture Disaggregated | Hyper-converged
Model Shared SAS
Resiliency
Options
Cluster nodes 2-4
Enclosures 1-4
MPIO Yes
Max pool size 84 disks
SSDs to repair 0-1 per pool
HDDs to repair 0-4 per pool
Virtual disks/node 2-4
Fault domains Disk or enclosure
Dual-site clusters
w/Storage Replica No
2-4 2*-16 (64?)
1-4 0-1
Yes No
84 disks 240
0-1 per pool 0
0-4 per pool 0-2 per node
2-4
Disk or enclosure Cluster node, chassis, rack
2+ nodes per site 3-4+ nodes per site
Windows Server 2016
Disaggregated | Hyper-converged
Shared SAS Storage Space Direct
OS Windows Server 2012 R2
Architecture Disaggregated | Hyper-converged
Model Shared SAS
•
•
•
•
•
•
•
Network
Options
Bandwidth >= 1Gbit/s
Protocol Ethernet, IB3
Interfaces/Server 2
NIC Teaming Not w/RDMA
SET Teaming -
RSS Recommended
RSC Recommended
RDMA1 Preferred
PacketDirect -
In-box drivers
Not recommended for
production2
>=10Gib/s
Ethernet, IB3
2
Yes
Recommended
Recommended
Recommended
Preferred Recommended
Preferred
Not supported2
1 Either iWARP or RoCE; however, RoCE requires additional server and switch configuration/support (PFC and DCB) and VLANs
2 Use approved NIC driver from solution vendor
3 Don’t use Infiniband w/hyper-converged
Windows Server 2016
Disaggregated | Hyper-converged
Shared SAS Storage Space Direct
OS Windows Server 2012 R2
Architecture Disaggregated | Hyper-converged
Model Shared SAS
Windows Server 2016
Disaggregated | Hyper-converged
Shared SAS Storage Space Direct
OS Windows Server 2012 R2
Architecture Disaggregated | Hyper-converged
Model Shared SAS
RAM for
storage caches
None
Total RAM 64 GB
Extra RAM for VMs 32-128 GB if hyper-converged
CPU
2 6-core CPUs typical
(add for extra SSDs, Data
Deduplication, hyper-converged)
16 GB CSV cache?
10 GB per 1 TB of flash cache
16 GB CSV cache?
64 GB 64 GB
32-128 GB if hyper-converged
2 6-core CPUs typical
(add for extra SSDs, Data Deduplication, hyper-converged)
Windows Server 2016
Disaggregated | Hyper-converged
Shared SAS Storage Space Direct
OS Windows Server 2012 R2
Architecture Disaggregated | Hyper-converged
Model Shared SAS
Solution vendors
Dell
DataOn
RAID Inc.
HP
Dell
DataOn
RAID Inc.
HP
Dell
Cisco
Fujotsu
HP
Intel
Lenovo
Quanta
Quanta D51PH

HP Apollo 2000 System
Dell PowerEdge R730xdCisco UCS C3260 Rack Server
Intel® Server Board S2600WT-Based Systems Lenovo System x3650 M5
Fujitsu Primergy RX2540 M1
Node
File-Delete
Notification (TRIM)
Disable w/FSUtil
Storage Spaces
registry changes
Optimize Storage Spaces repair
settings on Scale-Out File
Servers
MPIO policy Optimize with MPClaim
Physical disk caches Disable with Diskcache.exe
CSV read cache Max 64 TB; by-passed w/tiers
Pool
RetireMissingPhysic
alDisks
Set with Set-StoragePool
Volume
Column count Calculate w/spreadsheet
Volume size
10 TB if using Volsnap for VSS
snapshots
UseLargeFRS Set with Format-Volume
Use defaults
Use defaults
Use defaults N/A
Use defaults
Supported
Use defaults (handled by Health Service)
Use defaults 
>10TB if using ReFS snapshots
10 TB if using Volsnap for VSS snapshots
Use defaults
Windows Server 2016
Disaggregated | Hyper-converged
Shared SAS Storage Space Direct
OS Windows Server 2012 R2
Architecture Disaggregated | Hyper-converged
Model Shared SAS
Storage Spaces Storage Pool
Storage Spaces Virtual Disks
Scale-Out File Server
CSVFS Cluster File System
Software Storage Bus
Virtual Machines
1 2
ReFS On-Disk File System
Virtual Machines
ClusPort
SpacePort
Virtual Disks
File System
Cluster Shared Volumes File System (CSVFS)
Application
Node 1
Block over SMB
ClusBflt
Node 2
Physical Devices
ClusPort
SpacePort
Virtual Disks
File System
ClusBflt
Physical Devices
Storage
Configuration
Caching
devices
Capacity
devices
Caching
behavior
SSD + HDD SSD HDD Write + Read
SSD + SSD SSD SSD Write only
Caching Devices
Capacity Devices
TP5
New-Cluster
Enable-ClusterS2D
New-Volume
New-Cluster
Enable-ClusterS2D
New-StoragePool
Set-PhysicalDisk -Usage Journal
New-StorageTier -ResiliencySettingName Mirror
New-StorageTier -ResiliencySettingName Parity
New-Volume
$mt = New-StorageTier -FriendlyName Mirror -MediaType SSD -ResiliencySettingName Mirror
$pt = New-StorageTier -FriendlyName Parity -MediaType SSD -ResiliencySettingName Parity
New-Volume -FriendlyName SQlServer -StorageTiers $mt -StorageTierSizes 1TB
New-Volume -FriendlyName GenericVM -StorageTiers $mt,$pt -StorageTierSizes 100GB,900GB
Mirror Parity
$mt = New-StorageTier -FriendlyName Mirror -MediaType SSD -ResiliencySettingName Mirror
$pt = New-StorageTier -FriendlyName Parity -MediaType HDD -ResiliencySettingName Parity
New-Volume -FriendlyName SQlServer -StorageTiers $mt -StorageTierSizes 1TB
New-Volume -FriendlyName GenericVM -StorageTiers $mt,$pt -StorageTierSizes 100GB,900GB
Mirror Parity
Potential metadata
devices
Virtual Disk Mirror Parity Mixed-Resiliency
Optimized for Performance Capacity Balanced
Use case All data is hot All data is cold Mix of hot and cold
Efficiency Least (33%) Most (57+%) Depends on mix
File System ReFS (or NTFS) ReFS (or NTFS) Requires ReFS
Node requirement 2+ 4+ 4+
Mirror tier Parity tier
Server 2 Server 3 Server 4Server 1
A A’ A’’B B’ B’’Mirror
Parity
X1 X2 Px
Y1 Y2 Py
Q
Nodes Mirror tier
Efficiency
Parity Tier
Efficiency
Resiliency
4 33% 57% 1 node
8 33% 80% 1 node
16 33% 84% 1 node
LRC Erasure Coding in Storage Spaces
Mirror tier Parity tier
W
ReFS
•
•
•
•
•
•
Storage spaces direct webinar
Storage spaces direct webinar

More Related Content

What's hot

Enterprise Storage NAS - Dual Controller
Enterprise Storage NAS - Dual ControllerEnterprise Storage NAS - Dual Controller
Enterprise Storage NAS - Dual ControllerFernando Barrientos
 
Exchange 2010 New England Vmug
Exchange 2010 New England VmugExchange 2010 New England Vmug
Exchange 2010 New England Vmugcsharney
 
Storage Enhancements in Windows 2012 R2
Storage Enhancements in Windows 2012 R2Storage Enhancements in Windows 2012 R2
Storage Enhancements in Windows 2012 R2Michael Rüefli
 
VMware Performance Troubleshooting
VMware Performance TroubleshootingVMware Performance Troubleshooting
VMware Performance Troubleshootingglbsolutions
 
Varrow madness 2013 virtualizing sql presentation
Varrow madness 2013 virtualizing sql presentationVarrow madness 2013 virtualizing sql presentation
Varrow madness 2013 virtualizing sql presentationpittmantony
 
Simplifying Ceph Management with Virtual Storage Manager (VSM)
Simplifying Ceph Management with Virtual Storage Manager (VSM)Simplifying Ceph Management with Virtual Storage Manager (VSM)
Simplifying Ceph Management with Virtual Storage Manager (VSM)Ceph Community
 
IMCSummit 2015 - Day 1 Developer Session - The Science and Engineering Behind...
IMCSummit 2015 - Day 1 Developer Session - The Science and Engineering Behind...IMCSummit 2015 - Day 1 Developer Session - The Science and Engineering Behind...
IMCSummit 2015 - Day 1 Developer Session - The Science and Engineering Behind...In-Memory Computing Summit
 
Xen Virtualization 2008
Xen Virtualization 2008Xen Virtualization 2008
Xen Virtualization 2008mwlang88
 
State of the Art Thin Provisioning
State of the Art Thin ProvisioningState of the Art Thin Provisioning
State of the Art Thin ProvisioningStephen Foskett
 
VMware Advance Troubleshooting Workshop - Day 4
VMware Advance Troubleshooting Workshop - Day 4VMware Advance Troubleshooting Workshop - Day 4
VMware Advance Troubleshooting Workshop - Day 4Vepsun Technologies
 
ProfessionalVMware BrownBag VCP5 Section3: Storage
ProfessionalVMware BrownBag VCP5 Section3: StorageProfessionalVMware BrownBag VCP5 Section3: Storage
ProfessionalVMware BrownBag VCP5 Section3: StorageProfessionalVMware
 
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best Practices
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best PracticesVMworld 2014: Advanced SQL Server on vSphere Techniques and Best Practices
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best PracticesVMworld
 
Developing a Ceph Appliance for Secure Environments
Developing a Ceph Appliance for Secure EnvironmentsDeveloping a Ceph Appliance for Secure Environments
Developing a Ceph Appliance for Secure EnvironmentsCeph Community
 
Data core makes_ha_nas_practical_20mar12
Data core makes_ha_nas_practical_20mar12Data core makes_ha_nas_practical_20mar12
Data core makes_ha_nas_practical_20mar12jelenaveskovic
 
Vm13 vnx mixed workloads
Vm13 vnx mixed workloadsVm13 vnx mixed workloads
Vm13 vnx mixed workloadspittmantony
 

What's hot (19)

Enterprise Storage NAS - Dual Controller
Enterprise Storage NAS - Dual ControllerEnterprise Storage NAS - Dual Controller
Enterprise Storage NAS - Dual Controller
 
Exchange 2010 New England Vmug
Exchange 2010 New England VmugExchange 2010 New England Vmug
Exchange 2010 New England Vmug
 
TDS-16489U - Dual Processor
TDS-16489U - Dual ProcessorTDS-16489U - Dual Processor
TDS-16489U - Dual Processor
 
Storage Enhancements in Windows 2012 R2
Storage Enhancements in Windows 2012 R2Storage Enhancements in Windows 2012 R2
Storage Enhancements in Windows 2012 R2
 
Scale2014
Scale2014Scale2014
Scale2014
 
VMware Performance Troubleshooting
VMware Performance TroubleshootingVMware Performance Troubleshooting
VMware Performance Troubleshooting
 
Varrow madness 2013 virtualizing sql presentation
Varrow madness 2013 virtualizing sql presentationVarrow madness 2013 virtualizing sql presentation
Varrow madness 2013 virtualizing sql presentation
 
Simplifying Ceph Management with Virtual Storage Manager (VSM)
Simplifying Ceph Management with Virtual Storage Manager (VSM)Simplifying Ceph Management with Virtual Storage Manager (VSM)
Simplifying Ceph Management with Virtual Storage Manager (VSM)
 
IMCSummit 2015 - Day 1 Developer Session - The Science and Engineering Behind...
IMCSummit 2015 - Day 1 Developer Session - The Science and Engineering Behind...IMCSummit 2015 - Day 1 Developer Session - The Science and Engineering Behind...
IMCSummit 2015 - Day 1 Developer Session - The Science and Engineering Behind...
 
Xen Virtualization 2008
Xen Virtualization 2008Xen Virtualization 2008
Xen Virtualization 2008
 
Redis Persistence
Redis  PersistenceRedis  Persistence
Redis Persistence
 
State of the Art Thin Provisioning
State of the Art Thin ProvisioningState of the Art Thin Provisioning
State of the Art Thin Provisioning
 
VMware Advance Troubleshooting Workshop - Day 4
VMware Advance Troubleshooting Workshop - Day 4VMware Advance Troubleshooting Workshop - Day 4
VMware Advance Troubleshooting Workshop - Day 4
 
ProfessionalVMware BrownBag VCP5 Section3: Storage
ProfessionalVMware BrownBag VCP5 Section3: StorageProfessionalVMware BrownBag VCP5 Section3: Storage
ProfessionalVMware BrownBag VCP5 Section3: Storage
 
110629 nexenta- andy bennett
110629   nexenta- andy bennett110629   nexenta- andy bennett
110629 nexenta- andy bennett
 
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best Practices
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best PracticesVMworld 2014: Advanced SQL Server on vSphere Techniques and Best Practices
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best Practices
 
Developing a Ceph Appliance for Secure Environments
Developing a Ceph Appliance for Secure EnvironmentsDeveloping a Ceph Appliance for Secure Environments
Developing a Ceph Appliance for Secure Environments
 
Data core makes_ha_nas_practical_20mar12
Data core makes_ha_nas_practical_20mar12Data core makes_ha_nas_practical_20mar12
Data core makes_ha_nas_practical_20mar12
 
Vm13 vnx mixed workloads
Vm13 vnx mixed workloadsVm13 vnx mixed workloads
Vm13 vnx mixed workloads
 

Viewers also liked

Windows Server "10": что нового в виртуализации
Windows Server "10": что нового в виртуализацииWindows Server "10": что нового в виртуализации
Windows Server "10": что нового в виртуализацииВиталий Стародубцев
 
APP-V 5.0 Service Pack 3, виртуализация Microsoft Office
APP-V 5.0 Service Pack 3, виртуализация Microsoft OfficeAPP-V 5.0 Service Pack 3, виртуализация Microsoft Office
APP-V 5.0 Service Pack 3, виртуализация Microsoft OfficeВиталий Стародубцев
 
SCRUM - разработка без начальника
SCRUM - разработка без начальникаSCRUM - разработка без начальника
SCRUM - разработка без начальникаRealSpeaker 2.0
 
Эволюция программно-аппаратного обеспечения хранения фотографий в Badoo / Дми...
Эволюция программно-аппаратного обеспечения хранения фотографий в Badoo / Дми...Эволюция программно-аппаратного обеспечения хранения фотографий в Badoo / Дми...
Эволюция программно-аппаратного обеспечения хранения фотографий в Badoo / Дми...Ontico
 

Viewers also liked (7)

Windows Server "10": что нового в виртуализации
Windows Server "10": что нового в виртуализацииWindows Server "10": что нового в виртуализации
Windows Server "10": что нового в виртуализации
 
APP-V 5.0 Service Pack 3, виртуализация Microsoft Office
APP-V 5.0 Service Pack 3, виртуализация Microsoft OfficeAPP-V 5.0 Service Pack 3, виртуализация Microsoft Office
APP-V 5.0 Service Pack 3, виртуализация Microsoft Office
 
Презентация RDS & App-V, VDI
Презентация RDS & App-V, VDIПрезентация RDS & App-V, VDI
Презентация RDS & App-V, VDI
 
Advanced Directory Services Windows Server 2012
Advanced Directory Services Windows Server 2012Advanced Directory Services Windows Server 2012
Advanced Directory Services Windows Server 2012
 
System Center 2012 Operations Manager + SP1
System Center 2012 Operations Manager + SP1 System Center 2012 Operations Manager + SP1
System Center 2012 Operations Manager + SP1
 
SCRUM - разработка без начальника
SCRUM - разработка без начальникаSCRUM - разработка без начальника
SCRUM - разработка без начальника
 
Эволюция программно-аппаратного обеспечения хранения фотографий в Badoo / Дми...
Эволюция программно-аппаратного обеспечения хранения фотографий в Badoo / Дми...Эволюция программно-аппаратного обеспечения хранения фотографий в Badoo / Дми...
Эволюция программно-аппаратного обеспечения хранения фотографий в Badoo / Дми...
 

Similar to Storage spaces direct webinar

Storage, San And Business Continuity Overview
Storage, San And Business Continuity OverviewStorage, San And Business Continuity Overview
Storage, San And Business Continuity OverviewAlan McSweeney
 
Ceph Day San Jose - Red Hat Storage Acceleration Utlizing Flash Technology
Ceph Day San Jose - Red Hat Storage Acceleration Utlizing Flash TechnologyCeph Day San Jose - Red Hat Storage Acceleration Utlizing Flash Technology
Ceph Day San Jose - Red Hat Storage Acceleration Utlizing Flash TechnologyCeph Community
 
Windows Server 2012 R2 Software-Defined Storage
Windows Server 2012 R2 Software-Defined StorageWindows Server 2012 R2 Software-Defined Storage
Windows Server 2012 R2 Software-Defined StorageAidan Finn
 
Red Hat Ceph Storage Acceleration Utilizing Flash Technology
Red Hat Ceph Storage Acceleration Utilizing Flash Technology Red Hat Ceph Storage Acceleration Utilizing Flash Technology
Red Hat Ceph Storage Acceleration Utilizing Flash Technology Red_Hat_Storage
 
Red Hat Storage Day Dallas - Red Hat Ceph Storage Acceleration Utilizing Flas...
Red Hat Storage Day Dallas - Red Hat Ceph Storage Acceleration Utilizing Flas...Red Hat Storage Day Dallas - Red Hat Ceph Storage Acceleration Utilizing Flas...
Red Hat Storage Day Dallas - Red Hat Ceph Storage Acceleration Utilizing Flas...Red_Hat_Storage
 
DUG'20: 10 - Storage Orchestration for Composable Storage Architectures
DUG'20: 10 - Storage Orchestration for Composable Storage ArchitecturesDUG'20: 10 - Storage Orchestration for Composable Storage Architectures
DUG'20: 10 - Storage Orchestration for Composable Storage ArchitecturesAndrey Kudryavtsev
 
Varrow datacenter storage today and tomorrow
Varrow   datacenter storage today and tomorrowVarrow   datacenter storage today and tomorrow
Varrow datacenter storage today and tomorrowpittmantony
 
JetStor NAS ZFS based 716U 724U Network Attached Storage
JetStor NAS ZFS based 716U 724U Network Attached StorageJetStor NAS ZFS based 716U 724U Network Attached Storage
JetStor NAS ZFS based 716U 724U Network Attached StorageGene Leyzarovich
 
Planning & Best Practice for Microsoft Virtualization
Planning & Best Practice for Microsoft VirtualizationPlanning & Best Practice for Microsoft Virtualization
Planning & Best Practice for Microsoft VirtualizationLai Yoong Seng
 
Webinar NETGEAR - ReadyNAS, le novità hardware e software
Webinar NETGEAR - ReadyNAS, le novità hardware e softwareWebinar NETGEAR - ReadyNAS, le novità hardware e software
Webinar NETGEAR - ReadyNAS, le novità hardware e softwareNetgear Italia
 
Webinar NETGEAR - ReadyNAS come File Server: condivisione, collaborazione e ...
Webinar NETGEAR - ReadyNAS come File Server:  condivisione, collaborazione e ...Webinar NETGEAR - ReadyNAS come File Server:  condivisione, collaborazione e ...
Webinar NETGEAR - ReadyNAS come File Server: condivisione, collaborazione e ...Netgear Italia
 
VMware - Virtual SAN - IT Changes Everything
VMware - Virtual SAN - IT Changes EverythingVMware - Virtual SAN - IT Changes Everything
VMware - Virtual SAN - IT Changes EverythingVMUG IT
 
openSUSE storage workshop 2016
openSUSE storage workshop 2016openSUSE storage workshop 2016
openSUSE storage workshop 2016Alex Lau
 
VMworld Europe 2014: Virtual SAN Best Practices and Use Cases
VMworld Europe 2014: Virtual SAN Best Practices and Use CasesVMworld Europe 2014: Virtual SAN Best Practices and Use Cases
VMworld Europe 2014: Virtual SAN Best Practices and Use CasesVMworld
 
Demystifying Storage - Building large SANs
Demystifying  Storage - Building large SANsDemystifying  Storage - Building large SANs
Demystifying Storage - Building large SANsDirecti Group
 
Storage essentials (by Merlin Ran)
Storage essentials (by Merlin Ran)Storage essentials (by Merlin Ran)
Storage essentials (by Merlin Ran)gigix1980
 
Taking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout SessionTaking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout SessionSplunk
 
Demystifying Storage
Demystifying  StorageDemystifying  Storage
Demystifying Storagebhavintu79
 

Similar to Storage spaces direct webinar (20)

Storage, San And Business Continuity Overview
Storage, San And Business Continuity OverviewStorage, San And Business Continuity Overview
Storage, San And Business Continuity Overview
 
Ceph Day San Jose - Red Hat Storage Acceleration Utlizing Flash Technology
Ceph Day San Jose - Red Hat Storage Acceleration Utlizing Flash TechnologyCeph Day San Jose - Red Hat Storage Acceleration Utlizing Flash Technology
Ceph Day San Jose - Red Hat Storage Acceleration Utlizing Flash Technology
 
Windows Server 2012 R2 Software-Defined Storage
Windows Server 2012 R2 Software-Defined StorageWindows Server 2012 R2 Software-Defined Storage
Windows Server 2012 R2 Software-Defined Storage
 
Red Hat Ceph Storage Acceleration Utilizing Flash Technology
Red Hat Ceph Storage Acceleration Utilizing Flash Technology Red Hat Ceph Storage Acceleration Utilizing Flash Technology
Red Hat Ceph Storage Acceleration Utilizing Flash Technology
 
Red Hat Storage Day Dallas - Red Hat Ceph Storage Acceleration Utilizing Flas...
Red Hat Storage Day Dallas - Red Hat Ceph Storage Acceleration Utilizing Flas...Red Hat Storage Day Dallas - Red Hat Ceph Storage Acceleration Utilizing Flas...
Red Hat Storage Day Dallas - Red Hat Ceph Storage Acceleration Utilizing Flas...
 
DUG'20: 10 - Storage Orchestration for Composable Storage Architectures
DUG'20: 10 - Storage Orchestration for Composable Storage ArchitecturesDUG'20: 10 - Storage Orchestration for Composable Storage Architectures
DUG'20: 10 - Storage Orchestration for Composable Storage Architectures
 
Varrow datacenter storage today and tomorrow
Varrow   datacenter storage today and tomorrowVarrow   datacenter storage today and tomorrow
Varrow datacenter storage today and tomorrow
 
JetStor NAS ZFS based 716U 724U Network Attached Storage
JetStor NAS ZFS based 716U 724U Network Attached StorageJetStor NAS ZFS based 716U 724U Network Attached Storage
JetStor NAS ZFS based 716U 724U Network Attached Storage
 
Planning & Best Practice for Microsoft Virtualization
Planning & Best Practice for Microsoft VirtualizationPlanning & Best Practice for Microsoft Virtualization
Planning & Best Practice for Microsoft Virtualization
 
Webinar NETGEAR - ReadyNAS, le novità hardware e software
Webinar NETGEAR - ReadyNAS, le novità hardware e softwareWebinar NETGEAR - ReadyNAS, le novità hardware e software
Webinar NETGEAR - ReadyNAS, le novità hardware e software
 
NVMe over Fabric
NVMe over FabricNVMe over Fabric
NVMe over Fabric
 
Webinar NETGEAR - ReadyNAS come File Server: condivisione, collaborazione e ...
Webinar NETGEAR - ReadyNAS come File Server:  condivisione, collaborazione e ...Webinar NETGEAR - ReadyNAS come File Server:  condivisione, collaborazione e ...
Webinar NETGEAR - ReadyNAS come File Server: condivisione, collaborazione e ...
 
VMware - Virtual SAN - IT Changes Everything
VMware - Virtual SAN - IT Changes EverythingVMware - Virtual SAN - IT Changes Everything
VMware - Virtual SAN - IT Changes Everything
 
openSUSE storage workshop 2016
openSUSE storage workshop 2016openSUSE storage workshop 2016
openSUSE storage workshop 2016
 
VMworld Europe 2014: Virtual SAN Best Practices and Use Cases
VMworld Europe 2014: Virtual SAN Best Practices and Use CasesVMworld Europe 2014: Virtual SAN Best Practices and Use Cases
VMworld Europe 2014: Virtual SAN Best Practices and Use Cases
 
DAS RAID NAS SAN
DAS RAID NAS SANDAS RAID NAS SAN
DAS RAID NAS SAN
 
Demystifying Storage - Building large SANs
Demystifying  Storage - Building large SANsDemystifying  Storage - Building large SANs
Demystifying Storage - Building large SANs
 
Storage essentials (by Merlin Ran)
Storage essentials (by Merlin Ran)Storage essentials (by Merlin Ran)
Storage essentials (by Merlin Ran)
 
Taking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout SessionTaking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout Session
 
Demystifying Storage
Demystifying  StorageDemystifying  Storage
Demystifying Storage
 

More from Виталий Стародубцев

FirstLook: Экзамен 70-333 и 70-334 (на основе курсов 20333, 20334)
FirstLook: Экзамен 70-333 и 70-334 (на основе курсов 20333, 20334)FirstLook: Экзамен 70-333 и 70-334 (на основе курсов 20333, 20334)
FirstLook: Экзамен 70-333 и 70-334 (на основе курсов 20333, 20334)Виталий Стародубцев
 
Windows Server «10»: Что нового в виртуализации
Windows Server «10»: Что нового в виртуализацииWindows Server «10»: Что нового в виртуализации
Windows Server «10»: Что нового в виртуализацииВиталий Стародубцев
 
Обзор администрирования и подготовка к сертификации по Office 365
Обзор администрирования и подготовка к сертификации по Office 365 Обзор администрирования и подготовка к сертификации по Office 365
Обзор администрирования и подготовка к сертификации по Office 365 Виталий Стародубцев
 
Подготовка к Экзамену Microsoft 70-411 (MCSA: Windows Server 2012)
Подготовка к Экзамену Microsoft 70-411 (MCSA: Windows Server 2012)Подготовка к Экзамену Microsoft 70-411 (MCSA: Windows Server 2012)
Подготовка к Экзамену Microsoft 70-411 (MCSA: Windows Server 2012)Виталий Стародубцев
 
Подготовка к Экзамену Microsoft 70-410 с А. Кибкало
Подготовка к Экзамену Microsoft 70-410 с А. КибкалоПодготовка к Экзамену Microsoft 70-410 с А. Кибкало
Подготовка к Экзамену Microsoft 70-410 с А. КибкалоВиталий Стародубцев
 

More from Виталий Стародубцев (8)

FirstLook: Экзамен 70-333 и 70-334 (на основе курсов 20333, 20334)
FirstLook: Экзамен 70-333 и 70-334 (на основе курсов 20333, 20334)FirstLook: Экзамен 70-333 и 70-334 (на основе курсов 20333, 20334)
FirstLook: Экзамен 70-333 и 70-334 (на основе курсов 20333, 20334)
 
Windows Server «10»: Что нового в виртуализации
Windows Server «10»: Что нового в виртуализацииWindows Server «10»: Что нового в виртуализации
Windows Server «10»: Что нового в виртуализации
 
Обзор администрирования и подготовка к сертификации по Office 365
Обзор администрирования и подготовка к сертификации по Office 365 Обзор администрирования и подготовка к сертификации по Office 365
Обзор администрирования и подготовка к сертификации по Office 365
 
Будущее Directory Services (в облаках)
Будущее Directory Services (в облаках) Будущее Directory Services (в облаках)
Будущее Directory Services (в облаках)
 
Аренда класса
Аренда классаАренда класса
Аренда класса
 
Подготовка к Экзамену Microsoft 70-411 (MCSA: Windows Server 2012)
Подготовка к Экзамену Microsoft 70-411 (MCSA: Windows Server 2012)Подготовка к Экзамену Microsoft 70-411 (MCSA: Windows Server 2012)
Подготовка к Экзамену Microsoft 70-411 (MCSA: Windows Server 2012)
 
Подготовка к Экзамену Microsoft 70-410 с А. Кибкало
Подготовка к Экзамену Microsoft 70-410 с А. КибкалоПодготовка к Экзамену Microsoft 70-410 с А. Кибкало
Подготовка к Экзамену Microsoft 70-410 с А. Кибкало
 
Windows Server 2012 R2! Что нового в Hyper-V?
Windows Server 2012 R2! Что нового в Hyper-V?Windows Server 2012 R2! Что нового в Hyper-V?
Windows Server 2012 R2! Что нового в Hyper-V?
 

Recently uploaded

UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...Sayali Powar
 
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdfDanh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdfQucHHunhnh
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfTamralipta Mahavidyalaya
 
Application of Matrices in real life. Presentation on application of matrices
Application of Matrices in real life. Presentation on application of matricesApplication of Matrices in real life. Presentation on application of matrices
Application of Matrices in real life. Presentation on application of matricesRased Khan
 
The Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve ThomasonThe Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve ThomasonSteve Thomason
 
Solid waste management & Types of Basic civil Engineering notes by DJ Sir.pptx
Solid waste management & Types of Basic civil Engineering notes by DJ Sir.pptxSolid waste management & Types of Basic civil Engineering notes by DJ Sir.pptx
Solid waste management & Types of Basic civil Engineering notes by DJ Sir.pptxDenish Jangid
 
Fish and Chips - have they had their chips
Fish and Chips - have they had their chipsFish and Chips - have they had their chips
Fish and Chips - have they had their chipsGeoBlogs
 
How to Break the cycle of negative Thoughts
How to Break the cycle of negative ThoughtsHow to Break the cycle of negative Thoughts
How to Break the cycle of negative ThoughtsCol Mukteshwar Prasad
 
Salient features of Environment protection Act 1986.pptx
Salient features of Environment protection Act 1986.pptxSalient features of Environment protection Act 1986.pptx
Salient features of Environment protection Act 1986.pptxakshayaramakrishnan21
 
Advances in production technology of Grapes.pdf
Advances in production technology of Grapes.pdfAdvances in production technology of Grapes.pdf
Advances in production technology of Grapes.pdfDr. M. Kumaresan Hort.
 
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdfINU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdfbu07226
 
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptxslides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptxCapitolTechU
 
Accounting and finance exit exam 2016 E.C.pdf
Accounting and finance exit exam 2016 E.C.pdfAccounting and finance exit exam 2016 E.C.pdf
Accounting and finance exit exam 2016 E.C.pdfYibeltalNibretu
 
Benefits and Challenges of Using Open Educational Resources
Benefits and Challenges of Using Open Educational ResourcesBenefits and Challenges of Using Open Educational Resources
Benefits and Challenges of Using Open Educational Resourcesdimpy50
 
Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345beazzy04
 
How to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS ModuleHow to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS ModuleCeline George
 
Jose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptx
Jose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptxJose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptx
Jose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptxricssacare
 
Palestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptxPalestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptxRaedMohamed3
 

Recently uploaded (20)

UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
 
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdfDanh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
 
Application of Matrices in real life. Presentation on application of matrices
Application of Matrices in real life. Presentation on application of matricesApplication of Matrices in real life. Presentation on application of matrices
Application of Matrices in real life. Presentation on application of matrices
 
Mattingly "AI & Prompt Design: Limitations and Solutions with LLMs"
Mattingly "AI & Prompt Design: Limitations and Solutions with LLMs"Mattingly "AI & Prompt Design: Limitations and Solutions with LLMs"
Mattingly "AI & Prompt Design: Limitations and Solutions with LLMs"
 
The Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve ThomasonThe Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve Thomason
 
Solid waste management & Types of Basic civil Engineering notes by DJ Sir.pptx
Solid waste management & Types of Basic civil Engineering notes by DJ Sir.pptxSolid waste management & Types of Basic civil Engineering notes by DJ Sir.pptx
Solid waste management & Types of Basic civil Engineering notes by DJ Sir.pptx
 
Fish and Chips - have they had their chips
Fish and Chips - have they had their chipsFish and Chips - have they had their chips
Fish and Chips - have they had their chips
 
How to Break the cycle of negative Thoughts
How to Break the cycle of negative ThoughtsHow to Break the cycle of negative Thoughts
How to Break the cycle of negative Thoughts
 
Salient features of Environment protection Act 1986.pptx
Salient features of Environment protection Act 1986.pptxSalient features of Environment protection Act 1986.pptx
Salient features of Environment protection Act 1986.pptx
 
Advances in production technology of Grapes.pdf
Advances in production technology of Grapes.pdfAdvances in production technology of Grapes.pdf
Advances in production technology of Grapes.pdf
 
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdfINU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
 
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptxslides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
 
Accounting and finance exit exam 2016 E.C.pdf
Accounting and finance exit exam 2016 E.C.pdfAccounting and finance exit exam 2016 E.C.pdf
Accounting and finance exit exam 2016 E.C.pdf
 
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
Operations Management - Book1.p  - Dr. Abdulfatah A. SalemOperations Management - Book1.p  - Dr. Abdulfatah A. Salem
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
 
Benefits and Challenges of Using Open Educational Resources
Benefits and Challenges of Using Open Educational ResourcesBenefits and Challenges of Using Open Educational Resources
Benefits and Challenges of Using Open Educational Resources
 
Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345
 
How to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS ModuleHow to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS Module
 
Jose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptx
Jose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptxJose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptx
Jose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptx
 
Palestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptxPalestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptx
 

Storage spaces direct webinar

  • 1.
  • 2.   your choice  reduce File Based StorageBlock StorageSAN AlternativeHyper-Converged Cloud Fabric SMB3
  • 3. Hyper-V cluster with local storage
  • 4. Hyper-V Cluster Scale-Out File Server Cluster Hyper-V Cluster(s) SMB3 Storage Network Fabric
  • 5.
  • 6.
  • 7.
  • 8.  Software-Defined Storage Design Calculator  Software-Defined Storage Design Considerations Guide 
  • 10.
  • 11. • • • • When compute + storage needs grow together • Great for small clusters
  • 12. • • • • • • • • 3-node Windows Server scale-out file server Shared SAS storage enclosures SAS cables Node w/disks Node w/disks Node w/disks Node w/disks SMB Network w/RDMA
  • 13. Servers Enclosures Disk/Enclosure Resiliency Raw/Net Capacity (TB) $/GB 2 1 4x 800GB SSD, 8x 6TB HDD 2-way mirror 50 / 23 1.41 2 2 12x 1.6TB SSD, 48x 8TB HDD 3-way mirror 770 / 230 1.14 4 4 12x 1.6TB SSD, 48x 8TB HDD 3-way w/ EA 1500 / 450 1.18 Servers Disks (local) Resiliency Raw/Net Capacity (TB) $/GB 2 4x 800GB SSD, 12x 4TB HDD 2-way mirror 96 / 42 1.02 3 3-way mirror 144 / 42 1.53 4 3-way mirror/dual-parity 192 / 90 0.95 8 384 / 217 0.79 12 576 / 368 0.70 16 768 / 505 0.68
  • 14. Storage Options Flash support SAS SSD Flash caches 1 GB WBC per virtual disk Performance tier Mirror Capacity tier Mirror Tier optimization Nightly (default) Flash sizing 5-6% of raw capacity SAS SSD SAS or SATA SSD, NVMe 1 GB WBC per virtual disk ReFS read cache (40% of SSD capacity) Read/Write cache Mirror Replaced by flash cache Parity Mirror (~10%) Parity (~90) Real-time w/ReFS 5-6% of raw capacity Windows Server 2016 Disaggregated | Hyper-converged Shared SAS w/ReFS* Storage Space Direct* OS Windows Server 2012 R2 Architecture Disaggregated | Hyper-converged Model Shared SAS
  • 15. Resiliency Options Cluster nodes 2-4 Enclosures 1-4 MPIO Yes Max pool size 84 disks SSDs to repair 0-1 per pool HDDs to repair 0-4 per pool Virtual disks/node 2-4 Fault domains Disk or enclosure Dual-site clusters w/Storage Replica No 2-4 2*-16 (64?) 1-4 0-1 Yes No 84 disks 240 0-1 per pool 0 0-4 per pool 0-2 per node 2-4 Disk or enclosure Cluster node, chassis, rack 2+ nodes per site 3-4+ nodes per site Windows Server 2016 Disaggregated | Hyper-converged Shared SAS Storage Space Direct OS Windows Server 2012 R2 Architecture Disaggregated | Hyper-converged Model Shared SAS
  • 17. Network Options Bandwidth >= 1Gbit/s Protocol Ethernet, IB3 Interfaces/Server 2 NIC Teaming Not w/RDMA SET Teaming - RSS Recommended RSC Recommended RDMA1 Preferred PacketDirect - In-box drivers Not recommended for production2 >=10Gib/s Ethernet, IB3 2 Yes Recommended Recommended Recommended Preferred Recommended Preferred Not supported2 1 Either iWARP or RoCE; however, RoCE requires additional server and switch configuration/support (PFC and DCB) and VLANs 2 Use approved NIC driver from solution vendor 3 Don’t use Infiniband w/hyper-converged Windows Server 2016 Disaggregated | Hyper-converged Shared SAS Storage Space Direct OS Windows Server 2012 R2 Architecture Disaggregated | Hyper-converged Model Shared SAS
  • 18. Windows Server 2016 Disaggregated | Hyper-converged Shared SAS Storage Space Direct OS Windows Server 2012 R2 Architecture Disaggregated | Hyper-converged Model Shared SAS RAM for storage caches None Total RAM 64 GB Extra RAM for VMs 32-128 GB if hyper-converged CPU 2 6-core CPUs typical (add for extra SSDs, Data Deduplication, hyper-converged) 16 GB CSV cache? 10 GB per 1 TB of flash cache 16 GB CSV cache? 64 GB 64 GB 32-128 GB if hyper-converged 2 6-core CPUs typical (add for extra SSDs, Data Deduplication, hyper-converged)
  • 19.
  • 20.
  • 21.
  • 22. Windows Server 2016 Disaggregated | Hyper-converged Shared SAS Storage Space Direct OS Windows Server 2012 R2 Architecture Disaggregated | Hyper-converged Model Shared SAS Solution vendors Dell DataOn RAID Inc. HP Dell DataOn RAID Inc. HP Dell Cisco Fujotsu HP Intel Lenovo Quanta
  • 23. Quanta D51PH  HP Apollo 2000 System Dell PowerEdge R730xdCisco UCS C3260 Rack Server Intel® Server Board S2600WT-Based Systems Lenovo System x3650 M5 Fujitsu Primergy RX2540 M1
  • 24.
  • 25. Node File-Delete Notification (TRIM) Disable w/FSUtil Storage Spaces registry changes Optimize Storage Spaces repair settings on Scale-Out File Servers MPIO policy Optimize with MPClaim Physical disk caches Disable with Diskcache.exe CSV read cache Max 64 TB; by-passed w/tiers Pool RetireMissingPhysic alDisks Set with Set-StoragePool Volume Column count Calculate w/spreadsheet Volume size 10 TB if using Volsnap for VSS snapshots UseLargeFRS Set with Format-Volume Use defaults Use defaults Use defaults N/A Use defaults Supported Use defaults (handled by Health Service) Use defaults  >10TB if using ReFS snapshots 10 TB if using Volsnap for VSS snapshots Use defaults Windows Server 2016 Disaggregated | Hyper-converged Shared SAS Storage Space Direct OS Windows Server 2012 R2 Architecture Disaggregated | Hyper-converged Model Shared SAS
  • 26.
  • 27. Storage Spaces Storage Pool Storage Spaces Virtual Disks Scale-Out File Server CSVFS Cluster File System Software Storage Bus Virtual Machines 1 2 ReFS On-Disk File System Virtual Machines
  • 28. ClusPort SpacePort Virtual Disks File System Cluster Shared Volumes File System (CSVFS) Application Node 1 Block over SMB ClusBflt Node 2 Physical Devices ClusPort SpacePort Virtual Disks File System ClusBflt Physical Devices
  • 29. Storage Configuration Caching devices Capacity devices Caching behavior SSD + HDD SSD HDD Write + Read SSD + SSD SSD SSD Write only Caching Devices Capacity Devices
  • 31. $mt = New-StorageTier -FriendlyName Mirror -MediaType SSD -ResiliencySettingName Mirror $pt = New-StorageTier -FriendlyName Parity -MediaType SSD -ResiliencySettingName Parity New-Volume -FriendlyName SQlServer -StorageTiers $mt -StorageTierSizes 1TB New-Volume -FriendlyName GenericVM -StorageTiers $mt,$pt -StorageTierSizes 100GB,900GB Mirror Parity
  • 32. $mt = New-StorageTier -FriendlyName Mirror -MediaType SSD -ResiliencySettingName Mirror $pt = New-StorageTier -FriendlyName Parity -MediaType HDD -ResiliencySettingName Parity New-Volume -FriendlyName SQlServer -StorageTiers $mt -StorageTierSizes 1TB New-Volume -FriendlyName GenericVM -StorageTiers $mt,$pt -StorageTierSizes 100GB,900GB Mirror Parity
  • 33.
  • 35. Virtual Disk Mirror Parity Mixed-Resiliency Optimized for Performance Capacity Balanced Use case All data is hot All data is cold Mix of hot and cold Efficiency Least (33%) Most (57+%) Depends on mix File System ReFS (or NTFS) ReFS (or NTFS) Requires ReFS Node requirement 2+ 4+ 4+
  • 36. Mirror tier Parity tier Server 2 Server 3 Server 4Server 1 A A’ A’’B B’ B’’Mirror Parity X1 X2 Px Y1 Y2 Py Q Nodes Mirror tier Efficiency Parity Tier Efficiency Resiliency 4 33% 57% 1 node 8 33% 80% 1 node 16 33% 84% 1 node LRC Erasure Coding in Storage Spaces
  • 37. Mirror tier Parity tier W ReFS
  • 38.