Hybrid Storage Pools         Using Disk and Flash with ZFS       (Now with the benefit of hindsight!)Adam Leventhal @ahl
Flash	  Emerges	  •  Storage	  medium	  invented	  in	  1980	        –    Very	  fast	  reads	  (~50us)	        –    Fast	...
Disk	  is	  dead…	  just	  like	  tape	  •  Many	  predicted	  the	  death	  of	  disk	  or	  relegaSon	  of	  disk	  to	 ...
ZFS	  circa	  2007	  •    Sun	  was	  developing	  a	  ZFS-­‐based	  storage	  appliance	  (Fishworks)	  •    ZFS:	  enter...
Hybrid	  Storage	  Pool	  (HSP)	  •  Use	  flash	  as	  a	  storage	  Ser	  •  Between	  DRAM	  and	  disk	  in	  cost,	  c...
Hybrid	  Storage	  Pool	  Example	  
Hybrid	  Storage	  Pool	  Example	  
ZFS	  Caching	  •  AdapSve	  Replacement	  Cache	  (ARC)	  as	  the	  primary	  DRAM	  cache	  •  L2ARC	  developed	  by	 ...
L2ARC	  Problems	  •  Non-­‐persistent	        –  Aeer	  a	  reboot	  or	  fatal	  system	  failure,	  the	  cache	  is	  ...
Changing	  Landscape	  •    DRAM	  prices	  have	  dropped	  dramaScally	  •    Large	  memory	  systems	  available	  (3T...
What	  to	  do	  today?	  •  The	  L2ARC	  can	  help	        –    The	  SSD	  space	  is	  large	  and	  highly	  varied	...
Next	  for	  ZFS	  •  For	  the	  L2ARC	  to	  be	  viable,	  it	  needs	  to	  be	  persistent	  •  Lots	  of	  performan...
Questions?
Upcoming SlideShare
Loading in …5
×

Hybrid Storage Pools (Now with the benefit of hindsight!)

2,034 views

Published on

http://dtrace.org/blogs/ahl/?p=1209
by Adam Leventhal

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,034
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
19
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Hybrid Storage Pools (Now with the benefit of hindsight!)

  1. 1. Hybrid Storage Pools Using Disk and Flash with ZFS (Now with the benefit of hindsight!)Adam Leventhal @ahl
  2. 2. Flash  Emerges  •  Storage  medium  invented  in  1980   –  Very  fast  reads  (~50us)   –  Fast  writes  (~300us)   –  High  IOPS  /  low  latency   –  Limited  number  of  write  cycles  •  2004:  flash  cost  as  much  as  DRAM  •  2007:  flash  cost  was  right  between  DRAM  and  disk  
  3. 3. Disk  is  dead…  just  like  tape  •  Many  predicted  the  death  of  disk  or  relegaSon  of  disk  to  backup  •  Didn’t  happen  •  All-­‐flash  soluSons  sSll  trying  to  gain  mass  adopSon  
  4. 4. ZFS  circa  2007  •  Sun  was  developing  a  ZFS-­‐based  storage  appliance  (Fishworks)  •  ZFS:  enterprise  class  storage  on  commodity  hardware  •  Problem:  enterprise  storage  was  a  lot  faster  •  Looked  at  tradiSonal  soluSons   –  NV-­‐DRAM  to  accelerate  writes   –  Massive  DRAM  to  cache  reads  •  But  it  was  just  the  right  Sme  for  flash…  
  5. 5. Hybrid  Storage  Pool  (HSP)  •  Use  flash  as  a  storage  Ser  •  Between  DRAM  and  disk  in  cost,  capacity,  latency,  throughput  •  Use  commodity  disks   –  7200  RPM   –  Good  throughput   –  Great  $/GB  and  wa_s/GB  •  Combine  disk,  flash,  DRAM  into  a  hybrid  pool  •  In  ZFS:   –  ZFS  intent  log  (ZIL)  for  write  acceleraSon   –  L2ARC  to  extend  the  reach  of  the  ZFS  cache  
  6. 6. Hybrid  Storage  Pool  Example  
  7. 7. Hybrid  Storage  Pool  Example  
  8. 8. ZFS  Caching  •  AdapSve  Replacement  Cache  (ARC)  as  the  primary  DRAM  cache  •  L2ARC  developed  by  Brendan  Gregg  to  use  external  (flash)  devices  •  Takes  into  account  opSmal  IO  pa_erns  for  flash   –  Random,  small  writes  =  hastened  failure   –  SequenSal,  large  writes  =  happy  SSDs   –  Thro_les  writes  to  preserve  longevity  •  Uses  predicSve  evicSon  to  idenSfy  blocks  to  cache  
  9. 9. L2ARC  Problems  •  Non-­‐persistent   –  Aeer  a  reboot  or  fatal  system  failure,  the  cache  is  empty  •  Slow  to  warm  up   –  Will  only  write  to  one  device  at  a  Sme  -­‐>  best  case  1TB  /  hour   –  Real  world  example  2TB  in  24  hours  •  Conceptually  most  of  the  way  there  •  No  real  way  to  tune  it  to  a  workload  •  Not  much  real-­‐world  tesSng  and  tuning  done  
  10. 10. Changing  Landscape  •  DRAM  prices  have  dropped  dramaScally  •  Large  memory  systems  available  (3TB+)  •  NAND  flash  is  geing  trickier  to  build  around  •  Endurance  and  performance  decrease  as  lithography  and  price  decrease   –  MLC  and  “TLC”  (volume  flash)  have  parScularly  short  lives  •  Running  into  size  limitaSons   –  32nm  in  2008   –  19nm  today   –  Supposed  floor  around  11nm  •  SSDs  are  becoming  increasingly  complex  
  11. 11. What  to  do  today?  •  The  L2ARC  can  help   –  The  SSD  space  is  large  and  highly  varied   –  Generally  cheap,  laptop  SSDs  suffice  for  the  L2ARC   –  Give  it  enough  Sme  to  warm  up  (hours  or  days)   –  Measure  the  impact  on  your  actual  workload  •  The  ARC  is  great  and  relaSvely  simple   –  Load  up  on  DRAM  
  12. 12. Next  for  ZFS  •  For  the  L2ARC  to  be  viable,  it  needs  to  be  persistent  •  Lots  of  performance  work  needed   –  Run  it  through  a  bunch  of  real-­‐world  use  cases   –  Make  it  easy  to  collect  coherent,  relevant  data   –  Create  the  right  knobs  for  users  to  turn  •  There  are  a  few  companies  using  the  L2ARC  •  Hopefully  they  will  take  up  the  mantle  
  13. 13. Questions?

×