Comparing Enterprise Server And Storage Networking Options

686 views

Published on

DOWNLOAD ORIGINAL FROM IBM: http://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=WH&infotype=SA&appname=STGE_QC_QC_USEN&htmlfid=QCL12384USEN&attachment=QCL12384USEN.PDF

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
686
On SlideShare
0
From Embeds
0
Number of Embeds
23
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Comparing Enterprise Server And Storage Networking Options

  1. 1. Test Validation ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.       Comparing  Enterprise  Server   and  Storage  Networking  Options   Enabling  you  to  make  the  best  technology  decisions   Author: Russ Fellows June 5, 2014
  2. 2.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  1   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Table  of  Contents   Executive  Summary  ......................................................................................................  3   Findings  Summary  .....................................................................................................................  3   Evaluation  Overview  .....................................................................................................  4   Comparing  Storage  Network  Choices  ........................................................................................  5   IBM  FC  Environment  ........................................................................................................................  7   Cisco  FCoE  to  FC  Environment  .........................................................................................................  7   Storage  Equipment  ..........................................................................................................................  7   Validation  Objectives  ....................................................................................................  7   Scope  .........................................................................................................................................  8   Configuration  Choices  ...............................................................................................................  8   Test  Approach  ...........................................................................................................................  8   Application  Software  .......................................................................................................................  9   Test  Results  ..................................................................................................................  9   Cabling  and  Management  .......................................................................................................  10   Application  Workload  Test  ......................................................................................................  11   Application  Test  Results  .................................................................................................................  11   Storage  Network  Saturation  Test  ............................................................................................  12   Saturation  Test  Results  ..................................................................................................................  13   Evaluation  Summary  ....................................................................................................  15   Performance  ............................................................................................................................  15   Final  Observations  ...................................................................................................................  16    
  3. 3.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  2   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Appendix  A  –  Configuration  Overview  .........................................................................  17   VMware  .........................................................................................................................................  17   IBM  Environment  ...........................................................................................................................  17   Cisco  Environment  .........................................................................................................................  17   Appendix  B  -­‐  IBM  Flex  System  Environment  .................................................................  18   Appendix  C  -­‐  Cisco  UCS  Environment  ............................................................................  19   Appendix  D  -­‐  Application  Workloads  ............................................................................  21     Enabling  you  to  make  the  best  technology  decisions  
  4. 4.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  3   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Executive Summary Data   center   design   and   architecture   remain   important   considerations   for   companies   of   all   size.     Optimizing   applications   to   achieve   the   greatest   performance   at   the   lowest   cost   is   a   goal   of   all   companies.    With  an  increased  number  of  technologies  and  alternatives,  IT  architects  now  have  more   options  for  optimizing  their  architectures.       Some  of  the  technologies  having  the  greatest  impact  include  physical  consolidation  of  computing,  the   consolidation  of  workloads  using  server  virtualization  and  solid-­‐state  flash  storage  systems.    Both  the   increased   CPU   cores   per   socket   and   Blade   systems   are   increasing   server   density,   driving   higher   utilization  of  connections  between  elements  and  reducing  the  number  of  cables  required  for  servers.       Coinciding  with  these  trends  are  the  capabilities  of  fabric  enhanced  SAN  and  LAN  networking,  which   enable  transparent  scaling  by  aggregating  multiple  connections.    Both  FC  and  Ethernet  provide  fabric   technologies,   with   Ethernet   requiring   use   of   new   Ethernet   technologies   known   as   DCB   (Data   Center   Bridging).     To   date,   Fibre   Channel   has   been   the   most   popular   choice   for   storage   connectivity,   in   particular   when   high   performance,   dedicated   bandwidth   is   required.       According   to   Evaluator   Group   data,  FC  attached  storage  accounts  for  42%  of  connectivity  compared  to  FCoE,  which  accounts  for  3%  of   storage  connectivity.1       Evaluator   Group   has   found   that   storage   is   a   bottleneck   limiting   performance   for   many   demanding   applications.    In  particular,  OLTP  and  virtualized  applications  often  exhibit  random  I/O  and  require  lower   latency  in  order  to  improve  the  applications  performance.    For  these  reasons,  solid-­‐state  storage  using   NAND  flash  has  become  a  popular  option  for  improving  IO-­‐intensive  application  performance.       In  order  to  provide  real-­‐world  results  to  enable  IT  professionals  to  make  informed  decisions,  Evaluator   Group  compared  two  alternative  configurations.    A  demanding  set  of  applications  was  chosen  to  run  in  a   virtualized  environment  using  the  VMware  hypervisor  on  two  competing  platforms.    This  test  compared   an   IBM   Flex   System   using   end-­‐to-­‐end   FC   connectivity   to   a   Cisco   UCS   with   FCoE   to   FC   storage   connectivity.    In  both  instances,  the  same  high-­‐speed  IBM  FlashSystem  840  solid-­‐state  storage  system   was  used  as  the  storage  target.     Findings Summary The  tested  configuration  showed  the  following  results:   • IBM  with  FC  delivered  up  to  4x  faster  response  times  as  workloads  surpassed  80%  utilization   • IBM  with  FC  provided  higher  performance  with  50%  fewer  connections  than  FCoE                                                                                                                   1  Data  based  on  IDC  and  Evaluator  Group  sources  for  storage  connections  at  the  end  of  2013  
  5. 5.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  4   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Evaluation Overview This   testing   was   done   to   provide   IT   architects   with   actual   performance   data   points   to   help   them   evaluate  and  consider  the  tradeoffs  of  these  two  approaches  to  utilizing  high-­‐speed  solid-­‐state  storage   in  virtual  server  environments.       IBM  offers  a  number  of  technologies,  designed  to  enable  IT  architects  to  construct  optimal  solutions  to   their   particular   problems.     The   storage   connectivity   options   available   by   IBM   include   SAN,   LAN   and   converged   networking   technologies,   including   support   for   storage   attachment   to   IBM   Flex   via   iSCSI,   FCoE  and  FC  SAN.         IBM  commissioned  Evaluator  Group  to  compare  two  competing  enterprise  configurations,  assessing  the   operational  efficiency  and  performance  features  of  these  two  alternative  technologies.    The  tests  were   designed  to  recreate  a  high  performance  environment  that  leveraged  the  full  benefits  of  a  high-­‐speed  all   solid-­‐state  IBM  FlashSystem  storage  device.       All  testing  was  performed  by  Evaluator  Group,  using  a  combination  of  IBM  and  partner  equipment  for   the  testing.    Test  results  indicate  that  as  storage  network  utilization  rates  increase,  the  Fibre  Channel   configuration   delivered   consistent   performance,   while   providing   better   performance   during   high   utilization.    This  report  details  the  testing  process,  equipment  and  other  findings.       Testing   occurred   in   April   and   May   2014,   focusing   on   performance,   along   with   operational   efficiency   characteristics.    The  workloads  utilized  were  designed  to  recreate  actual  enterprise  applications  rather   than   using   synthetic   workload   generation   tools   in   order   to   create   an   accurate   assessment   of   the   configurations  used  in  enterprise  environments.    Performance  was  the  primary  evaluation  criteria:     o Performance  of  configurations  as  application  workloads  scale   o Relative  performance  of  the  two  configuration     Performance  is  an  important  consideration  for  some  workloads.    In  particular,  for  demanding   applications  that  utilize  high-­‐speed  storage,  such  as  the  IBM  FlashSystem  840,  application  delays  are   critically  important.    The  IBM  FlashSystem  840  storage  array  is  well  suited  to  these  environments,  with   latencies  as  low  as  100  us  (based  on  IBM  specs),  providing  high  I/O  rates  for  demanding  workloads  and   virtualized  applications.  
  6. 6.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  5   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Comparing Storage Network Choices Current  market  data  shows  that  FC  accounts  for  approximately  45%  of  enterprise  storage  connections,   while  FCoE  accounts  for  approximately  3%.2    Additionally,  of  the  storage  connectivity  options  including   FC,  iSCSI,  NAS,  IB  and  FCoE;  FCoE  is  projected  to  have  the  lowest  connection  rate  and  the  lowest  growth   rate  over  the  next  5  years.3   In  terms  of  technology,  Fibre  Channel  over  Ethernet  (FCoE)  has  become  a  proven  technology  that  works   well  as  a  server  to  Top  of  Rack  connectivity  method.  However,  due  to  the  underlying  networking   technologies,  multiple  FCoE  links  are  less  efficient  than  a  single  16  Gb  FC  link.    Typically,  FCoE  is   deployed  over  10  Gb  Ethernet  and  then  bridged  into  8  Gb  FC.     Evaluator  Group  comments:    It  should  be  noted  that  the  performance  efficiency  of  two  10  Gb   FCoE  links,  bridged  into  8  Gb  FC  is  at  least  21%  less  efficient  than  a  single  16  Gb  FC  link,  and   in  some  cases  may  be  up  to  45%  less  efficient.    The  test  results  would  be  similar  if  two  8  Gb   FC  were  used.    These  are  mathematical  facts  based  on  protocol  efficiency,  queuing  theory   and  other  scientific  methods.    4   Thus,  for  high  performance  environments  an  end-­‐to-­‐end  FC  deployment  should  be  expected  to   outperform  an  FCoE  to  FC  environment,  while  using  fewer  cables  and  I/O  connectivity.    Even  when   accounting  for  the  separate  Ethernet  connections  required  by  the  FC  only  environment,  a  dedicated  SAN   environment  can  outperform  a  converged  DCB  Ethernet  environment  with  fewer  network  cables.                                                                                                                           2  Data  based  on  IDC  and  Evaluator  Group  sources  for  storage  connections  at  the  end  of  2013.   3  IBID   4  Evaluator  Group  Blog  “Why  2  *  8  !=  16”  
  7. 7.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  6   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     This  evaluation  highlights  the  differences  found  when  running  identical  workloads  on  two  configurations,   shown  in  Figure  1,  using  the  following  products:   1. Configuration  1:  IBM  with  FC   a. IBM  Flex  System  with  embedded  FC5022  -­‐  8/16  Gbps  SAN  Scalable  Switch   b. IBM  SAN48B-­‐5,  16  Gbps  FC  switch     2. Configuration  2:  Cisco  UCS  with  FCoE   a. Cisco  5108  UCS  blade  system  with  2208  Fabric  Extender  and  a  6248  Fabric  Interconnect     b. Cisco  MDS  9710  FC  switch  for  Cisco  UCS  FCoE  configuration   3. Common  Storage  used  for  both  configuration  1  and  2   a. IBM  FlashSystem  840  solid-­‐state  storage  system  with  16  Gb  FC  connectivity     Figure  1:  Test  Configuration  (IBM  Flex  System  vs.  Cisco  UCS)  
  8. 8.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  7   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     IBM FC Environment IBM  Flex  System  with  IBM  System  Networking  FC  SAN  Switch   • IBM  Flex  System  blade  enclosure:   o 2  x  IBM  x240  blade  servers,  each  with  dual,  quad  core  CPU  and  128  GB  RAM   o 1  x  IBM  EN4023  embedded  Ethernet  switch  for  LAN  connectivity  (10  GbE)   o IBM  FC5172  embedded  2  port,  16  Gb  FC  mezzanine  HBA   o 1  x  IBM  FC  5022  embedded  switch  for  SAN  connectivity  (28  x  16  Gb)   • IBM  IMM  for  single  point  management  of  blades  and  I/O  connectivity   • 1  x  external  IBM  SAN48B-­‐5  FC  SAN  switch  for  storage  connectivity     Cisco FCoE to FC Environment Cisco  Unified  Compute  Server  with  FCoE   • Cisco  5108  blade  chassis:   o 2  x  B200  M3  blade  servers,  each  with  dual,  quad  core  CPU  and  128  GB  RAM   o 2  x  2208  XP  Fabric  Extenders  (with  1240  VIC  -­‐  virtual  interface  connection  to  blades)   o 1  x  external  6248  UP  Fabric  Interconnect  for  SAN  /  LAN  connectivity  and  FCoE  to  FC     • Cisco  UCS  Manager  for  unified  management  of  server,  unified  fabric  extenders  along  with  LAN   and  FCoE  SAN  host  ports   • 1  x  external  Cisco  MDS  9710  FC  SAN  switch  for  storage  connectivity     Storage Equipment Common  Equipment   • 1  x  FC  attached,  IBM  FlashSystem  840  (attached  to  both  Cisco  and  IBM  configurations)   o 16  Gb  FC  attached  solid-­‐state  storage  system   o Dual,  HA  controllers,  each  with  a  single  FC  connection  to  FC  switch  (Cisco  or  IBM)   Validation Objectives • Compare  and  contrast  an  end-­‐to-­‐end  FC  configuration  with  an  FCoE  to  FC  configuration   o Configurations  were  equivalent  when  possible   o Best  practice  configurations  utilized  with  single  path  setup   o Focus  on  real-­‐world  scenarios  and  common  deployments   • Show  performance  of  both  alternatives   • Evaluate  cabling  differences  if  significant     The  testing  was  both  quantitative  and  qualitative  in  nature.    The  quantitative  testing  measured  specific   values  for  performance  and  time  to  accomplish  major  tasks.    
  9. 9.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  8   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     This  report  highlights  similarities  and  differences  in  test  results.    Evaluator  Group  commentary  provides   context  and  a  narrative  assessment  of  the  results  as  experienced  by  Evaluator  Group  personnel.       Scope • Performance   o Application  latency   o Application  throughput   o Maximum  number  of  VMs  supportable       Configuration Choices 1. Both  configurations  utilized  a  single  path  due  to  equipment  constraints.    Typical  production   environments  would  use  redundant  components.    This  also  simplified  the  test  harness  to  collect   the  performance  results.    (The  tested  configurations  represent  one-­‐half  of  a  fully  HA   configuration).    Performance  was  not  impacted  by  this  choice.   2. VMware  settings  were  utilized  to  optimize  storage  performance  and  minimize  any  queuing   delays;  these  settings  provided  performance  benefits  to  both  configurations.    In  particular  they   benefited  the  Cisco  UCS  with  FCoE  configuration  more,  which  experiences  significant  queuing   delays  without  them.    Specifically  the  SATP  was  set  to  “VMW_SATP_SVC”  and  the  PSP  set  to   “Round  Robin”  with  the  I/O  count  set  to  “1”.       3. The  test  configuration  utilized  FC  attached  storage,  rather  than  native  FCoE  storage  due  to  the   fact  that  FC  attached  storage  is  the  predominant  connection  technology  and  end-­‐to-­‐end  FCoE   deployments  are  limited  due  to  a  number  of  factors.5   4. The  bottleneck  was  chosen  to  be  the  SAN  connection  between  the  server  and  the  switch  for  both   configurations,  with  the  nominal  bandwidth  limited  to  16  Gb.    For  the  IBM  FC  configuration  this   was  a  single  16  Gb  FC  link  between  the  IBM  Flex  and  the  FC  switch.  For  the  Cisco  FCoE   configuration  the  limitation  was  two,  8  Gb  FC  links  between  the  6248  and  the  MDS  9710.       Test Approach The   tests   were   designed   to   assess   two   common   enterprise   configurations   using   equivalent   blade   systems,  storage  networking  and  high  performance  solid-­‐state  storage.    The  difference  in  protocols  and   vendors  for  the  two  solutions  highlights  the  comparison  between  FCoE  and  Fibre  Channel.  IBM’s  Flex   System  has  the  capability  to  use  multiple  protocols  and  evolve  as  needed  while  Cisco’s  UCS  solution  is   focused  on  using  FCoE.    The  test  is  designed  to  capture  the  positives  and  negatives  of  this  decision.                                                                                                                   5  Based  on  previously  cited  IDC,  Gartner  and  Evaluator  Group  data  
  10. 10.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  9   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     The  two  configurations  were  chosen  to  be  as  close  to  identical  as  possible  with  the  primary  difference   being  the  protocol  used  for  storage  networking.       The  application  workload  set  was  chosen  that  would  fully  utilize  the  server  and  storage  components  of   the  solutions.  The  IOmark  suite6  of  workload  tools  was  used  to  generate  application  storage  workloads   that  recreate  actual  storage  workloads  without  the  CPU  or  memory  component.     Evaluator  Group  comments:    Both  test  environments  utilized  the  same  IBM  F840  storage   system.    The  testing  utilized  enterprise  FC  attached  storage,  since  a  majority  of  enterprises   choose  FC  storage  as  their  preferred  storage  connectivity  with  Fibre  Channel  attached   storage  continuing  to  be  the  most  predominate  connectivity  method  as  previously  noted.       Application Software The  performance  limits  of  the  environment  were  found  by  running  multiple  applications  instances  on   both  configurations.    By  adding  successively  more  application  workloads  and  instances,  the  maximum   number  of  applications  supported  was  determined  for  both  infrastructures.       Multiple   application   workloads   were   used   as   I/O   test   load   generators.     A   total   of   four   categories   of   application  workloads  were  generated  including  read-­‐only  video  rendering,  NL  video  editing  of  HD,  VDI   desktop   applications   and   two   industry   standard   databases,   DVD   store   database   and   Exchange   mail   server.       The  applications  categories  were:   1. NL  Video  render     2. VDI  Desktop  applications  (each  with  MS  Office,  Acrobat,  IE  and  other  apps)   3. DVD  data  store     4. Exchange  data  base   Test Results The  test  findings  are  derived  from  the  testing  outlined,  with  results  detailed  in  the  remainder  of  this   report.    The  configuration  details  for  both  environments  are  provided  in  Appendix  A.    The  primary   difference  between  the  configurations  was  storage  network  connectivity.                                                                                                                       6  Description  of  the  IOmark  suite  of  storage  application  benchmarks  at  -­‐  www.iomark.org  
  11. 11.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  10   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Cabling and Management Cabling  and  connectivity  is  another  consideration.    In  particular  reduced  cabling  is  one  of  the  claimed   benefits  made  by  vendors  who  promote  converged  SAN  and  LAN  on  enterprise  Ethernet.     During  testing,  the  end-­‐to-­‐end  FC  using  IBM  Flex  System  required  fewer  cables  than  did  the  FCoE  to  FC   system  using  Cisco  UCS  when  including  both  the  LAN  and  SAN  connections.    As  noted  previously,  both   environments  were  configured  as  one  side  of  an  HA  configuration,  which  would  include  dual,  redundant   storage  and  server  connectivity.    The  reduced  cabling  advantage  for  the  IBM  configuration  would  remain   in  either  instance.     Evaluator  Group  comments:    When  including  both  LAN  and  SAN  connectivity,  the  all  FC   environment  required  fewer  connections  while  providing  higher  performance.    In  comparing   only  the  storage  connectivity,  the  IBM  with  FC  system  required  50%  fewer  cables  than  did   Cisco  with  FCoE  configuration.     IBM  Flex  System  Environment   The  IBM  environment  had  5  total  connections  as  follows:   • 1  x  10  Gbps  Ethernet  LAN  connection  to  network   • 1  x  1  Gb  IBM  Flex  CMM  management  network   • 1  x  16  Gbps  FC  SAN  connection  from  IBM  Flex  System  to  IBM  System  Storage  SAN48B-­‐5   • 2  x  16  Gbps  FC  SAN  connections  from  IBM  System  Storage  SAN48B-­‐5to  IBM  Flash  System  storage   Cisco  UCS  Environment   The  Cisco  environment  had  8  total  connections  as  follows:       • 3  x  10  Gbps  converged  Ethernet  connections  from  UCS  2208  FEX  to  6248UP  interconnect   • 1  x  10  Gbps  Ethernet  connection  from  6248  to  management  network   • 2  x  8  Gbps  connections  from  6248UP  to  Cisco  MDS  9710  FC  switch   • 2  x  16  Gbps  FC  SAN  connections  from  MDS  FC  switch  to  IBM  Flash  System  storage   Note:  For  this  test,  a  single  16  Gbps  FC  connection  from  the  Flex  System  to  the  IBM  System  Storage   SAN48B-­‐5was  used,  compared  to  2  x  8  Gbps  FC  connections  from  the  6248  interconnect  to  the  FC  switch.     This  was  done  to  provide  equivalent  nominal  performance  and  was  not  representative  of  a  production   configuration.    The  use  of  3  x  10  Gb  connections  from  the  UCS  to  the  6248  was  used  to  provide  the  same   SAN  and  LAN  bandwidth  as  that  available  with  the  IBM  Flex  configuration.           Additional  Note:    The  testing  used  4  x  10  Gb  connections  from  the  UCS  chassis  to  the  6248  fabric   interconnect,  however,  only  3  are  accounted  for  in  this  comparison  to  avoid  a  cable  count  advantage  for   the  IBM  FC  configuration.      
  12. 12.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  11   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     As  shown,  the  FC  environment  delivered  higher  performance  using  fewer  cables  with  reduced   management  complexity.     Evaluator  Group  Comments:    It  is  important  to  note  that  the  FC  environment  did  not  use  an   HA  setup  in  order  to  provide  the  FCoE  environment  with  equivalent  nominal  performance.   Application Workload Test As  detailed  previously,  hundreds  of  applications  workloads  were  configured  to  run  simultaneously   against  multiple  solid-­‐state  FC  attached  LUNs.    Each  workload  was  run  for  a  1  hour  time  period  with   successive  instances  of  the  same  workload  started  after  a  delay  of  up  to  15  minutes.      The  measurement   period  for  the  complete  set  of  workloads  was  a  30-­‐minute  interval  during  which  all  workloads  were   running.       Specifically  the  following  applications  were  tested:   Workload   Total  Instances   Non-­‐Linear  Video  Editing  (50  MB/s  -­‐  50%)   10   Video  Streaming  (at  6  MB/s  -­‐  100%  read)   72   IOmark-­‐VM  :  DVD  Store  DB   105   IOmark-­‐VM  :  Exchange  Mail  (1000  User)   105   IOmark-­‐VDI  :  Standard  User   640   Table  1:  Workload  Specifications     These  applications  are  both  representative  of  common  applications  within  enterprise  environments,  but   also  represent  a  workload  that  requires  more  than  80%  of  a  16  Gbps  FC  link  for  both  read  and  write   operations.    The  tested  workloads  vary  over  time,  similar  to  how  enterprise  workloads  behave.   Application Test Results The  application  workload  results  for  the  end-­‐to-­‐end  FC  configuration  with  IBM  Flex  were  substantially   better  than  those  for  the  FCoE  to  FC  configuration  using  Cisco  UCS.    Specifically,  the  Cisco  FCoE   configuration  was  not  able  to  complete  the  tests,  with  one  of  the  tests  failing  during  each  of  multiple   attempts.    The  reason  for  the  failures  was  too  high  of  response  times,  causing  excessive  I/O  queuing.     The  test  was  run  3  times,  with  at  least  one  application  failing  each  time.      
  13. 13.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  12   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     In  contrast,  the  end-­‐to-­‐end  FC  configuration  was  run  several  times,  and  no  application  failed  on  any  of   the  test  runs.    Additionally,  the  response  time  results  for  the  remaining  workloads  were  also   substantially  lower  for  the  FC  environment  with  IBM  Flex  server  and  FlashSystem.    Shown  below  in   Figure  2  are  a  summary  of  the  application  workload  response  times  for  the  different  applications   running  on  the  IBM  Flex  System  and  the  Cisco  UCS.       Evaluator  Group  Comments:    NOTE  -­‐  For  the  Cisco,  FCoE  to  FC  environment,  the  DS2DB   application  failed.    In  multiple  test  runs,  one  or  more  test  instances  failed  with  FCoE  due  to   excessively  high  latency.    As  a  result,  the  remaining  workloads  running  on  the  FCoE   configuration  were  less  than  that  of  the  FC  environment  with  IBM  Flex.    The  all  FC   environment  achieved  significantly  lower  response  times,  3.5  ms  vs.  over  26.0  ms  for  the   FCoE  to  FC  configuration.       Figure  2:  Mixed  Application  Workload  Performance  (FCoE  includes  Failures)   Storage Network Saturation Test Comparing  how  a  single  16  Gbps  FC  connection  performed  under  increasing  loads  compared  to  two  10   Gbps  FCoE  links  was  another  performance  test.    In  this  second  test,  the  configuration  was  identical  to   the  previous  test,  with  the  FCoE  environment  using  twice  as  many  connections  at  the  point  of   contention  as  the  end-­‐to-­‐end  FC  environment  with  a  single  16  Gb  link  at  the  point  of  contention.     4.62   4.06   50.00   3.39   45.17   2.88   4.68   3.82   26.12   3.54   0.00   10.00   20.00   30.00   40.00   50.00   60.00   FCoE   FC   Avg.  Response  Time  (in  ms)  per  Workload   Mixed  Workload  Response  Times   Average   VDI   Exchng   DS2DB   Video  
  14. 14.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  13   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     This  test  was  designed  to  saturate  the  network  and  utilized  a  block  size  of  32  KB  for  both  read  and  write   operations.    The  size  of  32  KB  was  chosen  due  to  it  being  the  approximate  average  size  of  the  I/O   operations  occurring  on  the  mixed  workload  testing  for  the  DVD  Store  database,  the  Exchange  mail   application  and  the  VDI  instances.       Using  the  IEEE,  16  Gb  FC  specification  a  maximum  data  rate  of  1600  MBps  is  possible  for  a  single  16  Gb   connection.    Workloads  consisting  of  partial  ratios  of  this  were  used,  ranging  from  10%  up  to  90%  (1,440   MBps)  were  generated  with  response  times  shown  for  each  workload  below  in  figure  3.   Saturation Test Results The  results  for  a  50%  read,  50%  write  workload  at  32  KB  are  shown  below  in  Figure  3  for  select  workload   levels.    As  depicted  in  Figure  3  below,  the  response  times  for  the  Cisco  configuration  began  to  increase   significantly  above  those  for  the  IBM  system  beyond  80%  of  maximum  saturation  levels.       Evaluator  Group  comments:    Comparison  for  a  single  16  Gbps  FC  connection  used  for  the   end-­‐to-­‐end  FC  with  IBM  configuration  to  two,  10  Gbps  FCoE  connections  for  the  Cisco  UCS.       We  found  the  end-­‐to-­‐end  FC  configuration  performed  increasingly  better  than  the  FCoE  to  FC   configuration.    Specifically,  at  80%  SAN  utilization  rates,  the  FC  environment  was  50%  faster   as  measured  by  application  I/O  response  times.    As  workloads  increased  to  90%,  latencies   were  4X  faster    (or  1/4th  of  the  latency)  for  the  FC  environment  compared  to  FCoE.         Figure  3:  Response  time  at  32  KB  I/O  size  (Note:  lower  is  better)   Common  Link  Saturation  Examples   Link  saturation  can  occur  in  real-­‐world  usage.    This  would  typically  occur  when  multiple  applications   experience  I/O  peaks  at  the  same  time.    While  the  probability  of  such  an  event  may  seem  low,  it  is  in   fact  quite  common.    This  is  because  many  applications  are  inter-­‐related.    That  is,  many  applications   0.00   2.00   4.00   6.00   8.00   10.00   50%   60%   70%   80%   90%   Latency  in  ms   SAN  Network  Bandwidth   Application  Latency   FCoE   FC  
  15. 15.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  14   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     impact  other  applications,  and  a  spike  in  one  can  drive  spikes  in  other  applications.    For  example,  a  web   application  server  can  drive  usage  in  a  business  application  server,  which  in  turn  can  drive  workloads  on   a  back-­‐end  database  server.       Additionally,  applications  exhibit  peaks,  or  spikes  in  activity  and  do  not  generate  constant  I/O  patterns.     This  can  be  clearly  shown  in  the  testing  by  observing  measured  I/O,  as  shown  in  Appendix  D  on  page  22.   Response  Time   As  shown  above  in  Figure  3,  the  average  response  times  for  transactions  were  similar  for  the  all  FC   configuration  and  the  FCoE  to  FC  configurations  while  the  network  load  was  less  than  70%  of  maximum.     Above  the  70%  level,  the  FCoE  response  times  began  increasing  faster  than  the  FC  response  times.     Above  80%  network  load,  the  rates  diverged  even  further  with  the  FCoE  configuration  experiencing  over   4X  higher  latencies.       Predictability   Variance  and  standard  deviation  are  statistical  methods  for  measuring  predictability.    The  greater  the   variance,  the  greater  the  range  is  for  the  results.    The  variance  increased  as  the  response  times   increased.    Overall  the  FC  environment  using  end-­‐to-­‐end  16  Gb  FC  provided  more  consistent  I/O   response  times  leading  to  more  predictable  application  performance.     An  analysis  of  the  standard  deviation  of  response  time  results  also  showed  differences.    The  Cisco   environment  using  FCoE  to  FC  had  more  variation  than  did  the  IBM  environment  using  end-­‐to-­‐end  FC  at   all  network  loads.    The  greatest  differences  were  found  during  the  mixed  application  workload  testing,   which  produced  highly  variable  results  for  several  workloads,  particularly  the  DVD  Store  database   application.    As  noted  previously,  of  the  100+  instances  started,  21  of  those  jobs  failed  when  using  FCoE,   with  the  remaining  DVD  Store  database  workloads  experiencing  response  times  of  over  1  second  in   many  instances.       Evaluator  Group  comments:    A  higher  variation  in  response  time  for  the  Cisco  environment   lead  to  decreased  predictability  of  response  times.    The  IBM  environment  provided  more   deterministic  results,  which  in  turn  provides  more  predicable  quality  of  service.    These  factors   all  help  IT  users  deliver  higher  service  levels,  or  SLAs  by  using  FC  connectivity.        
  16. 16.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  15   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Evaluation Summary As  previously  noted,  FC  is  the  predominant  connectivity  method  for  storage  systems,  with  over  10x   more  native  connections  than  FCoE  currently  based  on  revenue.    In  terms  of  petabyte  shipments,  FC   has   over   100X   more   connections   than   FCoE   according   to   multiple   sources.7     As   such,   FC   attached   storage   is   the   most   realistic   option   for   enterprise   environments   to   consider   for   their   high-­‐speed   virtualized  environments.   This  evaluation  provided  an  opportunity  to  test  two  competing  configurations  head  to  head,  with  an   IBM  Flex  System  using  FC  connectivity  to  storage,  compared  to  a  Cisco  UCS  solution  using  FCoE  to  FC   connected  to  the  same  storage  system,  in  order  to  provide  comparisons  for  two  viable  alternatives.   These  results  did  not  uncover  a  problem  or  failure  with  Cisco  or  FCoE;  rather  it  shows  that  for  heavily   utilized  storage  environments  with  flash-­‐based,  solid-­‐state  storage;  the  IBM  solution  with  end-­‐to-­‐end   FC  provided  higher  performance  and  reduced  cable  management  when  compared  to  the  Cisco  FCoE  to   FC  configuration.       The  tested  configuration,  showed  the  following  results:   • IBM  Flex  with  FC  delivered  up  to  4x  faster  response  times  as  SAN  utilization  increased   • IBM  Flex  with  FC  provided  higher  performance  with  50%  fewer  connections  than  FCoE   Performance Evaluator   Group   tested   two   leading   enterprise   configurations   from   multiple   perspectives.     The   comparison   was   made   using   application   based   tests,   with   both   configured   as   nearly   identical   as   possible   with   the   primary   difference   being   the   storage   networking   connectivity.   The   superior   performance   of   the   Fibre   Channel   configuration   was   evident   when   compared   to   the   FCoE   environment,  while  FCoE  claims  of  reduced  cabling  were  not  accurate  when  performance  is  a  primary   consideration,  as  was  true  with  the  tested  configuration.       The  Cisco  UCS  with  FCoE  environment  required  more  cables  than  did  the  IBM  Flex  with  FC  environment   including  both  the  LAN  and  SAN  connections.    The  increased  bandwidth  of  the  IBM  Flex  with  16  Gbps   FC  consolidated  multiple  8  Gbps  FC  connections  and  eliminated  the  extra  bridged  SAN  connections   required  by  the  Cisco  UCS  between  the  6248  interconnect  and  the  Cisco  MDS  FC  switch.    With  large   configurations,  the  degree  of  savings  become  significant  due  to  reduced  cabling  of  16  Gb  FC  compared   to  the  converged  FCoE  configuration.                                                                                                                       7  Based  on  IDC  and  Evaluator  Group  data  for  storage  at  the  end  of  2013.  
  17. 17.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  16   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     By  its  nature,  storage  and  therefore  storage  networking  traffic  is  bursty  in  nature,  due  in  part  to  the   interdependence   of   applications.     As   one   portion   of   an   application   increases   demand,   it   places   additional  demands  on  other  portions  of  the  application.    While  the  FCoE  configuration  had  the  same   nominal   bandwidth,   it   would   require   three,   8   Gbps   FCoE   to   FC   connections   to   provide   the   same   effective  bandwidth  as  a  single,  16  Gbps  FC  connection  can  provide.    Thus,  the  tradeoff  between  cost   and   complexity   for   performance   always   remain   and   should   be   evaluated   for   each   scenario   and   workload.   Final Observations Fibre   Channel   technology   remains   the   dominant   choice   for   storage   connectivity   in   the   majority   of   large  enterprises.    This  is  due  to  its  proven  performance,  manageability  and  high  availability  features.   FCoE  is  a  recent  alternative  that  enables  the  FC  protocol  to  run  over  Ethernet  connectivity.    As  a  result,   a  single  enterprise  Ethernet  connection  is  able  to  support  converged  SAN  and  LAN  traffic.       The   promise   of   a   converged   server   and   storage   networking   has   resulted   in   some   IT   organizations   considering   the   use   of   FCoE   between   their   servers   and   Top-­‐of-­‐Rack   switches.     This   evaluation   commonly  occurs  when  upgrading  existing  IT  environments,  or  when  deploying  new  applications.     The   testing   performed   shows   that   as   network   utilization   rates   increase,   Fibre   Channel   delivers   consistent  performance,  while  providing  better  performance  during  high  bandwidth  utilization.       As  a  result,  Evaluator  Group  believes  that  Fibre  Channel  connectivity  is  required  in  order  to  achieve  the   full  benefits  that  solid-­‐state  storage  is  able  to  provide.      
  18. 18.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  17   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Appendix A – Configuration Overview VMware For  both  test  configurations,  VMware  was  utilized.    The  workloads  ran  as  VM’s  within  the  vSphere   environment,  hosted  on  their  respective  IBM  or  Cisco  blades.    Configuration  details  include:   • VMware  vSphere  5.5  Enterprise  Plus,  version  5.5  with  VMware  vCenter  5.5  for  management   • VM’s  hosted  on  either  IBM  or  Cisco  blades,  moved  via  vMotion  for  identical  test  environment   • Storage  access  for  VM’s  enabled  by  shared  SAN  access  of  LUNs  across  the  IBM  and  Cisco  config   • VMW  SATP  was  set  to  recommended  “VMware_SATP_SVC”  for  IBM  FlashSystem  840   • Path  Selection  (PSP)  set  to  “Round  Robin”  with  I/O  count=1  to  both  configurations   IBM Environment The  IBM  environment  consisted  of  the  following:   • Server   o IBM  Flex  System  Chassis   o 2  -­‐  x240  blades,  w/  2x  E2600  CPUs  and  128  GBPS  RAM   o IBM  branded  mezzanine  2600  FC  HBA   o 1  x  IBM  branded  embedded  Gen  5,  16  Gbps  FC  switch   • SAN   o 1  x  IBM48B-­‐5  -­‐  16  Gbps  FC  Switch   • Storage   o IBM  FlashSystem  840,  dual  controllers,  each  with  1  @  16  Gbps  FC  SAN  connection   Cisco Environment The  Cisco  UCS  system  was  chosen  as  a  commonly  available  competing  system.    Due  to  the  lack  of  native   FC  connectivity,  FCoE  from  the  UCS  chassis  was  used  to  a  Cisco  6248  fabric  interconnect.    This  bridged   the  FCoE  to  native  FC,  connected  to  a  Cisco  MDS  9710  FC  director.    A  native  Cisco  virtual  interface  card   (VIC)  was  used  as  the  FCoE  interface.       • Server   o Cisco  UCS  5108  Blade  System   o 2  @  B200  M3  blades,  w/  2x  E2600  CPUs  and  128  GBPS  RAM   o 2  @  Cisco  2208  Fabric  Extender  (FEX)  with  Cisco  VIC  1240  (virtual  interconnect)   o 1  @  Cisco  6248UP  Fabric  Interconnect  (FCoE  to  FC  bridging)  750W,     • SAN   o 1  -­‐  Cisco  MDS  9710  with  16  Gbps  FC  Switch     • Storage   o IBM  FlashSystem  840,  dual  controllers,  each  with  1  @  16  Gbps  FC  SAN  connection  
  19. 19.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  18   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Appendix B - IBM Flex System Environment The  chosen  FC  environment  was  constructed  using  the  following  components:   • IBM  Flex  System  blade  enclosure   • IBM  x240  blades  (quantity  2),  each  with:   o 2  @  E5  2609  CPU’s  /  blade   o 128  GB  RAM  /  blade  (8  x  16  GB)   • IBM  Flex  System  FC5022  SAN  Scalable  Switch   • IBM  FC5172  embedded  2  port,  16  Gb  FC  mezzanine  HBA     Shown  below  is  a  screenshot  of  the  IBM  CMM  Manager.           Figure  4:  IBM  CMM  GUI  (Source:  EGI  Labs)    
  20. 20.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  19   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Appendix C - Cisco UCS Environment The  chosen  FCoE  environment  was  constructed  using  the  following  components:   • Cisco  UCS  5108  blade  enclosure   • Cisco  B200  M3  blades  (2  each)   o 2  @  E5  2609  CPU’s  /  blade   o 128  Gbps  RAM  /  blade  (8  x  16  GB)   • Cisco  2208  -­‐  Fabric  extender  (FEX)       • Cisco  6248UP  Fabric  interconnect     Shown  below  is  a  screenshot  of  the  Cisco  UCS  manager  used  during  testing  (note,  only  2  blades  used).         Figure  5:  Cisco  UCS  Manager  GUI  (Source:  EGI  labs)  
  21. 21.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  20   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.       Figure  6:  UCS  Cabling  Configuration  (UCS  Manager  Screenshot)     Above  is  a  screenshot  capture  of  the  Cisco  UCS  manager  showing  the  internal  connectivity.    The  two   highlighted  ports  of  27  and  28  were  attached  via  FC  to  the  external  Cisco  9710  MDS  for  storage   connectivity.      
  22. 22.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  21   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.     Appendix D - Application Workloads The  application  workloads  were  generated  using  I/O  traces  of  actual  application  workloads.       Shown  below  is  a  trace  for  each  datastore  on  a  single  ESXi  host  (blade  server).    Each  datastore   represents  a  single  LUN  mapped  from  a  solid-­‐state  storage  system  used  as  a  test  target  device.    As  seen   below,  each  workload  varied  over  time,  and  was  not  a  static  workload  as  is  typically  found  with  synthetic   workloads.       One  important  consideration  for  real  world  environments  is  the  dynamic  nature  of  workloads.    As  seen   below,  each  workload  varies  over  time.    Additionally,  some  workloads  are  positively  correlated  with   other  workloads.    That  is,  an  increased  I/O  rate  for  one  application  may  in  turn  generate  an  increased   I/O  rate  for  other  applications.       The  workload  shown  below  is  identical  for  both  the  FC  and  FCoE  environment,  since  this  graph  depicts   each  workloads  I/O  rate,  which  is  the  same  regardless  of  supporting  storage  networking  technology.   Evaluator  Group  Comments:    All  solid-­‐state  storage  was  utilized  for  both  tests,  enabling   higher  transactional  throughput  that  conventional  storage.    Each  of  the  I/O  traces  shown   below  represents  one  or  more  applications.    The  total  I/O  required  is  the  sum  of  each  of  the   individual  traces.    As  seen  below,  peaks  can  occur,  creating  potential  bottlenecks  for   environments  without  sufficient  bandwidth.    For  the  FCoE  environment,  bottlenecks  were   experienced  during  the  periods  of  high  total  I/O  activity.     Figure  7:  vCenter  trace  of  ESXi  host  running  mixed  application  workload  test    
  23. 23.     Test  Validation  –  IBM  Flex  System  vs.  Cisco  UCS   Russ  Fellows   p.  22   of  22     ©  2014  Evaluator  Group,  Inc.    All  rights  reserved.    Reproduction  of  this  publication  in  any  form     without  prior  written  permission  is  prohibited.                                 About Evaluator Group Evaluator  Group  Inc.  is  dedicated  to  helping  IT  professionals  and  vendors  create  and  implement  strategies  that  make  the  most  of  the  value   of  their  storage  and  digital  information.  Evaluator  Group  services  deliver  in-­‐depth,  unbiased  analysis  on  storage  architectures,   infrastructures  and  management  for  IT  professionals.    Since  1997  Evaluator  Group  has  provided  services  for  thousands  of  end  users  and   vendor  professionals  through  product  and  market  evaluations,  competitive  analysis  and  education.    www.evaluatorgroup.com  Follow  us   on  Twitter  @evaluator_group   Copyright 2014 Evaluator Group, Inc. All rights reserved. No  part  of  this  publication  may  be  reproduced  or  transmitted  in  any  form  or  by  any  means,  electronic  or  mechanical,  including  photocopying  and  recording,   or  stored  in  a  database  or  retrieval  system  for  any  purpose  without  the  express  written  consent  of  Evaluator  Group  Inc.    The  information  contained  in  this   document  is  subject  to  change  without  notice.  Evaluator  Group  assumes  no  responsibility  for  errors  or  omissions.    Evaluator  Group  makes  no  expressed  or   implied  warranties  in  this  document  relating  to  the  use  or  operation  of  the  products  described  herein.    In  no  event  shall  Evaluator  Group  be  liable  for  any   indirect,  special,  inconsequential  or  incidental  damages  arising  out  of  or  associated  with  any  aspect  of  this  publication,  even  if  advised  of  the  possibility  of   such  damages.    The  Evaluator  Series  is  a  trademark  of  Evaluator  Group,  Inc.    All  other  trademarks  are  the  property  of  their  respective  companies.     This  document  was  developed  with  IBM  funding.  Although  the  document  may  utilize  publicly  available  material  from   various  vendors,  including  IBM,  it  does  not  necessarily  reflect  the  positions  of  such  vendors  on  the  issues  addressed  in  this   document.  

×