• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
OSDC 2014: Christopher Kunz - Software defined networking in an open-source compute cloud
 

OSDC 2014: Christopher Kunz - Software defined networking in an open-source compute cloud

on

  • 136 views

Big networking vendors have discovered network virtualization for themselves. However, not only hardware appliances, but also open-source solutions have various means of virtualising networks.Hosting ...

Big networking vendors have discovered network virtualization for themselves. However, not only hardware appliances, but also open-source solutions have various means of virtualising networks.Hosting an IaaS cloud, you are faced with the challenge of isolating VMs, implementing private internal networks, billing and accounting, firewalls and shaping. And all these challenges should not affect the rest of your (non-virtualized) network. Using OpenVSwitch, you can tackle many of these tasks. In this session, we show you the caveats, but also the exciting possibilities of open-source network virtualization in practical examples.

Statistics

Views

Total Views
136
Views on SlideShare
124
Embed Views
12

Actions

Likes
0
Downloads
5
Comments
0

1 Embed 12

http://www.netways.de 12

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    OSDC 2014: Christopher Kunz - Software defined networking in an open-source compute cloud OSDC 2014: Christopher Kunz - Software defined networking in an open-source compute cloud Presentation Transcript

    • Heartbleed ...and why yours should, too
    • You are in the right session _ This  is  an  emergency  service  announcement   _ Due  to  events  that  transpired  on  Tuesday   _ I  thought  it‘d  be  good  to  have  some  info   10.04.14 OSDC 2014 2  
    • About me _ Dr.  Christopher  Kunz   _ Studied  CompSci  in  Hannover,  PhD   in  2012   _ Works  as  a  hoster  for  15  years   _ Some  admin  experience   _ Used  to  do  a  lot  of  PHP   _ Author,  „PHP-­‐Sicherheit“,  ed.  1-­‐3   _ And  don‘t  get  me  started  about   swords!   10.04.14 OSDC 2014 3  
    • About filoo _ hQps://www.filoo.de   _ Quickly-­‐growing  hosVng  company   _ Data  center  in  Frankfurt,  Germany   _ Developed  own  IaaS  middleware   _ QEMU/KVM,  OVS,  Ceph   _ Offer  hosVng,  co-­‐locaVon,  cloud  services   _ 100%  subsidiary  of  Thomas-­‐Krenn.AG   _ Visit  their  booth!   10.04.14 OSDC 2014 4  
    • Heartbleed in a nutshell _ A  bug  with  a  cute  name   _ ...and  not  so  cute  effects   _ Pre-­‐auth,  pre-­‐logging   universal  TLS/SSL  bug   _ Introduced  in  OpenSSL   1.0.1a  (2012)   _ Allows  to  make  64kb   memory  dumps  of  the   server‘s  memory   10.04.14 OSDC 2014 5  
    • Wait. What? _ Yes,  remote  memory  dumps   _ Due  to  an  unchecked  buffer  length,  a  TLS  enabled   server  may  dump  memory  contents  to  the  client   _ Limit  of  64k  per  reply   _ MulVple  replies  possible   _ Memdump  may  contain...   _ URLs  and  GET  /  POST  variables   _ Random  excerpts  from  whatever   _ Source  code  of  scripts/whatever  else   _ SSL  cerVficate  private  keys   10.04.14 OSDC 2014 6  
    • About DTLS heartbeats _ RFC  6520,  Transport  Layer  Security  (TLS)  and  Datagram   Transport  Layer  Security  (DTLS)  Heartbeat  Extension   _ Provides  a  heartbeat  for  TLS  (TCP)  and  DTLS  (mostly   UDP)  sessions   _ Intended  to  add  stability  to  unstable  connecVons  and   prevent  renegoVaVons   _ Implemented  in  OpenSSL  as  part  of  a  PhD  thesis   _ Patch  commiQed  Dec  15,  2011     10.04.14 OSDC 2014 7  
    • What this bug is not _ This  is  not  a  crypto  bug   _ At  least  not  in  its  primary  funcVon   _ This  is  not  a  fully  arbitrary  mem  disclosure   _ Only  memory  belonging  to  aQacked  daemon  can  be  dumped   _ This  is  not  a  remote  root  hole   _ Hence  the  relaVvely  low  CVE  score  of  5.0   10.04.14 OSDC 2014 8  
    • Anatomy of the bug 1 struct { HeartbeatMessageType type; uint16 payload_length; opaque payload[HeartbeatMessage.payload_length]; opaque padding[padding_length]; } HeartbeatMessage; _ From RFC6520: _ payload_length: The length of the payload. _ payload: The payload consists of arbitrary content. 10.04.14 OSDC 2014 9  
    • Anatomy of the bug 2 _ ssl/d1_both.c,  line  1474+:   buffer = OPENSSL_malloc(1 + 2 + payload + padding); bp = buffer; [..] memcpy(bp, pl, payload); _ From: https://github.com/openssl/openssl/commit/ 4817504d069b4c5082161b02a22116ad75f822b1 10.04.14 OSDC 2014 10  
    • Anatomy of the bug _ The  heartbeat  extension  allocates  payload+19  bytes   of  memory   _ Copies  pl  bytes  of  arbitrary  user-­‐supplied  data   payload  via  memcpy()  to  construct  response   _ Client  sets  pl  to  65535   _ Client  sends  only  1  byte  of  data  in  payload _ Response  contains  1  byte  of  client-­‐supplied  payload   _ ...and  64K  of  RAM  from  the  memcpy()  call   _ Analysis  in:  hQp://blog.existenValize.com/diagnosis-­‐of-­‐ the-­‐openssl-­‐heartbleed-­‐bug.html   10.04.14 OSDC 2014 11  
    • Test vulnerability _ Python  script  at:  hQps://gist.github.com/takeshixx/ 10107280   _ Can  test  any  SSL/TLS  enabled  TCP  service   _ Has  support  for  StartTLS  (-­‐s  opVon)   _ Conveniently  dumps  64kb  of  memory  for  you   10.04.14 OSDC 2014 12   00d0: 10 00 11 00 23 00 00 00 0F 00 01 01 33 41 31 25 ....#.......3A1% 00e0: 32 43 25 32 32 5F 6D 6F 64 65 25 32 32 25 33 41 2C%22_mode%22%3A 00f0: 25 32 32 6A 73 6F 6E 25 32 32 25 32 43 25 32 32 %22json%22%2C%22 0100: 5F 69 64 25 32 32 25 33 41 25 32 32 70 5F 33 30 _id%22%3A%22p_30 0110: 33 34 35 38 31 38 25 32 32 25 32 43 25 32 32 5F 345818%22%2C%22_ 0120: 63 6F 6E 74 61 69 6E 65 72 25 32 32 25 33 41 30 container%22%3A0 0130: 25 32 43 25 32 32 5F 61 63 74 69 6F 6E 25 32 32 %2C%22_action%22 0140: 25 33 41 25 32 32 76 69 65 77 25 32 32 25 32 43 %3A%22view%22%2C
    • Memdump _ From:  hQps://twiQer.com/markloman/status/ 453502888447586304   10.04.14 OSDC 2014 13  
    • Memdump _ Memory  contents  is  non-­‐determinisVc   _ SomeVmes  exciVng,  mostly  boring   _ while true
 do python hb-test.py yahoo.com | grep -C 2 login >> /tmp/out; 
 sleep 1; 
 done" _ Profit!   10.04.14 OSDC 2014 14  
    • Detect exploitation _ No  logging  on  the  machine   _ All  exploitaVon  is  pre-­‐logging,  pre-­‐applicaVon   _ IDS  vendors  are  pushing  out  signatures  already   10.04.14 OSDC 2014 15  
    • Affected services _ Above  all,  SSL-­‐enabled  web  servers   _ Any  that  uses  OpenSSL,  anyway   _ Mail  servers   _ IMAP  over  SSL,  POP  over  SSL,  SMTP  over  SSL,  StartTLS   _ VPN  tunnels   _ OpenVPN  when  using  cert  auth  (maybe?)   _ PotenVally  others   _ IRC  servers,  XMPP,  FTP  over  TLS   _ Android  4.1.1  is  vulnerable   _ OpenSSH  is  not  vulnerable   10.04.14 OSDC 2014 16  
    • Linux versions affected _ OpenSSL  1.0.1  a  thru  f   _ Debian  Wheezy,  Jessie,  Sid   _ Fixed  for  Wheezy  &  Sid   _ Ubuntu  10.04,  12.04,  12.10,  13.10,  14.04   _ Fixed  packages  exist   _ RHEL  6   _ Patch  exists   _ And  all  others  that  ship  OpenSSL   _ Clients  are  also  vulnerable!     10.04.14 OSDC 2014 17  
    • Other affected stuff _ Cisco  devices:  „We  use  Cisco  SSL  which  is  not   OpenSSL.“;  SSL  VPN  products  potenVally  affected   _ Juniper  has  released  fixes  for  their  SSL  VPN,  none  for  J-­‐ Web  etc.  yet   _ Big  IP?  Kemp?  Fritz.Box?  Your  home  NAS?     _ More  info  (hopefully)  here:  hQp://www.kb.cert.org/ vuls/byvendor?searchview&Query=FIELD +Reference=720951&SearchOrder=4   10.04.14 OSDC 2014 18  
    • Mitigation & cleanup _ First,  upgrade  to  fixed  openssl   _ apt-­‐get  install  openssl  libssl-­‐1.0.0   _ Next,  restart  all  services  that  load  old  lib   _ Use  checkrestart  or  lsof  –n  |  grep  DEL  |  grep  ssl   _ If  you  use  staVc  binaries,  recompile  everything   _ If  you  use  Google‘s  mod_spdy  on  Apache2.2,  don‘t   _ It  has  its  own  staVcally  linked  mod_ssl  which  is  shamefully   out  of  date   10.04.14 OSDC 2014 19  
    • What about certs? _ It  is  possible  that  privkeys   have  leaked   _ If  so,  you  need  to   revoke&reissue  certs   _ Some  CAs  offer  free   reissue   _ If  you  don‘t  have  PFS,  you   have  a  problem   _ AQackers  who  sniffed  your   traffic  might  be  able  to   decode  it   10.04.14 OSDC 2014 20  
    • Thank you _ .Do  not  despair,  there  is  hope!   _ ...and  now,  back  to  our  regular  scheduled  programme!   10.04.14 OSDC 2014 21   hQp://xkcd.com/1353/  
    • Software-defined Networking In an open-source cloud
    • Agenda _ High-­‐Level  overview:  What  is  this  about?   _ The  use  case  –  virtualized  networks  for  IaaS   _ Intro  to  OpenVSwitch   _ How-­‐to:  Deploy  OpenVSwitch   _ Frontnet,  Backnet,  public  net   _ Firewalling   _ Tying  it  all  together   10.04.14 OSDC 2014 23  
    • So what‘s the hype? _ Sovware-­‐Defined  Networking  is  the  hype   _ I‘m  not  good  with  hype   _ Networking  is  decoupled  from  bare  metal   _ EssenVally  you  virtualize  parts  of  your  network   _ Control  and  data  plane  are  decoupled   _ Many  vendors  jumped  on  the  train   _ HP,  Cisco,  VMWare,  you  name  it   10.04.14 OSDC 2014 24  
    • OpenFlow _ ImperaVve  control   _ Switches  are  dumb  –  they  only  forward  according  to   rules   _ OpenFlow  controllers  make  the  rules   _ First  packet  of  each  type  is  sent  thru  OpenFlow   controller   _ Subsequent  ones  go  directly  through  switch   10.04.14 OSDC 2014 25  
    • OpFlex _ Cisco‘s  answer  to  OpenFlow   _ Other  vendors  on  board:  Citrix,  MSFT,  RHAT,  Canonical   _ Not  on  board:  J,  HP,  Huawei,  vmWare   _ Balance  intelligence  between  switch  and  controller   _ „DeclaraVve  control“;  just  declare  how  you  want  it  and  the   switch  interprets  that  rule   _ IETF  proposed  standard   _ Drav-­‐smith-­‐opflex   _ Open  APIs   _ AltruisVc  goal:  Eliminate  SPOF  (the  controller)   _ EgoisVc  goal:  Sell  smarter  (=$++)  switches   10.04.14 OSDC 2014 26  
    • The OSS Contender _ OpenVSwitch   _ Openvswitch.org   _ Open  Source   _ Apache  2.0  license,  non-­‐viral   _ GPLv2   _ MulVlayer  (2,3)  virtual  switch   _ Supports  lots  of  interesVng  features   _ VLANs,  Ne{low,  sFlow,  LACP,  filtering,  ...   10.04.14 OSDC 2014 27  
    • OVS Overview _ Shamelessly  lived  from  [1]   10.04.14 OSDC 2014 28   ovs-vswitchd OVS Kernel Module Control Cluster ovsdb-server Off-box User Kernel Management Protocol (6632/TCP) OpenFlow (6633/TCP) Netlink
    • OSVDB _ Database  holds  configuraVon  items   _ DefiniVons  for  bridges,  tunnels,  interfaces   _ Controller  addresses   _ ConfiguraVon  is  reboot-­‐safe   _ Custom  database  system,  not  MySQLiteMongoDB     _ Speaks  custom  protocol  (OSVDB)   _ Log  based   _ osvdb-­‐tool  show-­‐log  shows  all  changes     _ Nivy  for  debug  /  change  management!   10.04.14 OSDC 2014 29  
    • How ovs works _ ImperaVve  control   _ All  intelligence  is  in  the  controller   _ Data  path  only  carries  out  instrucVons   _ Data  Path   _ Kernel  module     _ Licensed  under  GPLv2   _ Controller   _ Lives  in  userland   _ Licensed  under  Apache  2.0   10.04.14 OSDC 2014 30  
    • Flow flow _ Everything  is  a  flow   _ CombinaVon  of  input  port,  VLAN,  MAC,  IP,  TCP/UDP  port   10.04.14 OSDC 2014 31  
    • OVS management _ Command-­‐line  tools   _ Ovs-­‐vsctl  for  switch  management   _ Ovs-­‐ofctl  for  flow  management   _ Ovsdb-­‐tool  for  database  management   10.04.14 OSDC 2014 32  
    • What‘s our angle here? _ filoo  is  a  hoster.   _ We  host  VMs.   _ VMs  need  networking.   _ See  where  this  goes?   10.04.14 OSDC 2014 33  
    • What we wanted _ Internet-­‐facing  front-­‐net  interface   _ Private  LAN  for  VMs   _ VM  isolaVon   _ Firewalling   _ Traffic  shaping   _ Fine-­‐grained  accounVng   _ Live  migraVon   10.04.14 OSDC 2014 34  
    • Overview - physical 10.04.14 OSDC 2014 35   Back-­‐end  switch   Front-­‐end  switch  
    • Overview - virtual 10.04.14 OSDC 2014 36   Firewall   Firewall   Firewall  
    • Overview – OVS stack 10.04.14 OSDC 2014 37   OVS   OVS   OVS  
    • Let‘s get started _ We  usually  compile  ovs  ourselves   _ There  are  also  packages  in  apt   _ Those  might  work  or  not   _ Download  &  compile  OVS   _ Latest  stable:  2.1.0,  latest  LTS:  1.9.3   _ ./boot.sh  &&  ./configure  &&  make  &&  make  install   _ Kernel  module  from  3.3+   _ Enable  in  Kernel  Networking  -­‐>  OpVons  -­‐>  Open  Vswitch   _ modprobe  openvswitch   10.04.14 OSDC 2014 38  
    • Let‘s get started 2 _ Set  up  ovs  db   _ Ovsdb-­‐tool  create  conf.db  vswitch.ovsschema   _ Conf.db  is  in  /usr/localetc/openvswitch   _ /usr/src/openvswitch-­‐1.9.3/vswitchd/vswitch.ovsschema     _ Make  sure    ovs-­‐vswitchd  and  ovsdb-­‐server  start  before   networking   _ Add  startup  entries  to  rc.local   _ Remove  networking  from  rc.d   _ start  networking  in  rc.local   10.04.14 OSDC 2014 39  
    • Initial bridges _ Front-­‐net  vlan:  199   _ Same  procedure  for  back-­‐net  VLAN   _ Add  bridges   _ ovs-­‐vsctl  add-­‐br  vmbr1   _ ovs-­‐vsctl  add-­‐port  vmbr1  vlan199  tag=199   _ ovs-­‐vsctl  set  interface  vlan199  type=internal   _ Log  in  via  IPMI   _ ovs-­‐vsctl  add-­‐port  vmbr1  eth1   _ Machine  is  offline  now   _ Modify  physical  switching   10.04.14 OSDC 2014 40  
    • VM networking _ We  use  KVM/QEMU   _ Add  the  TAP  interface   _ /sbin/ip  tuntap  add  dev  tap1i0d0  mode  tap  user  fcms   _ qemu-­‐system-­‐x86_64  ...  -­‐device   rtl8139,mac=00:F1:70:00:00:10,netdev=vlan0d0  -­‐netdev   type=tap,id=vlan0d0,ifname=tap1i0d0   _ Bring  up  the  port   _ /usr/local/bin/ovs-­‐vsctl  add-­‐port  vmbr0  tap1i0d0  199   other_config:stp-­‐enable=false   10.04.14 OSDC 2014 41  
    • From TAP to port to flow _ We  have  a  tap  interface  tap1i0d0   _ Find  the  corresponding  bridge  port:   _ ovs-­‐ofctl  show  vmbr0  |  grep  tap1i0d0     _ 1820(tap1i0d0):  addr:fa:7a:67:e3:5d:€     _ Now  we  have  a  port  number:  1820   _ We  use  this  port  for  flow  management   10.04.14 OSDC 2014 42  
    • Multiple interfaces _ Add  more  TAP  interfaces   _ Assign  one  VLAN  per  customer   _ Internal  network  across  VMs  on  same  node     _ Make  VLAN  known  on  inter-­‐node  switches   _ Via  whatever  switch  automaVon  you  have   _ Cross-­‐node  internal  networking   _ VLAN  limits  apply  –  hard  cut  at  ~4090     _ Overlay  networks  to  the  rescue   10.04.14 OSDC 2014 43  
    • Prevent MAC spoofing _ PORT=1820   _ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  arp   idle_Vmeout=0  priority=39500  acVon=resubmit("$ {PORT}",2)“   _ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  table=2   arp  priority=200  idle_Vmeout=0   arp_sha=00:F1:70:00:00:10  nw_src=192.168.1.1   acVon=normal"     _ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  table=2   priority=100  idle_Vmeout=0  acVon=drop"   10.04.14 OSDC 2014 44   We  know  this  MAC   because  we  control   the  hypervisor!   We  know  this   address  too!  
    • Caveats for MAC/ARP _ SomeVmes  you  want  customers  to  spoof   _ HA  soluVons  that  switch  „cluster  IP  addresses“   _ You  can  cater  for  this  in  case  you  know  the   corresponding  MACs   _ Assign  sequenVal  MACs  and  wildcard   _ Or  set  specific  rules   _ OpVonal  „HA  feature“  for  VMs   _ Never  allow  customers  to  wildcard  here!   10.04.14 OSDC 2014 45  
    • Firewalling with flows _ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  table=1   tcp  idle_Vmeout=0  nw_dst=192.168.12.13/32   nw_src=192.168.1.123/32  tp_dst="80"  priority=38000   acVon=drop“   _ From  192.168.1.123     _ To  192.168.12.13   _ Port  80   _ Drop   10.04.14 OSDC 2014 46  
    • Port ranges _ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  table=1   tcp  idle_Vmeout=0  nw_src=192.168.1.123/32   nw_dst=192.168.12.13/24  tp_src="0x05E8/0xFFFC"   priority=37960  acVon=drop“   _ Source  192.168.1.123   _ DesVnaVon  192.168.12.0/24   _ Source  port  =  0x05E8/0xFFFC   _ 0x05E8/0xFFFC  =  1512/65532   _ Port  1512  –  1516   _ OVS  1.11  supports  „Megaflows“,  i.e  universal   wildcarding   10.04.14 OSDC 2014 47  
    • Default accept _ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  table=1   priority=100  acVon=normal“   _ Fallthru  rule   _ Match  everything  else   10.04.14 OSDC 2014 48  
    • Accounting _ We  grab  interface  counters  from  the  tap  interfaces   _ You  can  also  use  Ne{low/sFlow  or  ipfix     _ We  didn‘t  go  there  yet,  experiences  welcome   10.04.14 OSDC 2014 49  
    • Shaping _ Simple  shaping:   _ ovs-­‐vsctl  set  Interface  tap0  ingress_policing_rate=100000   _ ovs-­‐vsctl  set  Interface  tap0  ingress_policing_burst=1000   _ QoS  policies:   _ ovs-­‐vsctl  set  port  eth1  qos=@newqos     id=@newqos  create  qos  type=linux-­‐htb     other-­‐config:max-­‐rate=200000000  queues=0=@q0,1=@q1     _ We  don‘t  do  QoS  policies,  shaping  works  mostly  as   intended   10.04.14 OSDC 2014 50  
    • Live migration _ We  don‘t  actually  do  OVS‘s  own  live  migraVon   _ Start  VM  on  target  host  in  suspend-­‐to-­‐RAM  mode   _ Stop  VM  on  losing  host;  down  interface   _ Resume  VM  on  target  host   _ There  are  live  migraVon  mechanisms  in  OVS   _ L2  based     _ Inter-­‐OVS  GRE  tunnel   _ Honestly,  I  have  no  clue.   10.04.14 OSDC 2014 51  
    • Thank you _ I  hope  you  learned  something   _ If  not,  I  hope  you  had  a  laugh  at  my  expense   _ If  neither,  I‘m  really  sorry.  Beer?   _ QuesVons?   10.04.14 OSDC 2014 52  
    • Literature _ [1]  hQp://openvswitch.org/slides/ OpenStack-­‐131107.pdf  –  OVS  Deep  Dive   _ OVS  IntroducVon:  hQp://horms.net/projects/ openvswitch/2010-­‐10/openvswitch.en.pdf   10.04.14 OSDC 2014 53