Herding networking cats:
Integrating Linux routing with
         FusionCLI™

       Stephen Hemminger
     shemminger@vyatta.com
Outline

  Introduction
  FusionCLI™
  QoS configuration
  Issues
My background

  OSDL – Linux Foundation
    Linux 2.5 network infrastructure → bridging
    TCP congestion control → netem → iproute
    Network performance → Marvell replacement drivers
  Vyatta
    Kernel performance
    Quality Of Service support
Vyatta versions

      http://vyatta.org      http://vyatta.com
    Free download         Subscription
      livecd
    Update 2x year        Update 4x year

    Community forums
                          Software or Hardware
                         Phone and Email
                          Support
       Same source and features
Vyatta Distribution
   Debian GNU/Linux
      Current version based on testing → lenny
   Linux kernel 2.6.24 + bugfix
      Filesystems: unionfs, squashfs
      Serial driver: wanpipe
   Additional packages
      CLI infrastructure
      Configuration templates
   Updated routing related packages
      Quagga, SNMP, SNORT, ...
Linux cats
CLI Requirements

  Router look & feel
 Command completion
 Roles: Administrator, operator
 Configure mode
 Extensible
 Text based
 Language neutral
 Integrated with operating system
Vyatta




 Proprietary                                                  Open
  System                                                     System

  Monolithic SW

                                                      Graphical User Interface
                                                                                                        Network Apps
                                                            FusionCLITM
       Internet Protocols




                                                                                                           WAN Optimization
                                                         Open API
                                                                                                       Load Balancing
                            Scalable Routing       NW Functions            Security
                                                                                          Extensible
                            Internet Protocols   DHCP, NAT, Radius…       Firewall, VPN                            Anti-X
                                                                                                         PBX
                                                           Linux Kernel                                          IDS
     Unique
      HW                                                                                                         Massive
                                                                                                               open-source
                                                          Standard HW                                          ecosystem…
CLI architecture
Demo 1: Basic interface

  $ show interfaces 
  Interface    IP Address         State       Link   Description
  eth0         192.168.111.132/24 up          up
  eth1         ­                  up          up 
  lo           127.0.0.1/8        up          up  
  lo           ::1/128            up          up

  $ show interfaces et<tab>hernet <tab>   
  detail  eth0    eth1    

  $ show interfaces ethernet eth0 
  eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
    state UNKNOWN qlen 1000
      link/ether 00:0c:29:f6:20:9e brd ff:ff:ff:ff:ff:ff
      inet 192.168.111.132/24 brd 192.168.111.255 scope global eth0
      inet6 fe80::20c:29ff:fef6:209e/64 scope link 
         valid_lft forever preferred_lft forever
  ...
Template hierarchy

  /opt/vyatta/share/vyatta­op/templates/show/interfaces/ethernet
  |­­ detail
  |   `­­ node.def
  |­­ node.def
  `­­ node.tag
      |­­ brief
      |   `­­ node.def
      |­­ capture
      |   |­­ node.def
      |   |­­ not
      |   |   |­­ node.def
      |   |   `­­ port
      |   |       |­­ node.def
      |   |       `­­ node.tag
      |   |           `­­ node.def
      |   `­­ port
      |       |­­ node.def
      |       `­­ node.tag
      |           `­­ node.def
      |­­ identify
      |   `­­ node.def
      |­­ node.def
      |­­ physical
      |   `­­ node.def
Operational template

           show/interfaces/ethernet/node.tag/node.def
  help: Show specified ethernet interface information
  allowed: for dev in /sys/class/net/*; 
           do if [[ ­L $dev/device ]]
              then if [[ $(cat $dev/type) ­eq 1 ]]
                   then echo ­n ${dev##*/} " "
                   fi
              fi
           done
  run: vyatta­show­interfaces.pl ­­intf="$4"
Demo 2: Configuration

  $ configure
  [edit]
  # set interfaces ethernet eth0 description 'Vmware NAT'
  [edit]
  # show interfaces ethernet eth0
   address dhcp
  +description "Vmware NAT"
   hw­id 00:0c:29:f6:20:9e
  [edit]

  # mount
  ...
  unionfs on /opt/vyatta/config/tmp/new_config_5035 type unionfs    
    (rw,dirs=/tmp/changes_only_5035=rw:/opt/vyatta/config/active=r
    o)
  [edit]
  vyatta@vc3­1­3# commit
Configuration templates

  multi:
  type: txt
  help: Set an IP address for this interface
  syntax:expression: exec "/opt/vyatta/sbin/vyatta­interfaces.pl 
      ­­valid­addr $VAR(@) ­­dev $VAR(../@)" 
     ; "Invalid IP address/prefix [$VAR(@)] for interface $VAR(../@)"

  update: /opt/vyatta/sbin/vyatta­interfaces.pl 
     ­­eth­addr­update $VAR(@) ­­dev $VAR(../@)

  delete: /opt/vyatta/sbin/vyatta­interfaces.pl 
      ­­eth­addr­delete $VAR(@) ­­dev $VAR(../@)

  allowed: echo "dhcp <>"
  comp_help:Possible completions:
    <x.x.x.x/x>     Set the IP address and prefix length
    <h:h:h:h:h:h:h:h/x>   Set the IPv6 address and prefix length
    dhcp        Set the IP address and prefix length via DHCP
Configuration save restore

  interfaces {
      ethernet eth0 {
          address dhcp
          duplex auto
          hw­id 00:0c:29:f6:20:9e
          speed auto
      }
      loopback lo {
      }
  }
  service {
      ssh {
          port 22
          protocol­version v2
      }
  }
Quality of Service (QoS) usage models

  Real time services
       VOIP
  Network control plane
       BGP, OSPF, STP
  Fairness

  Throttle batch services
       P2P, backup,
Vyatta QoS

  Organized by policy types
    Fair queue          => sfq
    Traffic shaper      => htb
    Drop tail           => fifo
    Rate limiter        => tbf
    Traffic limiter   => ingress
    ...
fair-queue

  # set qos­policy fair­queue fq
  [edit]
  # set interfaces ethernet eth0 qos­policy out fq
  [edit]
  # commit
  [edit]
  # run show queueing   
  Output queues:
  Interface  Qos­Policy             Sent    Dropped   Overlimit
  eth0       fair­queue             4578          0          0
  eth1       default                 468          0          0
fair-queue template

              set/qos-policy/fair-queue/node.def
  tag:
  type: txt
  help: Set fair queueing policy
  syntax:expression: pattern $VAR(@) "^[[:alnum:]][­_[:alnum:]]*$"
                     ; "only alpha­numeric policy name allowed"
  update: /opt/vyatta/sbin/vyatta­qos.pl ­­create­policy "$VAR(.)" 
    "$VAR(@)"
  delete: /opt/vyatta/sbin/vyatta­qos.pl ­­delete­policy "$VAR(@)"
Qos on Ethernet Interface

    set/interfaces/ethernet/node.tag/qos-policy/out/node.def
  type: txt
  help: Set outbound QOS policy for specified ethernet interface
  allowed: /opt/vyatta/sbin/vyatta­qos.pl ­­list­policy
  update: /opt/vyatta/sbin/vyatta­qos.pl 
     ­­update­interface $VAR(../../@) $VAR(.) $VAR(@)
  delete: /opt/vyatta/sbin/vyatta­qos.pl 
     ­­delete­interface $VAR(../../@) $VAR(.)
Internals - perl code

  sub update_interface {
      my ($interface, $direction, $name ) = @_;
      my $config = new VyattaConfig;

      ( $direction eq "out" ) or die "Only out direction 
    supported";

      $config­>setLevel('qos­policy');
      foreach my $type ( $config­>listNodes() ) {
          if ( $config­>exists("$type $name") ) {
            my $shaper = make_policy($config, $type, $name);
            delete_interface($interface, $direction);
            open my $out, "|­" or exec qw:sudo /sbin/tc ­batch ­:;
            $shaper­>commands($out, $interface);
            if (! close $out) {
                delete_interface($interface, $direction);
            }
      }

      die "Unknown qos­policy $namen";
  }
QoS traffic-shaper

  # edit qos­policy traffic­shaper lartc

  [edit qos­policy traffic­shaper lartc]

  # set class 2 bandwidth 100%

  [edit qos­policy traffic­shaper lartc]

  # set class 2 match www ip destination port 80

  [edit qos­policy traffic­shaper lartc]

  # set class 3 bandwidth 3mbit

  [edit qos­policy traffic­shaper lartc]

  # set class 3 ceiling 5mbit  

  [edit qos­policy traffic­shaper lartc]

  # set class 3 match smtp ip destination port 25

  [edit qos­policy traffic­shaper lartc]

  # exit
Traffic-shaper continue

  [edit]

  # commit

  qos­policy traffic­shaper lartc configuration not complete: missing 
     default class

  Commit failed

  # set qos­policy traffic­shaper lartc default bandwidth 1

  [edit]

  # commit

  [edit]

  # set interfaces ethernet eth0 qos­policy out
Result

  # run show queueing ethernet eth0         

  eth0 Output queue:
  Class      Qos­Policy             Sent    Dropped   Overlimit
  1:        traffic­shaper        11438          0          0
    4       fair­queue            11438          0          0
    2       fair­queue                0          0          0
    3       fair­queue                0          0          0
  [edit]
Issues

  Developer documentation
    Linux Kongress paper
         Watch this space
  Vyatta ↔ Linux management
    Vyatta config ignores other changes
    CLI stays loosely coupled
  Vyatta package changes
    Every distribution is a fork
    All changes are fed to upstream
SPC-FLOSS: orphaned projects

  Users want support for orphaned projects
    Multicast routing
    MPLS
    RSTP
    IPV6
Future
Richer features
  QoS+, bonding, …
GUI
Multi-queue
 Performance

Integrating Linux routing with FusionCLI™

  • 1.
    Herding networking cats: IntegratingLinux routing with FusionCLI™ Stephen Hemminger shemminger@vyatta.com
  • 2.
    Outline Introduction FusionCLI™ QoS configuration Issues
  • 3.
    My background OSDL – Linux Foundation Linux 2.5 network infrastructure → bridging TCP congestion control → netem → iproute Network performance → Marvell replacement drivers Vyatta Kernel performance Quality Of Service support
  • 7.
    Vyatta versions http://vyatta.org http://vyatta.com Free download Subscription livecd Update 2x year Update 4x year Community forums Software or Hardware Phone and Email Support Same source and features
  • 8.
    Vyatta Distribution Debian GNU/Linux Current version based on testing → lenny Linux kernel 2.6.24 + bugfix Filesystems: unionfs, squashfs Serial driver: wanpipe Additional packages CLI infrastructure Configuration templates Updated routing related packages Quagga, SNMP, SNORT, ...
  • 10.
  • 11.
    CLI Requirements Router look & feel Command completion Roles: Administrator, operator Configure mode Extensible Text based Language neutral Integrated with operating system
  • 12.
    Vyatta Proprietary Open System System Monolithic SW Graphical User Interface Network Apps FusionCLITM Internet Protocols WAN Optimization Open API Load Balancing Scalable Routing NW Functions Security Extensible Internet Protocols DHCP, NAT, Radius… Firewall, VPN Anti-X PBX Linux Kernel IDS Unique HW Massive open-source Standard HW ecosystem…
  • 13.
  • 14.
    Demo 1: Basicinterface $ show interfaces  Interface    IP Address         State       Link   Description eth0         192.168.111.132/24 up          up eth1         ­                  up          up  lo           127.0.0.1/8        up          up   lo           ::1/128            up          up $ show interfaces et<tab>hernet <tab>    detail  eth0    eth1     $ show interfaces ethernet eth0  eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast  state UNKNOWN qlen 1000     link/ether 00:0c:29:f6:20:9e brd ff:ff:ff:ff:ff:ff     inet 192.168.111.132/24 brd 192.168.111.255 scope global eth0     inet6 fe80::20c:29ff:fef6:209e/64 scope link         valid_lft forever preferred_lft forever ...
  • 15.
    Template hierarchy /opt/vyatta/share/vyatta­op/templates/show/interfaces/ethernet |­­ detail |   `­­ node.def |­­ node.def `­­ node.tag     |­­ brief     |   `­­ node.def     |­­ capture     |   |­­ node.def     |   |­­ not     |   |   |­­ node.def     |   |   `­­ port     |   |       |­­ node.def     |   |       `­­ node.tag     |   |           `­­ node.def     |   `­­ port     |       |­­ node.def     |       `­­ node.tag     |           `­­ node.def     |­­ identify     |   `­­ node.def     |­­ node.def     |­­ physical     |   `­­ node.def
  • 16.
    Operational template show/interfaces/ethernet/node.tag/node.def help: Show specified ethernet interface information allowed: for dev in /sys/class/net/*;           do if [[ ­L $dev/device ]]             then if [[ $(cat $dev/type) ­eq 1 ]]                  then echo ­n ${dev##*/} " "                  fi             fi          done run: vyatta­show­interfaces.pl ­­intf="$4"
  • 17.
    Demo 2: Configuration $ configure [edit] # set interfaces ethernet eth0 description 'Vmware NAT' [edit] # show interfaces ethernet eth0  address dhcp +description "Vmware NAT"  hw­id 00:0c:29:f6:20:9e [edit] # mount ... unionfs on /opt/vyatta/config/tmp/new_config_5035 type unionfs     (rw,dirs=/tmp/changes_only_5035=rw:/opt/vyatta/config/active=r o) [edit] vyatta@vc3­1­3# commit
  • 18.
    Configuration templates multi: type: txt help: Set an IP address for this interface syntax:expression: exec "/opt/vyatta/sbin/vyatta­interfaces.pl      ­­valid­addr $VAR(@) ­­dev $VAR(../@)"     ; "Invalid IP address/prefix [$VAR(@)] for interface $VAR(../@)" update: /opt/vyatta/sbin/vyatta­interfaces.pl     ­­eth­addr­update $VAR(@) ­­dev $VAR(../@) delete: /opt/vyatta/sbin/vyatta­interfaces.pl      ­­eth­addr­delete $VAR(@) ­­dev $VAR(../@) allowed: echo "dhcp <>" comp_help:Possible completions:   <x.x.x.x/x> Set the IP address and prefix length   <h:h:h:h:h:h:h:h/x>   Set the IPv6 address and prefix length   dhcp Set the IP address and prefix length via DHCP
  • 19.
    Configuration save restore interfaces {     ethernet eth0 {         address dhcp         duplex auto         hw­id 00:0c:29:f6:20:9e         speed auto     }     loopback lo {     } } service {     ssh {         port 22         protocol­version v2     } }
  • 21.
    Quality of Service(QoS) usage models Real time services VOIP Network control plane BGP, OSPF, STP Fairness Throttle batch services P2P, backup,
  • 22.
    Vyatta QoS Organized by policy types Fair queue => sfq Traffic shaper => htb Drop tail => fifo Rate limiter => tbf Traffic limiter => ingress ...
  • 23.
    fair-queue # set qos­policy fair­queue fq [edit] # set interfaces ethernet eth0 qos­policy out fq [edit] # commit [edit] # run show queueing    Output queues: Interface  Qos­Policy             Sent    Dropped   Overlimit eth0       fair­queue             4578          0          0 eth1       default                 468          0          0
  • 24.
    fair-queue template set/qos-policy/fair-queue/node.def tag: type: txt help: Set fair queueing policy syntax:expression: pattern $VAR(@) "^[[:alnum:]][­_[:alnum:]]*$"                    ; "only alpha­numeric policy name allowed" update: /opt/vyatta/sbin/vyatta­qos.pl ­­create­policy "$VAR(.)"  "$VAR(@)" delete: /opt/vyatta/sbin/vyatta­qos.pl ­­delete­policy "$VAR(@)"
  • 25.
    Qos on EthernetInterface set/interfaces/ethernet/node.tag/qos-policy/out/node.def type: txt help: Set outbound QOS policy for specified ethernet interface allowed: /opt/vyatta/sbin/vyatta­qos.pl ­­list­policy update: /opt/vyatta/sbin/vyatta­qos.pl  ­­update­interface $VAR(../../@) $VAR(.) $VAR(@) delete: /opt/vyatta/sbin/vyatta­qos.pl  ­­delete­interface $VAR(../../@) $VAR(.)
  • 26.
    Internals - perlcode sub update_interface {     my ($interface, $direction, $name ) = @_;     my $config = new VyattaConfig;     ( $direction eq "out" ) or die "Only out direction  supported";     $config­>setLevel('qos­policy');     foreach my $type ( $config­>listNodes() ) {         if ( $config­>exists("$type $name") ) {           my $shaper = make_policy($config, $type, $name);           delete_interface($interface, $direction);           open my $out, "|­" or exec qw:sudo /sbin/tc ­batch ­:;           $shaper­>commands($out, $interface);           if (! close $out) {               delete_interface($interface, $direction);           }     }     die "Unknown qos­policy $namen"; }
  • 27.
    QoS traffic-shaper # edit qos­policy traffic­shaper lartc [edit qos­policy traffic­shaper lartc] # set class 2 bandwidth 100% [edit qos­policy traffic­shaper lartc] # set class 2 match www ip destination port 80 [edit qos­policy traffic­shaper lartc] # set class 3 bandwidth 3mbit [edit qos­policy traffic­shaper lartc] # set class 3 ceiling 5mbit   [edit qos­policy traffic­shaper lartc] # set class 3 match smtp ip destination port 25 [edit qos­policy traffic­shaper lartc] # exit
  • 28.
    Traffic-shaper continue [edit] # commit qos­policy traffic­shaper lartc configuration not complete: missing  default class Commit failed # set qos­policy traffic­shaper lartc default bandwidth 1 [edit] # commit [edit] # set interfaces ethernet eth0 qos­policy out
  • 29.
    Result # run show queueing ethernet eth0          eth0 Output queue: Class      Qos­Policy             Sent    Dropped   Overlimit 1:        traffic­shaper        11438          0          0   4       fair­queue            11438          0          0   2       fair­queue                0          0          0   3       fair­queue                0          0          0 [edit]
  • 30.
    Issues Developerdocumentation Linux Kongress paper Watch this space Vyatta ↔ Linux management Vyatta config ignores other changes CLI stays loosely coupled Vyatta package changes Every distribution is a fork All changes are fed to upstream
  • 31.
    SPC-FLOSS: orphaned projects Users want support for orphaned projects Multicast routing MPLS RSTP IPV6
  • 32.
    Future Richer features QoS+, bonding, … GUI Multi-queue Performance

Editor's Notes

  • #2 Today, I am going to talk about Vyatta&apos;s Command Line Interface. The CLI tries to integrate existing Linux projects which as they say in the US is a bit like herding cats.. Maybe that is a US expression so let me explain...
  • #9 Vyatta Debian mirror repository Does include all the networking packages on livecd Does not include X / Gnome /KD Can use packages from Debian except overlaps.
  • #11 Start with Linux + ipv6 + serial + bonding + bridging + vlan + QoS + Quagga + Snort + clam + ...
  • #12 Command completion with TAB and ? Admin is like root Operator is the guy you only partially trust Vyatta has 4 modes: root, admin, operator, normal
  • #13 IOS is monolithic. Not extensible Explain layers Marketing != reality
  • #15 Use slogin to VM Show system kernel-messages Cd /opt/vyatta/share/vyatta-op/templates Cat show/kernel-messages/node.def (Explain template hierarchy and run tag) Show interfaces ethernet eth0 Cat show/interfaces/ethernet/node.tag/node.def (Explain tag nodes and allowed, etc)
  • #18 Cd # home Show how values are stored Show how snapshots are done; mount Emphasize that this is internals (like .git) Explain save/load If that was all CLI could do this would be...
  • #22 Use QoS as example, most familiar to me, less to audience
  • #23 Mapping from policy to internals
  • #28 Example from LARTC, Linux server with total of 10Mbit available bandwidth. You want to limit webserver traffic to 5Mbit, SMTP traffic to 3Mbit and everything else (unclassified traffic) to 1Kbit. In case there is unused bandwidth, you want to share it between SMTP and unclassified traffic. The &amp;quot;total bandwidth&amp;quot; implies one top-level class with maximum bandwidth of 5Mbit. Under the top-level class, there are three child classes. set qos-policy traffic-shaper lartc bandwidth 10mbit edit qos-policy traffic-shaper lartc set class 2 bandwidth 50% set class 2 match www ip destination port 80 set class 3 bandwidth 3mbit set class 3 ceiling 100% set class 3 match smtp ip destination port 25 Commit # Fix missing default bandwidth class for unclassified traffic is allowed to use 1K but must not exceed 5Mbit. set default bandwidth 1 set default ceiling 5mbit Exit # edit mode commit # Apply to eth0 set interfaces ethernet eth0 qos-policy out lartc commit
  • #31 These are projects customers want, but upstream project is abandoned IPV6 is not really abandoned but we want to make sure it has a good home after the end of the WIDE project
  • #33 Current release (3.1.3) aka Hollywood is finally out GUI is in next release along with lots more little pieces Excited about opportunity offered by MQ...