meets

         Tremashark
           Jan. 26, 2013
         Yasunobu Chiba
Part-time Developer, Trema Project


          Copyright © 2013 NEC Corporation
Network debugging today
                                         Log     Log      Log    Log
                       Collect logs
                                         Log     Log      Log    Log


      DUT/SUT                                                    Combine logs into a single
                 Log
                                                                 event series
                                          0.00         Sent packet A to …
  Log                  Log                0.12         Sent packet B to …       Product Manual
                                          0.15         Sent packet A to …    Protocol Specification
                                          0.29         Validate packet A …
                                          0.52         Sent packet B to …
Log                     Log               0.81         Receive packet A …


                Log                                              Find out focusing events
        Log            Log
                                          0.00         Sent packet A to …           Sent packet A to …
                                          0.15         Sent packet A to …           Sent packet A to …
                                          0.29         Validate packet A …          Validate packet A …
                                          0.81         Receive packet A …           Receive packet A …
                                                                Compare with expectation



                              Copyright © 2013 NEC Corporation
Problems
• Multiple mechanisms depending on network
  elements are required for collecting
  event/state logs
  – Syslog, SNMP, Console log, sFlow …
• Event/state collection mechanisms seriously
  interfere in operation status
  – Heavily consume CPU/memory resource etc.
• Real-time event/state log collection is hard

                  Copyright © 2013 NEC Corporation
Possible solution
Packet Capture


Syslog Outputs


 Console Log


    SNMP



    sFlow



 Collect                      Store                  Understand
                  Copyright © 2013 NEC Corporation
Possible solution – cont’d
Packet Capture


Syslog Outputs


 Console Log


    SNMP



    sFlow



 Collect                       Store                  Understand
                 Big Data !!!
                   Copyright © 2013 NEC Corporation
Solution we love really…
Packet Capture


Syslog Outputs


 Console Log


    SNMP



    sFlow



 Collect                     Store                  Understand
            Tremashark
                 Copyright © 2013 NEC Corporation
Why Wireshark?
• Events happening in networks usually relate
  to/contain packets
  – Wireshark can analyze packets
• Wireshark has powerful realtime event
  (packet) filtering function
  – Perfect platform for developing log viewer
• We run Wireshark everyday 


                   Copyright © 2013 NEC Corporation
Use case?

                                                       Packet capture
                                                       Inter-function
                                                       module events
                                                       Packet capture
                                                       Syslog message


                                      OpenFlow Controller

                                            learning_switch
                                   29. Packet-In       30. Packet-Out
                                   28. Packet-In       31. Packet-Out
                                               switch.abc


                                   27. Packet-In      32. Packet-Out

                                                      34. Error !!!



Copyright © 2013 NEC Corporation
FAQ
• Q: Tremashark captures IPC messages among
  Trema modules?
  – A: No. Such function is not implemented.
• Q: Tremashark is a wireshark plugin for
  Trema?
  – A: No. It includes wireshark plugin but it is not
    Tremashark.
• Q: Tremashark is a debugger?
  – A: No. It helps debugging but not a debugger.

                    Copyright © 2013 NEC Corporation
Implementation
 Trema-based OpenFlow      Notifications through Trema IPC (messenger)        Developer / Operator
       Controller
                                                                                               Real-time /
                                                                                               Off-line
                                                                                               Monitoring
      Switch      Syslog



                      Syslog                                              Event Viewer/Analyzer
       Host           Relay
                                           Event
                                          Collector                           Wireshark w/ plugin
Network Interface /
      Tap
                                          Circular Buffer
                       Packet
                      Capture                                  Serialized
                                                              Notifications
                                                                                   Pcap File
Any Text String
                       Stdin
                       Relay

                                   Copyright © 2013 NEC Corporation
Implementation – cont’d
• Event Collector
   – [trema]/src/tremashark/tremashark.c
• Syslog Relay
   – [trema]/src/tremashark/syslog_relay.c
• Packet Capture
   – [trema]/src/tremashark/packet_capture.c
• Stdin Relay
   – [trema]/src/tremashark/stdin_relay.c
• Wireshark Plugin
   – [trema]/src/tremashark/plugin/packet-trema
• Trema (libtrema)
   – [trema]/src/lib/messenger.[ch] – Any IPC event can be notified.


                        Copyright © 2013 NEC Corporation
How to build and install?
$ git clone git://github.com/trema/trema.git
$ cd trema
$ ./build.rb
$ cd src/tremashark/plugin
$ wget http://www.wireshark.org/download/src/wireshark-
1.6.12.tar.bz2
$ tar xvf wireshark-1.6.12.tar.bz2
$ ln -s wireshark-1.6.12 wireshark
$ cd wireshark
$ sudo apt-get install wireshark flex bison libgtkmm-2.4-dev
$ ./configure


                      Copyright © 2013 NEC Corporation
How to build and install? – cont’d
$ cd ../packet-trema
$ make
$ mkdir -p ~/.wireshark/plugins
$ cp packet-trema.so ~/.wireshark/plugins
$ cp ../user_dlts ~/.wireshark/




                     Copyright © 2013 NEC Corporation
Use case #1
• Scenario
  – Capture control channel connection between
    OpenFlow controller and switch
  – At the same time, capture all log output generated
    by the controller
   OpenFlow Controller

       learning_switch
                                    Syslog
                                    Relay                     Event
           Trema
                                    Packet                   Collector
                                   Capture



                          Copyright © 2013 NEC Corporation
Use case #1 – cont’d
• Setup syslog (rsyslog)
$ sudo sh -c 'echo “local6.* @127.0.0.1:10514” >
/etc/rsyslog.d/60-tremashark.conf‘
$ sudo service rsyslog restart




                      Copyright © 2013 NEC Corporation
Use case #1 – cont’d
• Run controller and Tremashark
 $ cd [trema]
 $ ./trema run -c ls.conf -d
 $ export TREMA_HOME=`pwd`                       -p: UDP port
                                                 -d: Daemonize
 $ ./objects/tremashark/syslog_relay -p 10514 -d
 $ sudo -E ./objects/tremashark/packet_capture -i lo -d port
 6633                                                -i: Interface
                                                     -d: Daemonize
 $ ./objects/tremashark/tremashark

      ls.conf   run {
                  path "./objects/examples/learning_switch/learning_switch"
                  options "-l debug", "-g", "-f local6"
                }
                         Copyright © 2013 NEC Corporation
Use case #1 – cont’d




     Copyright © 2013 NEC Corporation
Use case #2
• Scenario
  – In addition to use case #1, capture IPC events
    happening in a Trema-based OpenFlow controller

   OpenFlow Controller

       learning_switch
                                    Syslog
                                    Relay                     Event
           Trema
                                    Packet                   Collector
                                   Capture




                          Copyright © 2013 NEC Corporation
Use case #2 – cont’d
• Run controller and Tremashark
 $ cd [trema]
 $ ./trema run -c ls.conf -d
 $ export TREMA_HOME=`pwd`
 $ ./objects/tremashark/syslog_relay -p 10514 -d
 $ sudo -E ./objects/tremashark/packet_capture -i lo -d port
 6633
  Enable IPC event notification
 $ kill –USR2 `cat tmp/pid/learning_switch.pid`
 $ ./objects/tremashark/tremashark



                              Copyright © 2013 NEC Corporation
Use case #2 – cont’d




     Copyright © 2013 NEC Corporation
Use case #3
• Scenario
  – In the same scenario as use case #1, save events
    into the circular buffer
  – Dump the circular buffer as needed
   OpenFlow Controller
                                                                         Circular
       learning_switch
                                    Syslog
                                    Relay                                 Buffer
                                                              Event
           Trema
                                    Packet                   Collector
                                   Capture




                          Copyright © 2013 NEC Corporation
Use case #3 – cont’d
• Run controller and Tremashark
 $ cd [trema]
 $ ./trema run -c ls.conf -d
 $ export TREMA_HOME=`pwd`
 $ ./objects/tremashark/syslog_relay -p 10514 -d
 $ sudo -E ./objects/tremashark/packet_capture -i lo -d port
 6633
 $ ./objects/tremashark/tremashark -p -c 1000 -w log.cap -d
                                                                  -p: Do not launch wireshark
                                                                  -c: # of buffers
  Dump the circular buffer to the file specified.
                                                                  -w: File to output
 $ kill –USR2 `cat tmp/pid/tremashark.pid`                        -d: Daemonize



                               Copyright © 2013 NEC Corporation
Summary
• Tremashark is general-purpose event/state
  collector + viewer/analyzer for network
  debugging
• Event viewer/analyzer leverages Wireshark
• Useful tool for debugging Trema-based
  OpenFlow controllers
• We love


                 Copyright © 2013 NEC Corporation

Tremashark

  • 1.
    meets Tremashark Jan. 26, 2013 Yasunobu Chiba Part-time Developer, Trema Project Copyright © 2013 NEC Corporation
  • 2.
    Network debugging today Log Log Log Log Collect logs Log Log Log Log DUT/SUT Combine logs into a single Log event series 0.00 Sent packet A to … Log Log 0.12 Sent packet B to … Product Manual 0.15 Sent packet A to … Protocol Specification 0.29 Validate packet A … 0.52 Sent packet B to … Log Log 0.81 Receive packet A … Log Find out focusing events Log Log 0.00 Sent packet A to … Sent packet A to … 0.15 Sent packet A to … Sent packet A to … 0.29 Validate packet A … Validate packet A … 0.81 Receive packet A … Receive packet A … Compare with expectation Copyright © 2013 NEC Corporation
  • 3.
    Problems • Multiple mechanismsdepending on network elements are required for collecting event/state logs – Syslog, SNMP, Console log, sFlow … • Event/state collection mechanisms seriously interfere in operation status – Heavily consume CPU/memory resource etc. • Real-time event/state log collection is hard Copyright © 2013 NEC Corporation
  • 4.
    Possible solution Packet Capture SyslogOutputs Console Log SNMP sFlow Collect Store Understand Copyright © 2013 NEC Corporation
  • 5.
    Possible solution –cont’d Packet Capture Syslog Outputs Console Log SNMP sFlow Collect Store Understand Big Data !!! Copyright © 2013 NEC Corporation
  • 6.
    Solution we lovereally… Packet Capture Syslog Outputs Console Log SNMP sFlow Collect Store Understand Tremashark Copyright © 2013 NEC Corporation
  • 7.
    Why Wireshark? • Eventshappening in networks usually relate to/contain packets – Wireshark can analyze packets • Wireshark has powerful realtime event (packet) filtering function – Perfect platform for developing log viewer • We run Wireshark everyday  Copyright © 2013 NEC Corporation
  • 8.
    Use case? Packet capture Inter-function module events Packet capture Syslog message OpenFlow Controller learning_switch 29. Packet-In 30. Packet-Out 28. Packet-In 31. Packet-Out switch.abc 27. Packet-In 32. Packet-Out 34. Error !!! Copyright © 2013 NEC Corporation
  • 9.
    FAQ • Q: Tremasharkcaptures IPC messages among Trema modules? – A: No. Such function is not implemented. • Q: Tremashark is a wireshark plugin for Trema? – A: No. It includes wireshark plugin but it is not Tremashark. • Q: Tremashark is a debugger? – A: No. It helps debugging but not a debugger. Copyright © 2013 NEC Corporation
  • 10.
    Implementation Trema-based OpenFlow Notifications through Trema IPC (messenger) Developer / Operator Controller Real-time / Off-line Monitoring Switch Syslog Syslog Event Viewer/Analyzer Host Relay Event Collector Wireshark w/ plugin Network Interface / Tap Circular Buffer Packet Capture Serialized Notifications Pcap File Any Text String Stdin Relay Copyright © 2013 NEC Corporation
  • 11.
    Implementation – cont’d •Event Collector – [trema]/src/tremashark/tremashark.c • Syslog Relay – [trema]/src/tremashark/syslog_relay.c • Packet Capture – [trema]/src/tremashark/packet_capture.c • Stdin Relay – [trema]/src/tremashark/stdin_relay.c • Wireshark Plugin – [trema]/src/tremashark/plugin/packet-trema • Trema (libtrema) – [trema]/src/lib/messenger.[ch] – Any IPC event can be notified. Copyright © 2013 NEC Corporation
  • 12.
    How to buildand install? $ git clone git://github.com/trema/trema.git $ cd trema $ ./build.rb $ cd src/tremashark/plugin $ wget http://www.wireshark.org/download/src/wireshark- 1.6.12.tar.bz2 $ tar xvf wireshark-1.6.12.tar.bz2 $ ln -s wireshark-1.6.12 wireshark $ cd wireshark $ sudo apt-get install wireshark flex bison libgtkmm-2.4-dev $ ./configure Copyright © 2013 NEC Corporation
  • 13.
    How to buildand install? – cont’d $ cd ../packet-trema $ make $ mkdir -p ~/.wireshark/plugins $ cp packet-trema.so ~/.wireshark/plugins $ cp ../user_dlts ~/.wireshark/ Copyright © 2013 NEC Corporation
  • 14.
    Use case #1 •Scenario – Capture control channel connection between OpenFlow controller and switch – At the same time, capture all log output generated by the controller OpenFlow Controller learning_switch Syslog Relay Event Trema Packet Collector Capture Copyright © 2013 NEC Corporation
  • 15.
    Use case #1– cont’d • Setup syslog (rsyslog) $ sudo sh -c 'echo “local6.* @127.0.0.1:10514” > /etc/rsyslog.d/60-tremashark.conf‘ $ sudo service rsyslog restart Copyright © 2013 NEC Corporation
  • 16.
    Use case #1– cont’d • Run controller and Tremashark $ cd [trema] $ ./trema run -c ls.conf -d $ export TREMA_HOME=`pwd` -p: UDP port -d: Daemonize $ ./objects/tremashark/syslog_relay -p 10514 -d $ sudo -E ./objects/tremashark/packet_capture -i lo -d port 6633 -i: Interface -d: Daemonize $ ./objects/tremashark/tremashark ls.conf run { path "./objects/examples/learning_switch/learning_switch" options "-l debug", "-g", "-f local6" } Copyright © 2013 NEC Corporation
  • 17.
    Use case #1– cont’d Copyright © 2013 NEC Corporation
  • 18.
    Use case #2 •Scenario – In addition to use case #1, capture IPC events happening in a Trema-based OpenFlow controller OpenFlow Controller learning_switch Syslog Relay Event Trema Packet Collector Capture Copyright © 2013 NEC Corporation
  • 19.
    Use case #2– cont’d • Run controller and Tremashark $ cd [trema] $ ./trema run -c ls.conf -d $ export TREMA_HOME=`pwd` $ ./objects/tremashark/syslog_relay -p 10514 -d $ sudo -E ./objects/tremashark/packet_capture -i lo -d port 6633 Enable IPC event notification $ kill –USR2 `cat tmp/pid/learning_switch.pid` $ ./objects/tremashark/tremashark Copyright © 2013 NEC Corporation
  • 20.
    Use case #2– cont’d Copyright © 2013 NEC Corporation
  • 21.
    Use case #3 •Scenario – In the same scenario as use case #1, save events into the circular buffer – Dump the circular buffer as needed OpenFlow Controller Circular learning_switch Syslog Relay Buffer Event Trema Packet Collector Capture Copyright © 2013 NEC Corporation
  • 22.
    Use case #3– cont’d • Run controller and Tremashark $ cd [trema] $ ./trema run -c ls.conf -d $ export TREMA_HOME=`pwd` $ ./objects/tremashark/syslog_relay -p 10514 -d $ sudo -E ./objects/tremashark/packet_capture -i lo -d port 6633 $ ./objects/tremashark/tremashark -p -c 1000 -w log.cap -d -p: Do not launch wireshark -c: # of buffers Dump the circular buffer to the file specified. -w: File to output $ kill –USR2 `cat tmp/pid/tremashark.pid` -d: Daemonize Copyright © 2013 NEC Corporation
  • 23.
    Summary • Tremashark isgeneral-purpose event/state collector + viewer/analyzer for network debugging • Event viewer/analyzer leverages Wireshark • Useful tool for debugging Trema-based OpenFlow controllers • We love Copyright © 2013 NEC Corporation