Developing productionOpenFlow controller with TremaApr. 20, 2013Yasunobu ChibaPart-time developer, Trema ProjectCopyright ...
Agenda• Provide tips and typical mistakes whendeveloping OpenFlow controller with Trema Clibrary• Explain a production Ope...
THINGS TO KNOW ABOUT OPENFLOWAND TREMA BEFORE DEVELOPINGOPENFLOW CONTROLLERCopyright (C) 2013 NEC Corporation
Trema is not an OpenFlow controller• Trema is a programming framework and just abuilding block for implementing OpenFlowco...
OpenFlow switch is not a database• OpenFlow switch is not a database and does nothave ACID properties• Error messages are ...
Asynchronous events happen beforefeatures request/reply exchange• You may receive asynchronous OpenFlowevents such as Pack...
send_openflow_message() does notsend message actually• send_openflow_message() just pushes a givenmessage to an appropriat...
libtrema is not fully thread-safe• The followings may be thread-safe but notguaranteed for all functions– buffer, doubly l...
Length of send queue is limited• The maximum length of send queue (forsending OpenFlow messages) is limited andfixed• send...
Do not call flush_messenger()• flush_messenger() flushes all send queues andreceive queues immediately andsynchronously• C...
start_trema() may call fork(2)• start_trema() calls fork(2) if your controller isto be daemonized• Use caution if you call...
CONTROLLER DEVELOPMENT USECASECopyright (C) 2013 NEC Corporation
Requirements• Functional Requirements– Manage association among virtual networks (basedon non-OpenFlow technology) and swi...
Requirements – cont’dCopyright (C) 2013 NEC CorporationSwitch #1 Switch #2 Switch #3 Switch #XVirtual Network #NVirtual Ne...
Design• Load balancer + three-tiered architecture + αCopyright (C) 2013 NEC CorporationDispatcher(Load Balancer)WorkerWork...
ImplementationCopyright (C) 2013 NEC CorporationREST I/FController ClusterOpenFlow Load Balancer (LVS)Virtual Network Mana...
Evaluation setupCopyright (C) 2013 NEC CorporationVnet Manager+ Config.Frontend#1Vnet Manager+ Config.Frontend#2Vnet Manag...
Evaluation items and results• # of switches that can be managed– 410 - 412 switches per a single Virtual NetworkManager we...
Evaluation items and results• # of virtual networks that can be managed– 16384 virtual networks that have 8 ports (hosts)e...
To be continued…Copyright (C) 2013 NEC Corporation
Upcoming SlideShare
Loading in...5
×

Developing production OpenFlow controller with Trema

3,298

Published on

Published in: Technology

Transcript of "Developing production OpenFlow controller with Trema"

  1. 1. Developing productionOpenFlow controller with TremaApr. 20, 2013Yasunobu ChibaPart-time developer, Trema ProjectCopyright (C) 2013 NEC Corporation
  2. 2. Agenda• Provide tips and typical mistakes whendeveloping OpenFlow controller with Trema Clibrary• Explain a production OpenFlow controllerdevelopment use caseCopyright (C) 2013 NEC Corporation
  3. 3. THINGS TO KNOW ABOUT OPENFLOWAND TREMA BEFORE DEVELOPINGOPENFLOW CONTROLLERCopyright (C) 2013 NEC Corporation
  4. 4. Trema is not an OpenFlow controller• Trema is a programming framework and just abuilding block for implementing OpenFlowcontrollers• You may use any other building blocks as wellas Trema– For high availability, scalability, network protocolsupport, and etc.Copyright (C) 2013 NEC Corporation
  5. 5. OpenFlow switch is not a database• OpenFlow switch is not a database and does nothave ACID properties• Error messages are returned from switchasynchronously• Send a Barrier Request to complete a previouslysent Flow Mod (or any other) operation• See and reuse Transaction Manager:– https://github.com/trema/apps/tree/master/transaction_managerCopyright (C) 2013 NEC Corporation
  6. 6. Asynchronous events happen beforefeatures request/reply exchange• You may receive asynchronous OpenFlowevents such as Packet-In from unknownswitches that features reply has not beenreceived• In Trema, Switch Ready event is triggeredwhen a features reply is receivedCopyright (C) 2013 NEC Corporation
  7. 7. send_openflow_message() does notsend message actually• send_openflow_message() just pushes a givenmessage to an appropriate send queue• The send queue is automatically flushed insidethe main loop• Message ordering is NOT guaranteed amongswitchesCopyright (C) 2013 NEC Corporation
  8. 8. libtrema is not fully thread-safe• The followings may be thread-safe but notguaranteed for all functions– buffer, doubly linked list, hash, linked list, log,match table, OpenFlow message, packet info,packet parser, stat, utility, wrapper• OpenFlow application interface is NOT thread-safe• Call send_openflow_message() from the mainthread that trema_run() is runningCopyright (C) 2013 NEC Corporation
  9. 9. Length of send queue is limited• The maximum length of send queue (forsending OpenFlow messages) is limited andfixed• send_openflow_message() may return false ifthe send queue is fullCopyright (C) 2013 NEC Corporation
  10. 10. Do not call flush_messenger()• flush_messenger() flushes all send queues andreceive queues immediately andsynchronously• Calling it in an OpenFlow event handler mayrecursively call the event handler!• Call only if you known how it works very wellCopyright (C) 2013 NEC Corporation
  11. 11. start_trema() may call fork(2)• start_trema() calls fork(2) if your controller isto be daemonized• Use caution if you call functions that may beaffected by fork(2)Copyright (C) 2013 NEC Corporation
  12. 12. CONTROLLER DEVELOPMENT USECASECopyright (C) 2013 NEC Corporation
  13. 13. Requirements• Functional Requirements– Manage association among virtual networks (basedon non-OpenFlow technology) and switch ports– Associate a switch port with MAC addresses locatedon the switch port– All operations above can be done via REST interface• Non-functional Requirements– 1K+ switches must be handled– 10K+ active virtual networks must be managed– 10K+ end hosts must be connected to virtual networksCopyright (C) 2013 NEC Corporation
  14. 14. Requirements – cont’dCopyright (C) 2013 NEC CorporationSwitch #1 Switch #2 Switch #3 Switch #XVirtual Network #NVirtual Network #2Virtual Network #110,000+ ActiveVirtual Networks1,000+ Switches10,000+ End HostsPhysical NetworkVirtual Networks
  15. 15. Design• Load balancer + three-tiered architecture + αCopyright (C) 2013 NEC CorporationDispatcher(Load Balancer)WorkerWorkerWorkerBackendDatabaseConfigurationInterfaceExternal ApplicationOpenFlow SwitchRESTOpenFlow+ RESTOpenFlow+ REST
  16. 16. ImplementationCopyright (C) 2013 NEC CorporationREST I/FController ClusterOpenFlow Load Balancer (LVS)Virtual Network Manager Virtual Network Manager Virtual Network ManagerConfiguration Frontend Configuration Frontend Configuration FrontendREST Interface Load Balancer (LVS)Backend DB (MySQL)WatchdogTrema Trema TremaOpenFlowVirtual NetworkAgentRESTVMLegacyNetworkOpenFlow SwitchREST OpenFlowVXLAN Tunnel End-PointVirtual NetworksLegacy L2/L3 NetworkVMLegacyNetworkVirtual NetworkAgentOpenFlow SwitchVXLAN Tunnel End-Point
  17. 17. Evaluation setupCopyright (C) 2013 NEC CorporationVnet Manager+ Config.Frontend#1Vnet Manager+ Config.Frontend#2Vnet Manager+ Config.Frontend#NBackend DBLoadBalancereth0 eth0 eth0 eth0eth1 eth2 eth2 eth2 eth2Database Network192.168.17.0/24192.168.16.1 192.168.16.2 192.168.16.N192.168.48.16192.168.17.64 192.168.17.1 192.168.17.2 192.168.17.N 192.168.16.254eth180/tcp6633/tcpREST Clienteth1 eth1 eth1InternalControlNetwork192.168.16.0/24192.168.64.17 192.168.64.18 192.168.64.19 192.168.64.16eth0ManagementNetwork192.168.64.0/20192.168.64.(17+N)ControlandData Network192.168.48.0/20eth0 - 192.168.65.1VXLAN TEPVnetAgentVM Host #1Open vSwitchhp001sp001eth1 - 192.168.49.1hp128sp128hp002sp002eth0 - 192.168.65.NVXLAN TEPVnetAgentVM Host #NOpen vSwitchhp001sp001eth1 - 192.168.49.Nhp128sp128hp002sp002192.168.63.253
  18. 18. Evaluation items and results• # of switches that can be managed– 410 - 412 switches per a single Virtual NetworkManager were connected and initialized properly• Switch daemons were not able to run due toinsufficient memory (system memory was 2 GB)– 1024 switches were connected and initialized withthree Virtual Network ManagersCopyright (C) 2013 NEC Corporation
  19. 19. Evaluation items and results• # of virtual networks that can be managed– 16384 virtual networks that have 8 ports (hosts)each were successfully created with 1024 switchesand three Virtual Network Managers• Virtual network setup time– Setup time did not increase even if we have anumber of virtual networks– Database access time was constant and a minorfactorCopyright (C) 2013 NEC Corporation
  20. 20. To be continued…Copyright (C) 2013 NEC Corporation

×