Uploaded byYasunobu Chiba
Developing production OpenFlow controller with Trema
This document provides tips and common mistakes to avoid when developing an OpenFlow controller using the Trema framework. It discusses key things to know about OpenFlow and Trema, such as Trema being a programming framework rather than a full controller itself. It then presents a use case of developing a production OpenFlow controller to manage virtual networks across thousands of switches and hosts. The design uses a load balancer and three-tier architecture. An evaluation showed the controller could manage over 400 switches and 16,000 virtual networks per controller instance.
More Related Content
Similar to Developing production OpenFlow controller with Trema
Developing production OpenFlow controller with Trema
- 1. Developing production OpenFlow controllerwith Trema Apr. 20, 2013 Yasunobu Chiba Part-time developer, Trema Project Copyright (C) 2013 NEC Corporation
- 2. Agenda • Provide tipsand typical mistakes when developing OpenFlow controller with Trema C library • Explain a production OpenFlow controller development use case Copyright (C) 2013 NEC Corporation
- 3. THINGS TO KNOWABOUT OPENFLOW AND TREMA BEFORE DEVELOPING OPENFLOW CONTROLLER Copyright (C) 2013 NEC Corporation
- 4. Trema is notan OpenFlow controller • Trema is a programming framework and just a building block for implementing OpenFlow controllers • You may use any other building blocks as well as Trema – For high availability, scalability, network protocol support, and etc. Copyright (C) 2013 NEC Corporation
- 5. OpenFlow switch isnot a database • OpenFlow switch is not a database and does not have ACID properties • Error messages are returned from switch asynchronously • Send a Barrier Request to complete a previously sent Flow Mod (or any other) operation • See and reuse Transaction Manager: – https://github.com/trema/apps/tree/master/transacti on_manager Copyright (C) 2013 NEC Corporation
- 6. Asynchronous events happenbefore features request/reply exchange • You may receive asynchronous OpenFlow events such as Packet-In from unknown switches that features reply has not been received • In Trema, Switch Ready event is triggered when a features reply is received Copyright (C) 2013 NEC Corporation
- 7. send_openflow_message() does not sendmessage actually • send_openflow_message() just pushes a given message to an appropriate send queue • The send queue is automatically flushed inside the main loop • Message ordering is NOT guaranteed among switches Copyright (C) 2013 NEC Corporation
- 8. libtrema is notfully thread-safe • The followings may be thread-safe but not guaranteed for all functions – buffer, doubly linked list, hash, linked list, log, match table, OpenFlow message, packet info, packet parser, stat, utility, wrapper • OpenFlow application interface is NOT thread- safe • Call send_openflow_message() from the main thread that trema_run() is running Copyright (C) 2013 NEC Corporation
- 9. Length of sendqueue is limited • The maximum length of send queue (for sending OpenFlow messages) is limited and fixed • send_openflow_message() may return false if the send queue is full Copyright (C) 2013 NEC Corporation
- 10. Do not callflush_messenger() • flush_messenger() flushes all send queues and receive queues immediately and synchronously • Calling it in an OpenFlow event handler may recursively call the event handler! • Call only if you known how it works very well Copyright (C) 2013 NEC Corporation
- 11. start_trema() may callfork(2) • start_trema() calls fork(2) if your controller is to be daemonized • Use caution if you call functions that may be affected by fork(2) Copyright (C) 2013 NEC Corporation
- 12. CONTROLLER DEVELOPMENT USE CASE Copyright(C) 2013 NEC Corporation
- 13. Requirements • Functional Requirements –Manage association among virtual networks (based on non-OpenFlow technology) and switch ports – Associate a switch port with MAC addresses located on the switch port – All operations above can be done via REST interface • Non-functional Requirements – 1K+ switches must be handled – 10K+ active virtual networks must be managed – 10K+ end hosts must be connected to virtual networks Copyright (C) 2013 NEC Corporation
- 14. Requirements – cont’d Copyright(C) 2013 NEC Corporation Switch #1 Switch #2 Switch #3 Switch #X Virtual Network #N Virtual Network #2 Virtual Network #1 10,000+ Active Virtual Networks 1,000+ Switches 10,000+ End Hosts Physical Network Virtual Networks
- 15. Design • Load balancer+ three-tiered architecture + α Copyright (C) 2013 NEC Corporation Dispatcher (Load Balancer) Worker Worker Worker Backend Database Configuration Interface External Application OpenFlow Switch REST OpenFlow + REST OpenFlow + REST
- 16. Implementation Copyright (C) 2013NEC Corporation REST I/FController Cluster OpenFlow Load Balancer (LVS) Virtual Network Manager Virtual Network Manager Virtual Network Manager Configuration Frontend Configuration Frontend Configuration Frontend REST Interface Load Balancer (LVS) Backend DB (MySQL) Watchdog Trema Trema Trema OpenFlow Virtual Network Agent REST VM Legacy Network OpenFlow Switch REST OpenFlow VXLAN Tunnel End-Point Virtual Networks Legacy L2/L3 Network VM Legacy Network Virtual Network Agent OpenFlow Switch VXLAN Tunnel End-Point
- 17. Evaluation setup Copyright (C)2013 NEC Corporation Vnet Manager + Config. Frontend #1 Vnet Manager + Config. Frontend #2 Vnet Manager + Config. Frontend #N Backend DB Load Balancer eth0 eth0 eth0 eth0 eth1 eth2 eth2 eth2 eth2 Database Network192.168.17.0/24 192.168.16.1 192.168.16.2 192.168.16.N 192.168.48.16 192.168.17.64 192.168.17.1 192.168.17.2 192.168.17.N 192.168.16.254 eth1 80/tcp 6633/tcp REST Client eth1 eth1 eth1 InternalControlNetwork192.168.16.0/24 192.168.64.17 192.168.64.18 192.168.64.19 192.168.64.16 eth0 ManagementNetwork192.168.64.0/20 192.168.64.(17+N) ControlandData Network192.168.48.0/20 eth0 - 192.168.65.1 VXLAN TEP Vnet Agent VM Host #1 Open vSwitch hp001 sp001 eth1 - 192.168.49.1 hp128 sp128 hp002 sp002 eth0 - 192.168.65.N VXLAN TEP Vnet Agent VM Host #N Open vSwitch hp001 sp001 eth1 - 192.168.49.N hp128 sp128 hp002 sp002 192.168.63.253
- 18. Evaluation items andresults • # of switches that can be managed – 410 - 412 switches per a single Virtual Network Manager were connected and initialized properly • Switch daemons were not able to run due to insufficient memory (system memory was 2 GB) – 1024 switches were connected and initialized with three Virtual Network Managers Copyright (C) 2013 NEC Corporation
- 19. Evaluation items andresults • # of virtual networks that can be managed – 16384 virtual networks that have 8 ports (hosts) each were successfully created with 1024 switches and three Virtual Network Managers • Virtual network setup time – Setup time did not increase even if we have a number of virtual networks – Database access time was constant and a minor factor Copyright (C) 2013 NEC Corporation
- 20. To be continued… Copyright(C) 2013 NEC Corporation