This document provides tips and common mistakes to avoid when developing an OpenFlow controller using the Trema framework. It discusses key things to know about OpenFlow and Trema, such as Trema being a programming framework rather than a full controller itself. It then presents a use case of developing a production OpenFlow controller to manage virtual networks across thousands of switches and hosts. The design uses a load balancer and three-tier architecture. An evaluation showed the controller could manage over 400 switches and 16,000 virtual networks per controller instance.

1. Developing production
OpenFlow controller with Trema
Apr. 20, 2013
Yasunobu Chiba
Part-time developer, Trema Project
Copyright (C) 2013 NEC Corporation

2. Agenda
• Provide tips and typical mistakes when
developing OpenFlow controller with Trema C
library
• Explain a production OpenFlow controller
development use case
Copyright (C) 2013 NEC Corporation

3. THINGS TO KNOW ABOUT OPENFLOW
AND TREMA BEFORE DEVELOPING
OPENFLOW CONTROLLER
Copyright (C) 2013 NEC Corporation

4. Trema is not an OpenFlow controller
• Trema is a programming framework and just a
building block for implementing OpenFlow
controllers
• You may use any other building blocks as well
as Trema
– For high availability, scalability, network protocol
support, and etc.
Copyright (C) 2013 NEC Corporation

5. OpenFlow switch is not a database
• OpenFlow switch is not a database and does not
have ACID properties
• Error messages are returned from switch
asynchronously
• Send a Barrier Request to complete a previously
sent Flow Mod (or any other) operation
• See and reuse Transaction Manager:
– https://github.com/trema/apps/tree/master/transacti
on_manager
Copyright (C) 2013 NEC Corporation

6. Asynchronous events happen before
features request/reply exchange
• You may receive asynchronous OpenFlow
events such as Packet-In from unknown
switches that features reply has not been
received
• In Trema, Switch Ready event is triggered
when a features reply is received
Copyright (C) 2013 NEC Corporation

7. send_openflow_message() does not
send message actually
• send_openflow_message() just pushes a given
message to an appropriate send queue
• The send queue is automatically flushed inside
the main loop
• Message ordering is NOT guaranteed among
switches
Copyright (C) 2013 NEC Corporation

8. libtrema is not fully thread-safe
• The followings may be thread-safe but not
guaranteed for all functions
– buffer, doubly linked list, hash, linked list, log,
match table, OpenFlow message, packet info,
packet parser, stat, utility, wrapper
• OpenFlow application interface is NOT thread-
safe
• Call send_openflow_message() from the main
thread that trema_run() is running
Copyright (C) 2013 NEC Corporation

9. Length of send queue is limited
• The maximum length of send queue (for
sending OpenFlow messages) is limited and
fixed
• send_openflow_message() may return false if
the send queue is full
Copyright (C) 2013 NEC Corporation

10. Do not call flush_messenger()
• flush_messenger() flushes all send queues and
receive queues immediately and
synchronously
• Calling it in an OpenFlow event handler may
recursively call the event handler!
• Call only if you known how it works very well
Copyright (C) 2013 NEC Corporation

11. start_trema() may call fork(2)
• start_trema() calls fork(2) if your controller is
to be daemonized
• Use caution if you call functions that may be
affected by fork(2)
Copyright (C) 2013 NEC Corporation

12. CONTROLLER DEVELOPMENT USE
CASE
Copyright (C) 2013 NEC Corporation

13. Requirements
• Functional Requirements
– Manage association among virtual networks (based
on non-OpenFlow technology) and switch ports
– Associate a switch port with MAC addresses located
on the switch port
– All operations above can be done via REST interface
• Non-functional Requirements
– 1K+ switches must be handled
– 10K+ active virtual networks must be managed
– 10K+ end hosts must be connected to virtual networks
Copyright (C) 2013 NEC Corporation

14. Requirements – cont’d
Copyright (C) 2013 NEC Corporation
Switch #1 Switch #2 Switch #3 Switch #X
Virtual Network #N
Virtual Network #2
Virtual Network #1
10,000+ Active
Virtual Networks
1,000+ Switches
10,000+ End Hosts
Physical Network
Virtual Networks

15. Design
• Load balancer + three-tiered architecture + α
Copyright (C) 2013 NEC Corporation
Dispatcher
(Load Balancer)
Worker
Worker
Worker
Backend
Database
Configuration
Interface
External Application
OpenFlow Switch
REST
OpenFlow
+ REST
OpenFlow
+ REST

16. Implementation
Copyright (C) 2013 NEC Corporation
REST I/FController Cluster
OpenFlow Load Balancer (LVS)
Virtual Network Manager Virtual Network Manager Virtual Network Manager
Configuration Frontend Configuration Frontend Configuration Frontend
REST Interface Load Balancer (LVS)
Backend DB (MySQL)
Watchdog
Trema Trema Trema
OpenFlow
Virtual Network
Agent
REST
VM
Legacy
Network
OpenFlow Switch
REST OpenFlow
VXLAN Tunnel End-Point
Virtual Networks
Legacy L2/L3 Network
VM
Legacy
Network
Virtual Network
Agent
OpenFlow Switch
VXLAN Tunnel End-Point

17. Evaluation setup
Copyright (C) 2013 NEC Corporation
Vnet Manager
+ Config.
Frontend
#1
Vnet Manager
+ Config.
Frontend
#2
Vnet Manager
+ Config.
Frontend
#N
Backend DB
Load
Balancer
eth0 eth0 eth0 eth0
eth1 eth2 eth2 eth2 eth2
Database Network192.168.17.0/24
192.168.16.1 192.168.16.2 192.168.16.N
192.168.48.16
192.168.17.64 192.168.17.1 192.168.17.2 192.168.17.N 192.168.16.254
eth1
80/tcp
6633/tcp
REST Client
eth1 eth1 eth1
InternalControlNetwork192.168.16.0/24
192.168.64.17 192.168.64.18 192.168.64.19 192.168.64.16
eth0
ManagementNetwork192.168.64.0/20
192.168.64.(17+N)
ControlandData Network192.168.48.0/20
eth0 - 192.168.65.1
VXLAN TEP
Vnet
Agent
VM Host #1
Open vSwitch
hp001
sp001
eth1 - 192.168.49.1
hp128
sp128
hp002
sp002
eth0 - 192.168.65.N
VXLAN TEP
Vnet
Agent
VM Host #N
Open vSwitch
hp001
sp001
eth1 - 192.168.49.N
hp128
sp128
hp002
sp002
192.168.63.253

18. Evaluation items and results
• # of switches that can be managed
– 410 - 412 switches per a single Virtual Network
Manager were connected and initialized properly
• Switch daemons were not able to run due to
insufficient memory (system memory was 2 GB)
– 1024 switches were connected and initialized with
three Virtual Network Managers
Copyright (C) 2013 NEC Corporation

19. Evaluation items and results
• # of virtual networks that can be managed
– 16384 virtual networks that have 8 ports (hosts)
each were successfully created with 1024 switches
and three Virtual Network Managers
• Virtual network setup time
– Setup time did not increase even if we have a
number of virtual networks
– Database access time was constant and a minor
factor
Copyright (C) 2013 NEC Corporation

20. To be continued…
Copyright (C) 2013 NEC Corporation