About RUNOS OpenFlow controller: historical versions, current architecture, and vision

1. RUNOS OpenFlow controller
Alexander Shalimov
http://arccn.ru/
ashalimov@arccn.ru
@alex_shali
@arccnnews

2. What is SDN?
Basic principles
• Physical separation between forwarding plane and control plane.
• Centralized management.
• Programmability.
• Open standards, vendor-neutral control interface
Benefits
• Simplify network
operation (OPEX)
• Reducing equipment
cost (CAPEX)
• New innovative
services
Deployments
...
“SDN means thinking differently about networking”
SDN = Software Defined Networking

3. SDN/OpenFlow basics
A
B
• First packet is sent to the controller (OF_PACKET_IN).
• Controller calculates the best route across the network (the lowest cost and
according to the routing policies).
• Appropriate OpenFlow rules are sent to switches with optional reverse route too
(OF_PACKET_OUT/FLOW_MOD).
A
B
A -> B
OpenFlow
controller
host/client
host/client
switch 1 switch 2
switch 3 switch 4

4. SDN/OpenFlow basics
A
B
• First packet is sent to the controller (OF_PACKET_IN).
• Controller calculates the best route across the network (the lowest cost and
according to the routing policies).
• Appropriate OpenFlow rules are sent to switches with optional reverse route too
(OF_PACKET_OUT/FLOW_MOD).
• Dynamic reconfiguration in case of network error.
A
B
OpenFlow
controller
host/client
host/client
switch 1 switch 2
switch 3 switch 4

5. Requirements to SDN controller
• Programmability
– Functionality: applications and services
– Programming interface
• Performance
– Bandwidth
– Latency
• Reliability and safety
– 24/7
• DC requires processing
>10M events per second
• Reactive controllers are
more “sensitive”

6. Results of study (2013)
• Maximum performance is 7 000 000 events per second.
• Minimum latency is from 50 to 75us.
• Shortcomings:
– Reliability of controllers caused questions
– Performance was insufficient
[1] A. Shalimov, D. Zuikov, D. Zimarina, V. Pashkov, R. Smeliansky, "Advanced Study of SDN/OpenFlow controllers",
Proceedings of the CEE-SECR '13: Central & Eastern European Software Engineering Conference in Russia, ACM SIGSOFT,
October 23-25, 2013

7. RUNOS
• It is a historical series of SDN/OpenFlow controllers
– In-kernel – super fast, hard to develop apps
– Fusion – userspace memory control interface to the kernel
controller
– Userspace (Easy) [main]– fully userspace controller with
high functionality, easy to develop your apps, relatively
high performance comparing with existing controllers
– Distributed – HA version of the userspace controller
• The project is in the open source arccn.github.io/runos
(currently only userspace version)
[main meaning] RUNOS = Run (fast) Operating System
RUNOS = RUssian Network Operating System

8. In-kernel controller
We implemented in-kernel OpenFlow controller [2]:
• It is super fast
– No context switching on network communication
– Less time consuming memory translation procedure
• It is very hard to develop your own programs
– low-level programming language
– limited number of libraries and tools
– high risk to corrupt the whole system
[2] P. Ivashchenko, A. Shalimov, R. Smeliansky "High performance in-kernel SDN/OpenFlow controller", Proceedings of
the 2014 Open Networking Summit Research Track, USENIX, March 3-5, 2014 Santa Clara, USA

9. Performance (in_kernel)
• Performance is 30M fps
• Latency is 45us

10. Fusion: Program interface to In-kernel version
10
User space
Kernel space
Applications
Services
OpenFlow
Applications
Services
OpenFlow
Applications
Services
OpenFlow
NOX, Pox, Floodlight,
OpenDaylight, MUL,
etc
ARCCN
+: wide range
of applications
-: low
performance
+: fast performance
-: no applications
What is in the middle?
Offload architecture:
• Apps are in userspace
• Frequently used services
is in kernel (e.g., topo)
• Communication interface
+: wide range of
applications
+: fast performance
[3] Shalimov A., Ivashchenko P. In‐kernel
offloading of an SDN/OpenFlow Controller
Proceedings of the Modern Networking
Technologies (MoNeTec), IEEE, Moscow, Russia,
2014

11. Experimental evaluation
• The performance is 15M per second (2.5x
faster).
• Latency is 50us.
11
I/O throughput (cbench + l2learning), fps Response time, us

12. Programming interface
Interface through shared memory:
• /dev/ctrl – open kernel
• ioctl() – subscribe to events
• mmap() – get access to shared
memory
• poll() – read info from the
kernel
– POLLIN, POOLRDNORM,
POLLRDBAND, POLLOUT
• write() – flush output buffers
l2 learning example code
Interface is still difficult:
• Low-level C
• Applications require explicit
synchronization among themselves
• Even despite that now it is possible to
use standard libraries

13. Easy RUNOS = RUNOS
• Project objective
– “Could an OpenFlow controller be both easy to
develop applications for and also high
performance?”
– To develop system which will be convenient for
development of new network applications
– Remember about the speed and about further
integration with other versions of the controller
[UNDER ACTIVE DEVELOPMENT]
[the main controller for now, this is what we call runos today]

14. RUNOS goals
• The existing OpenFlow controllers have problems
with running multiple apps independently:
– Connect third party apps in one chain, hardcode for
new cases, need third party apps modifications
– No built-in conflict resolution system: apps might send
overlapping rules (see example on the next slide)
– …
RUNOS is intended to solve these problems and
simplify developing complex OpenFlow
applications.

15. Possible problems in OpenFlow controllers
Forwarding SPAN
OpenFlow
ip_dst:10.0.0.1
Rule 1
Rule 2
Flow table
Example of the problem with running several apps independently:
• Forwarding and Span apps. First app sends a flow over port 1, while second ones
sends the same flow over port 5. Rules intersect with each other.
• Final rules order in the flow table is unknown.
• Packets will go using only the first rule. Thus, only one app will work. Conflict!
• We may to resolve such conflicts and some others. Just ip_src:10.0.0.1 -> output:1,5!
Rule 1: ip_src:10.0.0.1 -> output:1
Rule 2: ip_src:10.0.0.1 -> output:5
New
packet
never used

16. Launch options
• The number of threads of the
controller set
– for interaction with switches
– for applications
• List of applications
– its parameters (eg. poll-interval)
– to pin the thread of execution in
exclusive use (pin-to-thread,
own-thread)
Config (json):
“controller”: {
“threads”: 4
},
“loader”: {
“threads”: 3
},
“link discovery”: {
“poll-interval” : 10,
“pin-to-thread” : 2
},
“learning switch”: {
}
…

17. Architecture
Controller
TraceTree
Workers
Thread pool
Initialization of the controller:
1. Starting the desired number of
threads
2. Starting the service components
3. Starting applications and their
distribution by threads
4. Determination of the order of event
processing by applications
Apps
Logical pipelines

18. RUNOS Details
• Subscription on events
– Explicitly on packet_in
– Implicitly on switch up/down, port_description, stats.
• Two pipelines of execution
– run-to-completion
• sorting the applications are running on the basis of dependences
– deferred execution
• TraceTree - conflict resolution and auto rules
generation
– Arrangement of priorities of rules, combining of rules
– LOAD, MATCH, READ abstractions
– MAPLE based

19. Features
• Graphical user interface - enterprise network management
system [4]
• Wide range of applications:
– L2/L3 forwarding, QoS, multipath forwarding, network virtualization,
anti-DDOS, monitoring, load balancer, ACL, firewall, authentication,
SPAN, NAT, ARP, DNS, DHCP, BGP, verification and troubleshooting, WiFi,
OpenStack
[4] A. Shalimov, D. Morkovnik, S. Nizovtsev, R. Smeliansky EasyWay: Simplifying and automating enterprise network
management with SDN/OpenFlow// 10th Central and Eastern European Software Engineering Conference in Russia,
CEE-SECR 2014б, ACM SIGSOFT, Moscow, Russia.

20. Performance (easy)
• Performance is 10M fps (based on libfluid).
• Latency is 55us.
I/O throughput (cbench + l2learning), fps

21. Implementation details
Keywords: C++11, QT
Major third-party components:
• libfluid project (_base, _msg)
– for interaction with switches and parsing the OpenFlow 1.3 messages
• libtins
– parsing of packets in OpenFlow messages
• glog (google log)
– logging, multithreaded
• tcmalloc (google performance tools)
– alternative faster implementation of malloc/free
• json11
– parsing of the configuration file
• boost graph
– storage of topology, calculate the route

22. Releases roadmap
• Now version 0.2
– controller’s core
– building topology
– building a route across the network
– the first version of the rules generation subsystem
• Next version 0.3
– Rest API (Floodlight compatible)
– WebUI
• 0.4
– Applications ARP, DHCP
– Optimization of performance
– Statistics, metering
• 0.5
– Advanced routing applications , qos
– Load balancing
– Firewall/ACL
…
• 1.0
– Management system corporate network EasyWay
• 2.0
– Integration with In-kernel version (transparent for applications)

23. Distrib Runos: SDN for backbone networks
Distributed OpenFlow controller ARCCN
• Self-adjusting and self-organizing distributed controller for
DC and WAN.
– Self-adjusting as possible to ensure the specified quality of
network management through scalable performance within a
cluster and between clusters.
– Self-organizing possible redistribution network segment
between adjacent networks in case of unavailability of a
dedicated cluster controllers.
• Specification:
– OpenFlow 1.3
– 100M flows per second
– Latency is 500us
– Supports up to 10000 switches

24. Distrib: Distributed OpenFlow controller
Main features:
• Reliability and fault tolerance (redundancy within a cluster and between clusters)
• Load balancing (adding new nodes to the controller, depending on the load)
• Centralized management and vision of the entire network
• Working with distributed network applications
• Safety and protections against external loads

25. Organization scheme
• Synchronization/reservation
– via DB
– Special protocol
– Cold, hot, warm
• Additional functions
– Placement of nodes of the
controller in network
– Redistribution of loading
• migration of switches
• Characteristics
– Switching on the reserve
controller borrows on average
50ms
– Throughput decreases by 10%
(for NOX with 30K to 25K)
[5] Pashkov V, Shalimov A., Smeliansky R. Controller Failover for SDN Enterprise Networks // Proceedings of the Modern
Networking Technologies (MoNeTec), IEEE, Moscow, Russia, 2014,

26. Conclusion
• Series of OpenFlow controllers:
– From fast to programmable and distributed
• The project RUNOS is in the open source
– OpenFlow controller arccn.github.io/runos.
• RUNOS is already used in universities and in
industry (service providers, telecom operators,
DCs).
• We invite everyone to join the development!
http://arccn.ru/ ashalimov@arccn.ru
@alex_shali
@arccnnews

27. Video about SDN
• A bit of humour
– SDN from different points of view
– http://www.youtube.com/watch?v=GRVygzcXrM0