Solo Prize Winner - 6WIND Speed Matters: The Challenge Contest
Ostinato is a network packet and traffic generator and analyzer with a friendly GUI. It aims to be "Wireshark in Reverse" and thus become complementary to Wireshark. It is useful for both functional and performance testing. (GPL, Linux/BSD/OSX/Win32)
Accompanying code: https://github.com/pstavirs/dpdk-ostinato
5. http://ostinato.org/
Multi-core Processor
DPDK Programming Model
Core
Packet
Processing
Software
Engine
Core
Packet
Processing
Software
Engine
Core
Launch
and
Control
of Engines
Core
Packet
Processing
Software
Engine
DPDK Libraries + Environment Abstraction Layer (EAL)
SlavesMaster
Run to completion
model for engines
Alternatively,
they can collaborate
in a pipeline model
9. http://ostinato.org/
Port Class Interface (1/2)
class AbstractPort {
public:
// Common functionality for all ports implemented by AbstractPort
int id();
const char* name();
bool modify(const OstProto::Port &port);
int streamCount();
StreamBase* streamAtIndex(int index);
StreamBase* stream(int streamId);
bool addStream(StreamBase *stream);
bool deleteStream(int streamId);
void updatePacketList();
(more)
12. http://ostinato.org/
Key Point for transmitted packets
AbstractPort
computes the minimum set of unique packets
required to be transmitted
pre-builds this minimum unique packet set
during Tx, this packet set is looped over and over
No packet buffers are allocated or built
during transmit
13. http://ostinato.org/
DpdkPort Engines (1/2)
Port Rx Ring Polling
while (!stopRxPolling) {
rte_eth_rx_burst(rxPkts)
foreach pkt in rxPkts rte_pktmbuf_free(pkt)
}
Note: Ostinato, being a traffic generator, does not need to process Rx
packets
Port Tx
while (!stopTransmit) {
rte_eth_tx_burst(txPkts) // txPkts is pre-built
rte_delay_us(timeTillNextPktOrBurst)
}
Note: ensure rte_eth_tx_burst() does not free mbufs by bumping their
refcnt via rte_pktmbuf_clone() during packet list pre-building time so we
can keep transmitting the same mbuf again and again without
realloc/rebuild
14. http://ostinato.org/
DpdkPort Engines (2/2)
Port Rx Capture
while (!stopCapture) {
rte_eth_rx_burst(rxPkts)
foreach pkt in rxPkts
rte_memcpy(capBuf++, pkt)
rte_pktmbuf_free(pkt)
}
Port Statistics Rx/Tx
while (!stopStatsPolling) {
rte_eth_stats_get()
sleep(1)
}
15. http://ostinato.org/
Engine - Core assignment strategies
DpdkPort engines to be assigned to cores
Always On (core assignment at init)
Drone core functionality
RPC, Protocol Builders, PacketList Builder etc.
Port Rx Ring Polling
Port Statistics Rx/Tx
On-Demand (core reserved at init or assigned on-demand)
Port Tx (of pre-built packets)
Port Rx Capture
16. http://ostinato.org/
Core assignment example strategy #1
Core Assignment
Master: Drone core functionality + DPDK port stats (one thread polls all ports)
Slave 0: Poll Rx Rings for all ports, Capture Rx for requested ports
Slave 1: Port 1 Tx
Slave 2: Port 2 Tx
…
Slave n: Port n Tx
Pros
Dedicated core to a port Tx
Optimal for port with only 1 Tx queue
Cons
2 cores are potentially under-utilized (Master, Slave 0)
Port Rx + Capture Rx may exceed core capacity leading to packet drops
Slave 1 – n cores may be under-utilized depending on Tx Rate (or if Tx off)
#ports = (#cores – 2)
Quad-core processor can support only 2 ports
17. http://ostinato.org/
Core assignment example strategy #2
Core Assignment
Master: Drone core functionality + DPDK port stats (one thread polls all ports)
Slave 0: Poll Rx Rings for all ports
Slave 1: Port 1 Tx + Port 1 Capture Rx
Slave 2: Port 2 Tx + Port 2 Capture Rx
…
Slave n: Port n Tx + Port n Capture Rx
Pros
More Rx Capture Bandwidth for simultaneous capture on multiple ports
Optimal when packets transmitted from one port are captured on another
Cons
2 cores are potentially under-utilized (Master, Slave 0)
Port Tx rate may be affected by the multiplexed capture functionality
#ports = (#cores – 2)
Quad-core processor can support only 2 ports
18. http://ostinato.org/
Core assignment example strategy #3
Core Assignment
Master: Drone core functionality + DPDK port stats (one thread polls all ports)
Slave 0: Poll Rx Rings for all ports, Capture Rx for requested ports
Slave 1: Port Tx Q1 for all ports
Slave 2: Port Tx Q2 for all ports
…
Slave n: Port Tx Qn for all ports
Pros
Multiple cores (equal to no. of supported Tx queues) dedicated to a port Tx
Optimal for ports with multiple Tx queues
All ports supported
Cons
2 cores are potentially under-utilized (Master, Slave 0)
Port Rx + Capture Rx may exceed core capacity leading to packet drops
Slave 1 – n may be under/over-utilized depending on supported Tx queues
and Tx rates
19. http://ostinato.org/
The “core” problem
Engine - Core assignment strategy will determine
Max number of supported ports
Max packet transmit/capture rate on a port
No one strategy fits all use cases
Optimal solution requires knowledge of
system configuration and capacity
use case
Implement multiple strategies and allow
end-user to override the default strategy
20. http://ostinato.org/
DpdkPort Performance
Unfortunately, we don’t have access to DPDK
supported hardware NICs and a server-grade multi-
core processor
Current development is being done on emulated
82545EM NICs on a VirtualBox VM running on a
quad-core laptop
On this development platform, for 64-byte packets,
DpdkPort generates a max of ~190Kpps against
PcapPort’s ~15Kpps
21. http://ostinato.org/
DPDK Potential Improvements
DPDK Build Environment
Not straight-forward to integrate into existing applications
having their own build environment
A traffic generator doesn’t need active Rx Ring polling
If capture is not enabled on port, can PMD/NIC
automatically empty rx ring and free mbuf?
Note: Rx stats still need to be maintained and collected
Nano-second resolution Delay API
rte_delay_ns()
22. http://ostinato.org/
Source Code
Repo URL: https://code.google.com/r/pstavirs-dpdk/
Caveats
Code is work in progress (aka alpha quality)
Not everything discussed here is implemented as of today
(Aug 1, 2014)
23. http://ostinato.org/
Build Instructions
DPDK
Download DPDK - http://dpdk.org/download
Follow build and verification steps from
http://dpdk.org/doc/quick-start
Drone
hg clone https://code.google.com/r/pstavirs-dpdk/ ostinato
Export environment variables RTE_SDK and RTE_TARGET
Follow build steps from
http://ostinato.org/wiki/BuildingFromSource
25. http://ostinato.org/
Support
For programmers who know their “stuff”
Contact the author Srivats P. <pstavirs@gmail.com>
Everyone else
Please wait till the code is ready for “General Availability”
(aka beta quality)