TIPC is a networking protocol that provides container/namespace support, group messaging, 128-bit node identities, encryption, and wormhole communication between containers on the same host. It has seen many improvements in functionality, performance, scalability, and code quality in recent Linux kernels. The roadmap for 2021 includes exploring 128-bit addressing and testing TIPC in Kubernetes for inter- and intra-pod messaging.

1. TIPC
Status and Roadmap 2021
by Jon Maloy

2. STATUS OCTOBER 2020

3. FUNCTIONALITY
Container/Name Space support
→ From Linux 4.0
Autoadaptive Hierarchical Neighbor Node monitoring
→ Scales up to 1000 nodes with 1 second failure discovery time
→ From Linux 4.7
Group Messaging
→ User instantiated brokerless message bus
→ Allows for lossless, flow controlled, broadcast/multicast/anycast/unicast within a group
→ From Linux 4.14
128-bit self-assigned node identity
→ Can optionally be assigned by user: integer number, UUID, host name…
→ From Linux 4.16
Moved away from <Zone,Cluster,Node> network and addressing hierarchy
→ A node now only has an identity with no assumed internal structure
→ All node using the same cluster identity will detect each other and form a cluster
→ From Linux 4.16
Extensive Tracing Support
→ From Linux 4.20

4. FUNCTIONALITY
Introduced 'Smart Nagle' mechanism
→ Switches itself on/off depending on the traffic characteristics
→ Doubles stream socket throughput for small message traffic
→ From Linux 5.4
Built-in AEAD-based Encryption/Authentication Support
→ AES-GCM (Like in TLS 1.3)
→ Transparent to user, no application change needed
→ Works even for multicast, without requiring message replication
→ From Linux 5.4
‘Wormhole' inter-name space communication mechanism
→ Messages take a shortcut, bypassing network interfaces, between containers located on the same host.
→ Gives inter-container traffic the same performance as normal intra-container traffic
→ Much better throughput and latency than TCP
→ 100% secure
→ From Linux 5.4
Introduced variable window link flow control mechanism
→ Based on TCP’s Reno algorithm
→ Gives 25% improvement of max link throughput. (Jon Maloy/Xin Long, Linux 5.5
→ From Linux 5.5
Broadcast Binding Table Updates
→ When network infrastructure so permits
→ From Linux 5.8

5. API
Low-level socket API
→ Needs some effort to learn and use
Higher-level socket C API exists
→ As static libtipc library
→ Still only available by downloading tipcutils tar file
APIs for Python, Java, GO, Perl, Ruby, D
Support for TIPC in ZeroMQ
→ Not yet with full features

6. AVAILABILITY
Management tool now in package iproute2
→ “tipc” replaces previously used “tipc-config”
→ Available in all new distros after May 2015
Kernel module available in all major distros
→ Inclusive RHEL 8

7. IMPLEMENTATION
Significant effort to improve quality and maintainability over the last years
→ Reduced code bloat
→ Reduced structure interdependencies
→ Improved locking policies
→ Fewer locks, RCU locks instead of RW locks…
→ Eliminated all known risks of deadlock
→ Buffer handling
→ Much more use of sk_buff lists and other linux features
→ Improved and simplified fragmentation/reassembly
→ Redesigned and simplified broadcast link resulting in a significant performance improvement
→ Simplified and de-generalized topology server
Linuxification of code and coding style
→ Adapting to kernel naming conventions
→ Using RB tree instead of expanding arrays in service binding table

8. FLOW CONTROL
New connection level flow control
→ Now based on roundup(1k) size block instead of messages
→ Reduces need for socket receive buffer with factor 30
Link level flow control still uses a small fix window by default
→ Not optimal from performance viewpoint
→ Configurable
Datagram flow control
→ Resolved as part of the Group Communication feature

9. SCALABILITY
Introduced the “Overlapping Ring Supervision” algorithm in Linux 4.7
→ A combination of Gossip and Ring Supervision
→ Largest cluster we have tested had 800 fully-meshed nodes
→ But more is possible
→ Works flawlessly
New flat address space
→ Moved from <Zone.Cluster.Node> address scheme to a flat 128-bit identity space
# tipc node set addr 1.1.1 (Deprecated. Mandatory before Linux 4.18.)
# tipc node set identity myhostname (Optional. From Linux 4.18. Self assigned if not set.)

10. PERFORMANCE
Latency times better than on TCP
→ ~33 % faster inter-node
→ 2 to 7 times faster intra-node messaging (depends on message size)
→ We don’t use the loopback interface
→ From Linux 5.4 we do this even with inter container messages on same host
→ “Wormhole” technique
→ Just as secure as going via an interface
→ Kubernetes Pods will benefit greatly from this
Throughput lower than TCP on bare metal and between VMs
→ ~65-90% of max TCP throughput inter-node
→ Seems to be environment dependent, notably due to TCP TSO
Thoughput 25-400% better than TCP intra-node/inter-container on same host
→ 4 times better than TCP when using “wormhole” technique between containers on same host
→ New “smart nagle” feature from Linux 5.4 improves small message throughput with 50-100%
→ Turns itself on an off depending on traffic pattern
→ No ACK delay problem as in TCP

11. MANAGEMENT
Netlink based API
→ Replaces old ascii-based commands (also via netlink)
→ Uses more standard features such as socket buffers, attribute nesting, sanity checks etc.
→ Scales much better when clusters grow
New user space tool “tipc”
→ Syntax inspired by “ip” tool
→ Modular design inspired by git
→ Uses libmnl
→ Replaces deprecated “tipc-config” tool
→ Part of iproute2 package
→ Along with “ip”, “tc” and others

12. ROADMAP 2021

13. FUNCTIONALITY
128-bit service and socket addresses
→ All fields 128 bits long
→ Possible, but intrusive change
Try it out in Kubernetes
→ Inter- and intra-pod messaging
Built-in consensus protocol for topology events
→ RAFT or Blockchain based?
struct tipc_addr{
char type[16]; // zero: socket addess, otherwise random
char instance[16]; // e.g. uuid, port number if socket address
char node[16]; // zero: whole cluster is lookup scope
};

14. MORE INFORMATION
TIPC Overview
https://www.slideshare.net/JonMaloy/tipc-overview-93483595
TIPC Communication Groups
https://www.slideshare.net/JonMaloy/tipc-communication-groups
TIPC Overlapping Ring Neighbor Monitoring
https://www.youtube.com/watch?v=ni-iNJ-njPo
TIPC home page
http://tipc.io
TIPC project page
https://sourceforge.net/projects/tipc
TIPC protocol specification (needs update)
http://tipc.sourceforge.net/doc/draft-spec-tipc-10.html
TIPC programmer’s guide (needs update)
http://tipc.sourceforge.net/doc/tipc_2.0_prog_guide.html
TIPC demo and test programs package
https://sourceforge.net/projects/tipc/files/?source=navbar

15. THANKYOU