This document discusses network communication in Unix systems. It describes how the networking infrastructure abstracts different network architectures and consists of network protocols, address families, and additional facilities. It also summarizes the network subsystem layers, memory management using mbufs, data flow between sockets and the network, common network protocols, network interfaces, routing, and protocol control blocks.

1. Lab 12: Network Communication
Advanced Operating Systems
Zubair Nabi
zubair.nabi@itu.edu.pk
April 24, 2013

2. Introduction
• In *nix systems, the networking infrastructure abstracts away
many network architectures

3. Introduction
• In *nix systems, the networking infrastructure abstracts away
many network architectures
• Each network architecture consists of
• Network-communication protocols

4. Introduction
• In *nix systems, the networking infrastructure abstracts away
many network architectures
• Each network architecture consists of
• Network-communication protocols
• The protocol family

5. Introduction
• In *nix systems, the networking infrastructure abstracts away
many network architectures
• Each network architecture consists of
• Network-communication protocols
• The protocol family
• Conventions of naming end-points

6. Introduction
• In *nix systems, the networking infrastructure abstracts away
many network architectures
• Each network architecture consists of
•
•
•
•
Network-communication protocols
The protocol family
Conventions of naming end-points
The address family or address format

7. Introduction
• In *nix systems, the networking infrastructure abstracts away
many network architectures
• Each network architecture consists of
•
•
•
•
•
Network-communication protocols
The protocol family
Conventions of naming end-points
The address family or address format
Additional facilities

8. Introduction
• In *nix systems, the networking infrastructure abstracts away
many network architectures
• Each network architecture consists of
•
•
•
•
•
Network-communication protocols
The protocol family
Conventions of naming end-points
The address family or address format
Additional facilities
• Network facilities are accessed through the socket abstraction

9. Network Subsystem
Consists of three layers:
1
Transport layer: In charge of sockets-amenable addressing
structure and protocol mechanisms, such as ordering, reliability,
etc.

10. Network Subsystem
Consists of three layers:
1
Transport layer: In charge of sockets-amenable addressing
structure and protocol mechanisms, such as ordering, reliability,
etc.
2
Network layer: Responsible for delivery of data across the
network (must maintain a routing database)

11. Network Subsystem
Consists of three layers:
1
Transport layer: In charge of sockets-amenable addressing
structure and protocol mechanisms, such as ordering, reliability,
etc.
2
Network layer: Responsible for delivery of data across the
network (must maintain a routing database)
3
Link layer: Responsible for shipping messages between hosts
connected to a common transmission medium

12. Network Subsystem (2)
• The layering is just logical layering

13. Network Subsystem (2)
• The layering is just logical layering
• The network service itself might choose to use more or fewer
layers based on its requirements

14. Network Subsystem (2)
• The layering is just logical layering
• The network service itself might choose to use more or fewer
layers based on its requirements
• For instance, raw sockets use a null implementation at one or
more layers

15. Network Subsystem (2)
• The layering is just logical layering
• The network service itself might choose to use more or fewer
layers based on its requirements
• For instance, raw sockets use a null implementation at one or
more layers
• Similarly, tunneling of one protocol through another requires
additional implementations of multiple layers

16. Memory Management
• Memory management for communication protocols is different
than regular entities as memory is required in widely varying sizes

17. Memory Management
• Memory management for communication protocols is different
than regular entities as memory is required in widely varying sizes
• A special-purpose memory-management facility exists for IPC
and networking systems

18. Memory Management
• Memory management for communication protocols is different
than regular entities as memory is required in widely varying sizes
• A special-purpose memory-management facility exists for IPC
and networking systems
• The unit of allocation is an mbuf (skbuff in Linux), which is
128 bytes long with 100 or 108 bytes reserved for data

19. Memory Management
• Memory management for communication protocols is different
than regular entities as memory is required in widely varying sizes
• A special-purpose memory-management facility exists for IPC
and networking systems
• The unit of allocation is an mbuf (skbuff in Linux), which is
128 bytes long with 100 or 108 bytes reserved for data
• A chain of mbufs can be linked together (m_next) to hold an
arbitrary quantity of data

20. Memory Management
• Memory management for communication protocols is different
than regular entities as memory is required in widely varying sizes
• A special-purpose memory-management facility exists for IPC
and networking systems
• The unit of allocation is an mbuf (skbuff in Linux), which is
128 bytes long with 100 or 108 bytes reserved for data
• A chain of mbufs can be linked together (m_next) to hold an
arbitrary quantity of data
• For instance, a chain of mbufs is used to represent packets by
network protocols

21. mbuf

22. Data Flow
• Socket-to-network-subsystem
• Calls the transport-layer modules that support the socket
abstraction

23. Data Flow
• Socket-to-network-subsystem
• Calls the transport-layer modules that support the socket
abstraction
• Typically started by system calls

24. Data Flow
• Socket-to-network-subsystem
• Calls the transport-layer modules that support the socket
abstraction
• Typically started by system calls
• Network-subsystem-to-socket
• Flows up the stack and is placed in the receive queue of the
destination socket

25. Data Flow
• Socket-to-network-subsystem
• Calls the transport-layer modules that support the socket
abstraction
• Typically started by system calls
• Network-subsystem-to-socket
• Flows up the stack and is placed in the receive queue of the
destination socket
• Asynchronously received and added to the per-protocol input
message queue

26. Upwards Dataflow

27. Network Protocols
• Defined by a set of conventions, including packet formats, states,
and state transitions

28. Network Protocols
• Defined by a set of conventions, including packet formats, states,
and state transitions
• Each communication-protocol module implements a particular
protocol and is made up of a collection of procedures and private
data structures

29. Network Protocols
• Defined by a set of conventions, including packet formats, states,
and state transitions
• Each communication-protocol module implements a particular
protocol and is made up of a collection of procedures and private
data structures
• The external interface of a module is described by a
protocol-switch structure

30. Network Protocols
• Defined by a set of conventions, including packet formats, states,
and state transitions
• Each communication-protocol module implements a particular
protocol and is made up of a collection of procedures and private
data structures
• The external interface of a module is described by a
protocol-switch structure
• This interface is used by the socket layer for all interaction

31. Network Protocols
• Defined by a set of conventions, including packet formats, states,
and state transitions
• Each communication-protocol module implements a particular
protocol and is made up of a collection of procedures and private
data structures
• The external interface of a module is described by a
protocol-switch structure
• This interface is used by the socket layer for all interaction
• The address of this structure is present within the socket’s
so_proto field

32. Network Protocols
• Defined by a set of conventions, including packet formats, states,
and state transitions
• Each communication-protocol module implements a particular
protocol and is made up of a collection of procedures and private
data structures
• The external interface of a module is described by a
protocol-switch structure
• This interface is used by the socket layer for all interaction
• The address of this structure is present within the socket’s
so_proto field
• Each time a socket is created the protocol is selected based on
the type of socket (pr_type)

33. Network Protocols
• Defined by a set of conventions, including packet formats, states,
and state transitions
• Each communication-protocol module implements a particular
protocol and is made up of a collection of procedures and private
data structures
• The external interface of a module is described by a
protocol-switch structure
• This interface is used by the socket layer for all interaction
• The address of this structure is present within the socket’s
so_proto field
• Each time a socket is created the protocol is selected based on
the type of socket (pr_type)
• Also in charge of mbuf storage management

34. Network Interfaces
• Each interface defines a link-layer path through which messages
can be sent and received

35. Network Interfaces
• Each interface defines a link-layer path through which messages
can be sent and received
• Typically, a hardware device is represented by this interface

36. Network Interfaces
• Each interface defines a link-layer path through which messages
can be sent and received
• Typically, a hardware device is represented by this interface
• The loopback interface is in software which is used to route
traffic to local sockets

37. Network Interfaces
• Each interface defines a link-layer path through which messages
can be sent and received
• Typically, a hardware device is represented by this interface
• The loopback interface is in software which is used to route
traffic to local sockets
• Also in charge of encapsulation and decapsulation of link-layer
protocol headers

38. Network Interfaces
• Each interface defines a link-layer path through which messages
can be sent and received
• Typically, a hardware device is represented by this interface
• The loopback interface is in software which is used to route
traffic to local sockets
• Also in charge of encapsulation and decapsulation of link-layer
protocol headers
• Typically implemented as a separate layer that is shared by
various hardware drivers

39. Network Interfaces
• Each interface defines a link-layer path through which messages
can be sent and received
• Typically, a hardware device is represented by this interface
• The loopback interface is in software which is used to route
traffic to local sockets
• Also in charge of encapsulation and decapsulation of link-layer
protocol headers
• Typically implemented as a separate layer that is shared by
various hardware drivers
• The selection of the interface is taken care of by the network-layer
protocol

40. Network Interfaces
• Each interface defines a link-layer path through which messages
can be sent and received
• Typically, a hardware device is represented by this interface
• The loopback interface is in software which is used to route
traffic to local sockets
• Also in charge of encapsulation and decapsulation of link-layer
protocol headers
• Typically implemented as a separate layer that is shared by
various hardware drivers
• The selection of the interface is taken care of by the network-layer
protocol
• Represented by an ifnet structure

41. Socket-to-Protocol Interface
• Enabled by two routines: 1) User request, pr_usrreq() and
2) Control output, pr_ctloutput()

42. Socket-to-Protocol Interface
• Enabled by two routines: 1) User request, pr_usrreq() and
2) Control output, pr_ctloutput()
• These methods are present in the protocol-switch table for each
protocol

43. Socket-to-Protocol Interface
• Enabled by two routines: 1) User request, pr_usrreq() and
2) Control output, pr_ctloutput()
• These methods are present in the protocol-switch table for each
protocol
• Control-output: Implements getsockopt and setsockopt
system calls

44. Socket-to-Protocol Interface
• Enabled by two routines: 1) User request, pr_usrreq() and
2) Control output, pr_ctloutput()
• These methods are present in the protocol-switch table for each
protocol
• Control-output: Implements getsockopt and setsockopt
system calls
• User-request: Implements all other operations

45. Protocol-to-Network-Interface Interface
• Lowest layer in the protocol family must interact with one or more
interfaces to send and receive packets

46. Protocol-to-Network-Interface Interface
• Lowest layer in the protocol family must interact with one or more
interfaces to send and receive packets
• Obviously a routing decision must have already chosen the
outgoing interface

47. Code: Packet Sending
error = (*ifp->if_output)(ifp, m, dst, rt);
struct
struct
struct
struct
ifnet *ifp;
mbuf *m;
sockaddr *dst;
rtentry *rt;

48. Packet Sending
• Packet m is transmitted to destination dst via interface ifp

49. Packet Sending
• Packet m is transmitted to destination dst via interface ifp
• Steps:
• Validation of the destination address

50. Packet Sending
• Packet m is transmitted to destination dst via interface ifp
• Steps:
• Validation of the destination address
• Queuing of the packet on the send queue

51. Packet Sending
• Packet m is transmitted to destination dst via interface ifp
• Steps:
• Validation of the destination address
• Queuing of the packet on the send queue
• If the interface is not busy, using an interrupt-driven routine to
transmit the packet

52. Packet Sending
• Packet m is transmitted to destination dst via interface ifp
• Steps:
• Validation of the destination address
• Queuing of the packet on the send queue
• If the interface is not busy, using an interrupt-driven routine to
transmit the packet
• The link-layer address is chosen by ARP in case of Ethernet

53. Packet Receiving
• Incoming packets are queued in the corresponding protocol’s
input packet queue

54. Packet Receiving
• Incoming packets are queued in the corresponding protocol’s
input packet queue
• A software interrupt is posted to initiate network-layer processing

55. Code: Packet Receiving
if (IF_QFULL(&ipintrq)) {
IF_DROP(&ipintrq);
ifp->if_iqdrops++;
m_freem(m);
} else {
schednetisr(NETISR_IP);
IF_ENQUEUE(&ipintrq, m)
}

56. Routing
• The routing system has two components; one within the kernel
and one in user-space

57. Routing
• The routing system has two components; one within the kernel
and one in user-space
• The routing mechanism is present within the kernel while routing
policies are defined in user-space

58. Routing
• The routing system has two components; one within the kernel
and one in user-space
• The routing mechanism is present within the kernel while routing
policies are defined in user-space
• The routing mechanism involves a table lookup to get a first-hop
for a given destination

59. Routing
• The routing system has two components; one within the kernel
and one in user-space
• The routing mechanism is present within the kernel while routing
policies are defined in user-space
• The routing mechanism involves a table lookup to get a first-hop
for a given destination
• Routing policies include components that help in choosing
first-hop routes

60. Kernel Routing Mechanism
• Implements a routing table for first/next hop lookup

61. Kernel Routing Mechanism
• Implements a routing table for first/next hop lookup
• Two distinct portions:
1
A data structure with routing entries, one per specific route

62. Kernel Routing Mechanism
• Implements a routing table for first/next hop lookup
• Two distinct portions:
1
2
A data structure with routing entries, one per specific route
A lookup algorithm to locate the correct route for each possible
destination

63. Kernel Routing Mechanism
• Implements a routing table for first/next hop lookup
• Two distinct portions:
1
2
A data structure with routing entries, one per specific route
A lookup algorithm to locate the correct route for each possible
destination
• Each destination is represented by a sockaddr structure

64. Kernel Routing Mechanism
• Implements a routing table for first/next hop lookup
• Two distinct portions:
1
2
A data structure with routing entries, one per specific route
A lookup algorithm to locate the correct route for each possible
destination
• Each destination is represented by a sockaddr structure
• Routes are either:
1
Host or network

65. Kernel Routing Mechanism
• Implements a routing table for first/next hop lookup
• Two distinct portions:
1
2
A data structure with routing entries, one per specific route
A lookup algorithm to locate the correct route for each possible
destination
• Each destination is represented by a sockaddr structure
• Routes are either:
1
2
Host or network
Direct or indirect

66. User-space Routing Policies
• Policies add, delete, or modify kernel routing table entries

67. User-space Routing Policies
• Policies add, delete, or modify kernel routing table entries
• A number of routing policies exist, including the Routing
Information Protocol (RIP)

68. Protocol Control Blocks
• For each TCP or UDP socket, an Internet protocol control block
(inpcb) is created to hold address, ports, routing information,
and pointers to any additional data structures

69. Protocol Control Blocks
• For each TCP or UDP socket, an Internet protocol control block
(inpcb) is created to hold address, ports, routing information,
and pointers to any additional data structures
• TCP in addition creates a TCP control block (tcpcb) to hold
implementation-specific information

70. Protocol Control Blocks
• For each TCP or UDP socket, an Internet protocol control block
(inpcb) is created to hold address, ports, routing information,
and pointers to any additional data structures
• TCP in addition creates a TCP control block (tcpcb) to hold
implementation-specific information
• TCP and UDP protocol modules each have a private doubly
linked list of inpcbs

71. Protocol Control Blocks
• For each TCP or UDP socket, an Internet protocol control block
(inpcb) is created to hold address, ports, routing information,
and pointers to any additional data structures
• TCP in addition creates a TCP control block (tcpcb) to hold
implementation-specific information
• TCP and UDP protocol modules each have a private doubly
linked list of inpcbs
• Common routines are used by the modules to manipulate these
lists

72. Protocol Control Blocks
• For each TCP or UDP socket, an Internet protocol control block
(inpcb) is created to hold address, ports, routing information,
and pointers to any additional data structures
• TCP in addition creates a TCP control block (tcpcb) to hold
implementation-specific information
• TCP and UDP protocol modules each have a private doubly
linked list of inpcbs
• Common routines are used by the modules to manipulate these
lists
• Traffic is multiplexed by the IP layer on the basis of the protocol
identifier in the protocol and passed on to the individual transport
protocol

73. Protocol Control Blocks
• For each TCP or UDP socket, an Internet protocol control block
(inpcb) is created to hold address, ports, routing information,
and pointers to any additional data structures
• TCP in addition creates a TCP control block (tcpcb) to hold
implementation-specific information
• TCP and UDP protocol modules each have a private doubly
linked list of inpcbs
• Common routines are used by the modules to manipulate these
lists
• Traffic is multiplexed by the IP layer on the basis of the protocol
identifier in the protocol and passed on to the individual transport
protocol
• Each protocol is then responsible for passing a direct message to
the appropriate socket

74. Today’s task
• Design a network subsystem for xv6

75. Reading(s)
• Chapter 12 and 13 from “The Design and Implementation of the
4.4BSD Operating System” by Marshall Kirk McKusick, Keith
Bostic, Michael J. Karels, and John S. Quarterman.