My foray into Go began a few years ago when I started working at DigitalOcean. While building an abstraction layer on top of Kubernetes and familiarizing myself with the language, I began to love it. Syntactically simple, with amazing concurrency primitives and a wonderful community, Go was an excellent choice for a cloud-hosting company with a variety of low-level, server-side microservices. In the last year, however, I've joined the software-defined networking team and learned of another application of Go; networking services. The networking team at DigitalOcean uses Go for a variety of purposes - from DHCP servers to IP address management services..to even wrappers around virtual switch tooling. Intrigued, I decided to also investigate how Go could be used to build other services such as port scanners and load-balancers. This session will highlight my networking journey via Go. I will discuss useful packages, key learnings, and even struggles faced while building a variety networking services within and outside of DigitalOcean. I will discuss both relevant packages within the standard library and open source packages used to implement key network protocols. As a result, listeners will gain an understanding of how to specifically leverage Go for their own networking needs.

2. digitalocean.com
Networking and Go
An Engineer’s Journey

3. digitalocean.com
Hi!
I’m Sneha.
I’m a software engineer at DigitalOcean.

4. digitalocean.com
DigitalOcean
cloud-hosting company
12 data centers
1.15 million droplets
k8saas, dbaas, storage, VPC

5. digitalocean.com
My Journey

6. digitalocean.com
2016:
Joined DigitalOcean
“What is Go?”

7. digitalocean.com

8. digitalocean.com
Designed as a systems-language, used
in a lot of verticals.

9. digitalocean.com
Go is a statically-typed, memory-safe,
compiled programming language.

10. digitalocean.com
Excellent concurrency support

11. digitalocean.com
Goroutines
lightweight processes multiplexed
onto threads

12. digitalocean.com
SYNC
package with concurrency primitives
such as mutexes, thread-safe maps,
actor goroutine groups

13. digitalocean.com
CHANNELS
a conduit through which you can
send and receive data with a typed
operator

14. digitalocean.com
Syntactically simple

15. digitalocean.com
Enforces a coding style

16. digitalocean.com
2016:
Joined DigitalOcean
“What is go?”
2017:
Developer tooling
k8s, Prometheus

17. digitalocean.com
Microservices are a small, loosely-coupled
and independently-deployable collection
of services.

18. digitalocean.com
Let’s containerize these services.

19. digitalocean.com
Let’s manage and schedule these
containers.

20. digitalocean.com
DOCC: an abstraction layer on top of
kubernetes (k8s).

21. digitalocean.com
Okay, we need some observability.

22. digitalocean.com
Tracing, logging, and metrics.

23. digitalocean.com
Pandora: a functionally sharded,
data-center wide Prometheus setup.

24. digitalocean.com

25. digitalocean.com

26. digitalocean.com

27. digitalocean.com

28. digitalocean.com
2016:
Joined DigitalOcean
“What is go?”
2017:
Developer tooling
k8s, Prometheus
2018:
Networking Team
DHCP, Gateways

29. digitalocean.com
What is the OSI model?

30. digitalocean.com
What is a socket?

31. digitalocean.com
DHCP, ARP, NDP Protocols

32. digitalocean.com
2016:
Joined DigitalOcean
“What is go?”
2017:
Developer tooling
k8s, Prometheus
2018:
Networking Team
DHCP, Gateways
2019:
MORE
LB, port scanner

33. digitalocean.com

34. digitalocean.com
How and when to use
Go to build networking
primitives.

35. digitalocean.com
The plan for today

36. digitalocean.com
▣ why use Go
⬚ the networking stack
⬚ networking primitives
LB: Layer 7 vs. 4
Port scanner: Layer 4
DHCP server

37. digitalocean.com
Why use Go for networking services?

38. digitalocean.com
Concurrency
EASY TO FUZZ-TEST
SERVER-SIDE
GREAT FOR CLI
TOOLS
EASY TO WRITE
REST/RPC SERVICES

39. digitalocean.com
But when might Go NOT be the answer?

40. digitalocean.com
High-performance packet-processing

41. digitalocean.com

42. digitalocean.com
Why did we use Go?

43. digitalocean.com
DigitalOcean has a monorepo and
shared packages.

44. digitalocean.com
hvflowd
hvaddrd
OvS
br0
hvflowctl
RNS
AddFlows
OpenFlow
SetParameters
bond0
ethX
ethX addr0
bolt
DHCPv4 NDP
gRPC
DHCPv6
tapX dropletX
hvaddrd traffic
hvaddrctl
AddFlows
internet traffic
Hypervisor
main
Using gRPC and CLI tools

45. digitalocean.com
Performance was already good enough.

46. digitalocean.com
▣ why use Go
▣ the networking stack
⬚ networking primitives
LB: Layer 7 vs. 4
Port scanner: Layer 4
DHCP server

47. digitalocean.com
OSI: Open-systems Interconnect Model

48. digitalocean.comSource: https://www.researchgate.net/figure/OSI-model-seven-layer-protocol-stack-28_fig6_322568288

49. digitalocean.com
TCP/IP Model

50. digitalocean.com
Source: https://searchnetworking.techtarget.com/definition/TCP-IP

51. digitalocean.com
The Layers: Data Encapsulation

52. digitalocean.com
Source: https://www.aboutdebian.com/network.htm

53. digitalocean.com
socket: internal endpoint for sending or
receiving data on a computer network

54. digitalocean.com
How do we read a segment?

55. digitalocean.com
TCP (stream socket): provides a
connection-oriented, sequenced flow of data

56. digitalocean.com
UDP (datagram socket): provides a
connectionless flow of data

57. digitalocean.com
How do we read a packet?
How do we read a frame?

58. digitalocean.com
raw socket: a socket that allows sending and
receiving of data without protocol-specific
transport layer formatting

59. digitalocean.com
raw socket: IP layer (3)
socket(AF_INET,SOCK_RAW,protocol)
packet socket: link layer (2)
socket(AF_PACKET,SOCK_RAW,protocol)

60. digitalocean.com
bpf (berkeley packet filter) interface:
raw link-layer interface on most
unix-based systems

61. digitalocean.com
bpf filters also used with raw sockets
or tcpdump

62. digitalocean.com
Networking Protocols

63. digitalocean.com
Source: https://www.quora.com/What-are-some-recommendation-to-fully-understand-networking-protocols

64. digitalocean.com

65. digitalocean.com

66. digitalocean.com
“The best way to learn is simply
by doing.”
--- Me

67. digitalocean.com
▣ when to use Go
▣ the networking stack
▣ networking primitives
LB: Layer 7 vs. 4
Port scanner: Layer 4
DHCP server

68. digitalocean.com
Layer 7: Loadbalancer

69. digitalocean.com
Loadbalancer (LB)
A device distributing traffic
across a number of backends.
network definition
Source: https://gbhackers.com/load-balancer-reverse-proxy/

70. digitalocean.com
Building a layer-7 LB
1. Use HTTP protocol.
2. Accept client-side connections.
3. Pass client-side request to one of the
backends.
4. Return server-response back to the client.

71. digitalocean.com
HTTP:
HyperText Transfer Protocol
An application layer (7) protocol:
● request-response based
● stateless
● Has different methods (GET,
PUT, etc)
● Uses TCP or UDP sockets
under-the-hood
network definition
Source:
https://www.ntu.edu.sg/home/ehchua/programming/webprogramming/HTTP_Basic
s.html

72. digitalocean.comGO FEATURE
net/http

73. digitalocean.com
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
...
})
log.Fatal(http.ListenAndServe(":8080", nil))
1. ListenAndServe

74. digitalocean.com
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
n := rand.Intn(len(backends))
r.URL.Host = backends[n]
r.URL.Scheme = "https"
req, err := http.NewRequest(r.Method, r.URL.String(), r.Body)
...
res, err := client.Do(req)
...
}
2. Read and send request to backend

75. digitalocean.com
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
...
w.WriteHeader(res.StatusCode)
_, err = io.Copy(w, res.Body)
if err != nil {
log.Printf("error writing response to client: %v", err)
}
...
}
3. Return response to client

76. digitalocean.com
Layer 4: Loadbalancer

77. digitalocean.com
Building a layer-4 LB:
1. Use TCP protocol (via a streaming socket).
2. Accept client-side connections.
3. Open backend connection.
4. Use one goroutine to shuttle packets from the
client to the backend.
5. Use another goroutine to shuttle packets from
the backend to the client.

78. digitalocean.com
TCP: Transmission
Control Protocol
Connection-oriented protocol
with 3 phases:
● Connection establishment
(TCP handshake)
● Data transfer
● Connection termination
network definition
Source:
https://www.researchgate.net/figure/Three-way-Handshake-in-TCP-Connection-
Establishment-Process_fig7_313951935

79. digitalocean.comGO FEATURE
net

80. digitalocean.comGO FEATURE
github.com/oklog/run

81. digitalocean.com
listener, err := net.Listen("tcp","127.0.0.1:9090")
if err != nil {
log.Fatal(err)
}
for {
conn, err := listener.Accept()
if err != nil {
log.Printf("error accepting client conn: %v", err)
continue
}
go handleConn(conn)
}
1. Open and bind client TCP socket

82. digitalocean.com
func handleConn(clientConn net.Conn) {
n := rand.Intn(len(backends))
backendConn, err := net.Dial("tcp", backends[n])
if err != nil {
log.Printf("error %s: %v", backends[n], err)
return
}
...
}
2. Open and connect backend TCP socket

83. digitalocean.com
var g run.Group
{
g.Add(func() error {
return copy(clientConn, backendConn)
}, func(error) {
clientConn.Close()
backendConn.Close()
})
}
{
g.Add(func() error {
return copy(backendConn, clientConn)
}, ...)
}
err = g.Run()
3. Configure rungroups

84. digitalocean.com
func copy(from net.Conn, to net.Conn) error {
for {
readBytes := make([]byte, 1024)
n, err := from.Read(readBytes)
if err != nil {
...
}
if _, err = to.Write(readBytes[:n]); err != nil {
...
}
}
}
4. Read and copy

85. digitalocean.com
Loadbalancer IRL:
● Exact mechanism of handling connections and/or requests
depends on the layer.
● Layer-4 loadbalancers should be more efficient:
○ Won’t necessarily create a new backend connection per
client conn.
○ Use NAT and munging incoming packets.
● Better loadbalancing algorithms.
● Client stickiness.

86. digitalocean.com
Layer 4: Port Scanner

87. digitalocean.com
Nmap
Linux utility that scans hosts and
attempts to determine open
UDP/TCP ports
network definition
Source: https://commons.wikimedia.org/wiki/File:Screenshot_Nmap.png

88. digitalocean.comGO FEATURE
sync

89. digitalocean.com
Building a TCP port-scanner
1. Select range of ports to scan.
2. Try to open and connect a TCP socket to a
remote address (and port)..
3. Print results.

90. digitalocean.com
addr := fmt.Sprintf("%s:%d", hostname, port)
conn, err := net.DialTimeout("tcp", addr, timeout)
if err != nil {
fmt.Printf("port %d closed: %vn", port, err)
} else {
fmt.Printf("port %d openn", port)
conn.Close()
}
1. Open and connect TCP socket

91. digitalocean.com
conSema := make(chan struct{}, 10)
var wg sync.WaitGroup
for i := 1; i <= 65535; i++ {
wg.Add(1)
go func(port int) {
conSema <- struct{}{}
...
wg.Done()
<-conSema
}(i)
}
wg.Wait()
2. Use waitgroup and channels

92. digitalocean.com
Nmap IRL:
● Checks UDP and TCP ports on local or remote hosts.
● Host discovery
● OS detection
● Auditing security of a firewall

93. digitalocean.com

94. digitalocean.com
Layer 2: DHCP Server

95. digitalocean.com
DHCP is technically layer-7.

96. digitalocean.com
DHCP Protocol
● Dynamic Host Configuration
Protocol is used by routers to
allocate IP addresses to
network interfaces
● DHCPv6 uses NDP and
DHCPv4 uses ARP
network definition
Source: https://study-ccna.com/dhcp-dns/

97. digitalocean.com
Need link layer MAC addresses.

98. digitalocean.com
Building a DHCP server:
1. Open and bind a raw socket to an interface.
2. Read data from socket into bytes buffer.
3. Unmarshal into DHCP message and retrieve sender
hardware address.
4. Switch between handlers based on message type.
5. Validate DHCP request message and craft response.
6. Unicast response back to sender.

99. digitalocean.comGO FEATURE
github.com/mdlayher/raw

100. digitalocean.comGO FEATURE
filename := "eth-packet-socket"
typ := unix.SOCK_RAW
if cfg.LinuxSockDGRAM {
filename = "packet-socket"
typ = unix.SOCK_DGRAM
}
sock, err := unix.Socket(unix.AF_PACKET, typ, 0)
...
f := os.NewFile(uintptr(sock), filename)
sc, err := f.SyscallConn()
...
n, addr, err = unix.Recvfrom(int(fd), p, flags)
github.com/mdlayher/raw/raw_linux.go

101. digitalocean.comGO FEATURE
// Try to find an available BPF device
for i := 0; i <= 10; i++ {
bpfPath := fmt.Sprintf( "/dev/bpf%d", i)
f, err = os.OpenFile(bpfPath, os.O_RDWR, 0666)
if err == nil {
break
}
if perr, ok := err.(*os.PathError); ok {
if perr.Err.(syscall.Errno) == syscall.EBUSY
{
continue
}
}
return nil, err
}
github.com/mdlayher/raw/raw_bsd.go

102. digitalocean.comGO FEATURE
syscall
os
golang.org/x/sys/unix

103. digitalocean.com
ifi, err := net.InterfaceByName(iface)
if err != nil {
return nil, err
}
pc, err := raw.ListenPacket(ifi, uint16(ethernet.EtherTypeIPv4),
&raw.Config{
// Don't set any timeouts to avoid syscall busy
// loops, since this server will run forever anyway.
NoTimeouts: true,
})
1. Open and bind raw socket

104. digitalocean.com
b := make([]byte, 1500)
for {
n, from, err := s.pc.ReadFrom(b)
if err != nil {
...
continue
}
buf := make([]byte, n)
copy(buf, b)
workC <- request{
buf: buf,
from: from,
}
}
2. Read data into buffer

105. digitalocean.com
var workWG sync.WaitGroup
workWG.Add(Workers)
workC := make(chan request, Workers)
for i := 0; i < Workers; i++ {
go func() {
defer workWG.Done()
for r := range workC {
s.serve(r.buf, r.from)
}
}()
}
3. Worker handles request

106. digitalocean.com
func (h *handler) serveDHCPv4(ctx context.Context, req
*dhcp4.Packet, from *dhcp4conn.Addr) (*dhcp4.Packet, net.Addr)
{
...
if !from.EthernetSource.Equal(reqMAC) {
...
return h.shouldNAK(req.Type, to, broadcast)
}
...
}
4. Validate request

107. digitalocean.com
func (h *handler) serveDHCPv4(ctx context.Context, req
*dhcp4.Packet, from *dhcp4conn.Addr) (*dhcp4.Packet, net.Addr) {
...
res, err := h.buildResponse(ctx, params, req, from)
if err != nil {
topError(span, err, "failed to build response")
return h.shouldNAK(req.Type, to, broadcast)
}
res.Broadcast = broadcast
return res, to
...
}
5. Build and send response

108. digitalocean.com
DHCP Summary:
1. DHCP: dynamic host configuration protocol
used to dynamically assign IP address.
2. Use raw sockets.
3. Jump down to link-layer to do source MAC
validation.

109. digitalocean.com
Conclusion

110. digitalocean.com
Go can be good for building networking
primitives.

111. digitalocean.com

112. digitalocean.com
Layer 7:
net/http package

113. digitalocean.com
Layer 4:
net package

114. digitalocean.com
Layer 3/2:
raw package
os,syscall
golang.org/x/sys/unix

115. digitalocean.com
sync package
github.com/oklog/run package
goroutines
channels

117. THANKS :)
@mdlayher
@juliusvolz
@nicboul

118. digitalocean.com
Ohai!
@snehainguva
sneha.inguva@gmail.com

119. digitalocean.com
Sources
● Port Scanner: https://github.com/si74/portscanner
● Layer 7 loadbalancer: https://github.com/si74/layer7lb
● Layer 4 loadbalancer: https://github.com/si74/tcpproxy
● Raw package: https://github.com/mdlayher/raw
● Godocs: https://godoc.org/
● Golang: https://godoc.org/