Networking and Go: An Engineer's Journey (Strangeloop 2019)

1. digitalocean.com Networking and Go An Engineer’s Journey

2. digitalocean.com Hi! I’m Sneha. I’m a software engineer at DigitalOcean.

3. digitalocean.com DigitalOcean cloud-hosting company 12 data centers 1.15 million droplets k8saas, dbaas, storage, VPC

4. digitalocean.com My Journey

5. digitalocean.com 2016: Joined DigitalOcean “What is Go?”

6. digitalocean.com

7. digitalocean.com Designed as a systems-language, used in a lot of verticals.

8. digitalocean.com Go is a statically-typed, memory-safe, compiled programming language.

9. digitalocean.com Excellent concurrency support

10. digitalocean.com Goroutines lightweight processes multiplexed onto threads

11. digitalocean.com SYNC package with concurrency primitives such as mutexes, thread-safe maps, actor goroutine groups

12. digitalocean.com CHANNELS a conduit through which you can send and receive data with a typed operator

13. digitalocean.com Syntactically simple

14. digitalocean.com Enforces a coding style

15. digitalocean.com 2016: Joined DigitalOcean “What is go?” 2017: Developer tooling k8s, Prometheus

16. digitalocean.com Microservices are a small, loosely-coupled and independently-deployable collection of services.

17. digitalocean.com Let’s containerize these services.

18. digitalocean.com Let’s manage and schedule these containers.

19. digitalocean.com DOCC: an abstraction layer on top of kubernetes (k8s).

20. digitalocean.com Okay, we need some observability.

21. digitalocean.com Tracing, logging, and metrics.

22. digitalocean.com Pandora: a functionally sharded, data-center wide Prometheus setup.

23. digitalocean.com

27. digitalocean.com 2016: Joined DigitalOcean “What is go?” 2017: Developer tooling k8s, Prometheus 2018: Networking Team DHCP, Gateways

28. digitalocean.com What is the OSI model?

29. digitalocean.com What is a socket?

30. digitalocean.com DHCP, ARP, NDP Protocols

31. digitalocean.com 2016: Joined DigitalOcean “What is go?” 2017: Developer tooling k8s, Prometheus 2018: Networking Team DHCP, Gateways 2019: MORE LB, port scanner

33. digitalocean.com How and when to use Go to build networking primitives.

34. digitalocean.com The plan for today

35. digitalocean.com ▣ why use Go ⬚ the networking stack ⬚ networking primitives LB: Layer 7 vs. 4 Port scanner: Layer 4 DHCP server

36. digitalocean.com Why use Go for networking services?

37. digitalocean.com Concurrency EASY TO FUZZ-TEST SERVER-SIDE GREAT FOR CLI TOOLS EASY TO WRITE REST/RPC SERVICES

38. digitalocean.com But when might Go NOT be the answer?

39. digitalocean.com High-performance packet-processing

41. digitalocean.com Why did we use Go?

42. digitalocean.com DigitalOcean has a monorepo and shared packages.

43. digitalocean.com hvflowd hvaddrd OvS br0 hvflowctl RNS AddFlows OpenFlow SetParameters bond0 ethX ethX addr0 bolt DHCPv4 NDP gRPC DHCPv6 tapX dropletX hvaddrd traffic hvaddrctl AddFlows internet traffic Hypervisor main Using gRPC and CLI tools

44. digitalocean.com Performance was already good enough.

45. digitalocean.com ▣ why use Go ▣ the networking stack ⬚ networking primitives LB: Layer 7 vs. 4 Port scanner: Layer 4 DHCP server

46. digitalocean.com OSI: Open-systems Interconnect Model

47. digitalocean.comSource: https://www.researchgate.net/ﬁgure/OSI-model-seven-layer-protocol-stack-28_ﬁg6_322568288

48. digitalocean.com TCP/IP Model

49. digitalocean.com Source: https://searchnetworking.techtarget.com/deﬁnition/TCP-IP

50. digitalocean.com The Layers: Data Encapsulation

51. digitalocean.com Source: https://www.aboutdebian.com/network.htm

52. digitalocean.com socket: internal endpoint for sending or receiving data on a computer network

53. digitalocean.com How do we read a segment?

54. digitalocean.com TCP (stream socket): provides a connection-oriented, sequenced ﬂow of data

55. digitalocean.com UDP (datagram socket): provides a connectionless ﬂow of data

56. digitalocean.com How do we read a packet? How do we read a frame?

57. digitalocean.com raw socket: a socket that allows sending and receiving of data without protocol-speciﬁc transport layer formatting

58. digitalocean.com raw socket: IP layer (3) socket(AF_INET,SOCK_RAW,protocol) packet socket: link layer (2) socket(AF_PACKET,SOCK_RAW,protocol)

59. digitalocean.com bpf (berkeley packet ﬁlter) interface: raw link-layer interface on most unix-based systems

60. digitalocean.com bpf ﬁlters also used with raw sockets or tcpdump

61. digitalocean.com Networking Protocols

62. digitalocean.com Source: https://www.quora.com/What-are-some-recommendation-to-fully-understand-networking-protocols

65. digitalocean.com “The best way to learn is simply by doing.” --- Me

66. digitalocean.com ▣ when to use Go ▣ the networking stack ▣ networking primitives LB: Layer 7 vs. 4 Port scanner: Layer 4 DHCP server

67. digitalocean.com Layer 7: Loadbalancer

68. digitalocean.com Loadbalancer (LB) A device distributing traﬃc across a number of backends. network deﬁnition Source: https://gbhackers.com/load-balancer-reverse-proxy/

69. digitalocean.com Building a layer-7 LB 1. Use HTTP protocol. 2. Accept client-side connections. 3. Pass client-side request to one of the backends. 4. Return server-response back to the client.

70. digitalocean.com HTTP: HyperText Transfer Protocol An application layer (7) protocol: ● request-response based ● stateless ● Has diﬀerent methods (GET, PUT, etc) ● Uses TCP or UDP sockets under-the-hood network deﬁnition Source: https://www.ntu.edu.sg/home/ehchua/programming/webprogramming/HTTP_Basic s.html

71. digitalocean.comGO FEATURE net/http

72. digitalocean.com http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { ... }) log.Fatal(http.ListenAndServe(":8080", nil)) 1. ListenAndServe

73. digitalocean.com http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { n := rand.Intn(len(backends)) r.URL.Host = backends[n] r.URL.Scheme = "https" req, err := http.NewRequest(r.Method, r.URL.String(), r.Body) ... res, err := client.Do(req) ... } 2. Read and send request to backend

74. digitalocean.com http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { ... w.WriteHeader(res.StatusCode) _, err = io.Copy(w, res.Body) if err != nil { log.Printf("error writing response to client: %v", err) } ... } 3. Return response to client

75. digitalocean.com Layer 4: Loadbalancer

76. digitalocean.com Building a layer-4 LB: 1. Use TCP protocol (via a streaming socket). 2. Accept client-side connections. 3. Open backend connection. 4. Use one goroutine to shuttle packets from the client to the backend. 5. Use another goroutine to shuttle packets from the backend to the client.

77. digitalocean.com TCP: Transmission Control Protocol Connection-oriented protocol with 3 phases: ● Connection establishment (TCP handshake) ● Data transfer ● Connection termination network deﬁnition Source: https://www.researchgate.net/figure/Three-way-Handshake-in-TCP-Connection- Establishment-Process_fig7_313951935

78. digitalocean.comGO FEATURE net

79. digitalocean.comGO FEATURE github.com/oklog/run

80. digitalocean.com listener, err := net.Listen("tcp","127.0.0.1:9090") if err != nil { log.Fatal(err) } for { conn, err := listener.Accept() if err != nil { log.Printf("error accepting client conn: %v", err) continue } go handleConn(conn) } 1. Open and bind client TCP socket

81. digitalocean.com func handleConn(clientConn net.Conn) { n := rand.Intn(len(backends)) backendConn, err := net.Dial("tcp", backends[n]) if err != nil { log.Printf("error %s: %v", backends[n], err) return } ... } 2. Open and connect backend TCP socket

82. digitalocean.com var g run.Group { g.Add(func() error { return copy(clientConn, backendConn) }, func(error) { clientConn.Close() backendConn.Close() }) } { g.Add(func() error { return copy(backendConn, clientConn) }, ...) } err = g.Run() 3. Conﬁgure rungroups

83. digitalocean.com func copy(from net.Conn, to net.Conn) error { for { readBytes := make([]byte, 1024) n, err := from.Read(readBytes) if err != nil { ... } if _, err = to.Write(readBytes[:n]); err != nil { ... } } } 4. Read and copy

84. digitalocean.com Loadbalancer IRL: ● Exact mechanism of handling connections and/or requests depends on the layer. ● Layer-4 loadbalancers should be more eﬃcient: ○ Won’t necessarily create a new backend connection per client conn. ○ Use NAT and munging incoming packets. ● Better loadbalancing algorithms. ● Client stickiness.

85. digitalocean.com Layer 4: Port Scanner

86. digitalocean.com Nmap Linux utility that scans hosts and attempts to determine open UDP/TCP ports network deﬁnition Source: https://commons.wikimedia.org/wiki/File:Screenshot_Nmap.png

87. digitalocean.comGO FEATURE sync

88. digitalocean.com Building a TCP port-scanner 1. Select range of ports to scan. 2. Try to open and connect a TCP socket to a remote address (and port).. 3. Print results.

89. digitalocean.com addr := fmt.Sprintf("%s:%d", hostname, port) conn, err := net.DialTimeout("tcp", addr, timeout) if err != nil { fmt.Printf("port %d closed: %vn", port, err) } else { fmt.Printf("port %d openn", port) conn.Close() } 1. Open and connect TCP socket

90. digitalocean.com conSema := make(chan struct{}, 10) var wg sync.WaitGroup for i := 1; i <= 65535; i++ { wg.Add(1) go func(port int) { conSema <- struct{}{} ... wg.Done() <-conSema }(i) } wg.Wait() 2. Use waitgroup and channels

91. digitalocean.com Nmap IRL: ● Checks UDP and TCP ports on local or remote hosts. ● Host discovery ● OS detection ● Auditing security of a ﬁrewall

93. digitalocean.com Layer 2: DHCP Server

94. digitalocean.com DHCP is technically layer-7.

95. digitalocean.com DHCP Protocol ● Dynamic Host Configuration Protocol is used by routers to allocate IP addresses to network interfaces ● DHCPv6 uses NDP and DHCPv4 uses ARP network deﬁnition Source: https://study-ccna.com/dhcp-dns/

96. digitalocean.com Need link layer MAC addresses.

97. digitalocean.com Building a DHCP server: 1. Open and bind a raw socket to an interface. 2. Read data from socket into bytes buﬀer. 3. Unmarshal into DHCP message and retrieve sender hardware address. 4. Switch between handlers based on message type. 5. Validate DHCP request message and craft response. 6. Unicast response back to sender.

98. digitalocean.comGO FEATURE github.com/mdlayher/raw

99. digitalocean.comGO FEATURE filename := "eth-packet-socket" typ := unix.SOCK_RAW if cfg.LinuxSockDGRAM { filename = "packet-socket" typ = unix.SOCK_DGRAM } sock, err := unix.Socket(unix.AF_PACKET, typ, 0) ... f := os.NewFile(uintptr(sock), filename) sc, err := f.SyscallConn() ... n, addr, err = unix.Recvfrom(int(fd), p, flags) github.com/mdlayher/raw/raw_linux.go

100. digitalocean.comGO FEATURE // Try to find an available BPF device for i := 0; i <= 10; i++ { bpfPath := fmt.Sprintf( "/dev/bpf%d", i) f, err = os.OpenFile(bpfPath, os.O_RDWR, 0666) if err == nil { break } if perr, ok := err.(*os.PathError); ok { if perr.Err.(syscall.Errno) == syscall.EBUSY { continue } } return nil, err } github.com/mdlayher/raw/raw_bsd.go

101. digitalocean.comGO FEATURE syscall os golang.org/x/sys/unix

102. digitalocean.com ifi, err := net.InterfaceByName(iface) if err != nil { return nil, err } pc, err := raw.ListenPacket(ifi, uint16(ethernet.EtherTypeIPv4), &raw.Config{ // Don't set any timeouts to avoid syscall busy // loops, since this server will run forever anyway. NoTimeouts: true, }) 1. Open and bind raw socket

103. digitalocean.com b := make([]byte, 1500) for { n, from, err := s.pc.ReadFrom(b) if err != nil { ... continue } buf := make([]byte, n) copy(buf, b) workC <- request{ buf: buf, from: from, } } 2. Read data into buﬀer

104. digitalocean.com var workWG sync.WaitGroup workWG.Add(Workers) workC := make(chan request, Workers) for i := 0; i < Workers; i++ { go func() { defer workWG.Done() for r := range workC { s.serve(r.buf, r.from) } }() } 3. Worker handles request

105. digitalocean.com func (h *handler) serveDHCPv4(ctx context.Context, req *dhcp4.Packet, from *dhcp4conn.Addr) (*dhcp4.Packet, net.Addr) { ... if !from.EthernetSource.Equal(reqMAC) { ... return h.shouldNAK(req.Type, to, broadcast) } ... } 4. Validate request

106. digitalocean.com func (h *handler) serveDHCPv4(ctx context.Context, req *dhcp4.Packet, from *dhcp4conn.Addr) (*dhcp4.Packet, net.Addr) { ... res, err := h.buildResponse(ctx, params, req, from) if err != nil { topError(span, err, "failed to build response") return h.shouldNAK(req.Type, to, broadcast) } res.Broadcast = broadcast return res, to ... } 5. Build and send response

107. digitalocean.com DHCP Summary: 1. DHCP: dynamic host conﬁguration protocol used to dynamically assign IP address. 2. Use raw sockets. 3. Jump down to link-layer to do source MAC validation.

108. digitalocean.com Conclusion

109. digitalocean.com Go can be good for building networking primitives.

111. digitalocean.com Layer 7: net/http package

112. digitalocean.com Layer 4: net package

113. digitalocean.com Layer 3/2: raw package os,syscall golang.org/x/sys/unix

114. digitalocean.com sync package github.com/oklog/run package goroutines channels

115. THANKS :) @mdlayher @juliusvolz @nicboul

116. digitalocean.com Ohai! @snehainguva sneha.inguva@gmail.com

117. digitalocean.com Sources ● Port Scanner: https://github.com/si74/portscanner ● Layer 7 loadbalancer: https://github.com/si74/layer7lb ● Layer 4 loadbalancer: https://github.com/si74/tcpproxy ● Raw package: https://github.com/mdlayher/raw ● Godocs: https://godoc.org/ ● Golang: https://godoc.org/

Networking and Go: An Engineer's Journey (Strangeloop 2019)

Sneha Inguva

Networking and Go: An Engineer's Journey (Strangeloop 2019)