Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Networking and Go: An Engineer's Journey (Strangeloop 2019)

100 views

Published on

My foray into Go began a few years ago when I started working at DigitalOcean. While building an abstraction layer on top of Kubernetes and familiarizing myself with the language, I began to love it. Syntactically simple, with amazing concurrency primitives and a wonderful community, Go was an excellent choice for a cloud-hosting company with a variety of low-level, server-side microservices.

In the last year, however, I've joined the software-defined networking team and learned of another application of Go; networking services. The networking team at DigitalOcean uses Go for a variety of purposes - from DHCP servers to IP address management services..to even wrappers around virtual switch tooling. Intrigued, I decided to also investigate how Go could be used to build other services such as port scanners and load-balancers.

This session will highlight my networking journey via Go. I will discuss useful packages, key learnings, and even struggles faced while building a variety networking services within and outside of DigitalOcean. I will discuss both relevant packages within the standard library and open source packages used to implement key network protocols. As a result, listeners will gain an understanding of how to specifically leverage Go for their own networking needs.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Networking and Go: An Engineer's Journey (Strangeloop 2019)

  1. 1. digitalocean.com Networking and Go An Engineer’s Journey
  2. 2. digitalocean.com Hi! I’m Sneha. I’m a software engineer at DigitalOcean.
  3. 3. digitalocean.com DigitalOcean cloud-hosting company 12 data centers 1.15 million droplets k8saas, dbaas, storage, VPC
  4. 4. digitalocean.com My Journey
  5. 5. digitalocean.com 2016: Joined DigitalOcean “What is Go?”
  6. 6. digitalocean.com
  7. 7. digitalocean.com Designed as a systems-language, used in a lot of verticals.
  8. 8. digitalocean.com Go is a statically-typed, memory-safe, compiled programming language.
  9. 9. digitalocean.com Excellent concurrency support
  10. 10. digitalocean.com Goroutines lightweight processes multiplexed onto threads
  11. 11. digitalocean.com SYNC package with concurrency primitives such as mutexes, thread-safe maps, actor goroutine groups
  12. 12. digitalocean.com CHANNELS a conduit through which you can send and receive data with a typed operator
  13. 13. digitalocean.com Syntactically simple
  14. 14. digitalocean.com Enforces a coding style
  15. 15. digitalocean.com 2016: Joined DigitalOcean “What is go?” 2017: Developer tooling k8s, Prometheus
  16. 16. digitalocean.com Microservices are a small, loosely-coupled and independently-deployable collection of services.
  17. 17. digitalocean.com Let’s containerize these services.
  18. 18. digitalocean.com Let’s manage and schedule these containers.
  19. 19. digitalocean.com DOCC: an abstraction layer on top of kubernetes (k8s).
  20. 20. digitalocean.com Okay, we need some observability.
  21. 21. digitalocean.com Tracing, logging, and metrics.
  22. 22. digitalocean.com Pandora: a functionally sharded, data-center wide Prometheus setup.
  23. 23. digitalocean.com
  24. 24. digitalocean.com
  25. 25. digitalocean.com
  26. 26. digitalocean.com
  27. 27. digitalocean.com 2016: Joined DigitalOcean “What is go?” 2017: Developer tooling k8s, Prometheus 2018: Networking Team DHCP, Gateways
  28. 28. digitalocean.com What is the OSI model?
  29. 29. digitalocean.com What is a socket?
  30. 30. digitalocean.com DHCP, ARP, NDP Protocols
  31. 31. digitalocean.com 2016: Joined DigitalOcean “What is go?” 2017: Developer tooling k8s, Prometheus 2018: Networking Team DHCP, Gateways 2019: MORE LB, port scanner
  32. 32. digitalocean.com
  33. 33. digitalocean.com How and when to use Go to build networking primitives.
  34. 34. digitalocean.com The plan for today
  35. 35. digitalocean.com ▣ why use Go ⬚ the networking stack ⬚ networking primitives LB: Layer 7 vs. 4 Port scanner: Layer 4 DHCP server
  36. 36. digitalocean.com Why use Go for networking services?
  37. 37. digitalocean.com Concurrency EASY TO FUZZ-TEST SERVER-SIDE GREAT FOR CLI TOOLS EASY TO WRITE REST/RPC SERVICES
  38. 38. digitalocean.com But when might Go NOT be the answer?
  39. 39. digitalocean.com High-performance packet-processing
  40. 40. digitalocean.com
  41. 41. digitalocean.com Why did we use Go?
  42. 42. digitalocean.com DigitalOcean has a monorepo and shared packages.
  43. 43. digitalocean.com hvflowd hvaddrd OvS br0 hvflowctl RNS AddFlows OpenFlow SetParameters bond0 ethX ethX addr0 bolt DHCPv4 NDP gRPC DHCPv6 tapX dropletX hvaddrd traffic hvaddrctl AddFlows internet traffic Hypervisor main Using gRPC and CLI tools
  44. 44. digitalocean.com Performance was already good enough.
  45. 45. digitalocean.com ▣ why use Go ▣ the networking stack ⬚ networking primitives LB: Layer 7 vs. 4 Port scanner: Layer 4 DHCP server
  46. 46. digitalocean.com OSI: Open-systems Interconnect Model
  47. 47. digitalocean.comSource: https://www.researchgate.net/figure/OSI-model-seven-layer-protocol-stack-28_fig6_322568288
  48. 48. digitalocean.com TCP/IP Model
  49. 49. digitalocean.com Source: https://searchnetworking.techtarget.com/definition/TCP-IP
  50. 50. digitalocean.com The Layers: Data Encapsulation
  51. 51. digitalocean.com Source: https://www.aboutdebian.com/network.htm
  52. 52. digitalocean.com socket: internal endpoint for sending or receiving data on a computer network
  53. 53. digitalocean.com How do we read a segment?
  54. 54. digitalocean.com TCP (stream socket): provides a connection-oriented, sequenced flow of data
  55. 55. digitalocean.com UDP (datagram socket): provides a connectionless flow of data
  56. 56. digitalocean.com How do we read a packet? How do we read a frame?
  57. 57. digitalocean.com raw socket: a socket that allows sending and receiving of data without protocol-specific transport layer formatting
  58. 58. digitalocean.com raw socket: IP layer (3) socket(AF_INET,SOCK_RAW,protocol) packet socket: link layer (2) socket(AF_PACKET,SOCK_RAW,protocol)
  59. 59. digitalocean.com bpf (berkeley packet filter) interface: raw link-layer interface on most unix-based systems
  60. 60. digitalocean.com bpf filters also used with raw sockets or tcpdump
  61. 61. digitalocean.com Networking Protocols
  62. 62. digitalocean.com Source: https://www.quora.com/What-are-some-recommendation-to-fully-understand-networking-protocols
  63. 63. digitalocean.com
  64. 64. digitalocean.com
  65. 65. digitalocean.com “The best way to learn is simply by doing.” --- Me
  66. 66. digitalocean.com ▣ when to use Go ▣ the networking stack ▣ networking primitives LB: Layer 7 vs. 4 Port scanner: Layer 4 DHCP server
  67. 67. digitalocean.com Layer 7: Loadbalancer
  68. 68. digitalocean.com Loadbalancer (LB) A device distributing traffic across a number of backends. network definition Source: https://gbhackers.com/load-balancer-reverse-proxy/
  69. 69. digitalocean.com Building a layer-7 LB 1. Use HTTP protocol. 2. Accept client-side connections. 3. Pass client-side request to one of the backends. 4. Return server-response back to the client.
  70. 70. digitalocean.com HTTP: HyperText Transfer Protocol An application layer (7) protocol: ● request-response based ● stateless ● Has different methods (GET, PUT, etc) ● Uses TCP or UDP sockets under-the-hood network definition Source: https://www.ntu.edu.sg/home/ehchua/programming/webprogramming/HTTP_Basic s.html
  71. 71. digitalocean.comGO FEATURE net/http
  72. 72. digitalocean.com http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { ... }) log.Fatal(http.ListenAndServe(":8080", nil)) 1. ListenAndServe
  73. 73. digitalocean.com http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { n := rand.Intn(len(backends)) r.URL.Host = backends[n] r.URL.Scheme = "https" req, err := http.NewRequest(r.Method, r.URL.String(), r.Body) ... res, err := client.Do(req) ... } 2. Read and send request to backend
  74. 74. digitalocean.com http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { ... w.WriteHeader(res.StatusCode) _, err = io.Copy(w, res.Body) if err != nil { log.Printf("error writing response to client: %v", err) } ... } 3. Return response to client
  75. 75. digitalocean.com Layer 4: Loadbalancer
  76. 76. digitalocean.com Building a layer-4 LB: 1. Use TCP protocol (via a streaming socket). 2. Accept client-side connections. 3. Open backend connection. 4. Use one goroutine to shuttle packets from the client to the backend. 5. Use another goroutine to shuttle packets from the backend to the client.
  77. 77. digitalocean.com TCP: Transmission Control Protocol Connection-oriented protocol with 3 phases: ● Connection establishment (TCP handshake) ● Data transfer ● Connection termination network definition Source: https://www.researchgate.net/figure/Three-way-Handshake-in-TCP-Connection- Establishment-Process_fig7_313951935
  78. 78. digitalocean.comGO FEATURE net
  79. 79. digitalocean.comGO FEATURE github.com/oklog/run
  80. 80. digitalocean.com listener, err := net.Listen("tcp","127.0.0.1:9090") if err != nil { log.Fatal(err) } for { conn, err := listener.Accept() if err != nil { log.Printf("error accepting client conn: %v", err) continue } go handleConn(conn) } 1. Open and bind client TCP socket
  81. 81. digitalocean.com func handleConn(clientConn net.Conn) { n := rand.Intn(len(backends)) backendConn, err := net.Dial("tcp", backends[n]) if err != nil { log.Printf("error %s: %v", backends[n], err) return } ... } 2. Open and connect backend TCP socket
  82. 82. digitalocean.com var g run.Group { g.Add(func() error { return copy(clientConn, backendConn) }, func(error) { clientConn.Close() backendConn.Close() }) } { g.Add(func() error { return copy(backendConn, clientConn) }, ...) } err = g.Run() 3. Configure rungroups
  83. 83. digitalocean.com func copy(from net.Conn, to net.Conn) error { for { readBytes := make([]byte, 1024) n, err := from.Read(readBytes) if err != nil { ... } if _, err = to.Write(readBytes[:n]); err != nil { ... } } } 4. Read and copy
  84. 84. digitalocean.com Loadbalancer IRL: ● Exact mechanism of handling connections and/or requests depends on the layer. ● Layer-4 loadbalancers should be more efficient: ○ Won’t necessarily create a new backend connection per client conn. ○ Use NAT and munging incoming packets. ● Better loadbalancing algorithms. ● Client stickiness.
  85. 85. digitalocean.com Layer 4: Port Scanner
  86. 86. digitalocean.com Nmap Linux utility that scans hosts and attempts to determine open UDP/TCP ports network definition Source: https://commons.wikimedia.org/wiki/File:Screenshot_Nmap.png
  87. 87. digitalocean.comGO FEATURE sync
  88. 88. digitalocean.com Building a TCP port-scanner 1. Select range of ports to scan. 2. Try to open and connect a TCP socket to a remote address (and port).. 3. Print results.
  89. 89. digitalocean.com addr := fmt.Sprintf("%s:%d", hostname, port) conn, err := net.DialTimeout("tcp", addr, timeout) if err != nil { fmt.Printf("port %d closed: %vn", port, err) } else { fmt.Printf("port %d openn", port) conn.Close() } 1. Open and connect TCP socket
  90. 90. digitalocean.com conSema := make(chan struct{}, 10) var wg sync.WaitGroup for i := 1; i <= 65535; i++ { wg.Add(1) go func(port int) { conSema <- struct{}{} ... wg.Done() <-conSema }(i) } wg.Wait() 2. Use waitgroup and channels
  91. 91. digitalocean.com Nmap IRL: ● Checks UDP and TCP ports on local or remote hosts. ● Host discovery ● OS detection ● Auditing security of a firewall
  92. 92. digitalocean.com
  93. 93. digitalocean.com Layer 2: DHCP Server
  94. 94. digitalocean.com DHCP is technically layer-7.
  95. 95. digitalocean.com DHCP Protocol ● Dynamic Host Configuration Protocol is used by routers to allocate IP addresses to network interfaces ● DHCPv6 uses NDP and DHCPv4 uses ARP network definition Source: https://study-ccna.com/dhcp-dns/
  96. 96. digitalocean.com Need link layer MAC addresses.
  97. 97. digitalocean.com Building a DHCP server: 1. Open and bind a raw socket to an interface. 2. Read data from socket into bytes buffer. 3. Unmarshal into DHCP message and retrieve sender hardware address. 4. Switch between handlers based on message type. 5. Validate DHCP request message and craft response. 6. Unicast response back to sender.
  98. 98. digitalocean.comGO FEATURE github.com/mdlayher/raw
  99. 99. digitalocean.comGO FEATURE filename := "eth-packet-socket" typ := unix.SOCK_RAW if cfg.LinuxSockDGRAM { filename = "packet-socket" typ = unix.SOCK_DGRAM } sock, err := unix.Socket(unix.AF_PACKET, typ, 0) ... f := os.NewFile(uintptr(sock), filename) sc, err := f.SyscallConn() ... n, addr, err = unix.Recvfrom(int(fd), p, flags) github.com/mdlayher/raw/raw_linux.go
  100. 100. digitalocean.comGO FEATURE // Try to find an available BPF device for i := 0; i <= 10; i++ { bpfPath := fmt.Sprintf( "/dev/bpf%d", i) f, err = os.OpenFile(bpfPath, os.O_RDWR, 0666) if err == nil { break } if perr, ok := err.(*os.PathError); ok { if perr.Err.(syscall.Errno) == syscall.EBUSY { continue } } return nil, err } github.com/mdlayher/raw/raw_bsd.go
  101. 101. digitalocean.comGO FEATURE syscall os golang.org/x/sys/unix
  102. 102. digitalocean.com ifi, err := net.InterfaceByName(iface) if err != nil { return nil, err } pc, err := raw.ListenPacket(ifi, uint16(ethernet.EtherTypeIPv4), &raw.Config{ // Don't set any timeouts to avoid syscall busy // loops, since this server will run forever anyway. NoTimeouts: true, }) 1. Open and bind raw socket
  103. 103. digitalocean.com b := make([]byte, 1500) for { n, from, err := s.pc.ReadFrom(b) if err != nil { ... continue } buf := make([]byte, n) copy(buf, b) workC <- request{ buf: buf, from: from, } } 2. Read data into buffer
  104. 104. digitalocean.com var workWG sync.WaitGroup workWG.Add(Workers) workC := make(chan request, Workers) for i := 0; i < Workers; i++ { go func() { defer workWG.Done() for r := range workC { s.serve(r.buf, r.from) } }() } 3. Worker handles request
  105. 105. digitalocean.com func (h *handler) serveDHCPv4(ctx context.Context, req *dhcp4.Packet, from *dhcp4conn.Addr) (*dhcp4.Packet, net.Addr) { ... if !from.EthernetSource.Equal(reqMAC) { ... return h.shouldNAK(req.Type, to, broadcast) } ... } 4. Validate request
  106. 106. digitalocean.com func (h *handler) serveDHCPv4(ctx context.Context, req *dhcp4.Packet, from *dhcp4conn.Addr) (*dhcp4.Packet, net.Addr) { ... res, err := h.buildResponse(ctx, params, req, from) if err != nil { topError(span, err, "failed to build response") return h.shouldNAK(req.Type, to, broadcast) } res.Broadcast = broadcast return res, to ... } 5. Build and send response
  107. 107. digitalocean.com DHCP Summary: 1. DHCP: dynamic host configuration protocol used to dynamically assign IP address. 2. Use raw sockets. 3. Jump down to link-layer to do source MAC validation.
  108. 108. digitalocean.com Conclusion
  109. 109. digitalocean.com Go can be good for building networking primitives.
  110. 110. digitalocean.com
  111. 111. digitalocean.com Layer 7: net/http package
  112. 112. digitalocean.com Layer 4: net package
  113. 113. digitalocean.com Layer 3/2: raw package os,syscall golang.org/x/sys/unix
  114. 114. digitalocean.com sync package github.com/oklog/run package goroutines channels
  115. 115. THANKS :) @mdlayher @juliusvolz @nicboul
  116. 116. digitalocean.com Ohai! @snehainguva sneha.inguva@gmail.com
  117. 117. digitalocean.com Sources ● Port Scanner: https://github.com/si74/portscanner ● Layer 7 loadbalancer: https://github.com/si74/layer7lb ● Layer 4 loadbalancer: https://github.com/si74/tcpproxy ● Raw package: https://github.com/mdlayher/raw ● Godocs: https://godoc.org/ ● Golang: https://godoc.org/

×