Recently the interest in concurrent programming has grown dramatically. Unfortunately, parallel programs do not always have reproducible behavior. Even when they are run with the same inputs, their results can be radically different. In this talk I’ll show how to debug concurrency programs in Go. I’ll start from showing how you can debug your gorotines using delve and gdb debuggers. Then I’ll try to visualize goroutines using different scenarios, sometimes it helps to better understand how things work. Next part of the topic will be about dumping a goroutine stack trace of your application while it’s running and inspect what each goroutine is doing. And I’ll demonstrate how to debug leaking goroutines by tracing the process of how the scheduler runs goroutines on logical processors which are bound to a physical processor via the operating system thread that is attached. As a bonus i’ll cover debugging tips on how to find deadlocks and how to avoid race conditions in your application.

1. Debugging concurrent
programs in Go

2. Andrii Soldatenko
• Gopher, OSS
contributor
• father of 👧 👶
• debuggers fan
• Speaker at many
conferences
@a_soldatenko

3. Concurrency Programming
is Challenging!

4. Debugging
concurrent
programs is Hard!

5. @a_soldatenko
Debugging sequential
programs
dlv test -- -test.run TestFibonacciBig
(dlv) b main_test.go:6
Breakpoint 1 set at 0x115887f for github.com/andriisoldatenko/
debug_test.TestFibonacciBig() ./main_test.go:6
(dlv) c
> github.com/andriisoldatenko/debug_test.TestFibonacciBig() ./
main_test.go:6 (hits goroutine(17):1 total:1) (PC: 0x115887f)
1: package main
2:
3: import "testing"
4:
5: func TestFibonacciBig(t *testing.T) {
=> 6: var want int64 = 55
7: got := FibonacciBig(10)
8: if got.Int64() != want {
9: t.Errorf("Invalid Fibonacci value for N: %d, got: %d,
want: %d", 10, got.Int64(), want)
10: }
11: }
(dlv)

6. @a_soldatenko
Debugging concurrent
programs
package main
import (
"fmt"
"time"
)
func say(s string) {
for i := 0; i < 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
}
func main() {
go say("world")
say("hello")
}
gowayfest git:(master) ✗ go run main.go
hello
world
world
hello
hello
world
world
hello
world
hello
gowayfest git:(master) ✗ go run main.go
world
hello
hello
world
hello
world
world
hello
hello
world

8. What is a goroutine?
https://tpaschalis.github.io/goroutines-size/

9. Do you know why
gorotines?

10. How can I debug
concurrent Go program?

11. Playground
GOMAXPROCS is 1

12. visualize goroutines🐞 ?
func main() {
c := coloredgoroutine.Colors(os.Stdout)
fmt.Fprintln(c, "Hi, I am go routine", goid.ID(), "from main routine")
count := 10
var wg sync.WaitGroup
wg.Add(count)
for i := 0; i < count; i++ {
i := i
go func() {
fmt.Fprintln(c, "Hi, I am go routine", goid.ID(), "from
loop i =", i)
wg.Done()
}()
}
wg.Wait()
}

13. visualize goroutines🐞 ?
https://github.com/xiegeo/coloredgoroutine

14. visualize goroutines🐞 ?
https://divan.dev/posts/go_concurrency_visualize/

15. Print scheduling events?
$ GODEBUG=schedtrace=5000 <binary>

16. scheduling events

17. @a_soldatenko
- delve
- gdb
using debuggers:

18. Advanced debugging techniques
of Go code

19. @a_soldatenko
How to set breakpoint
inside goroutine?package main
import (
"fmt"
)
func say(s string, r chan string) {
fmt.Println(s)
r <- s
}
func main() {
chan1 := make(chan string)
chan2 := make(chan string)
go say("world", chan1)
go say("hello", chan2)
res1 := <-chan1
res2 := <-chan2
fmt.Printf("Channel 1: %snChannel 2: %sn", res1, res2)
}

20. @a_soldatenko
> main.say() ./main.go:9 (hits goroutine(7):1 total:1) (PC:
0x10c46c9)
4: "fmt"
5: )
6:
7: func say(s string, r chan string) {
8: fmt.Println(s)
=> 9: r <- s
10: }
11:
12: func main() {
13: chan1 := make(chan string)
14: chan2 := make(chan string)
How to set breakpoint
inside goroutine?

21. @a_soldatenko

22. @a_soldatenko
How to debug channel?
package main
func main() {
ch := make(chan int, 4)
ch <- 1
ch <- 2
ch <- 3
ch <- 4
close(ch)
}

23. @a_soldatenko

24. @a_soldatenko
How to debug channel?(dlv) p ch
chan int {
qcount: 0,
dataqsiz: 4,
buf: *[4]int [0,0,0,0],
elemsize: 8,
closed: 0,
elemtype: *runtime._type {size: 8, ptrdata: 0, hash: 4149441018, tflag: tflagUncommon|tflagExtraStar|
tflagNamed|tflagRegularMemory (15), align: 8, fieldAlign: 8, kind: 2, equal: runtime.memequal64, gcdata: *1,
str: 663, ptrToThis: 22432},
sendx: 0,
recvx: 0,
recvq: waitq<int> {
first: *sudog<int> nil,
last: *sudog<int> nil,},
sendq: waitq<int> {
first: *sudog<int> nil,
last: *sudog<int> nil,},
lock: runtime.mutex {key: 0},}
(dlv) n
> main.main() ./main_ch.go:7 (PC: 0x105e8d9)
2:
3:
4: func main() {
5: ch := make(chan int, 4)
6: ch <- 1
=> 7: ch <- 2
8: ch <- 3
9: ch <- 4
10: close(ch)
11: }
(dlv) p ch

25. @a_soldatenko
dlv send to channel value
similar to set variable.

26. @a_soldatenko
GDB
t.me/golang_for_two

27. @a_soldatenko
Gdb and golang
go build -ldflags=-compressdwarf=false -
gcflags=all="-N -l" -o main main.go

28. @a_soldatenko
Gdb and goroutines

29. @a_soldatenko
Gdb and goroutines

30. @a_soldatenko
Deadlocks happen and are
painful to debug.

31. @a_soldatenko
How to detect deadlocks
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
/Users/andrii/workspace/src/github.com/andriisoldatenko/
gowayfest/main_deadlock.go:5 +0x50
exit status 2

32. @a_soldatenko
Real world examples
complicated scenario & tools
https://github.com/sasha-s/go-deadlock
https://github.com/cockroachdb/cockroach/issues/7972

33. @a_soldatenko

34. @a_soldatenko
7 simple rules for debugging
concurrency applications
- Never assume a particular order of execution
- Implement concurrency at the highest level possible
- Don’t forget Go only detects when the program as a
whole freezes, not when a subset of goroutines get
stuck.
- STRACE

35. @a_soldatenko
7 simple rules for debugging
concurrency applications
- conditional breakpoints your best friend
- DEBUG=schedtrace=5000
- go-deadlock

36. @a_soldatenko
References 1
- https://github.com/golang/go/blob/release-
branch.go1.14/src/runtime/HACKING.md
- https://github.com/golang/go/wiki/LearnConcurrency
- https://rakyll.org/go-cloud/
- https://yourbasic.org/golang/detect-deadlock/

37. @a_soldatenko
References 2
- https://blog.minio.io/debugging-go-routine-leaks-
a1220142d32c
- https://golang.org/src/cmd/link/internal/ld/dwarf.go
- https://golang.org/src/runtime/runtime-gdb.py
- https://cseweb.ucsd.edu/~yiying/GoStudy-
ASPLOS19.pdf
- https://golang.org/doc/articles/race_detector.html

38. Telegram channel
https://t.me/golang_for_two

39. Slides:
@a_soldatenko

40. Thank You
@a_soldatenko

41. Questions ?
@a_soldatenko