Advanced debugging  techniques in different environments

Advanced debugging
techniques in different
environments

Andrii Soldatenko
• Contributor in Apache Airﬂow,
Golang, Pyhelm, python-sendgrid,
tutorial of aiohttp, ansible, requests,
etc.
• Speaker at many conferences
• blogger at t.me/golang_for_two
@a_soldatenko

Problem: how to debug my
program?

Debugging in practise:
package main
import (
"fmt"
)
func main() {
const name, age = "Andrii", 22
fmt.Println(name, "is", age, "years old.")
}

@a_soldatenko
Interesting fact:
Go version prior 1.0
included debugger OGLE

Current versions of Go
produce DWARF
Debugging Standard
http://dwarfstd.org/

@a_soldatenko
- delve
- GDB
Current state of go
debuggers:

Debug CLI util
.
├── github.com/me/foo
├── cmd
│ └── foo
│ └── main.go
├── pkg
│ └── baz
│ ├── bar.go
│ └── bar_test.go
$ dlv debug github.com/me/foo/cmd/foo -- -arg1
valueType
(dlv) break main.main
Breakpoint 1 set at 0x49ecf3 for main.main() ./test.go:5
(dlv) continue

@a_soldatenko
Debugging specific
unit test
go test -test.run TestFibonacciBig
dlv test -- -test.run TestFibonacciBig
t.me/golang_for_two

@a_soldatenko
Conditional breakpoint
package main
import "fmt"
func main() {
nums := []int{2, 3, 4}
for i, num := range nums {
if num == 3 {
fmt.Println(“BUG!!! index:”, i)
}
}
}
t.me/golang_for_two

Conditional breakpoint
(dlv) b b2 hello.go:8
Breakpoint b2 set at 0x10b71e2 for main.main() ./hello.go:8
(dlv) cond b2 num == 3
(dlv) c
> [b2] main.main() ./hello.go:8 (hits goroutine(1):1 total:
1) (PC: 0x10b71e2)
3: import "fmt"
4:
5: func main() {
6: nums := []int{2, 3, 4}
7: for i, num := range nums {
=> 8: if num == 3 {
9: fmt.Println("index:", i)
10: }
11: }
12: }
(dlv) p num
3
(dlv) n

@a_soldatenko
Debugging unit tests:
example
dlv test -- -test.run TestFibonacciBig
(dlv) b main_test.go:6
Breakpoint 1 set at 0x115887f for github.com/andriisoldatenko/
debug_test.TestFibonacciBig() ./main_test.go:6
(dlv) c
> github.com/andriisoldatenko/debug_test.TestFibonacciBig() ./
main_test.go:6 (hits goroutine(17):1 total:1) (PC: 0x115887f)
1: package main
2:
3: import "testing"
4:
5: func TestFibonacciBig(t *testing.T) {
=> 6: var want int64 = 55
7: got := FibonacciBig(10)
8: if got.Int64() != want {
9: t.Errorf("Invalid Fibonacci value for N: %d, got: %d,
want: %d", 10, got.Int64(), want)
10: }
11: }
(dlv)
t.me/golang_for_two

Debugging unit tests:
breakpoint
dlv test github.com/andriisoldatenko/debugging-containerized-go-
applications/
Type 'help' for list of commands.
(dlv) b TestCloud1
Breakpoint 1 set at 0x113efaf for github.com/andriisoldatenko/
debugging-containerized-go-applications.TestCloud1() ./
hello_test.go:16
(dlv) c
TestMy1
TestYour1
> github.com/andriisoldatenko/debugging-containerized-go-
applications.TestCloud1() ./hello_test.go:16 (hits goroutine(33):1
total:1) (PC: 0x113efaf)
11:
12: func TestYour1(t *testing.T) {
13: fmt.Println("TestYour1")
14: }
15:
=> 16: func TestCloud1(t *testing.T) {
17: fmt.Println("TestCloud1")
18: }

@a_soldatenko
How to set variable?
4:
=> 5: func main() {
6: a := 1
7: b := 2
8: fmt.Println(a, b)
9: }
(dlv) set a = 10
t.me/golang_for_two

Call function
(dlv) call myFunc(localVar, 4, "foo")
https://golang.org/cl/109699 runtime: support for debugger
function calls
Add ability to safely call functions #119 (go-delve/delve/issues/119)

Call function
(dlv) call t()
> main.dummy() ./hello.go:18 (hits
goroutine(1):1 total:1) (PC:
0x10b732d)
13: }
14: }
15:
16:
17: func dummy(){
=> 18: fmt.Println("func call")
19: }

@a_soldatenko
Debugging containerized
go apps
t.me/golang_for_two

Step 1: Dockerfile
$ cat Dockerfile
FROM golang:1.13
WORKDIR /go/src/app
COPY . .
RUN go get -u github.com/go-delve/delve/cmd/dlv
CMD ["app"]
$ docker build -t my-golang-app .
$ docker run -it --rm my-golang-app bash

@a_soldatenko
Step 2: operation not
permitted
t.me/golang_for_two
$ docker run -it --rm my-golang-app bash
$root@03c1977b1063:/go/src/app# dlv debug
main.go
could not launch process: fork/exec /go/src/app/
__debug_bin: operation not permitted

@a_soldatenko
Step 3: :tada:
t.me/golang_for_two
$ docker run -it --rm
—security-opt="apparmor=unconfined"
—cap-add=SYS_PTRACE my-golang-app bash
root@7dc3a7e8b3fc:/go/src/app# dlv debug
main.go
Type 'help' for list of commands.
(dlv)

@a_soldatenkot.me/golang_for_two
Removing debugging
information from the binary
go build ldflags=-w …

Remote debugging
containerized go apps

@a_soldatenko
LIVE DEMO A
t.me/golang_for_two

# Final stage
FROM alpine:3.7
# Port 8080 belongs to our application, 40000 belongs to Delve
EXPOSE 8080 40000
# Allow delve to run on Alpine based containers.
RUN apk add --no-cache libc6-compat
WORKDIR /
COPY --from=build-env /server /
COPY --from=build-env /go/bin/dlv /
# Run delve
CMD ["/dlv", "--listen=0.0.0.0:40000", "--
headless=true", "--api-version=2", "debug", "go/
src/hello", "--log"
Example of Dockerfile

#!/usr/bin/env bash
docker run
-p 8080:8080
-p 40000:40000
--security-opt="apparmor=unconfined"
--cap-add=SYS_PTRACE hello-debug
Docker run

$cat ~/.dlv/config.yml
substitute-path:
- {from: "/go/src/hello/", to: "/Users/
andrii/workspace/src/github.com/
andriisoldatenko/debugging-containerized-
go-applications"}
Let’s fix it to more
readable

Configure Delve
$HOME/.dlv/conﬁg.yml
Or
(dlv) conﬁg -list

dlv connect localhost:40000
(dlv) c
> main.main() /go/src/hello/hello.go:5 (hits
goroutine(1):1 total:1) (PC: 0x49b3d8)
1: package main
2:
3: import "fmt"
4:
=> 5: func main() {
6: a := 1
7: b := 2
8: fmt.Println(a, b)
9: }
(dlv)
Delve connect

@a_soldatenko
LIVE DEMO B
t.me/golang_for_two

@a_soldatenko
GDB
t.me/golang_for_two

Starting program: /x/y/foo
Unable to find Mach task port for process-id 28885: (os/kern) failure
(0x5).
(please check gdb is codesigned - see taskgated(8))
https://sourceware.org/gdb/wiki/PermissionsDarwin
check gdb is codesigned

@a_soldatenko
[New Thread 0xe03 of process 45828]
[New Thread 0x1003 of process 45828]
warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/
Objects-normal/x86_64/bsd.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/darwin_vers.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/dirstat.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/dirstat_collection.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/err.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/exception.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/init.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/mach.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/stdio.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/stdlib.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/string.o': can't open to read symbols: No such file or directory.
Objects-normal/x86_64/variant.o': can't open to read symbols: No such file or directory.
Thread 2 hit Breakpoint 1, 0x0000000100000fa4 in main ()
(gdb) x/i $rip
=> 0x100000fa4 <main+4>: xor %eax,%eax
(gdb) quit
A debugging session is active.
Inferior 1 [process 45828] will be killed.
Quit anyway? (y or n) y
can't open to read
symbols:

@a_soldatenko
GDB: few notes!
(gdb) b main.main
Breakpoint 1 at 0x10b70f0
(gdb) c
The program is not being run.
(gdb) run
Starting program: /Users/andrii/workspace/src/
github.com/andriisoldatenko/debugging-
containerized-go-applications/hello
[New Thread 0xf03 of process 5994]
^C^Z
[2] + 5911 suspended gdb hello

@a_soldatenko
GDB: few notes!
In Go 1.11, the debug information is compressed
for purpose of reduce binary size, and gdb on
the Mac does not understand compressed DWARF.
(link)

@a_soldatenko
GDB: few notes!
go build -ldflags=-compressdwarf=false -
gcflags=all="-N -l" -o hello hello.go
➜ go git:(master) find . -name 'runtime-gdb.py'
./src/runtime/runtime-gdb.py
(gdb) source /Users/andrii/work/go/src/runtime/
runtime-gdb.py
Loading Go Runtime support.
➜ debugging-containerized-go-applications git:
(master) ✗ strings hello | grep gdb
/usr/local/Cellar/go/1.12.7/libexec/src/runtime/
runtime-gdb.py

@a_soldatenko
GDB: few notes!
(gdb) n
6 a := 1
(gdb) n
7 b := 2
(gdb) n
8 fmt.Println(a, b)
(gdb) n
1 2
9 }
(gdb)

@a_soldatenko
GDB: internals of Go
slices

@a_soldatenko
Conclusion GDB:
- GDB does not understand Go programs
well.
- The stack management, threading, and
runtime contain aspects that differ enough
from the execution model GDB expects
- GDB can be useful in some situations (e.g.,
debugging Cgo code

@a_soldatenko
Future Reading:
- Internal Architecture of Delve - slides
- t.me/golang_for_two

Telegram channel
https://t.me/golang_for_two

Questions
? @a_soldatenkot.me/golang_for_two

Advanced debugging  techniques in different environments

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Advanced debugging  techniques in different environments

Similar to Advanced debugging  techniques in different environments (20)

More from Andrii Soldatenko

More from Andrii Soldatenko (12)

Recently uploaded

Recently uploaded (20)

Advanced debugging  techniques in different environments