Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CGo for fun and profit

38 views

Published on

Go is, for all its C-esque syntax, is a relatively high-level language. There’s garbage collection, a type hierarchy, and even advanced concurrency primitives. Sometimes, though, you need access to lower-level control. Whether for performance, interoperability, or just for fun, the Cgo API is here to help! I will explain what it is, how to use it, and perhaps most importantly, when to avoid it.

A lot of Go programmers only experience with Cgo is when compilation fails with some obscure error. There’s a lot more to learn than that, and a lot of opportunities! I’ve personally used Cgo for writing Postgres foreign wrappers, and as a consumer when compiling Kubernetes.

Using Cgo can be a fun, rewarding experience. There are a lot of existing projects and libraries written in C, and calling them means not needing to reinvent the wheel. On the same face, if you have an existing project with a C API, calling Go can get you some of the higher-level Go niceness on top of the jagged C edges.

Some things I touch on:
* Calling Go from C
* Calling C from Go
* Memory management with Go objects in C
* Using C libraries from Go
* Real-world Cgo examples
* The pitfalls Go and dynamic linking

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

CGo for fun and profit

  1. 1. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Foreign Functions for Fun and Profit When and how to use CGo
  2. 2. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta github.com/liztio/cgo-demo
  3. 3. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta
  4. 4. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta
  5. 5. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta @stillinbeta
  6. 6. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta
  7. 7. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta recurse.com
  8. 8. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Renee French. (http://reneefrench.blogspot.com/)
  9. 9. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Full disclosure? ...I don’t actually like Go very much.
  10. 10. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta but I use it a lot
  11. 11. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Why not Cgo?
  12. 12. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta cgo is not Go - Dave Cheney bit.ly/cgo-not-go
  13. 13. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Why not Cgo ● More complicated compilation ● Less portable ● No cross compilation ● More segfaults ● No backtraces ● Less tooling
  14. 14. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Why Cgo ● ...sometimes it’s the only way
  15. 15. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Calling C from Go
  16. 16. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta #include <stdint.h> uint32_t add_one(uint32_t val); addlib.h #include "addlib.h" uint32_t add_one(uint32_t val) { return val + 1; } addlib.c
  17. 17. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta uint32_t Unsigned integer 32 bit type
  18. 18. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta import "C"
  19. 19. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta // #include "addlib.h" import "C"
  20. 20. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta added := C.add_one(num)
  21. 21. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta main.go package main // #include "addlib.h" import "C" // “C” always gets its own line import "fmt" func main() { num := 10 added := C.add_one(num) fmt.Printf("%d + 1 is %d!n", num, added) }
  22. 22. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ go run github.com/liztio/cgo-demo/cmd/c_from_go
  23. 23. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ go run github.com/liztio/cgo-demo/cmd/c_from_go # github.com/liztio/cgo-demo/cmd/c_from_go cmd/c_from_go/main.go:10:27: cannot use num (type int) as type _Ctype_uint in argument to _Cfunc_add_one
  24. 24. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta added := C.add_one(C.uint32_t(num))
  25. 25. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ go run github.com/liztio/cgo-demo/cmd/c_from_go 10 + 1 is 11!
  26. 26. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Calling Go from C
  27. 27. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta //export Greet func Greet(chars *C.char) { str := C.GoString(chars) fmt.Printf("Hello from Go, %s!n", str) }
  28. 28. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta //export Greet func Greet(chars *C.char) { str := C.GoString(chars) fmt.Printf("Hello from Go, %s!n", str) }
  29. 29. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta //export Greet func Greet(chars *C.char) { str := C.GoString(chars) fmt.Printf("Hello from Go, %s!n", str) }
  30. 30. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta F O S D E M x00 Strings in C
  31. 31. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta F O S D E M x00 46 4F 53 44 45 4d 00 Strings in C
  32. 32. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta F O S D E M *ptr 6 Strings in Go
  33. 33. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta //export Greet func Greet(chars *C.char) { str := C.GoString(chars) fmt.Printf("Hello from Go, %s!n", str) }
  34. 34. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta package main import "C" import "fmt" //export Greet func Greet(chars *C.char) { str := C.GoString(chars) fmt.Printf("Hello from Go, %s!n", str) } func main() {} // required for main package greet.go
  35. 35. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta main.c #include "_cgo_export.h" int main() { char *name = "FOSDEM"; Greet(name); return 0; // exit code }
  36. 36. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta _cgo_export.h /* Code generated by cmd/cgo; DO NOT EDIT. */ /* package main */ #line 1 "cgo-builtin-prolog" #include <stddef.h> /* for ptrdiff_t below */ #ifndef GO_CGO_EXPORT_PROLOGUE_H #define GO_CGO_EXPORT_PROLOGUE_H typedef struct { const char *p; ptrdiff_t n; } _GoString_; #endif /* Start of preamble from import "C" comments. */ /* End of preamble from import "C" comments. */ /* Start of boilerplate cgo prologue. */
  37. 37. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta _cgo_export.h /* don't worry about it */
  38. 38. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta A brief introduction to Makefiles
  39. 39. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta A brief introduction to Makefiles all: output output: input1 input2 input3 command --output $@ --inputs $^
  40. 40. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta A brief introduction to Makefiles all: output output: input1 input2 input3 command --output output --inputs input1 input2 input3
  41. 41. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Our Makefile _cgo_export.h: greet.go go tool cgo -exportheader $@ $^ greet.a: greet.go go build -buildmode=c-archive $^ gofromc: main.c _cgo_export.h greet.a $(CC) -o $@ $^ -lpthread
  42. 42. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Our Makefile _cgo_export.h: greet.go go tool cgo -exportheader $@ $^
  43. 43. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Our Makefile greet.a: greet.go go build -buildmode=c-archive $^
  44. 44. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Our Makefile gofromc: main.c _cgo_export.h greet.a $(CC) -o $@ $^ -lpthread
  45. 45. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ make go tool cgo -exportheader _cgo_export.h greet.go go build -buildmode=c-archive greet.go cc -o gofromc main.c _cgo_export.h greet.a -lpthread
  46. 46. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Let’s try it out! $ ./gofromc Hello from Go, FOSDEM!
  47. 47. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Pointers and Memory
  48. 48. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Go Memory ● new() / &value ● Garbage Collected ● malloc() ● Manually Freed with free() C Memory
  49. 49. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Allocating in Go type Point struct { x, y float32 } func getPoint() *Point { return &Point{ x: 10, y: 12, } } typedef struct { float x; float y; } Point; Point *getPoint() { Point *pt = malloc(sizeof(Point)); pt->x = 10; pt->y = 12; return pt; } Allocating in C
  50. 50. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Go func usePoint(pt *Point) { fmt.Printf("x: %f, y: %fn", pt.x, pt.y) // Goes out of scope } void usePoint(Point *pt) { printf("x: %f, y: %fn", pt->x, pt->y); free(pt); } C
  51. 51. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta What happens if you forget to free? $ ./gofromc x: 10.000000, y: 12.000000
  52. 52. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta What happens if you forget to free? $ valgrind --leak-check=full ./gofromc ==7974== Memcheck, a memory error detector ==7974== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==7974== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info ==7974== Command: ./gofromc ==7974== x: 10.000000, y: 12.000000 ==7974== ==7974== HEAP SUMMARY: ==7974== in use at exit: 8 bytes in 1 blocks ==7974== total heap usage: 2 allocs, 1 frees, 1,032 bytes allocated ==7974== ==7974== 8 bytes in 1 blocks are definitely lost in loss record 1 of 1 ==7974== at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==7974== by 0x10869B: getPoint (in /home/liz/src/github.com/liztio/cgo-demo/src/gofromc/gofromc) ==7974== by 0x10871C: main (in /home/liz/src/github.com/liztio/cgo-demo/src/gofromc/gofromc) ==7974== ==7974== LEAK SUMMARY: ==7974== definitely lost: 8 bytes in 1 blocks ==7974== indirectly lost: 0 bytes in 0 blocks ==7974== possibly lost: 0 bytes in 0 blocks ==7974== still reachable: 0 bytes in 0 blocks ==7974== suppressed: 0 bytes in 0 blocks ==7974== ==7974== For counts of detected and suppressed errors, rerun with: -v ==7974== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
  53. 53. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta What happens if you forget to free? $ valgrind --leak-check=full ./gofromc ==7974== Memcheck, a memory error detector ==7974== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==7974== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info ==7974== Command: ./gofromc ==7974== x: 10.000000, y: 12.000000 ==7974== ==7974== HEAP SUMMARY: ==7974== in use at exit: 8 bytes in 1 blocks ==7974== total heap usage: 2 allocs, 1 frees, 1,032 bytes allocated ==7974== ==7974== 8 bytes in 1 blocks are definitely lost in loss record 1 of 1 ==7974== at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==7974== by 0x10869B: getPoint (in /home/liz/src/github.com/liztio/cgo-demo/src/gofromc/gofromc) ==7974== by 0x10871C: main (in /home/liz/src/github.com/liztio/cgo-demo/src/gofromc/gofromc) ==7974== ==7974== LEAK SUMMARY: ==7974== definitely lost: 8 bytes in 1 blocks ==7974== indirectly lost: 0 bytes in 0 blocks ==7974== possibly lost: 0 bytes in 0 blocks ==7974== still reachable: 0 bytes in 0 blocks ==7974== suppressed: 0 bytes in 0 blocks ==7974== ==7974== For counts of detected and suppressed errors, rerun with: -v ==7974== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
  54. 54. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta What happens if you forget to free? $ valgrind --leak-check=full ./gofromc ==7974== 8 bytes in 1 blocks are definitely lost in loss record 1 of 1 ==7974== at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==7974== by 0x10869B: getPoint (in /home/liz/src/github.com/liztio/cgo-demo/src/gofromc/gofromc) ==7974== by 0x10871C: main (in /home/liz/src/github.com/liztio/cgo-demo/src/gofromc/gofromc)
  55. 55. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Passing Pointers between C and Go
  56. 56. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Okay ● Passing Go pointers to C ● Passing C pointers to Go ● Storing Go pointers in C memory before returning ● A bunch of weird special cases involving Java and Darwin ● Passing Go pointers with nested pointers to C ● Storing Go pointers in Go memory from C ● Storing Go pointers in C memory ● Storing Go pointers in C memory after returning ● Go Functions called by C returning Go pointers Invalid
  57. 57. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta tl;dr ● Pass pointers carefully ● Read the Cgo docs (golang.org/cmd/cgo)
  58. 58. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta How about something nontrivial?
  59. 59. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta How about a little magick?
  60. 60. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta #include <wand/magick_wand.h> int main() { MagickWand *mw = NULL; MagickWandGenesis(); /* Create a wand */ mw = NewMagickWand(); /* Read the input image */ MagickReadImage(mw,"logo:"); /* write it */ MagickWriteImage(mw,"logo.jpg"); /* Tidy up */ if(mw) mw = DestroyMagickWand(mw); MagickWandTerminus(); } https://imagemagick.org/MagickWand/
  61. 61. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta package main // #cgo pkg-config: MagickWand // #include <wand/MagickWand.h> import "C" func main() { C.MagickWandGenesis() /* Create a wand */ mw := C.NewMagickWand() defer func() { /* Tidy up */ C.DestroyMagickWand(mw) C.MagickWandTerminus() }() /* Read the input image */ C.MagickReadImage(mw, C.CString("logo:")) /* write it */ C.MagickWriteImage(mw, C.CString("logo.jpg")) }
  62. 62. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta what’s this? // #cgo pkg-config: MagickWand
  63. 63. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ pkg-config MagickWand --cflags -fopenmp -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -fopenmp -DMAGICKCORE_ $ pkg-config MagickWand --libs -lMagickWand-6.Q16 -lMagickCore-6.Q16
  64. 64. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Expands to // #cgo CFLAGS: -I/usr/include/ImageMagick-6 -I/usr/include/x86_64-linux-gnu/ImageMagick-6 // #cgo LDFLAGS: -lMagickWand-6.Q16 -lMagickCore-6.Q16
  65. 65. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta C defer anyone? defer func() { /* Tidy up */ C.DestroyMagickWand(mw) C.MagickWandTerminus() }()
  66. 66. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Full resizing demo in the Github Repo! github.com/liztio/cgo-demo/cmd/imagemagick
  67. 67. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Dynamic Libraries
  68. 68. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta “Go binaries are static”
  69. 69. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Windows OSX Linux .dll .dylib .so
  70. 70. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta int localFunc() { return 0; } int main() { return localFunc(); }
  71. 71. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta main 0x0000000000000605 localFunction 0x00000000000005fa mybinary int localFunc() { return 0; } int main() { return localFunc(); }
  72. 72. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta #include <lib1.h> #include <lib2.h> void localFunction() { char *str = lib1Func1(); lib2Func2(str); } int main() { localFunction() }
  73. 73. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta main.main 0x00000000004891d0 localFunction 0x0000000000489985 lib1func1 lib2func2 mybinary #include <lib1.h> #include <lib2.h> void localFunction() { char *str = lib1Func1(); lib2Func2(str); } int main() { localFunction() }
  74. 74. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta main.main 0x00000000004891d0 localFunction 0x0000000000489985 lib1func1 lib2func2 lib1func0 0x00000000004891d0 lib1func1 0x0000000000489985 lib1func2 0x0000000000489985 lib2func0 0x0000000000455ea0 lib2func1 0x000000000073f598 lib2func2 0x000000000073f598 lib1.so lib2.so mybinary
  75. 75. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta LDD(1) Linux Programmer's Manual LDD(1) NAME ldd - print shared object dependencies SYNOPSIS ldd [option]... file...
  76. 76. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ ldd ~/bin/ginkgo linux-vdso.so.1 (0x00007ffc71493000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa2d36c4000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa2d32d3000) /lib64/ld-linux-x86-64.so.2 (0x00007fa2d38e3000)
  77. 77. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ ldd ~/bin/ginkgo linux-vdso.so.1 (0x00007ffc71493000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa2d36c4000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa2d32d3000) /lib64/ld-linux-x86-64.so.2 (0x00007fa2d38e3000)
  78. 78. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta // #cgo LDFLAGS: -laddlib // #include "addlib.h" import "C"
  79. 79. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ go install github.com/liztio/cgo-demo/cmd/addbin
  80. 80. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ export ADDLIB_DIR=/home/liz/src/github.com/liztio/cgo-demo/src/addlib $ export CGO_CFLAGS=-I$ADDLIB_DIR $ export CGO_LDFLAGS=-L$ADDLIB_DIR $ go install github.com/liztio/cgo-demo/cmd/addbin
  81. 81. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ export ADDLIB_DIR=/home/liz/src/github.com/liztio/cgo-demo/src/addlib $ export CGO_CFLAGS=-I$ADDLIB_DIR $ export CGO_LDFLAGS=-L$ADDLIB_DIR $ go install github.com/liztio/cgo-demo/cmd/addbin
  82. 82. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ ldd addbin linux-vdso.so.1 (0x00007ffd2b966000) libaddlib.so => not found libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f252a003000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2529c12000) /lib64/ld-linux-x86-64.so.2 (0x00007f252a222000)
  83. 83. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ ldd addbin linux-vdso.so.1 (0x00007ffd2b966000) libaddlib.so => not found libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f252a003000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2529c12000) /lib64/ld-linux-x86-64.so.2 (0x00007f252a222000)
  84. 84. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ addbin addbin: error while loading shared libraries: libaddlib.so: cannot open shared object file: No such file or directory
  85. 85. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta $ LD_LIBRARY_PATH=$ADDLIB_DIR addbin Go says: 11 + one is 12
  86. 86. github.com/liztio/cgo-demo // #FOSDEM 2019 // @stillinbeta Questions? Please play with the code! github.com/liztio/cgo-demo

×