Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Static Analysis in Go

2,995 views

Published on

The slides are for GolangUK Conference 2017.
https://www.golanguk.com/

Published in: Technology
  • Be the first to comment

Static Analysis in Go

  1. 1. The Go gopher was designed by Renée French. The gopher stickers was made by Takuya Ueda. Licensed under the Creative Commons 3.0 Attributions license. Static Analysis in Go @GolangUK Conference 18th Aug. 2017 1 Published on : https://goo.gl/ZFuc7R
  2. 2. Who am I? Takuya Ueda @tenntenn ➔ Work for & 2
  3. 3. Mercari Europe 3 https://www.mercari.com/uk/
  4. 4. Tech Commnities Go Beginners in Tokyo Go Conference in Tokyo 4
  5. 5. Agenda ➔ Where’s Gopher? ➔ Static Analysis ➔ Static Analysis in Go ➔ Static Analysis for Products 5
  6. 6. Where’s Gopher? Find Me!! Powered by https://gopherize.me 6
  7. 7. Where’s Gopher? Find Me!! Powered by https://gopherize.me 7
  8. 8. Where’s “Gopher”? type Gopher struct { Gopher string `json:"gopher"` } func main() { const gopher = "GOPHER" gogopher := GOPHER() gogopher.Gopher = gopher fmt.Println(gogopher) } func GOPHER() (gopher *Gopher) { gopher = &Gopher{ Gopher: "gopher" } return } 8
  9. 9. We love grep $ grep Gopher main.go type Gopher struct { Gopher string `json:"gopher"` } gogopher.Gopher = gopher func GOPHER() (gopher *Gopher) { gopher = &Gopher{ Gopher: "gopher" } We can search “Gopher” with grep. 9
  10. 10. Where’s “Gopher” TYPE? type Gopher struct { Gopher string `json:"gopher"` } func main() { const gopher = "GOPHER" gogopher := GOPHER() gogopher.Gopher = gopher fmt.Println(gogopher) } func GOPHER() (gopher *Gopher) { gopher = &Gopher{ Gopher: "gopher" } return } 10
  11. 11. How to search “Gopher” type ➔ grep can search only by text ➔ The text must be understood as Go source code ➔ We need “Static Analysis” 11
  12. 12. Static Analysis 12
  13. 13. Static Analysis & Dynamic Analysis ➔ Static Analysis ◆ Analyzing a program WITHOUT execution ◆ Analyzing structure of a program from source codes ◆ e.g. linter, code complement, code formatter ➔ Dynamic Analysis ◆ Analyzing a program WITH execution ◆ Investigating variables and result of functions ◆ e.g. race detector 13
  14. 14. Reflection ➔ Reflection ◆ Analyzing values and types at runtime. ➔ Reflection in Go ◆ reflect package ◆ It is only way to get struct tags at runtime ◆ encoding package uses reflection to encode/decode JSONs, XMLs and so on. 14
  15. 15. Static Analysis Tools for Go ➔ There are many static analysis tools for Go gofmt/goimports code formatter go vet/golint code checker, linter guru code comprehension tool gocode code complement tool errcheck error handlings checker gorename/gomvpkg refactoring tools 15
  16. 16. Go for Static Analysis ➔ Go is easy to static analyze ◆ Static typing ◆ Simple syntax ◆ Type inference ◆ No Implicit type conversion ◆ go package is provided as a standard package Static Analysis gives a lot of information 16
  17. 17. Sub packages of go package ast Package ast declares the types used to represent syntax trees for Go packages. build Package build gathers information about Go packages. constant Package constant implements Values representing untyped Go constants and their corresponding operations. doc Package doc extracts source code documentation from a Go AST. format Package format implements standard formatting of Go source. importer Package importer provides access to export data importers. parser Package parser implements a parser for Go source files. printer Package printer implements printing of AST nodes. scanner Package scanner implements a scanner for Go source text. token Package token defines constants representing the lexical tokens of the Go programming language and basic operations on tokens (printing, predicates). types Package types declares the data types and implements the algorithms for type-checking of Go packages. 17
  18. 18. Static Analysis in Go 18
  19. 19. Flow of Static Analysis Source Code Token Abstract Syntax Tree (AST) Type Info Parse Tokenize Type Check go/scanner go/token go/parser go/ast go/types go/constant 19
  20. 20. Tokenization - go/scanner,go/token IDENT ADD INT Tokens Source Code: v + 1 ➔ Dividing input string into tokens 20
  21. 21. Parsing - go/parser,go/ast ➔ Converting tokens to abstract syntax tree v + 1 IDENT ADD INT Source Code: + v 1 BinaryExpr Ident BasicLit Tokens: Abstract Syntax Tree: (AST) 21
  22. 22. AST of “Hello, World” package main import "fmt" func main() { fmt.Println("Hello, 世界") } Run on Playground *ast.File []ast.Decl *ast.GenDecl *ast.FuncDecl 22
  23. 23. Type Check - go/types,go/constant ➔ Extract type infomation from AST ◆ Identifier Resolution ◆ Type Detection ◆ Constant Evalution n := 100 + 200 m := n + 300 Constant Evalution = 300 Type Detection -> int Identifier Resolution 23
  24. 24. Getting AST from source code ➔ Use parser.Parse* function ◆ ParseExpr,ParseExprFrom ● Parsing expression ● ParseExpr is simple version of ParseExprFrom ◆ ParseFile ● Parsing a file ◆ ParseDir ● Parsing files in a directory ● Calling ParseFile in the function 24
  25. 25. Getting AST from an expression expr, err := parser.ParseExpr(`v + 1`) if err != nil { /* handling the error */ } /* use expr */ ➔ Parsing an expression 25
  26. 26. Getting AST from a file const src = ` package main var v = 100 func main() { fmt.Println(v+1) }` fs := token.NewFileSet() f, err := parser.ParseFile(fs, "my.go", src, 0) if err != nil { /* handling the error */ } /* use f */ file name which is opened when src is nil Source Code Parsing Mode 26
  27. 27. token.FileSet ➔ Recording positions on files ◆ The position (token.Pos) is represented by integer ◆ The value is unique among multiple files ◆ token.FileSet holds offset of each files ◆ The offsets are recorded at parsing time ◆ token.FileSet is passed to parsing function as an output parameter type Pos int 27
  28. 28. Demo 1: Parse and Dump an AST 28 https://youtu.be/lM1Pj6xYxZs Demo Code: https://gist.github.com/tenntenn/833207ce1715089da23936b5dfaebea5
  29. 29. Traversing AST - ast.Inspect ➔ Using ast.Inspect expr, _ := parser.ParseExpr(`v + 1`) ast.Inspect(expr, func(n ast.Node) bool { if n != nil { fmt.Printf("%Tn", n) } return true }) Traverse the AST *ast.BinaryExpr *ast.Ident *ast.BasicLit Run on Playground ast.Walk is more powerful and complex + v 1 BinaryExpr Ident BasicLit 29
  30. 30. Traversing AST - Recursively func traverse(n ast.Node) { switch n := n.(type) { case *ast.Ident: fmt.Println(n.Name) case *ast.BinaryExpr: traverse(n.X) traverse(n.Y) case *ast.UnaryExpr: traverse(n.X) default: fmt.Println(n) } } print idetifyer’s name traverse each terms recursively switching by types Run on Playground traverse a term recursively 30
  31. 31. Demo 2: Inspect identifiers 31 https://youtu.be/mpUgaaASvHo Demo Code: https://gist.github.com/tenntenn/299c226c540fccd8386b7148f55f0884
  32. 32. Type Checking /* initialize configs for type checking */ cfg := &types.Config{Importer: importer.Default()} info := &types.Info{ /* TODO: initialize maps which will hold results */ } pkg,err := cfg.Check("main", fs, []*ast.File{f}, info) if err != nil { /* handling the error */ } /* TODO: use pkg or info */ ➔ (*types.Config).Check do a type checking 32
  33. 33. types.Info holds results of type checking type Info struct { // Types maps expressions to their types, and for constant // expressions, also their values. Types map[ast.Expr]TypeAndValue // Defs maps identifiers to the objects they define. Defs map[*ast.Ident]Object // Uses maps identifiers to the objects they denote. Uses map[*ast.Ident]Object // Implicits maps nodes to their implicitly declared objects, if any. Implicits map[ast.Node]Object // Selections maps selector expressions (excluding qualified identifiers) // to their corresponding selections. Selections map[*ast.SelectorExpr]*Selection // Scopes maps ast.Nodes to the scopes they define. Scopes map[ast.Node]*Scope // InitOrder is the list of package-level initializers in the order in which // they must be executed. InitOrder []*Initializer } 33
  34. 34. Demo 3: Inspect Gopher type 34 https://youtu.be/AuSDtmiMaXI Demo Code: https://gist.github.com/tenntenn/134d6965f75efe5eb6f6edf7b0ebe509
  35. 35. Static Analysis for Products Case 1: gpath,go-httpdoc 35
  36. 36. go.mercari.io/go-httpdoc ➔ Generate API Docs from tests of handlers ◆ Just write tests of HTTP handlers ◆ gRPC and JSON RPC are supported 36
  37. 37. Tests for requests and responses ➔ Requests (Responses) can be test by specifying fields and expected values, document for each fields validator.RequestBody(t, []httpdoc.TestCase{ {"Name", "tenntenn", "User Name"}, {"Attribute.Email", "tenntenn@example.com", "e-mail address"}, }, &createUserRequest{}) 37
  38. 38. github.com/tenntenn/gpath ➔ Simplefy reflection of struct fields ◆ Reflecting by expression of selectors ◆ Easy to reflect complex struct value ◆ Supports maps and slices (but restricted) type Bar struct { N []int } type Foo struct { Bar *Bar } f := &Foo{ Bar: &Bar{ N: []int{100} }} v, _ := At(f, `Bar.N[0]`) fmt.Println(v) $ go run main.go 100 38
  39. 39. Create own static analysis tools ➔ Reduce costs of code reviews by own tools ◆ Custimized linter for your project ◆ Detecting typical bugs with own tools ◆ You should pay your time for more important things ● algorithm, design, performance and so on 39
  40. 40. Static Analysis in Production Case 2: Banner Tool 40 Demos:https://goo.gl/HzHTm6
  41. 41. Banner Tool ➔ A management tool of in-app banners In-App Banner A screenshot of our app (Mercari Atte) 41
  42. 42. Banner Tool ➔ A management tool of in-app banners Banner Tool ● Delivery Conditions ● Response Getting Banners w/ OS, API Version List of Delivered Banners w/ Image URL, Destination URL, etc... Apps 42
  43. 43. Evaluation of Delivery Conditions Getting Banners GET /banner/?os=1 String(os) == "1" Banner Image, etc... Delivery Condition: Banner Tool "1" == "1" true BIND EVAL ➔ Describing delivery conditions by a Go like expression ◆ Eval expressions with go package ◆ Using a function calling for describing a variable with a type 43
  44. 44. Conclutions ➔ Static Analysis is EASY in Go ◆ static typing, simple syntax, go package, etc... ➔ go package can ◆ tokenize, parse and type check ➔ Static analysis in production ◆ own static analysis tools (e.g. gpath, go-httpdoc) ◆ web apps (e.g. Banner Tool) 44
  45. 45. Thank you! twitter: @tenntenn Qiita: tenntenn connpass: tenntenn 45

×