Golang Project Layout and Practice

Go Project Layout
and Practice
Bo-Yi Wu
2019.08.29
ModernWeb

About me
• Software Engineer in Mediatek
• Member of Drone CI/CD Platform
• Member of Gitea Platform
• Member of Gin Golang Framework
• Teacher of Udemy Platform: Golang + Drone
https://blog.wu-boy.com

Agenda
• Go in Mediatek
• Go Project Layout
• Go Practices
• RESTful api and GraphQL
• Model testing (Postgres, SQLite, MySQL)
• Software Quality
• Data Metrics
• Go Testing

Tech Stack
• Initial Project using Go in 2018/01
• Golang
• Easy to Learn
• Performance
• Deployment

Repository folder
• api
• assets
• cmd
• conﬁgs
• docker
• pkg
├── api
├── assets
│   └── dist
├── cmd
│   └── ggz
├── configs
├── docker
│   ├── server
└── pkg
├── config
├── errors
├── fixtures
├── helper
├── middleware
│   ├── auth
│   └── header
├── model
├── module
│   ├── mailer
│   ├── metrics
│   └── storage
├── router
│   └── routes
├── schema
└── version

Root folder
• .drone.yml (deploy config)
• .revive.toml (golint config)
• docker-compose.yml (DB, Redis and UI)
• Makefile
• go module config (go.mod and go.sum)
• .env.example

Go Module
https://blog.golang.org/using-go-modules

Improve Deployment
Using Go Module Proxy
https://github.com/gomods/athens

save time
with proxy
97s -> 6s

Makeﬁle
Build, Testing, Deploy

GOFMT ?= gofmt "-s"
GO ?= go
TARGETS ?= linux darwin windows
ARCHS ?= amd64 386
BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
GOFILES := $(shell find . -name "*.go" -type f)
TAGS ?= sqlite sqlite_unlock_notify
ifneq ($(shell uname), Darwin)
  EXTLDFLAGS = -extldflags "-static" $(null)
else
  EXTLDFLAGS =
endif
ifneq ($(DRONE_TAG),)
  VERSION ?= $(subst v,,$(DRONE_TAG))
else
  VERSION ?= $(shell git describe --tags --always')
endif

GGZ_DB_DRIVER=mysql
GGZ_DB_USERNAME=root
GGZ_DB_PASSWORD=123456
GGZ_DB_NAME=ggz
GGZ_DB_HOST=127.0.0.1:3307
GGZ_SERVER_ADDR=:8080
GGZ_DEBUG=true
GGZ_SERVER_HOST=http://localhost:8080
GGZ_STORAGE_DRIVER=disk
GGZ_MINIO_ACCESS_ID=xxxxxxxx
GGZ_MINIO_SECRET_KEY=xxxxxxxx
GGZ_MINIO_ENDPOINT=s3.example.com
GGZ_MINIO_BUCKET=example
GGZ_MINIO_SSL=true
GGZ_AUTH0_DEBUG=true

db:
image: mysql
restart: always
volumes:
- mysql-data:/var/lib/mysql
environment:
MYSQL_USER: example
MYSQL_PASSWORD: example
MYSQL_DATABASE: example
MYSQL_ROOT_PASSWORD: example
minio:
image: minio/minio
restart: always
ports:
volumes:
- minio-data:/data
environment:
MINIO_ACCESS_KEY: minio123456
MINIO_SECRET_KEY: minio123456
command: server /data
Development

Productionapi:
image: foo/bar
restart: always
ports:
- 8080:8080
environment:
- GGZ_METRICS_TOKEN=test-prometheus-token
- GGZ_METRICS_ENABLED=true
labels:
- "traefik.enable=true"
- "traefik.basic.frontend.rule=Host:${WEB_HOST}"
- "traefik.basic.protocol=http"

VersionCompile version info into Go binary

Version
• -X github.com/go-ggz/ggz/pkg/
version.Version=$(VERSION)
• -X github.com/go-ggz/ggz/pkg/
version.BuildDate=$(BUILD_DATE)
go build -o bin/api -ldﬂags

var (
  // Version number for git tag.
  Version string
  // BuildDate is the ISO 8601 day drone was built.
  BuildDate string
)
// PrintCLIVersion print server info
func PrintCLIVersion() string {
  return fmt.Sprintf(
    "version %s, built on %s, %s",
    Version,
    BuildDate,
    runtime.Version(),
  )
}

BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
ifneq ($(DRONE_TAG),)
VERSION ?= $(subst v,,$(DRONE_TAG))
else
VERSION ?= $(shell git describe --tags --always | sed 's/-/+/' | sed
's/^v//')
endif

AssetsEmbed ﬁles in Go
https://github.com/UnnoTed/ﬁleb0x

func ReadSource(origPath string) (content []byte, err error) {
  content, err = ReadFile(origPath)
  if err != nil {
    log.Warn().Err(err).Msgf("Failed to read builtin %s file.", origPath)
  }
  if config.Server.Assets != "" && file.IsDir(config.Server.Assets) {
    origPath = path.Join(config.Server.Assets, origPath)
    if file.IsFile(origPath) {
      content, err = ioutil.ReadFile(origPath)
      if err != nil {
        log.Warn().Err(err).Msgf("Failed to read custom %s file", origPath)
      }
    }
  }
  return content, err
}
Debug Setting

// ViewHandler support dist handler from UI
func ViewHandler() gin.HandlerFunc {
  fileServer := http.FileServer(dist.HTTP)
  data := []byte(time.Now().String())
  etag := fmt.Sprintf("%x", md5.Sum(data))
  return func(c *gin.Context) {
    c.Header("Cache-Control", "public, max-age=31536000")
    c.Header("ETag", etag)
    if match := c.GetHeader("If-None-Match"); match != "" {
      if strings.Contains(match, etag) {
        c.Status(http.StatusNotModified)
        return
      }
    }
    fileServer.ServeHTTP(c.Writer, c.Request)
  }
}
File Server Handler

圖片來來源：https://developers.google.com/web/fundamentals/performance/optimizing-
content-efﬁciency/http-caching?hl=zh-tw

// Favicon represents the favicon.
func Favicon(c *gin.Context) {
  file, _ := dist.ReadFile("favicon.ico")
  etag := fmt.Sprintf("%x", md5.Sum(file))
  c.Header("ETag", etag)
  c.Header("Cache-Control", "max-age=0")
  if match := c.GetHeader("If-None-Match"); match != "" {
    if strings.Contains(match, etag) {
      c.Status(http.StatusNotModified)
      return
    }
  }
  c.Data(
    http.StatusOK,
    "image/x-icon",
    file,
  )
}
NO Cache

/healthz
• health check for load balancer
func Heartbeat(c *gin.Context) {
c.AbortWithStatus(http.StatusOK)
c.String(http.StatusOK, "ok")
}

Command line package
• Golang package: ﬂag
• urfave/cli
• spf13/cobra

├── agent
│   ├── config
│   │   └── config.go
│   └── main.go
├── notify
│   └── main.go
└── tcp-server
├── config
│   └── config.go
└── main.go

Conﬁg
Management
github.com/spf13/viper

Config management
• Load config from File
• .json
• .ini
• Load config from Environment Variables
• .env

var envfile string
flag.StringVar(&envfile, "env-file", ".env", "Read in a file of environment
variables")
flag.Parse()
godotenv.Load(envfile)
_ "github.com/joho/godotenv/autoload"

Logging struct {
    Debug bool ènvconfig:"GGZ_LOGS_DEBUG"`
    Level string ènvconfig:"GGZ_LOGS_LEVEL" default:"info"`
    Color bool ènvconfig:"GGZ_LOGS_COLOR"`
    Pretty bool ènvconfig:"GGZ_LOGS_PRETTY"`
    Text bool ènvconfig:"GGZ_LOGS_TEXT"`
  }
  // Server provides the server configuration.
  Server struct {
    Addr string ènvconfig:"GGZ_SERVER_ADDR"`
    Port string ènvconfig:"GGZ_SERVER_PORT" default:"12000"`
    Path string ènvconfig:”GGZ_SERVER_PATH" default:"data"`
  }
github.com/kelseyhightower/envconfig

config, err := config.Environ()
  if err != nil {
    log.Fatal().
      Err(err).
      Msg("invalid configuration")
  }
  initLogging(config)
  // check folder exist
  if !file.IsDir(config.Server.Path) {
    log.Fatal().
      Str("path", config.Server.Path).
      Msg("log folder not found")
  }
Load env from structure

/configs
Configuration file templates or default config

global:
scrape_interval: 5s
external_labels:
monitor: 'my-monitor'
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'ggz-server'
static_configs:
- targets: ['ggz-server:8080']
bearer_token: 'test-prometheus-token'

├── ggz-redirect
│ ├── Dockerfile.linux.amd64
│ ├── Dockerfile.linux.arm
│ ├── Dockerfile.linux.arm64
│ ├── Dockerfile.windows.amd64
│ └── manifest.tmpl
└── ggz-server
├── Dockerfile.linux.amd64
├── Dockerfile.linux.arm
├── Dockerfile.linux.arm64
├── Dockerfile.windows.amd64
└── manifest.tmpl

ctx := context.Background()
  req := testcontainers.ContainerRequest{
    Image: "goggz/ggz-server",
    ExposedPorts: []string{"8080/tcp"},
    WaitingFor: wait.ForLog("Starting shorten server on :8080")
  }
  ggzServer, err := testcontainers.GenericContainer(
    ctx,
    testcontainers.GenericContainerRequest{
      ContainerRequest: req,
      Started: true,
    })
  if err != nil {
    t.Fatal(err)
  }
github.com/testcontainers/testcontainers-go

├── conﬁg
├── errors
├── ﬁxtures
├── helper
├── middleware
│ ├── auth
│ └── header
├── model
├── module
│ ├── metrics
│ └── storage
│ ├── disk
│ └── minio
├── router
│ └── routes
├── schema
└── version

// Type deﬁnes the type of an error
type Type string
const (
  // Internal error
  Internal Type = "internal"
  // NotFound error means that a speciﬁc item does not exis
  NotFound Type = "not_found"
  // BadRequest error
  BadRequest Type = "bad_request"
  // Validation error
  Validation Type = "validation"
  // AlreadyExists error
  AlreadyExists Type = "already_exists"
  // Unauthorized error
  Unauthorized Type = "unauthorized"
)

// ENotExists creates an error of type NotExist
func ENotExists(msg string, err error, arg ...interface{}) error {
  return New(NotFound, fmt.Sprintf(msg, arg...), err)
}
// EBadRequest creates an error of type BadRequest
func EBadRequest(msg string, err error, arg ...interface{}) error {
  return New(BadRequest, fmt.Sprintf(msg, arg...), err)
}
// EAlreadyExists creates an error of type AlreadyExists
func EAlreadyExists(msg string, err error, arg ...interface{}) error {
  return New(AlreadyExists, fmt.Sprintf(msg, arg...), err)
}

Rails-like test fixtures
Write tests against a real database
github.com/go-testfixtures/testfixtures

ﬁxtures/
posts.yml
comments.yml
tags.yml
posts_tags.yml
users.yml

-
id: 1
email: test@gmail.com
full_name: test
avatar: http://example.com
avatar_email: test@gmail.com
-
id: 2
email: test1234@gmail.com
full_name: test1234
avatar: http://example.com
avatar_email: test1234@gmail.com

func TestMain(m *testing.M) {
// test program to do extra
setup or teardown before or after
testing.
os.Exit(m.Run())
}
https://golang.org/pkg/testing/#hdr-Main

func MainTest(m *testing.M, pathToRoot string) {
  var err error
  fixturesDir := filepath.Join(pathToRoot, "pkg", "fixtures")
  if err = createTestEngine(fixturesDir); err != nil {
    fatalTestError("Error creating test engine: %vn", err)
  }
  os.Exit(m.Run())
}
func createTestEngine(fixturesDir string) error {
  var err error
  x, err = xorm.NewEngine("sqlite3", "file::memory:?cache=shared")
  if err != nil {
    return err
  }
  x.ShowSQL(config.Server.Debug)
  return InitFixtures(&testfixtures.SQLite{}, fixturesDir)
}
Testing with SQLite

func TestIsUserExist(t *testing.T) {
  assert.NoError(t, PrepareTestDatabase())
  exists, err := IsUserExist(0, "test@gmail.com")
  assert.NoError(t, err)
  assert.True(t, exists)
  exists, err = IsUserExist(0, "test123456@gmail.com")
  assert.False(t, exists)
  assert.True(t, exists)
  assert.False(t, exists)
} go test -v -run=TestIsUserExist ./pkg/models/

Helper func
• Encrypt and Decrypt
• Regexp func
• IsEmail, IsUsername
• Zipﬁle

func Secure(c *gin.Context) {
  c.Header("Access-Control-Allow-Origin", "*")
  c.Header("X-Frame-Options", "DENY")
  c.Header("X-Content-Type-Options", "nosniff")
  c.Header("X-XSS-Protection", "1; mode=block")
  if c.Request.TLS != nil {
    c.Header("Strict-Transport-Security", "max-age=31536000")
  }
}

// +build sqlite
package model
import (
_ "github.com/mattn/go-sqlite3"
)
func init() {
EnableSQLite3 = true
}
go build -v -tags 'sqlite sqlite_unlock_notify'

├── cron
├── download
├── jwt
├── ldap
├── mailer
│ ├── ses
│ └── smtp
├── queue
├── metrics
├── redis
└── storage
├── disk
└── minio

Integration with
Prometheus + Grafana

func NewCollector() Collector {
  return Collector{
    Users: prometheus.NewDesc(
      namespace+"users",
      "Number of Users",
      nil, nil,
    ),
}
// Collect returns the metrics with values
func (c Collector) Collect(ch chan<- prometheus.Metric) {
  stats := model.GetStatistic()
  ch <- prometheus.MustNewConstMetric(
    c.Users,
    prometheus.GaugeValue,
    ﬂoat64(stats.Counter.User),
  )
}

func Metrics(token string) gin.HandlerFunc {
  h := promhttp.Handler()
  return func(c *gin.Context) {
    if token == "" {
      h.ServeHTTP(c.Writer, c.Request)
      return
    }
    header := c.Request.Header.Get("Authorization")
    if header == "" {
      c.String(http.StatusUnauthorized, errInvalidToken.Error())
      return
    }
    bearer := fmt.Sprintf("Bearer %s", token)
    if header != bearer {
      c.String(http.StatusUnauthorized, errInvalidToken.Error())
      return
    }
    h.ServeHTTP(c.Writer, c.Request)
  }
}

c := metrics.NewCollector()
    prometheus.MustRegister(c)
    if conﬁg.Metrics.Enabled {
      root.GET("/metrics", router.Metrics(conﬁg.Metrics.Token))
    }
Your prometheus token

RESTful vs GraphQL
See the Slide: GraphQL in Go

var rootQuery = graphql.NewObject(
  graphql.ObjectConfig{
    Name: "RootQuery",
    Description: "Root Query",
    Fields: graphql.Fields{
      "queryShortenURL": &queryShortenURL,
      "queryMe": &queryMe,
    },
  })
var rootMutation = graphql.NewObject(
  graphql.ObjectConfig{
    Name: "RootMutation",
    Description: "Root Mutation",
    Fields: graphql.Fields{
      "createUser": &createUser,
    },
  })
// Schema is the GraphQL schema served by the server.
var Schema, _ = graphql.NewSchema(
  graphql.SchemaConfig{
    Query: rootQuery,
    Mutation: rootMutation,
  })

assert.NoError(t, model.PrepareTestDatabase())
  t.Run("user not login", func(t *testing.T) {
    test := T{
      Query: `{
queryMe {
email
}
}`,
      Schema: Schema,
      Expected: &graphql.Result{
        Data: map[string]interface{}{
          "queryMe": nil,
        },
        Errors: []gqlerrors.FormattedError{
          {
            Message: errorYouAreNotLogin,
          },
        },
      },
    }
  })
}

Best Practice
Testing your Go code

Testable Code
• Code Quality
• Readability
• Maintainability
• Testability

#1. Testing in Go
func TestFooBar(t *testing.T) {}
func ExampleFooBar(t *testing.T) {}
func BenchmarkFooBar(b *testing.B) {}
go test package_name

#2. Benchmark Testing
Proﬁling: CPU, Memory, Goroutine Block

func BenchmarkPlaylyfeGraphQLMaster(b *testing.B) {
  for i := 0; i < b.N; i++ {
    context := map[string]interface{}{}
    variables := map[string]interface{}{}
    playlyfeExecutor.Execute(context, "{hello}", variables, "")
  }
}
func BenchmarkGophersGraphQLMaster(b *testing.B) {
  for i := 0; i < b.N; i++ {
    ctx := context.Background()
    variables := map[string]interface{}{}
    gopherSchema.Exec(ctx, "{hello}", "", variables)
  }
} http://bit.ly/2L0CG3Q

#3. Example Testing
Examples on how to use your code

func ExampleFooBar() {
  fmt.Println(strings.Compare("a", "b"))
  fmt.Println(strings.Compare("a", "a"))
  fmt.Println(strings.Compare("b", "a"))
  // Output:
  // -1
  // 0
  // 1
}

$ go test -v -tags=sqlite -run=ExampleFooBar ./pkg/model/...
=== RUN ExampleFooBar
--- PASS: ExampleFooBar (0.00s)
PASS
ok github.com/go-ggz/ggz/pkg/model 0.022s

#4. Subtests in Testing Package
func (t *T) Run(name string, f func(t *T)) bool {}
func (b *B) Run(name string, f func(b *B)) bool {}

tests := []struct {
    name string
    fields fields
    args args
  }{}
  for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) {
      c := Collector{
        Shortens: tt.fields.Shortens,
        Users: tt.fields.Users,
      }
      c.Describe(tt.args.ch)
    })
  }
}

package metrics
import (
  "os"
  "testing"
)
func TestSkip(t *testing.T) {
  if os.Getenv("DEBUG_MODE") == "true" {
    t.Skipf("test skipped")
  }
}

#6. Running Tests in Parallel
Speedup your CI/CD Flow
t.Parallel()

func TestFooBar01(t *testing.T) {
  t.Parallel()
  time.Sleep(time.Second)
}
  t.Parallel()
  time.Sleep(time.Second * 2)
}
  t.Parallel()
  time.Sleep(time.Second * 3)
}

Just only use
one package
github.com/stretchr/testify

https://www.udemy.com/course/golang-ﬁght/?couponCode=GOLANG2019

https://www.udemy.com/course/devops-oneday/?couponCode=DRONE2019

Golang Project Layout and Practice

More Related Content

What's hot

Similar to Golang Project Layout and Practice

More from Bo-Yi Wu

Recently uploaded

In this document

Golang Project Layout and Practice