Gokit is microservice tookit and use Service/Endpoint/Transport to strict separation of concerns design. This talk to use go-kit develop microservice application integrate with consul, zipkin, prometheus, etc service and deploy on Kubernetes.

2. Build go kit Microservices at
Kubernetes with ease
2019/12/1

3. Cage Chung
@CageChung @cage1016
cage.chung@gmail.com
GDG Cloud Taipei co-organizer,
Google Developer Expert (GCP)
*photo*

4. We heritage the enthusiasm of Google Cloud Platform User Group
to create Taipei chapter

6. GDG Cloud Taipei Meetup #50
2019/12/12 at Google Taipei 101 Office 14F
● 我們與Kubernetes的距離, Astley Chen / GDG Cloud Taipei co-organizer, DevOps
engineer, Cloud Solution Architect
● Build go kit Microservices at Kubernetes with ease, Cage Chung / GDG Cloud Taipei
co-organizer, GDE (GCP)

7. Agenda
gokitconsulk8s
Go-kit
Service Mesh

8. Gokitconsulk8s
go kit microservice demo with consul & zipkin at kubernetes

9. Gokitconsulk8s
3 microservice
Router
● HTTP
● GRPC
Addsvc
● sum
● concat
Foosvc
● foo
Consul (service discovery)
Tracing (trace service)
Prometheus (monitor)
Grafana (monitor)
Fluentd (log)
Elasticsearch (log)
Kibana (log)

10. Gokitconsulk8s
https://github.com/cage1016/gokitconsulk8s

11. Gokitconsulk8s - Consul
https://github.com/cage1016/gokitconsulk8s

12. Gokitconsulk8s - Tracing
https://github.com/cage1016/gokitconsulk8s

13. Gokitconsulk8s - Prometheus
https://github.com/cage1016/gokitconsulk8s

14. Gokitconsulk8s - Grafana
https://github.com/cage1016/gokitconsulk8s

15. Gokitconsulk8s - Kibana
https://github.com/cage1016/gokitconsulk8s

16. $ kubectl get po
NAME READY STATUS RESTARTS AGE
addsvc-64d7d45464-jxfp6 3/3 Running 0 36m
ch-consul-connect-injector-webhook-deployment-679fcb48f-lnmt6 1/1 Running 0 37m
ch-consul-f7cj6 1/1 Running 0 37m
ch-consul-m89hf 1/1 Running 0 37m
ch-consul-qv6ms 1/1 Running 0 37m
ch-consul-server-0 1/1 Running 0 37m
ch-consul-w2lqw 1/1 Running 0 37m
ch-consul-xg5rb 1/1 Running 0 37m
foosvc-5c468bd6dd-2fflp 3/3 Running 0 36m
ingress-grpc-648786cf85-ddqsr 2/2 Running 0 36m
ingress-http-7b7b9c68f-7kwbz 2/2 Running 0 36m
router-grpc-5db4f9699b-q9qcq 2/2 Running 0 36m
router-http-5b55cf45b-57zwn 2/2 Running 0 36m
website-554db99b57-9cq9d 2/2 Running 0 36m
zipkin-5bb9d87c78-84pmk 2/2 Running 0 36m

17. ├── cmd
│ ├── addsvc
│ │ └── main.go
│ ├── foosvc
│ │ └── main.go
│ └── router
│ └── main.go
├── consul-helm
├── deployments
│ ├── docker
│ │ ├── Dockerfile
│ │ ├── Dockerfile.cleanbuild
│ │ └── Dockerfile.debug
│ └── k8s
│ ├── website
│ ├── addsvc.yaml
│ ├── foosvc.yaml
│ ├── ingress-grpc.yaml
│ ├── ingress-http.yaml
│ ├── router-grpc.yaml
│ ├── router-http.yaml
│ ├── service-monitor.yaml
│ ├── website.yaml
│ ├── zipkin-deployment.yaml
│ └── zipkin-service.yaml
├── pkg
│ ├── addsvc
│ │ ├── endpoints
│ │ ├── service
│ │ └── transports
│ ├── foosvc
│ │ ├── endpoints
│ │ │ ├── endpoints.go
│ │ │ ├── middleware.go
│ │ │ ├── requests.go
│ │ │ └── responses.go
│ │ ├── service
│ │ │ ├── logging.go
│ │ │ └── service.go
│ │ └── transports
│ │ ├── errors.go
│ │ ├── grpc.go
│ │ └── http.go
│ └── router
│ └── transports
├── makefile
└── skaffold.yaml

18. ## gateway sum
$ curl -X "POST" "http://34.80.152.38:8080/api/addsvc/sum" -H 'Content-Type: application/json; charset=utf-8' -d '{ "a": 3, "b": 34}'
{"rs":37,"err":null}
## gateway concat
$ curl -X "POST" "http://34.80.152.38:8080/api/addsvc/concat" -H 'Content-Type: application/json; charset=utf-8' -d '{ "a": "3", "b":
"34"}'
{"rs":"334","err":null}
## gateway foo
$ curl -X "POST" "http://34.80.152.38:8080/api/foosvc/foo" -H 'Content-Type: application/json; charset=utf-8' -d '{"s": "3ddd"}'
{"res":"3dddbar","err":null}
## grpc sum
$ grpcurl -plaintext -proto ./pb/addsvc/addsvc.proto -d '{"a": 3, "b":5}' 35.221.190.225:8081 pb.Addsvc.Sum
{
"rs": "8"
}
## grpc concat
$ grpcurl -plaintext -proto ./pb/addsvc/addsvc.proto -d '{"a": "3", "b":"5"}' 35.221.190.225:8081 pb.Addsvc.Concat
{
"rs": "35"
}
## grpc foo
$ grpcurl -plaintext -proto ./pb/foosvc/foosvc.proto -d '{"s": "foo"}' 35.221.190.225:8081 pb.Foosvc.Foo
{
"res": "foobar"
}

19. Demo
https://github.com/cage1016/gokitconsulk8s

20. Go-kit
Toolkit for microservices

21. Microservice solve
organizational problems
~
Microservice cause
technical problems

22. Problem solved
1. Team are blocked on other teams, can’t make progress
2. Product velocity stalled

23. Problem cause
1. Testing becomes really hard
2. Build pipelines
3. Requires dev/ops culture: devs deploy and operate their work
4. Monitoring and instrumentation - tailing logs?
5. Distributed tracing
6. Security

24. Go-kit goals
1. Make Go a first-class citizen for business logic
2. Microservice architecture
3. RPC as the messaging pattern
4. Provide best practices, idioms, examples and patterns
Golang UK Conference 2015 - Peter Bourgon - Go Kit A Toolkit for Microservices - https://www.youtube.com/watch?v=aL6sd4d4hxk

25. Go-kit non-goals
1. Be a turnkey solution
2. Require any specific infrastructure
3. Have opinions about orchestration, configuration, etc.
Golang UK Conference 2015 - Peter Bourgon - Go Kit A Toolkit for Microservices - https://www.youtube.com/watch?v=aL6sd4d4hxk

26. Go HTTP Servers
package main
import (
"fmt"
"net/http"
)
func hello(w http.ResponseWriter, req *http.Request) {
fmt.Fprintf(w, "Hello Devfest Taipei 2019")
}
func main() {
http.HandleFunc("/", hello)
http.ListenAndServe(":8888", nil)
}

27. type Addsvc interface {
sum(a, b int64) (int64, error)
}
type SumRequest struct {
A int64 `json:"a"`
B int64 `json:"b"`
}
Add service: service and implement
func (s *addService) sum(a, b int64) (int64, error) {
return a + b, nil
}
type addService struct{}

28. Transports: Implement HTTP handler
func (s *addService) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodPost:
var req SumRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if res, err := s.sum(req.A, req.B); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
} else {
fmt.Fprint(w, res)
}
}
}

29. main
func main() {
http.ListenAndServe(":8888", &addService{})
}
$ go run main.go
$ curl -X "POST" "http://localhost:8888" -d '{ "a": 3, "b": 34}'
37
Addservice an HTTP handler, we can pass it to ListenAndServe

30. Logging: try to add logging
func (s *addService) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodPost:
var req SumRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
code := http.StatusBadRequest
http.Error(w, err.Error(), code)
log.Printf("%s: %s %s %d", r.RemoteAddr, r.Method, r.URL, code)
return
}
if res, err := s.sum(req.A, req.B); err != nil {
code := http.StatusInternalServerError
http.Error(w, err.Error(), code)
log.Printf("%s: %s %s %d", r.RemoteAddr, r.Method, r.URL, code)
return
} else {
log.Printf("%s: %s %s %d", r.RemoteAddr, r.Method, r.URL, 200)
fmt.Fprint(w, res)
}
}

31. Logging
func main() {
log.Printf("listening on :8888")
log.Fatal(http.ListenAndServe(":8888", &addService{}))
}
$ go run main.go
2019/11/29 15:24:29 listening on :8888
2019/11/29 15:24:32 [::1]:52200: POST / 200
$ curl -X "POST" "http://localhost:8888" -d '{ "a": 3, "b": 34}'
37

32. Endpoint
type Endpoint func(request) (response)
Endpoint is the fundamental building block of servers and clients.It represents a
single RPC method.
type Endpoint func(ctx context.Context, request interface{}) (response interface{}, err error)

33. Add service endpoint
func MakePostSumEndpoint(s addService) endpoint.Endpoint {
return func(_ context.Context, request interface{}) (response interface{}, err error) {
p := request.(SumRequest)
return s.sum(p.A, p.B)
}
}
Create endpoint by add service constructor and keep add service pure

34. Endpoint middleware
Middleware is a chainable behavior modifier for endpoints.
type Middleware func(Endpoint) Endpoint

35. Logging middleware
func loggingMiddleware(next endpoint.Endpoint) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (response interface{}, err error) {
begin := time.Now()
defer func() {
log.Printf("request took %s", time.Since(begin))
}()
return next(ctx, request)
}
}
s := addService{}
var e endpoint.Endpoint
e = MakePostSumEndpoint(s)
e = loggingMiddleware(e)
Create endpoint by add service constructor and keep add service pure

36. Http transport
func NewServer(
e endpoint.Endpoint,
dec DecodeRequestFunc,
enc EncodeResponseFunc,
options ...ServerOption,
) *Server { ... }
postSum := httptransport.NewServer(e, decodeHTTPSumRequest, httptransport.EncodeJSONResponse)
r := mux.NewRouter()
r.Methods(http.MethodPost).Handler(postSum)
log.Printf("listening on :8888")
log.Fatal(http.ListenAndServe(":8888", r))
Go kit comes with handy HTTP transport. NewServer constructs a new server,
which implements http.Handler and wraps the provided endpoint.

37. Design — How is a Go kit microservice modeled?
https://gokit.io/faq/#design-mdash-how-is-a-go-kit-microservice-modeled
Transport
Endpoint
Service

38. Go + microservices = Go kit - Speaker Deck - https://speakerdeck.com/peterbourgon/go-plus-microservices-equals-go-kit?slide=78

39. Golang UK Conference 2015 - Peter Bourgon - Go Kit A Toolkit for Microservices - https://youtu.be/aL6sd4d4hxk?t=1022

40. ● Authentication: Basic, casbin, JWT.
● Circuit Breaker: Hystrix, GoBreaker, and HandyBreaker.
● Logging: Provide an interface for structured logging. Recognizes
that logs are data. They need context and semantics to be useful
for analysis. Supported formats are logfmt and JSON.
● Metrics: Provides uniform interfaces for service instrumentation.
Comes with counters, gauges, and histograms. Has adapters for
CloudWatch, Prometheus, Graphite, DogStatsD, StatsD, expvar,
and more.
● Rate Limit: Uses Go's token bucket implementation.
● Service Discovery: Consul, DNS SRV, etcd, Eureka, ZooKeeper,
and more.
● Tracing: OpenCensus, OpenTracing, and Zipkin.
● Transport: AMQP, AWS Lambda, gRPC, HTTP, NATS, Thrift.
.
├── auth
├── circuitbreaker
├── cmd
├── endpoint
├── examples
├── log
├── metrics
├── ratelimit
├── sd
├── tracing
├── transport
├── util
├── .build.yml
├── .gitignore
├── .travis.yml

41. Demo
Gokit generator: squaresvc

42. Service mesh

43. Microservice cause
technical problems
~
Service Mesh solve technical
problems

44. What Is a Service Mesh? By NGINX
1. Container orchestration framework
2. Services and instances (Kubernetes pods).
3. Sidecar proxy
4. Service discovery
5. Load balancing
6. Encryption
7. Authentication and authorization
8. Support for the circuit breaker pattern
What Is a Service Mesh? - NGINX - https://www.nginx.com/blog/what-is-a-service-mesh/

45. The control plane in a service mesh distributes configuration across the sidecar
proxies in the data plane
What Is a Service Mesh? - NGINX - https://www.nginx.com/blog/what-is-a-service-mesh/

46. Consul by HashiCorp - https://www.consul.io/

47. Gokitconsulk8s
https://github.com/cage1016/gokitconsulk8s

48. addsvc.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: addsvc
name: addsvc
spec:
replicas: 1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: addsvc
annotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/connect-service": "addsvc"
"consul.hashicorp.com/connect-service-port": "8021"
"consul.hashicorp.com/connect-service-protocol": "grpc"
"consul.hashicorp.com/connect-service-upstreams": "zipkin:9411"

49. Consul vs. Istio
Consul Service Mesh实战【转】 - 简书 - https://www.jianshu.com/p/ad970554403d

50. Reference
1. Go + Microservices = Go Kit [I] - Peter Bourgon, Go Kit
2. Go kit
3. https://gokit.io/faq/
4. cage1016/gokitconsulk8s
5. cage1016/gk
6. What Is a Service Mesh?

51. THANK YOU

52. Q & A