Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

GDG Devfest 2019 - Build go kit microservices at kubernetes with ease

770 views

Published on

Gokit is microservice tookit and use Service/Endpoint/Transport to strict separation of concerns design. This talk to use go-kit develop microservice application integrate with consul, zipkin, prometheus, etc service and deploy on Kubernetes.

Published in: Software
  • Login to see the comments

GDG Devfest 2019 - Build go kit microservices at kubernetes with ease

  1. 1. Build go kit Microservices at Kubernetes with ease 2019/12/1
  2. 2. Cage Chung @CageChung @cage1016 cage.chung@gmail.com GDG Cloud Taipei co-organizer, Google Developer Expert (GCP) *photo*
  3. 3. We heritage the enthusiasm of Google Cloud Platform User Group to create Taipei chapter
  4. 4. GDG Cloud Taipei Meetup #50 2019/12/12 at Google Taipei 101 Office 14F ● 我們與Kubernetes的距離, Astley Chen / GDG Cloud Taipei co-organizer, DevOps engineer, Cloud Solution Architect ● Build go kit Microservices at Kubernetes with ease, Cage Chung / GDG Cloud Taipei co-organizer, GDE (GCP)
  5. 5. Agenda gokitconsulk8s Go-kit Service Mesh
  6. 6. Gokitconsulk8s go kit microservice demo with consul & zipkin at kubernetes
  7. 7. Gokitconsulk8s 3 microservice Router ● HTTP ● GRPC Addsvc ● sum ● concat Foosvc ● foo Consul (service discovery) Tracing (trace service) Prometheus (monitor) Grafana (monitor) Fluentd (log) Elasticsearch (log) Kibana (log)
  8. 8. Gokitconsulk8s https://github.com/cage1016/gokitconsulk8s
  9. 9. Gokitconsulk8s - Consul https://github.com/cage1016/gokitconsulk8s
  10. 10. Gokitconsulk8s - Tracing https://github.com/cage1016/gokitconsulk8s
  11. 11. Gokitconsulk8s - Prometheus https://github.com/cage1016/gokitconsulk8s
  12. 12. Gokitconsulk8s - Grafana https://github.com/cage1016/gokitconsulk8s
  13. 13. Gokitconsulk8s - Kibana https://github.com/cage1016/gokitconsulk8s
  14. 14. $ kubectl get po NAME READY STATUS RESTARTS AGE addsvc-64d7d45464-jxfp6 3/3 Running 0 36m ch-consul-connect-injector-webhook-deployment-679fcb48f-lnmt6 1/1 Running 0 37m ch-consul-f7cj6 1/1 Running 0 37m ch-consul-m89hf 1/1 Running 0 37m ch-consul-qv6ms 1/1 Running 0 37m ch-consul-server-0 1/1 Running 0 37m ch-consul-w2lqw 1/1 Running 0 37m ch-consul-xg5rb 1/1 Running 0 37m foosvc-5c468bd6dd-2fflp 3/3 Running 0 36m ingress-grpc-648786cf85-ddqsr 2/2 Running 0 36m ingress-http-7b7b9c68f-7kwbz 2/2 Running 0 36m router-grpc-5db4f9699b-q9qcq 2/2 Running 0 36m router-http-5b55cf45b-57zwn 2/2 Running 0 36m website-554db99b57-9cq9d 2/2 Running 0 36m zipkin-5bb9d87c78-84pmk 2/2 Running 0 36m
  15. 15. ├── cmd │ ├── addsvc │ │ └── main.go │ ├── foosvc │ │ └── main.go │ └── router │ └── main.go ├── consul-helm ├── deployments │ ├── docker │ │ ├── Dockerfile │ │ ├── Dockerfile.cleanbuild │ │ └── Dockerfile.debug │ └── k8s │ ├── website │ ├── addsvc.yaml │ ├── foosvc.yaml │ ├── ingress-grpc.yaml │ ├── ingress-http.yaml │ ├── router-grpc.yaml │ ├── router-http.yaml │ ├── service-monitor.yaml │ ├── website.yaml │ ├── zipkin-deployment.yaml │ └── zipkin-service.yaml ├── pkg │ ├── addsvc │ │ ├── endpoints │ │ ├── service │ │ └── transports │ ├── foosvc │ │ ├── endpoints │ │ │ ├── endpoints.go │ │ │ ├── middleware.go │ │ │ ├── requests.go │ │ │ └── responses.go │ │ ├── service │ │ │ ├── logging.go │ │ │ └── service.go │ │ └── transports │ │ ├── errors.go │ │ ├── grpc.go │ │ └── http.go │ └── router │ └── transports ├── makefile └── skaffold.yaml
  16. 16. ## gateway sum $ curl -X "POST" "http://34.80.152.38:8080/api/addsvc/sum" -H 'Content-Type: application/json; charset=utf-8' -d '{ "a": 3, "b": 34}' {"rs":37,"err":null} ## gateway concat $ curl -X "POST" "http://34.80.152.38:8080/api/addsvc/concat" -H 'Content-Type: application/json; charset=utf-8' -d '{ "a": "3", "b": "34"}' {"rs":"334","err":null} ## gateway foo $ curl -X "POST" "http://34.80.152.38:8080/api/foosvc/foo" -H 'Content-Type: application/json; charset=utf-8' -d '{"s": "3ddd"}' {"res":"3dddbar","err":null} ## grpc sum $ grpcurl -plaintext -proto ./pb/addsvc/addsvc.proto -d '{"a": 3, "b":5}' 35.221.190.225:8081 pb.Addsvc.Sum { "rs": "8" } ## grpc concat $ grpcurl -plaintext -proto ./pb/addsvc/addsvc.proto -d '{"a": "3", "b":"5"}' 35.221.190.225:8081 pb.Addsvc.Concat { "rs": "35" } ## grpc foo $ grpcurl -plaintext -proto ./pb/foosvc/foosvc.proto -d '{"s": "foo"}' 35.221.190.225:8081 pb.Foosvc.Foo { "res": "foobar" }
  17. 17. Demo https://github.com/cage1016/gokitconsulk8s
  18. 18. Go-kit Toolkit for microservices
  19. 19. Microservice solve organizational problems ~ Microservice cause technical problems
  20. 20. Problem solved 1. Team are blocked on other teams, can’t make progress 2. Product velocity stalled
  21. 21. Problem cause 1. Testing becomes really hard 2. Build pipelines 3. Requires dev/ops culture: devs deploy and operate their work 4. Monitoring and instrumentation - tailing logs? 5. Distributed tracing 6. Security
  22. 22. Go-kit goals 1. Make Go a first-class citizen for business logic 2. Microservice architecture 3. RPC as the messaging pattern 4. Provide best practices, idioms, examples and patterns Golang UK Conference 2015 - Peter Bourgon - Go Kit A Toolkit for Microservices - https://www.youtube.com/watch?v=aL6sd4d4hxk
  23. 23. Go-kit non-goals 1. Be a turnkey solution 2. Require any specific infrastructure 3. Have opinions about orchestration, configuration, etc. Golang UK Conference 2015 - Peter Bourgon - Go Kit A Toolkit for Microservices - https://www.youtube.com/watch?v=aL6sd4d4hxk
  24. 24. Go HTTP Servers package main import ( "fmt" "net/http" ) func hello(w http.ResponseWriter, req *http.Request) { fmt.Fprintf(w, "Hello Devfest Taipei 2019") } func main() { http.HandleFunc("/", hello) http.ListenAndServe(":8888", nil) }
  25. 25. type Addsvc interface { sum(a, b int64) (int64, error) } type SumRequest struct { A int64 `json:"a"` B int64 `json:"b"` } Add service: service and implement func (s *addService) sum(a, b int64) (int64, error) { return a + b, nil } type addService struct{}
  26. 26. Transports: Implement HTTP handler func (s *addService) ServeHTTP(w http.ResponseWriter, r *http.Request) { switch r.Method { case http.MethodPost: var req SumRequest if err := json.NewDecoder(r.Body).Decode(&req); err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } if res, err := s.sum(req.A, req.B); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } else { fmt.Fprint(w, res) } } }
  27. 27. main func main() { http.ListenAndServe(":8888", &addService{}) } $ go run main.go $ curl -X "POST" "http://localhost:8888" -d '{ "a": 3, "b": 34}' 37 Addservice an HTTP handler, we can pass it to ListenAndServe
  28. 28. Logging: try to add logging func (s *addService) ServeHTTP(w http.ResponseWriter, r *http.Request) { switch r.Method { case http.MethodPost: var req SumRequest if err := json.NewDecoder(r.Body).Decode(&req); err != nil { code := http.StatusBadRequest http.Error(w, err.Error(), code) log.Printf("%s: %s %s %d", r.RemoteAddr, r.Method, r.URL, code) return } if res, err := s.sum(req.A, req.B); err != nil { code := http.StatusInternalServerError http.Error(w, err.Error(), code) log.Printf("%s: %s %s %d", r.RemoteAddr, r.Method, r.URL, code) return } else { log.Printf("%s: %s %s %d", r.RemoteAddr, r.Method, r.URL, 200) fmt.Fprint(w, res) } }
  29. 29. Logging func main() { log.Printf("listening on :8888") log.Fatal(http.ListenAndServe(":8888", &addService{})) } $ go run main.go 2019/11/29 15:24:29 listening on :8888 2019/11/29 15:24:32 [::1]:52200: POST / 200 $ curl -X "POST" "http://localhost:8888" -d '{ "a": 3, "b": 34}' 37
  30. 30. Endpoint type Endpoint func(request) (response) Endpoint is the fundamental building block of servers and clients.It represents a single RPC method. type Endpoint func(ctx context.Context, request interface{}) (response interface{}, err error)
  31. 31. Add service endpoint func MakePostSumEndpoint(s addService) endpoint.Endpoint { return func(_ context.Context, request interface{}) (response interface{}, err error) { p := request.(SumRequest) return s.sum(p.A, p.B) } } Create endpoint by add service constructor and keep add service pure
  32. 32. Endpoint middleware Middleware is a chainable behavior modifier for endpoints. type Middleware func(Endpoint) Endpoint
  33. 33. Logging middleware func loggingMiddleware(next endpoint.Endpoint) endpoint.Endpoint { return func(ctx context.Context, request interface{}) (response interface{}, err error) { begin := time.Now() defer func() { log.Printf("request took %s", time.Since(begin)) }() return next(ctx, request) } } s := addService{} var e endpoint.Endpoint e = MakePostSumEndpoint(s) e = loggingMiddleware(e) Create endpoint by add service constructor and keep add service pure
  34. 34. Http transport func NewServer( e endpoint.Endpoint, dec DecodeRequestFunc, enc EncodeResponseFunc, options ...ServerOption, ) *Server { ... } postSum := httptransport.NewServer(e, decodeHTTPSumRequest, httptransport.EncodeJSONResponse) r := mux.NewRouter() r.Methods(http.MethodPost).Handler(postSum) log.Printf("listening on :8888") log.Fatal(http.ListenAndServe(":8888", r)) Go kit comes with handy HTTP transport. NewServer constructs a new server, which implements http.Handler and wraps the provided endpoint.
  35. 35. Design — How is a Go kit microservice modeled? https://gokit.io/faq/#design-mdash-how-is-a-go-kit-microservice-modeled Transport Endpoint Service
  36. 36. Go + microservices = Go kit - Speaker Deck - https://speakerdeck.com/peterbourgon/go-plus-microservices-equals-go-kit?slide=78
  37. 37. Golang UK Conference 2015 - Peter Bourgon - Go Kit A Toolkit for Microservices - https://youtu.be/aL6sd4d4hxk?t=1022
  38. 38. ● Authentication: Basic, casbin, JWT. ● Circuit Breaker: Hystrix, GoBreaker, and HandyBreaker. ● Logging: Provide an interface for structured logging. Recognizes that logs are data. They need context and semantics to be useful for analysis. Supported formats are logfmt and JSON. ● Metrics: Provides uniform interfaces for service instrumentation. Comes with counters, gauges, and histograms. Has adapters for CloudWatch, Prometheus, Graphite, DogStatsD, StatsD, expvar, and more. ● Rate Limit: Uses Go's token bucket implementation. ● Service Discovery: Consul, DNS SRV, etcd, Eureka, ZooKeeper, and more. ● Tracing: OpenCensus, OpenTracing, and Zipkin. ● Transport: AMQP, AWS Lambda, gRPC, HTTP, NATS, Thrift. . ├── auth ├── circuitbreaker ├── cmd ├── endpoint ├── examples ├── log ├── metrics ├── ratelimit ├── sd ├── tracing ├── transport ├── util ├── .build.yml ├── .gitignore ├── .travis.yml
  39. 39. Demo Gokit generator: squaresvc
  40. 40. Service mesh
  41. 41. Microservice cause technical problems ~ Service Mesh solve technical problems
  42. 42. What Is a Service Mesh? By NGINX 1. Container orchestration framework 2. Services and instances (Kubernetes pods). 3. Sidecar proxy 4. Service discovery 5. Load balancing 6. Encryption 7. Authentication and authorization 8. Support for the circuit breaker pattern What Is a Service Mesh? - NGINX - https://www.nginx.com/blog/what-is-a-service-mesh/
  43. 43. The control plane in a service mesh distributes configuration across the sidecar proxies in the data plane What Is a Service Mesh? - NGINX - https://www.nginx.com/blog/what-is-a-service-mesh/
  44. 44. Consul by HashiCorp - https://www.consul.io/
  45. 45. Gokitconsulk8s https://github.com/cage1016/gokitconsulk8s
  46. 46. addsvc.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: app: addsvc name: addsvc spec: replicas: 1 strategy: {} template: metadata: creationTimestamp: null labels: app: addsvc annotations: "consul.hashicorp.com/connect-inject": "true" "consul.hashicorp.com/connect-service": "addsvc" "consul.hashicorp.com/connect-service-port": "8021" "consul.hashicorp.com/connect-service-protocol": "grpc" "consul.hashicorp.com/connect-service-upstreams": "zipkin:9411"
  47. 47. Consul vs. Istio Consul Service Mesh实战【转】 - 简书 - https://www.jianshu.com/p/ad970554403d
  48. 48. Reference 1. Go + Microservices = Go Kit [I] - Peter Bourgon, Go Kit 2. Go kit 3. https://gokit.io/faq/ 4. cage1016/gokitconsulk8s 5. cage1016/gk 6. What Is a Service Mesh?
  49. 49. THANK YOU
  50. 50. Q & A

×