SlideShare a Scribd company logo
Building
Services With
,
and !
Martin Kess ♥ Software Engineer
github.com/mdkess
Get the code:
github.com/namely/codecamp-2018-go
Agenda
● Background About Namely
● Why Services?
● Protobufs and gRPC - Defining Interfaces
● JSON
● Docker and Docker Compose
● Questions
About Namely
● Mission: Build A Better Workplace
● HR, Benefits and Payroll
● 1200 customers
● ~$1 billion in payroll/month
● ~100 engineers
● ~40 services, more shipping every week
● Polyglot environment: React, C# (.NET Core), Go, Ruby and Python
● Modern infrastructure: Kubernetes, Istio, AWS, Docker, Spinnaker.
● Big believers in open-source. We've contributed to the official gRPC C# repo. We
open source a lot of the tools we build.
What Are
(Micro)Services?
And Why Build Them, Anyway?
A service is software that...
● is the source of truth for its data.
● is independently deployable.
● prevents coupling through use of API
contracts.
● adds business value and open up new
opportunities.
● has a clear definition of availability (an SLO).
Domain Ownership
Services don't mean containers or AWS or Kubernetes. It
means that pieces of software that own their domain.
Services own the reads and writes for their data. Access to
this data should be done through APIs (not a shared DB).
Don't build a distributed monolith or you'll get all of the
weaknesses of services and none of the benefits.
Why Namely Uses Services
● In a monolith, teams ended up stepping on each others
feet.
○ Accidentally releasing each other team's features.
○ Big changes touching lots of code accidentally break things.
○ Unclear ownership of large parts of the codebase or data.
● Services make teams think in terms of API contracts.
● Teams can use language and tools of their choice.
● Give teams ownership and mastery of their domain.
A Domain
To Model
Companies And Employees
A Company is a collection of Employee
objects and has an Office Location.
Every Employee has a name, works for a
Company and has a badge number.
Every Company has a CEO, who is also an
Employee.
Company
+ company_uuid: uuid
+ ceo_employee_uuid: uuid
+ office_location: Address
Employee
+ employee_uuid: uuid
+ company_uuid: uuid
+ name: string
+ badge_number: int32
A Problem
These models are almost certainly wrong.
Do all companies have a CEO? Do all companies have one CEO?
Do all companies have an office location? Do all companies have only one
office location? Are all companies based in America?
Do all employees have badge numbers? Is a single name field the best choice?
Of course not.
Good Software
Anticipates
Change
Anticipating Change
There is no perfect domain model, but our model might be good enough for
our current customers. Don't design for a future that might not exist. We want
to start with this model and iterate. But in doing so, some things to consider:
● What if you can’t force your old API clients to update?
● How do you release API clients and API servers separately?
○ Very important when doing slow rollouts of software.
● How do you avoid breaking updated API clients after a rollback?
● What if your data is stored on disk?
○ In a message queue, a file or a database.
Protocol Buffers
Use protocol buffers aka "protobufs"!
Message format invented by Google. Supports forward and backward
compatibility: newer servers can read messages from old clients and vice
versa.
A .proto file gets compiled into many languages (C#, Java, Go, Ruby, etc.)
Think fancy JSON with a schema.
A Simple Proto File
example.proto
Think of a message as a
C struct/POJO/POCO -
just data.
On each field in the
message is the field
number (i.e. = 4), this is
used when serializing
protos. It's not a (default)
value.
syntax = "proto3";
package examples;
message Employee {
string employee_uuid = 1;
string company_uuid = 2;
string name = 3;
int32 badge_number = 4;
}
message Address {
string address1 = 1;
string address2 = 2;
string zip = 3;
string state = 4;
}
message Company {
string company_uuid = 1;
Address office_location = 2;
string ceo_employee_uuid = 3;
}
github.com/namely/codecamp-2018-go
Compiling Protos
The protobuf compiler
turns protos into code for
your language.
On the right, we turn our
employee.proto from the
previous slide into Go
code.
Can also do C#, JS, Ruby,
Python, Java and many
other languages.
$ docker run 
-v `pwd`:/defs namely/protoc-all 
-f example.proto 
-l go
The above command runs the docker container
namely/protoc-all to compile the example.proto
file into Go code and output the results to `pwd` (the
current directory).
$ ls
example.proto gen/
$ ls gen/pb-go/
example.pb.go
github.com/namely/codecamp-2018-go
The Generated
Code
example.pb.go looks
something like the right.
This code is generated
automatically by the
namely/protoc-all
container.
Try running
namely/protoc-all with
-l python instead.
// Code generated by protoc-gen-go. DO NOT EDIT.
// source: example.proto
package examples
... snip ...
type Employee struct {
EmployeeUuid string
`protobuf:"bytes,1,opt,name=employee_uuid,json=employeeUuid"
json:"employee_uuid,omitempty"`
CompanyUuid string
`protobuf:"bytes,2,opt,name=company_uuid,json=companyUuid"
json:"company_uuid,omitempty"`
Name string
`protobuf:"bytes,3,opt,name=name" json:"name,omitempty"`
BadgeNumber int32
`protobuf:"varint,4,opt,name=badge_number,json=badgeNumber"
json:"badge_number,omitempty"`
}
func (m *Employee) Reset() { *m = Employee{} }
func (m *Employee) String() string { return
proto.CompactTextString(m) }
func (*Employee) ProtoMessage() {}
func (*Employee) Descriptor() ([]byte, []int) { return fileDescriptor0,
[]int{0} }
... snip ...
github.com/namely/codecamp-2018-go
Great! But how do
protobufs help us
write services?
We need a way for our services to talk
to each other.
Remote Procedure Calls (RPCs) are
function calls that can be made over
the network.
is an open-source RPC framework for
building language agnostic servers and
clients that can talk to each other.
This means your Go/Ruby/C# client can talk
to your Python/Java/C++ server (and more).
It uses protocol buffers as its message
format.
Adding Services to
example.proto
You can also define
services in your proto file.
These get compiled to
gRPC servers and clients
that can speak protocol
buffers to each other.
You can write your server
and client in any
supported language.
service EmployeeService {
rpc CreateEmployee(CreateEmployeeRequest)
returns (Employee) {}
rpc ListEmployees(ListEmployeesRequest)
returns (ListEmployeesResponse) {}
}
message CreateEmployeeRequest {
Employee employee = 1;
}
message ListEmployeesRequest {
string company_uuid = 1;
}
message ListEmployeesResponse {
repeated Employee employees = 1;
}
github.com/namely/codecamp-2018-go
Let's Write A
Server In
Application
Structure
The Company Service in
company/
The Employee Service in
employee/
The protobufs in
protos/
gen_protos.sh to
compile the protos
Check out the code!
$ git clone 
github.com/namely/codecamp-2018-go
$ ls
CODEOWNERS LICENSE README.md
docker-compose.yml
example.proto
gen_protos.sh
protos/
company/
employee/
github.com/namely/codecamp-2018-go
Diving Into
Employee Service
Diving into
employee/main.go.
The main() function
listens on a TCP port,
creates a new gRPC
server and registers our
server interface to handle
gRPC calls
func main() {
flag.Parse()
lis, err := net.Listen("tcp",
fmt.Sprintf("0.0.0.0:%d", *port))
if err != nil {
log.Fatalf("error listening: %v", err)
}
server := grpc.NewServer()
pb.RegisterEmployeeServiceServer(
server, newServer())
server.Serve(lis)
}
github.com/namely/codecamp-2018-go
The Employee
Server
The EmployeeServer
stores all of the
employees in memory.
For a real server you
would use a database.
It also creates a client
that talks to company
service to check that
companies exist.
type EmployeeServer struct {
companies map[string]*EmployeeCollection
conn *grpc.ClientConn
companyClient company_pb.CompanyServiceClient
}
func newServer() *EmployeeServer {
s := &EmployeeServer{}
s.companies =
make(map[string]*EmployeeCollection)
s.conn, _ = grpc.Dial(
*companyAddr, grpc.WithInsecure())
s.companyClient =
company_pb.NewCompanyServiceClient(s.conn)
return s
}
github.com/namely/codecamp-2018-go
Looking at a
Handler
Let's look at the
CreateEmployee handler
It does three things:
1. Validate the input.
2. Call company service
to make sure the
company exists
3. Saves the employee.
This is the signature of the CreateEmployee function on
the EmployeeServer.
Input is:
- the call's context
- a CreateEmployeeRequest proto - the same one we
defined in our proto file earlier!
func (s *EmployeeServer)
CreateEmployee(
ctx context.Context,
req *employee_pb.CreateEmployeeRequest)
(*employee_pb.Employee, error) {
....
}
Input
parameters
Return
Type
(A tuple)
github.com/namely/codecamp-2018-go
Looking at a
Handler
Let's look at the
CreateEmployee handler
It does three things:
1. Validate the input.
2. Call company service
to make sure the
company exists
3. Saves the employee.
Here we check that the employee's name is set. If not,
we return an Invalid Argument error to the client.
func (s *EmployeeServer) CreateEmployee(
ctx context.Context,
req *employee_pb.CreateEmployeeRequest)
(*employee_pb.Employee, error) {
// The employee must have a name.
if req.Employee.Name == "" {
return nil, status.Error(
codes.InvalidArgument, "employee must have name")
}
....
}
github.com/namely/codecamp-2018-go
Looking at a
Handler
Let's look at the
CreateEmployee handler
It does three things:
1. Validate the input.
2. Call company service
to make sure the
company exists
3. Saves the employee.
Next we call CompanyService.GetCompany with a
GetCompanyRequest to check that the employee's
company exists.
func (s *EmployeeServer) CreateEmployee(
ctx context.Context,
req *employee_pb.CreateEmployeeRequest)
(*employee_pb.Employee, error) {
....
_, err := s.companyClient.GetCompany(
ctx, &company_pb.GetCompanyRequest{
CompanyUuid: req.Employee.CompanyUuid,
})
if err != nil {
return nil, status.Error(
codes.InvalidArgument, "company does not exist")
}
....
}
github.com/namely/codecamp-2018-go
Looking at a
Handler
Let's look at the
CreateEmployee handler
It does three things:
1. Validate the input.
2. Call company service
to make sure the
company exists
3. Saves the employee.
Finally, we save the employee and return the saved
employee to the caller. In our example, we just save it
in memory, but in real life you'd want to use some data
storage for this (i.e. a database).
func (s *EmployeeServer) CreateEmployee(
ctx context.Context,
req *employee_pb.CreateEmployeeRequest)
(*employee_pb.Employee, error) {
....
// If we're here, we can save the employee.
return s.SaveEmployee(req.Employee), nil
}
github.com/namely/codecamp-2018-go
Packaging Your
Services
Docker lets you build your applications into
container (which is sort of like a
lightweight virtual machine).
This makes it easy to distribute your
software and run it anywhere.
You make containers by writing a
Dockerfile.
Dockerfiles
Package your application
in a container that can be
run in various cloud
infrastructure.
Makes it easy to
distribute applications.
Here's the Dockerfile for
employees.
Try building this with
$ docker build -t company .
The above command builds the Dockerfile in the current
directory, and gives the resulting container the name
"company".
FROM golang:alpine AS build
RUN apk add --no-cache git
WORKDIR /go/src/github.com/namely/codecamp-2018-go/employee
COPY . .
RUN go get -d -v ./...
RUN go install -v ./...
FROM alpine
COPY --from=build /go/bin/employee /usr/local/bin/
CMD ["employee"]
github.com/namely/codecamp-2018-go
Stitching
Applications With
Docker-Compose
Docker-Compose lets you run and configure
multiple docker containers.
It makes starting and stopping containers
easy.
It creates DNS names for your containers so
they can talk to each other.
docker-compose.yml
Defines two services
company and employee.
The build field tells
docker-compose how to
find your Dockerfile to
build your services.
github.com/namely/codecamp-2018-go
version: "3.6"
services:
company:
build: ./company
command: company -port 50051
ports:
- 50051:50051
employee:
build: ./employee
command: >
employee -port=50051
-company_addr=company:50051
ports:
- 50052:50051
depends_on:
- company
Bringing Everything Up
Build your services with:
$ docker-compose build
And start them up (in the background) with
$ docker-compose up -d
github.com/namely/codecamp-2018-go
Calling Your
Services With
gRPC CLI
Using the gRPC CLI
Namely provides a Docker container that contains the official gRPC CLI for
querying gRPC services. Get it with
$ docker pull namely/grpc-cli
Create some aliases to make calling it easier. docker.for.mac.localhost is how
the namely/grpc-cli reaches your local machine where the service is running.
$ alias company_call='docker run -v 
`pwd`/protos/company:/defs --rm -it namely/grpc-cli 
call docker.for.mac.localhost:50051'
$ alias employee_call='docker run -v 
`pwd`/protos/employee:/defs --rm -it namely/grpc-cli 
call docker.for.mac.localhost:50052'
docker.for.win.localhost
on Windows!
Creating a Company
Let's use the grpc_cli to call CompanyService.CreateCompany. We say
docker.for.mac.localhost to let the grpc-cli docker container find localhost on
your local machine (where we exposed the port in docker-compose)
$ company_call CompanyService.CreateCompany 
"" --protofiles=company.proto
company_uuid: "3ac4f180-9410-467f-92b7-06763db0a8f1"
Creating an Employee
We'll take the company_uuid from the
$ employee_call EmployeeService.CreateEmployee 
"employee: {name:'Martin', 
company_uuid: '3ac4f180-9410-467f-92b7-06763db0a8f1'}" 
--protofiles=employee.proto
employee_uuid: "10b286b2-247a-4864-afe5-f56163681af6"
company_uuid: "3ac4f180-9410-467f-92b7-06763db0a8f1"
name: "Martin"
Listing the Employees
$ employee_call EmployeeService.ListEmployees 
"company_uuid: '3ac4f180-9410-467f-92b7-06763db0a8f1'" 
--protofiles=employee.proto
employees {
employee_uuid: "3701a099-7c67-40b2-bfac-c0e627efd0f7"
company_uuid: "3ac4f180-9410-467f-92b7-06763db0a8f1"
name: "Martin"
}
What About
JSON?
The Web Doesn't Speak gRPC
Adding HTTP
Annotations
gRPC-Gateway creates an
HTTP-gRPC reverse proxy
for your gRPC server.
Annotations tell it how to
map HTTP requests to
your gRPC requests.
import "google/api/annotations.proto";
service CompanyService {
rpc CreateCompany(CreateCompanyRequest)
returns (Company) {
option (google.api.http) = {
put: "/companies/{company_uuid}",
body: "company"
};
}
...
}
github.com/namely/codecamp-2018-go
HTTP gRPC
gRPCHTTP
gRPC
Server
gRPC
Gateway
Client
HTTP
Request
New JSON
Server Code
Just Kidding No New Code!
Just run Namely's Docker container to generate a new server
$ docker run -v `pwd`:/defs namely/gen-grpc-gateway 
-f protos/company/company.proto -s CompaniesService
This generates a complete server in gen/grpc-gateway. Now build it. The
example repo has this in the docker-compose file as well.
$ docker build -t companies-gw 
-f gen/grpc-gateway/Dockerfile gen/grpc-gateway/
github.com/namely/codecamp-2018-go
Using cURL to try
our HTTP API
Let's wire these together
and use cURL to try out
our new API.
gRPC-Gateway makes it
easy to share your
services with a front-end
application.
Bring up our gateway
$ docker-compose up -d companies-gw companies
Create a company
$ curl -X POST 
-d '{"office_location":{"address1":"foo"}}' 
localhost:8082/companies
{"company_uuid":"d13ecefd-6b63-4919-9b33-e0006ee676ec"
,"office_location":{"address1":"foo"}}
Get that company
$ curl
localhost:8082/companies/d13ecefd-6b63-4919-9b33-e00
06ee676ec
{"company_uuid":"d13ecefd-6b63-4919-9b33-e0006ee676ec
","office_location":{"address1":"foo"}}
EASY!
github.com/namely/codecamp-2018-go
Organizing Protos
Namely uses a monorepo
of all of our protobufs.
Services add this as a git
submodule.
This lets everyone stay up
to date. It also serves as
a central point for design
and API discussions.
What Did We Learn?
● How to build services with gRPC, Docker and Go.
● Thinking about services and how to get value from them.
● The importance of backward compatibility and how protobufs help.
● How to compile protobufs using namely/docker-protoc.
● How to use namely/grpc-cli to call your services.
● Using namely/gen-grpc-gateway to create HTTP services for your APIs.
● Use Docker to build your services into containers.
● Using Docker-Compose to bring up multiple containers.
Questions?
github.com/mdkess
Reducing
Boilerplate With
Interceptors
GRPC Interceptors
Interceptors let you catch calls before they get to the handlers, and before
they're returned to the client.
1 2
34
RPC
Handler
Interceptor
Client
RPC
GRPC Interceptors
An interceptor is a function with the signature:
func(ctx context.Context, // Info about the call (i.e. deadline)
req interface{}, // Request (i.e. CreateCompanyRequest)
info *grpc.UnaryServerInfo, // Server info (i.e. RPC method name)
handler grpc.UnaryHandler // Your handler for the RPC.
)
(resp interface{}, err error) // The response to send, or an error
A Typical
Interceptor
Interceptors let you do some
cool stuff:
1. Transform the request before
your handler gets it.
2. Transform the response
before the client sees it.
3. Add logging and other tooling
around the request without
having to copy-paste code in all
of your handlers.
func MetricsInterceptor(
ctx context.Context, req interface{},
info *grpc.UnaryServerInfo,
handler grpc.UnaryHandler)
(resp interface{}, err error) {
// Get the RPC name (i.e. "CreateCompany")
name := info.FullMethod
// Start a timer to see how long things take.
start := time.Now()
// Actually call the handler - your function.
out, err := handler(ctx, req)
// Check for errors
stat, ok := status.FromError(err)
// Log to our metrics system (maybe statsd)
LogMethodTime(name, start, time.Now(), stat)
// Return to the client. We could also change
// the response, perhaps by stripping out PII or
// doing error normalization/sanitization.
return out, err
}
} github.com/namely/codecamp-2018-go
Wiring It Together
grpc.ServerOption(
grpc.WithUnaryInterceptor(
MetricsInterceptor))
Then add the interceptor to the server start code.
Testing
Techniques
Services Are Hard
Testing With
Docker-Compose
Mock Services
As you grow, you won't
want to bring up all of
your services.
With Go and Mockgen,
you can make your tests
act like a real service.
Combining Unit and Integration Tests
Your tests can be a hybrid of using unit testing techniques (Mocks) and
integration techniques.
Mock out some of the dependent services. This is very powerful when testing
gRPC servers since we can have tighter control over some dependencies.
Instead of bringing up everything, just bring up the dependencies in your
service's domain. For Employment, we bring up the database, but not
companies service.
Hybrid Integration
Tests
Bring up actual
implementations of main
services.
Use mocks for anything
out of the main flow that
are used for checks.
docker-compose run
--use-aliases
--service-ports
employment-tests
Employee
Tests
Employee
Service
Employees.CreateEmployee
Employee
DB
CompanyService.GetCompany
Calls your test (instead of the
real Company service) so that
you can control behavior.
Docker Compose
services:
# ... snip ...
employee-tests:
ports:
- 50052
environment:
- COMPANIES_PORT=50052
employee:
environment:
- COMPANIES_HOST=employee-tests
- COMPANIES_PORT=50052
How Does This
Look?
TODO Code

More Related Content

What's hot

Amazon EKS によるスマホゲームのバックエンド運用事例
Amazon EKS によるスマホゲームのバックエンド運用事例Amazon EKS によるスマホゲームのバックエンド運用事例
Amazon EKS によるスマホゲームのバックエンド運用事例
gree_tech
 
Linux Namespaces
Linux NamespacesLinux Namespaces
Linux Namespaces
Masami Ichikawa
 
Intro to vue.js
Intro to vue.jsIntro to vue.js
Intro to vue.js
TechMagic
 
k8s practice 2023.pptx
k8s practice 2023.pptxk8s practice 2023.pptx
k8s practice 2023.pptx
wonyong hwang
 
Express JS
Express JSExpress JS
Express JS
Alok Guha
 
What's new in Spring Batch 5
What's new in Spring Batch 5What's new in Spring Batch 5
What's new in Spring Batch 5
ikeyat
 
Node.jsで使えるファイルDB"NeDB"のススメ
Node.jsで使えるファイルDB"NeDB"のススメNode.jsで使えるファイルDB"NeDB"のススメ
Node.jsで使えるファイルDB"NeDB"のススメ
Isamu Suzuki
 
[오픈소스컨설팅] EFK Stack 소개와 설치 방법
[오픈소스컨설팅] EFK Stack 소개와 설치 방법[오픈소스컨설팅] EFK Stack 소개와 설치 방법
[오픈소스컨설팅] EFK Stack 소개와 설치 방법
Open Source Consulting
 
その Pod 突然落ちても大丈夫ですか!?(OCHaCafe5 #5 実験!カオスエンジニアリング 発表資料)
その Pod 突然落ちても大丈夫ですか!?(OCHaCafe5 #5 実験!カオスエンジニアリング 発表資料)その Pod 突然落ちても大丈夫ですか!?(OCHaCafe5 #5 実験!カオスエンジニアリング 発表資料)
その Pod 突然落ちても大丈夫ですか!?(OCHaCafe5 #5 実験!カオスエンジニアリング 発表資料)
NTT DATA Technology & Innovation
 
Creare PWA con Angular
Creare PWA con AngularCreare PWA con Angular
Creare PWA con Angular
Francesco Sciuti
 
"행복한 백발의 개발자"라는 제목으로 2024-03-06 어느 IT 업체에서 직책자로 승진한 분들을 대상으로 한...
"행복한 백발의 개발자"라는 제목으로 2024-03-06 어느 IT 업체에서 직책자로 승진한 분들을 대상으로 한..."행복한 백발의 개발자"라는 제목으로 2024-03-06 어느 IT 업체에서 직책자로 승진한 분들을 대상으로 한...
"행복한 백발의 개발자"라는 제목으로 2024-03-06 어느 IT 업체에서 직책자로 승진한 분들을 대상으로 한...
Myeongseok Baek
 
NGINX ADC: Basics and Best Practices
NGINX ADC: Basics and Best PracticesNGINX ADC: Basics and Best Practices
NGINX ADC: Basics and Best Practices
NGINX, Inc.
 
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison DowdneySetting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
Weaveworks
 
PHP5.6からPHP7.0への移行
PHP5.6からPHP7.0への移行PHP5.6からPHP7.0への移行
PHP5.6からPHP7.0への移行
Yasuo Ohgaki
 
Microservices with NGINX pdf
Microservices with NGINX pdfMicroservices with NGINX pdf
Microservices with NGINX pdf
Katherine Bagood
 
Everything you want to know about Ingress
Everything you want to know about IngressEverything you want to know about Ingress
Everything you want to know about Ingress
Janakiram MSV
 
Installing and Configuring NGINX Open Source
Installing and Configuring NGINX Open SourceInstalling and Configuring NGINX Open Source
Installing and Configuring NGINX Open Source
NGINX, Inc.
 
Interceptors: Into the Core of Pedestal
Interceptors: Into the Core of PedestalInterceptors: Into the Core of Pedestal
Interceptors: Into the Core of Pedestal
Kent Ohashi
 
Understanding gRPC Authentication Methods
Understanding gRPC Authentication MethodsUnderstanding gRPC Authentication Methods
Understanding gRPC Authentication Methods
Anthony Chow
 
Project onion - Project Architecture for .Net Core Application
Project onion - Project Architecture for .Net Core ApplicationProject onion - Project Architecture for .Net Core Application
Project onion - Project Architecture for .Net Core Application
Abhinav Jha
 

What's hot (20)

Amazon EKS によるスマホゲームのバックエンド運用事例
Amazon EKS によるスマホゲームのバックエンド運用事例Amazon EKS によるスマホゲームのバックエンド運用事例
Amazon EKS によるスマホゲームのバックエンド運用事例
 
Linux Namespaces
Linux NamespacesLinux Namespaces
Linux Namespaces
 
Intro to vue.js
Intro to vue.jsIntro to vue.js
Intro to vue.js
 
k8s practice 2023.pptx
k8s practice 2023.pptxk8s practice 2023.pptx
k8s practice 2023.pptx
 
Express JS
Express JSExpress JS
Express JS
 
What's new in Spring Batch 5
What's new in Spring Batch 5What's new in Spring Batch 5
What's new in Spring Batch 5
 
Node.jsで使えるファイルDB"NeDB"のススメ
Node.jsで使えるファイルDB"NeDB"のススメNode.jsで使えるファイルDB"NeDB"のススメ
Node.jsで使えるファイルDB"NeDB"のススメ
 
[오픈소스컨설팅] EFK Stack 소개와 설치 방법
[오픈소스컨설팅] EFK Stack 소개와 설치 방법[오픈소스컨설팅] EFK Stack 소개와 설치 방법
[오픈소스컨설팅] EFK Stack 소개와 설치 방법
 
その Pod 突然落ちても大丈夫ですか!?(OCHaCafe5 #5 実験!カオスエンジニアリング 発表資料)
その Pod 突然落ちても大丈夫ですか!?(OCHaCafe5 #5 実験!カオスエンジニアリング 発表資料)その Pod 突然落ちても大丈夫ですか!?(OCHaCafe5 #5 実験!カオスエンジニアリング 発表資料)
その Pod 突然落ちても大丈夫ですか!?(OCHaCafe5 #5 実験!カオスエンジニアリング 発表資料)
 
Creare PWA con Angular
Creare PWA con AngularCreare PWA con Angular
Creare PWA con Angular
 
"행복한 백발의 개발자"라는 제목으로 2024-03-06 어느 IT 업체에서 직책자로 승진한 분들을 대상으로 한...
"행복한 백발의 개발자"라는 제목으로 2024-03-06 어느 IT 업체에서 직책자로 승진한 분들을 대상으로 한..."행복한 백발의 개발자"라는 제목으로 2024-03-06 어느 IT 업체에서 직책자로 승진한 분들을 대상으로 한...
"행복한 백발의 개발자"라는 제목으로 2024-03-06 어느 IT 업체에서 직책자로 승진한 분들을 대상으로 한...
 
NGINX ADC: Basics and Best Practices
NGINX ADC: Basics and Best PracticesNGINX ADC: Basics and Best Practices
NGINX ADC: Basics and Best Practices
 
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison DowdneySetting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison Dowdney
 
PHP5.6からPHP7.0への移行
PHP5.6からPHP7.0への移行PHP5.6からPHP7.0への移行
PHP5.6からPHP7.0への移行
 
Microservices with NGINX pdf
Microservices with NGINX pdfMicroservices with NGINX pdf
Microservices with NGINX pdf
 
Everything you want to know about Ingress
Everything you want to know about IngressEverything you want to know about Ingress
Everything you want to know about Ingress
 
Installing and Configuring NGINX Open Source
Installing and Configuring NGINX Open SourceInstalling and Configuring NGINX Open Source
Installing and Configuring NGINX Open Source
 
Interceptors: Into the Core of Pedestal
Interceptors: Into the Core of PedestalInterceptors: Into the Core of Pedestal
Interceptors: Into the Core of Pedestal
 
Understanding gRPC Authentication Methods
Understanding gRPC Authentication MethodsUnderstanding gRPC Authentication Methods
Understanding gRPC Authentication Methods
 
Project onion - Project Architecture for .Net Core Application
Project onion - Project Architecture for .Net Core ApplicationProject onion - Project Architecture for .Net Core Application
Project onion - Project Architecture for .Net Core Application
 

Similar to Building Services With gRPC, Docker and Go

c++ referesher 1.pdf
c++ referesher 1.pdfc++ referesher 1.pdf
c++ referesher 1.pdf
AnkurSingh656748
 
Parse cloud code
Parse cloud codeParse cloud code
Parse cloud code維佋 唐
 
Programming in C [Module One]
Programming in C [Module One]Programming in C [Module One]
Programming in C [Module One]
Abhishek Sinha
 
Oop concept in c++ by MUhammed Thanveer Melayi
Oop concept in c++ by MUhammed Thanveer MelayiOop concept in c++ by MUhammed Thanveer Melayi
Oop concept in c++ by MUhammed Thanveer Melayi
Muhammed Thanveer M
 
[GEMINI EXTERNAL DECK] Introduction to Gemini.pptx
[GEMINI EXTERNAL DECK] Introduction to Gemini.pptx[GEMINI EXTERNAL DECK] Introduction to Gemini.pptx
[GEMINI EXTERNAL DECK] Introduction to Gemini.pptx
AhmedElbaloug
 
C++ Programming
C++ ProgrammingC++ Programming
C++ Programming
C++ ProgrammingC++ Programming
C++ Programming
Rounak Samdadia
 
A gentle intro to the Django Framework
A gentle intro to the Django FrameworkA gentle intro to the Django Framework
A gentle intro to the Django FrameworkRicardo Soares
 
Software Engineering
Software EngineeringSoftware Engineering
Software Engineering
Tharindu Weerasinghe
 
BreizhCamp 2013 - Pimp my backend
BreizhCamp 2013 - Pimp my backendBreizhCamp 2013 - Pimp my backend
BreizhCamp 2013 - Pimp my backend
Horacio Gonzalez
 
An Overview Of Python With Functional Programming
An Overview Of Python With Functional ProgrammingAn Overview Of Python With Functional Programming
An Overview Of Python With Functional ProgrammingAdam Getchell
 
Lab manual object oriented technology (it 303 rgpv) (usefulsearch.org) (usef...
Lab manual object oriented technology (it 303 rgpv) (usefulsearch.org)  (usef...Lab manual object oriented technology (it 303 rgpv) (usefulsearch.org)  (usef...
Lab manual object oriented technology (it 303 rgpv) (usefulsearch.org) (usef...
Make Mannan
 
Angular JS2 Training Session #1
Angular JS2 Training Session #1Angular JS2 Training Session #1
Angular JS2 Training Session #1
Paras Mendiratta
 
C++ programming
C++ programmingC++ programming
Scalable code Design with slimmer Django models .. and more
Scalable code  Design with slimmer Django models .. and moreScalable code  Design with slimmer Django models .. and more
Scalable code Design with slimmer Django models .. and more
Dawa Sherpa
 
Python and MongoDB
Python and MongoDBPython and MongoDB
Python and MongoDB
Christiano Anderson
 
The First C# Project Analyzed
The First C# Project AnalyzedThe First C# Project Analyzed
The First C# Project Analyzed
PVS-Studio
 
cpp-streams.ppt,C++ is the top choice of many programmers for creating powerf...
cpp-streams.ppt,C++ is the top choice of many programmers for creating powerf...cpp-streams.ppt,C++ is the top choice of many programmers for creating powerf...
cpp-streams.ppt,C++ is the top choice of many programmers for creating powerf...
bhargavi804095
 
An Introduction to TypeScript
An Introduction to TypeScriptAn Introduction to TypeScript
An Introduction to TypeScript
WrapPixel
 

Similar to Building Services With gRPC, Docker and Go (20)

c++ referesher 1.pdf
c++ referesher 1.pdfc++ referesher 1.pdf
c++ referesher 1.pdf
 
Parse cloud code
Parse cloud codeParse cloud code
Parse cloud code
 
Programming in C [Module One]
Programming in C [Module One]Programming in C [Module One]
Programming in C [Module One]
 
Oop concept in c++ by MUhammed Thanveer Melayi
Oop concept in c++ by MUhammed Thanveer MelayiOop concept in c++ by MUhammed Thanveer Melayi
Oop concept in c++ by MUhammed Thanveer Melayi
 
[GEMINI EXTERNAL DECK] Introduction to Gemini.pptx
[GEMINI EXTERNAL DECK] Introduction to Gemini.pptx[GEMINI EXTERNAL DECK] Introduction to Gemini.pptx
[GEMINI EXTERNAL DECK] Introduction to Gemini.pptx
 
C++ Programming
C++ ProgrammingC++ Programming
C++ Programming
 
C++ Programming
C++ ProgrammingC++ Programming
C++ Programming
 
A gentle intro to the Django Framework
A gentle intro to the Django FrameworkA gentle intro to the Django Framework
A gentle intro to the Django Framework
 
Software Engineering
Software EngineeringSoftware Engineering
Software Engineering
 
BreizhCamp 2013 - Pimp my backend
BreizhCamp 2013 - Pimp my backendBreizhCamp 2013 - Pimp my backend
BreizhCamp 2013 - Pimp my backend
 
An Overview Of Python With Functional Programming
An Overview Of Python With Functional ProgrammingAn Overview Of Python With Functional Programming
An Overview Of Python With Functional Programming
 
Lab manual object oriented technology (it 303 rgpv) (usefulsearch.org) (usef...
Lab manual object oriented technology (it 303 rgpv) (usefulsearch.org)  (usef...Lab manual object oriented technology (it 303 rgpv) (usefulsearch.org)  (usef...
Lab manual object oriented technology (it 303 rgpv) (usefulsearch.org) (usef...
 
Angular JS2 Training Session #1
Angular JS2 Training Session #1Angular JS2 Training Session #1
Angular JS2 Training Session #1
 
C++ programming
C++ programmingC++ programming
C++ programming
 
C tutorials
C tutorialsC tutorials
C tutorials
 
Scalable code Design with slimmer Django models .. and more
Scalable code  Design with slimmer Django models .. and moreScalable code  Design with slimmer Django models .. and more
Scalable code Design with slimmer Django models .. and more
 
Python and MongoDB
Python and MongoDBPython and MongoDB
Python and MongoDB
 
The First C# Project Analyzed
The First C# Project AnalyzedThe First C# Project Analyzed
The First C# Project Analyzed
 
cpp-streams.ppt,C++ is the top choice of many programmers for creating powerf...
cpp-streams.ppt,C++ is the top choice of many programmers for creating powerf...cpp-streams.ppt,C++ is the top choice of many programmers for creating powerf...
cpp-streams.ppt,C++ is the top choice of many programmers for creating powerf...
 
An Introduction to TypeScript
An Introduction to TypeScriptAn Introduction to TypeScript
An Introduction to TypeScript
 

Recently uploaded

DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Nexer Digital
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Adtran
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
BookNet Canada
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
James Anderson
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
mikeeftimakis1
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
ControlCase
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
sonjaschweigert1
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
Pierluigi Pugliese
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Paige Cruz
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 

Recently uploaded (20)

DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 

Building Services With gRPC, Docker and Go

  • 1. Building Services With , and ! Martin Kess ♥ Software Engineer github.com/mdkess
  • 3. Agenda ● Background About Namely ● Why Services? ● Protobufs and gRPC - Defining Interfaces ● JSON ● Docker and Docker Compose ● Questions
  • 4. About Namely ● Mission: Build A Better Workplace ● HR, Benefits and Payroll ● 1200 customers ● ~$1 billion in payroll/month ● ~100 engineers ● ~40 services, more shipping every week ● Polyglot environment: React, C# (.NET Core), Go, Ruby and Python ● Modern infrastructure: Kubernetes, Istio, AWS, Docker, Spinnaker. ● Big believers in open-source. We've contributed to the official gRPC C# repo. We open source a lot of the tools we build.
  • 5. What Are (Micro)Services? And Why Build Them, Anyway?
  • 6. A service is software that... ● is the source of truth for its data. ● is independently deployable. ● prevents coupling through use of API contracts. ● adds business value and open up new opportunities. ● has a clear definition of availability (an SLO).
  • 7. Domain Ownership Services don't mean containers or AWS or Kubernetes. It means that pieces of software that own their domain. Services own the reads and writes for their data. Access to this data should be done through APIs (not a shared DB). Don't build a distributed monolith or you'll get all of the weaknesses of services and none of the benefits.
  • 8. Why Namely Uses Services ● In a monolith, teams ended up stepping on each others feet. ○ Accidentally releasing each other team's features. ○ Big changes touching lots of code accidentally break things. ○ Unclear ownership of large parts of the codebase or data. ● Services make teams think in terms of API contracts. ● Teams can use language and tools of their choice. ● Give teams ownership and mastery of their domain.
  • 10. Companies And Employees A Company is a collection of Employee objects and has an Office Location. Every Employee has a name, works for a Company and has a badge number. Every Company has a CEO, who is also an Employee. Company + company_uuid: uuid + ceo_employee_uuid: uuid + office_location: Address Employee + employee_uuid: uuid + company_uuid: uuid + name: string + badge_number: int32
  • 11. A Problem These models are almost certainly wrong. Do all companies have a CEO? Do all companies have one CEO? Do all companies have an office location? Do all companies have only one office location? Are all companies based in America? Do all employees have badge numbers? Is a single name field the best choice? Of course not.
  • 13. Anticipating Change There is no perfect domain model, but our model might be good enough for our current customers. Don't design for a future that might not exist. We want to start with this model and iterate. But in doing so, some things to consider: ● What if you can’t force your old API clients to update? ● How do you release API clients and API servers separately? ○ Very important when doing slow rollouts of software. ● How do you avoid breaking updated API clients after a rollback? ● What if your data is stored on disk? ○ In a message queue, a file or a database.
  • 14. Protocol Buffers Use protocol buffers aka "protobufs"! Message format invented by Google. Supports forward and backward compatibility: newer servers can read messages from old clients and vice versa. A .proto file gets compiled into many languages (C#, Java, Go, Ruby, etc.) Think fancy JSON with a schema.
  • 15. A Simple Proto File example.proto Think of a message as a C struct/POJO/POCO - just data. On each field in the message is the field number (i.e. = 4), this is used when serializing protos. It's not a (default) value. syntax = "proto3"; package examples; message Employee { string employee_uuid = 1; string company_uuid = 2; string name = 3; int32 badge_number = 4; } message Address { string address1 = 1; string address2 = 2; string zip = 3; string state = 4; } message Company { string company_uuid = 1; Address office_location = 2; string ceo_employee_uuid = 3; } github.com/namely/codecamp-2018-go
  • 16. Compiling Protos The protobuf compiler turns protos into code for your language. On the right, we turn our employee.proto from the previous slide into Go code. Can also do C#, JS, Ruby, Python, Java and many other languages. $ docker run -v `pwd`:/defs namely/protoc-all -f example.proto -l go The above command runs the docker container namely/protoc-all to compile the example.proto file into Go code and output the results to `pwd` (the current directory). $ ls example.proto gen/ $ ls gen/pb-go/ example.pb.go github.com/namely/codecamp-2018-go
  • 17. The Generated Code example.pb.go looks something like the right. This code is generated automatically by the namely/protoc-all container. Try running namely/protoc-all with -l python instead. // Code generated by protoc-gen-go. DO NOT EDIT. // source: example.proto package examples ... snip ... type Employee struct { EmployeeUuid string `protobuf:"bytes,1,opt,name=employee_uuid,json=employeeUuid" json:"employee_uuid,omitempty"` CompanyUuid string `protobuf:"bytes,2,opt,name=company_uuid,json=companyUuid" json:"company_uuid,omitempty"` Name string `protobuf:"bytes,3,opt,name=name" json:"name,omitempty"` BadgeNumber int32 `protobuf:"varint,4,opt,name=badge_number,json=badgeNumber" json:"badge_number,omitempty"` } func (m *Employee) Reset() { *m = Employee{} } func (m *Employee) String() string { return proto.CompactTextString(m) } func (*Employee) ProtoMessage() {} func (*Employee) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{0} } ... snip ... github.com/namely/codecamp-2018-go
  • 18. Great! But how do protobufs help us write services?
  • 19. We need a way for our services to talk to each other. Remote Procedure Calls (RPCs) are function calls that can be made over the network.
  • 20. is an open-source RPC framework for building language agnostic servers and clients that can talk to each other. This means your Go/Ruby/C# client can talk to your Python/Java/C++ server (and more). It uses protocol buffers as its message format.
  • 21. Adding Services to example.proto You can also define services in your proto file. These get compiled to gRPC servers and clients that can speak protocol buffers to each other. You can write your server and client in any supported language. service EmployeeService { rpc CreateEmployee(CreateEmployeeRequest) returns (Employee) {} rpc ListEmployees(ListEmployeesRequest) returns (ListEmployeesResponse) {} } message CreateEmployeeRequest { Employee employee = 1; } message ListEmployeesRequest { string company_uuid = 1; } message ListEmployeesResponse { repeated Employee employees = 1; } github.com/namely/codecamp-2018-go
  • 23. Application Structure The Company Service in company/ The Employee Service in employee/ The protobufs in protos/ gen_protos.sh to compile the protos Check out the code! $ git clone github.com/namely/codecamp-2018-go $ ls CODEOWNERS LICENSE README.md docker-compose.yml example.proto gen_protos.sh protos/ company/ employee/ github.com/namely/codecamp-2018-go
  • 24. Diving Into Employee Service Diving into employee/main.go. The main() function listens on a TCP port, creates a new gRPC server and registers our server interface to handle gRPC calls func main() { flag.Parse() lis, err := net.Listen("tcp", fmt.Sprintf("0.0.0.0:%d", *port)) if err != nil { log.Fatalf("error listening: %v", err) } server := grpc.NewServer() pb.RegisterEmployeeServiceServer( server, newServer()) server.Serve(lis) } github.com/namely/codecamp-2018-go
  • 25. The Employee Server The EmployeeServer stores all of the employees in memory. For a real server you would use a database. It also creates a client that talks to company service to check that companies exist. type EmployeeServer struct { companies map[string]*EmployeeCollection conn *grpc.ClientConn companyClient company_pb.CompanyServiceClient } func newServer() *EmployeeServer { s := &EmployeeServer{} s.companies = make(map[string]*EmployeeCollection) s.conn, _ = grpc.Dial( *companyAddr, grpc.WithInsecure()) s.companyClient = company_pb.NewCompanyServiceClient(s.conn) return s } github.com/namely/codecamp-2018-go
  • 26. Looking at a Handler Let's look at the CreateEmployee handler It does three things: 1. Validate the input. 2. Call company service to make sure the company exists 3. Saves the employee. This is the signature of the CreateEmployee function on the EmployeeServer. Input is: - the call's context - a CreateEmployeeRequest proto - the same one we defined in our proto file earlier! func (s *EmployeeServer) CreateEmployee( ctx context.Context, req *employee_pb.CreateEmployeeRequest) (*employee_pb.Employee, error) { .... } Input parameters Return Type (A tuple) github.com/namely/codecamp-2018-go
  • 27. Looking at a Handler Let's look at the CreateEmployee handler It does three things: 1. Validate the input. 2. Call company service to make sure the company exists 3. Saves the employee. Here we check that the employee's name is set. If not, we return an Invalid Argument error to the client. func (s *EmployeeServer) CreateEmployee( ctx context.Context, req *employee_pb.CreateEmployeeRequest) (*employee_pb.Employee, error) { // The employee must have a name. if req.Employee.Name == "" { return nil, status.Error( codes.InvalidArgument, "employee must have name") } .... } github.com/namely/codecamp-2018-go
  • 28. Looking at a Handler Let's look at the CreateEmployee handler It does three things: 1. Validate the input. 2. Call company service to make sure the company exists 3. Saves the employee. Next we call CompanyService.GetCompany with a GetCompanyRequest to check that the employee's company exists. func (s *EmployeeServer) CreateEmployee( ctx context.Context, req *employee_pb.CreateEmployeeRequest) (*employee_pb.Employee, error) { .... _, err := s.companyClient.GetCompany( ctx, &company_pb.GetCompanyRequest{ CompanyUuid: req.Employee.CompanyUuid, }) if err != nil { return nil, status.Error( codes.InvalidArgument, "company does not exist") } .... } github.com/namely/codecamp-2018-go
  • 29. Looking at a Handler Let's look at the CreateEmployee handler It does three things: 1. Validate the input. 2. Call company service to make sure the company exists 3. Saves the employee. Finally, we save the employee and return the saved employee to the caller. In our example, we just save it in memory, but in real life you'd want to use some data storage for this (i.e. a database). func (s *EmployeeServer) CreateEmployee( ctx context.Context, req *employee_pb.CreateEmployeeRequest) (*employee_pb.Employee, error) { .... // If we're here, we can save the employee. return s.SaveEmployee(req.Employee), nil } github.com/namely/codecamp-2018-go
  • 31. Docker lets you build your applications into container (which is sort of like a lightweight virtual machine). This makes it easy to distribute your software and run it anywhere. You make containers by writing a Dockerfile.
  • 32. Dockerfiles Package your application in a container that can be run in various cloud infrastructure. Makes it easy to distribute applications. Here's the Dockerfile for employees. Try building this with $ docker build -t company . The above command builds the Dockerfile in the current directory, and gives the resulting container the name "company". FROM golang:alpine AS build RUN apk add --no-cache git WORKDIR /go/src/github.com/namely/codecamp-2018-go/employee COPY . . RUN go get -d -v ./... RUN go install -v ./... FROM alpine COPY --from=build /go/bin/employee /usr/local/bin/ CMD ["employee"] github.com/namely/codecamp-2018-go
  • 34. Docker-Compose lets you run and configure multiple docker containers. It makes starting and stopping containers easy. It creates DNS names for your containers so they can talk to each other.
  • 35. docker-compose.yml Defines two services company and employee. The build field tells docker-compose how to find your Dockerfile to build your services. github.com/namely/codecamp-2018-go version: "3.6" services: company: build: ./company command: company -port 50051 ports: - 50051:50051 employee: build: ./employee command: > employee -port=50051 -company_addr=company:50051 ports: - 50052:50051 depends_on: - company
  • 36. Bringing Everything Up Build your services with: $ docker-compose build And start them up (in the background) with $ docker-compose up -d github.com/namely/codecamp-2018-go
  • 38. Using the gRPC CLI Namely provides a Docker container that contains the official gRPC CLI for querying gRPC services. Get it with $ docker pull namely/grpc-cli Create some aliases to make calling it easier. docker.for.mac.localhost is how the namely/grpc-cli reaches your local machine where the service is running. $ alias company_call='docker run -v `pwd`/protos/company:/defs --rm -it namely/grpc-cli call docker.for.mac.localhost:50051' $ alias employee_call='docker run -v `pwd`/protos/employee:/defs --rm -it namely/grpc-cli call docker.for.mac.localhost:50052' docker.for.win.localhost on Windows!
  • 39. Creating a Company Let's use the grpc_cli to call CompanyService.CreateCompany. We say docker.for.mac.localhost to let the grpc-cli docker container find localhost on your local machine (where we exposed the port in docker-compose) $ company_call CompanyService.CreateCompany "" --protofiles=company.proto company_uuid: "3ac4f180-9410-467f-92b7-06763db0a8f1"
  • 40. Creating an Employee We'll take the company_uuid from the $ employee_call EmployeeService.CreateEmployee "employee: {name:'Martin', company_uuid: '3ac4f180-9410-467f-92b7-06763db0a8f1'}" --protofiles=employee.proto employee_uuid: "10b286b2-247a-4864-afe5-f56163681af6" company_uuid: "3ac4f180-9410-467f-92b7-06763db0a8f1" name: "Martin"
  • 41. Listing the Employees $ employee_call EmployeeService.ListEmployees "company_uuid: '3ac4f180-9410-467f-92b7-06763db0a8f1'" --protofiles=employee.proto employees { employee_uuid: "3701a099-7c67-40b2-bfac-c0e627efd0f7" company_uuid: "3ac4f180-9410-467f-92b7-06763db0a8f1" name: "Martin" }
  • 42. What About JSON? The Web Doesn't Speak gRPC
  • 43. Adding HTTP Annotations gRPC-Gateway creates an HTTP-gRPC reverse proxy for your gRPC server. Annotations tell it how to map HTTP requests to your gRPC requests. import "google/api/annotations.proto"; service CompanyService { rpc CreateCompany(CreateCompanyRequest) returns (Company) { option (google.api.http) = { put: "/companies/{company_uuid}", body: "company" }; } ... } github.com/namely/codecamp-2018-go
  • 46. Just Kidding No New Code! Just run Namely's Docker container to generate a new server $ docker run -v `pwd`:/defs namely/gen-grpc-gateway -f protos/company/company.proto -s CompaniesService This generates a complete server in gen/grpc-gateway. Now build it. The example repo has this in the docker-compose file as well. $ docker build -t companies-gw -f gen/grpc-gateway/Dockerfile gen/grpc-gateway/ github.com/namely/codecamp-2018-go
  • 47. Using cURL to try our HTTP API Let's wire these together and use cURL to try out our new API. gRPC-Gateway makes it easy to share your services with a front-end application. Bring up our gateway $ docker-compose up -d companies-gw companies Create a company $ curl -X POST -d '{"office_location":{"address1":"foo"}}' localhost:8082/companies {"company_uuid":"d13ecefd-6b63-4919-9b33-e0006ee676ec" ,"office_location":{"address1":"foo"}} Get that company $ curl localhost:8082/companies/d13ecefd-6b63-4919-9b33-e00 06ee676ec {"company_uuid":"d13ecefd-6b63-4919-9b33-e0006ee676ec ","office_location":{"address1":"foo"}} EASY! github.com/namely/codecamp-2018-go
  • 48. Organizing Protos Namely uses a monorepo of all of our protobufs. Services add this as a git submodule. This lets everyone stay up to date. It also serves as a central point for design and API discussions.
  • 49. What Did We Learn? ● How to build services with gRPC, Docker and Go. ● Thinking about services and how to get value from them. ● The importance of backward compatibility and how protobufs help. ● How to compile protobufs using namely/docker-protoc. ● How to use namely/grpc-cli to call your services. ● Using namely/gen-grpc-gateway to create HTTP services for your APIs. ● Use Docker to build your services into containers. ● Using Docker-Compose to bring up multiple containers.
  • 52. GRPC Interceptors Interceptors let you catch calls before they get to the handlers, and before they're returned to the client. 1 2 34 RPC Handler Interceptor Client RPC
  • 53. GRPC Interceptors An interceptor is a function with the signature: func(ctx context.Context, // Info about the call (i.e. deadline) req interface{}, // Request (i.e. CreateCompanyRequest) info *grpc.UnaryServerInfo, // Server info (i.e. RPC method name) handler grpc.UnaryHandler // Your handler for the RPC. ) (resp interface{}, err error) // The response to send, or an error
  • 54. A Typical Interceptor Interceptors let you do some cool stuff: 1. Transform the request before your handler gets it. 2. Transform the response before the client sees it. 3. Add logging and other tooling around the request without having to copy-paste code in all of your handlers. func MetricsInterceptor( ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) { // Get the RPC name (i.e. "CreateCompany") name := info.FullMethod // Start a timer to see how long things take. start := time.Now() // Actually call the handler - your function. out, err := handler(ctx, req) // Check for errors stat, ok := status.FromError(err) // Log to our metrics system (maybe statsd) LogMethodTime(name, start, time.Now(), stat) // Return to the client. We could also change // the response, perhaps by stripping out PII or // doing error normalization/sanitization. return out, err } } github.com/namely/codecamp-2018-go
  • 58. Mock Services As you grow, you won't want to bring up all of your services. With Go and Mockgen, you can make your tests act like a real service.
  • 59. Combining Unit and Integration Tests Your tests can be a hybrid of using unit testing techniques (Mocks) and integration techniques. Mock out some of the dependent services. This is very powerful when testing gRPC servers since we can have tighter control over some dependencies. Instead of bringing up everything, just bring up the dependencies in your service's domain. For Employment, we bring up the database, but not companies service.
  • 60. Hybrid Integration Tests Bring up actual implementations of main services. Use mocks for anything out of the main flow that are used for checks. docker-compose run --use-aliases --service-ports employment-tests Employee Tests Employee Service Employees.CreateEmployee Employee DB CompanyService.GetCompany Calls your test (instead of the real Company service) so that you can control behavior. Docker Compose services: # ... snip ... employee-tests: ports: - 50052 environment: - COMPANIES_PORT=50052 employee: environment: - COMPANIES_HOST=employee-tests - COMPANIES_PORT=50052