This half-day tutorial introduces Protocol Buffers, gRPC, and the open source tools that Google uses to publish and support some of the world's biggest APIs. We'll show how the Protocol Buffer language allows APIs to be described, reviewed, and implemented in a programming-language independent way, how gRPC enables high-performance streaming APIs, and how \ a few simple conventions can enable related tools to serve robust REST APIs and generate production-quality client libraries in seven popular programming languages. This is API publishing the Google way, but large teams aren't required. With shared open-source tooling, even the smallest developer can build scalable, usable APIs that delight. https://apistrat18.sched.com/event/FTR3/usable-apis-at-scale-with-protocol-buffers-and-grpc-tim-burks-andrew-gunsch-google

1. Welcome to Usable APIs at Scale!
We'll use a Google Compute Engine Virtual Machine in our
examples.
Get a free Google Cloud account at
https://cloud.google.com/free.
Install the Google Cloud SDK.
https://cloud.google.com/sdk/
Clone the Kiosk API repository.
% git clone github.com/googleapis/kiosk

2. Usable APIs at Scale
With Protocol Buffers and gRPC
Tim Burks, Joe Bolinger, Andrew Gunsch
Google
APIStrat 2018

3. We’re talking about Networked APIs
From the Google Cloud APIs Design Guide:
Networked APIs
Application Programming Interfaces that operate across a network of
computers. They communicate using network protocols including HTTP, and
are frequently produced by different organizations than the ones that
consume them.

4. We’re (mostly) talking about RPC APIs.

6. 10 billion+
API calls
every second

8. October 4, 2000
CL 23852 by jeff
ProtocolBuffer, a class for encoding hierarchical data in a compact
binary form. Initially to be used for sending raw posting list data
out from the qserver to clients that want to do their own scoring.
Co-designed and Rev. by Sanjay

9. Protocol Buffers is a language-neutral, platform-neutral, extensible mechanism
for serializing structured data.

10. “Protocol Buffers” means several things
1. A serialization mechanism
2. An interface description language
3. A methodology

11. Protocol Buffer Serialization
A message is just a stream of bytes
[field_number<<3 + wire_type] [length if necessary][data]...
$ hexdump /tmp/request.bin
0000000 0a 05 68 65 6c 6c 6f
0a is “0000 1010”, so
field_number = 1 and wire_type = 2

12. Protocol Buffer IDL
message EchoMessage {
string text = 1;
// other fields...
}

13. Protocol Buffer Methodology
% protoc echo.proto --go_out=.
# This runs the Protocol Buffer Compiler
# (https://github.com/protocolbuffers/protobuf/releases)
#
# with a plugin called protoc-gen-go
#
# The plugin generates a Go source file that implements
# the data structures defined in echo.proto and code
# for reading and writing them as serialized bytes.

14. Try it!
package main
import (
"fmt"
"github.com/golang/protobuf/proto"
echo "./generated"
)
func main() {
document := echo.EchoMessage{Text:"hello"}
bytes, _ := proto.Marshal(&document)
fmt.Printf("%+vn", bytes)
}

15. gRPC
Open-Source messaging system based on Google’s internal API architecture.
Used for code and documentation generation, API management, and support
services for APIs and microservices running at very large scale.

16. gRPC is owned by the Cloud Native Computing Foundation

17. gRPC Adoption
Microservices: in data centres
Streaming telemetry from network devices
Client Server communication/Internal APIs
Mobile Apps

18. gRPC
Transport Mechanism
Client → Server
Server → Client
Initial
Metadata
MsgMsg
End of
Stream
Status & Trailing
Metadata
Initial
Metadata
MsgMsg Msg
HTTP/2

20. Cross-Language Interoperability
Java
Service
Python
Service
GoLang
Service
C++
Service
gRPC
Service
gRPC
Stub
gRPC
Stub
gRPC
Stub
gRPC
Stub
gRPC
Service
gRPC
Service
gRPC
Service
gRPC
Stub

21. A sample application: the Kiosk API

22. Describing the API
Kiosks
// Describes a hardware device that can display signs.
message Kiosk {
string id = 1; // unique id
string device_name = 2; // name of device hardware
ScreenSize size = 3; // screen size
LatLng location = 4; // kiosk location
Timestamp create_time = 5;
}

23. Describing the API
ScreenSizes
// Represents the size of a screen in pixels.
message ScreenSize {
int32 width = 1; // screen width
int32 height = 2; // screen height
}

24. Describing the API
Signs
// Describes a digital sign.
// Signs can include text, images, or both.
message Sign {
string id = 1; // unique id
string text = 2; // text to display
bytes image = 3; // image to display
Timestamp create_time = 4;
}

25. Describing the API: RPCs
Operation
● GetSignForKiosk
● GetSignsForKiosk
Kiosks
● Create
● List
● Get
● Delete
Control
● SetSignForKiosks
Signs
● Create
● List
● Get
● Delete
called from
controllers
called from
kiosks
hardware
devices
displayed
on kiosks

26. Describing the API: RPCs
service Display {
rpc CreateKiosk(Kiosk) returns (Kiosk) {}
rpc ListKiosks(google.protobuf.Empty) returns (ListKiosksResponse) {}
rpc GetKiosk(GetKioskRequest) returns (Kiosk) {}
rpc DeleteKiosk(DeleteKioskRequest) returns (google.protobuf.Empty) {}
rpc CreateSign(Sign) returns (Sign) {}
rpc ListSigns(google.protobuf.Empty) returns (ListSignsResponse) {}
rpc GetSign(GetSignRequest) returns (Sign) {}
rpc DeleteSign(DeleteSignRequest) returns (google.protobuf.Empty) {}
rpc SetSignIdForKioskIds(SetSignIdForKioskIdsRequest)
returns (google.protobuf.Empty) {}
rpc GetSignIdForKioskId(GetSignIdForKioskIdRequest)
returns (GetSignIdResponse) {}
rpc GetSignIdsForKioskId(GetSignIdForKioskIdRequest)
returns (stream GetSignIdResponse) {}
}

27. Google API Design Guide

28. API Design Guide
• Resource oriented design
• Resources are named entities
• Methods attached to resources
• Standard: List, Get, Create, Update, Delete
• Custom: async, batch, media, streaming

29. API Standardization
• Types: google.protobuf, google.type
• Interfaces: google.longrunning, google.watcher
• Fields: region_code, display_name, labels
• Naming: create_time, node_count
• Repo: file and directory structure

30. Generate Kiosk Glue Code
git clone https://github.com/googleapis/api-common-protos
go get github.com/golang/protobuf/protoc-gen-go
go get google.golang.org/grpc
mkdir -p generated
protoc kiosk.proto
-I api-common-protos
-I .
--go_out=plugins=grpc:generated

31. Implement Kiosk Server Interface
type DisplayClient interface {
CreateKiosk(ctx context.Context, in *Kiosk,
opts ...grpc.CallOption) (*Kiosk, error)
ListKiosks(ctx context.Context, in *google_protobuf.Empty,
opts ...grpc.CallOption) (*ListKiosksResponse, error)
GetKiosk(ctx context.Context, in *GetKioskRequest,
opts ...grpc.CallOption) (*Kiosk, error)
DeleteKiosk(ctx context.Context, in *DeleteKioskRequest,
opts ...grpc.CallOption) (*google_protobuf.Empty, error)
...
GetSignIdsForKioskId(ctx context.Context, in *GetSignIdForKioskIdRequest,
opts ...grpc.CallOption) (Display_GetSignIdsForKioskIdClient, error)
}

32. Try the API with a CLI
$ k
Usage:
k create kiosk <name>
k list kiosks
k get kiosk <kiosk_id>
k delete kiosk <kiosk_id>
k create sign <name> [--text=<text>] [--image=<image>]
k list signs
k get sign <sign_id>
k delete sign <sign_id>
k set sign <sign_id> for kiosk <kiosk_id>
k set sign <sign_id> for all kiosks
k get sign for kiosk <kiosk_id>
k get signs for kiosk <kiosk_id>

33. Let’s try it.
Be sure that you have the Google Cloud SDK.
https://cloud.google.com/sdk/
% git clone github.com/googleapis/kiosk
% cd kiosk/gce
% ./SETUP.sh <project-id> <instance-name>

34. Generate and Publish API Client Libraries
“Make it easier to use Google APIs”
1. Hide implementation details from API consumers.
2. Develop API clients faster.
3. Improve API client quality (better performance, fewer bugs).

35. Recent API Client Work
2010 Google API Discovery Format, Automatically-generated clients
2014 Veneer (handwritten clients, measurably improved usability)
2015 Scalable Veneer/Veneer Toolkit
Centralized and monolithic code generation
7 languages
21 Cloud APIs by March 2018
2018 API Client Tools
Decentralized and standardized code generation
More languages
More APIs

36. Generated API Clients
a.k.a.
“GAPICs”

37. Customers
Google Cloud APIs
Google Ads APIs
… other Google teams
… external API producers?

38. GAPIC Features and Benefits
● Usability:
○ Trivial getting-started experience
○ Flattening
○ Natural paging
○ Long-running operations
○ Sample code for every method
○ Resource name types (e.g. for /projects/{project}/topics/{topic})
● Performance:
○ Batching
● Reliability:
○ Automatic retries

39. Trivial Getting-Started Experience
GAPIC client:
Channel channel =
NettyChannelBuilder.forAddress(API_ENTRY, API_PORT).build();
List<ClientInterceptor> interceptors = new ArrayList<>();
GoogleCredentials credentials =
GoogleCredentials.getApplicationDefault();
interceptors.add(
ChannelFactory.credentialInterceptor(credentials));
LoggingService stub =
LoggingServiceGrpc.newBlockingStub(channel, interceptors);
gRPC:
LoggingClient client = LoggingClient.create();

40. Automatic Retry and Flattening
GAPIC client:
LoggingService stub = ...;
// Note: this only does a simple retry, exponential backoff
// is more complex
DeleteLogRequest request = DeleteLogRequest.newBuilder()
.setLogName(LOG_NAME)
.build();
for (int i = 0; i < MAX_RETRY; i++) {
try {
stub.deleteLog(request);
} catch (RpcException e) {
if (i == MAX_RETRY) throw e;
}
}
gRPC:
LoggingClient client = LoggingClient.create();
client.deleteLog(LOG_NAME);

41. Natural Pagination
GAPIC client:
LoggingService stub = ...;
ListLogsRequest request = ListLogsRequest.newBuilder()
.setParentName("projects/" + PROJECT_ID)
.build();
while (true) {
ListLogsResponse response = stub.listLogs(request);
for (Log log : response.getLogsList()) {
System.out.printf("Log: %s%n", log.getDisplayName());
}
String nextPageToken = response.getNextPageToken();
if (nextPageToken != null) {
request = ListLogsRequest.newBuilder()
.setPageToken(nextPageToken).build();
} else {
break;
}
}
gRPC:
LoggingClient client = LoggingClient.create();
ParentNameOneOf parent =
ParentNameOneOf.from(ProjectName.create(PROJECT_ID));
for (Log log : client.listLogs(PARENT).iterateAll()) {
System.out.printf("Log: %s%n", log.getDisplayName());
}

42. Long-Running Operations
SpeechClient speechClient = SpeechClient.create();
OperationFuture<LongRunningRecognizeResponse> recognizeOperation =
speechClient.longRunningRecognizeAsync(config, audio);
…
LongRunningRecognizeResponse response = recognizeOperation.get();
● Java: OperationFuture
○ Polls the service until the LRO is done
○ Provides metadata as the LRO is in progress
○ Provides the final result

43. Resource Name Types
GAPIC client:
PublisherService stub = ...;
CreateTopicRequest request = CreateTopicRequest.newBuilder()
.setTopic("projects/my-project/topics/my-topic")
.build();
Topic response = stub.createTopic(request);
gRPC:
TopicAdminClient topicAdminClient = TopicAdminClient.create();
TopicName name = TopicName.create("my-project", "my-topic");
Topic response = topicAdminClient.createTopic(name);

44. Generated Sample Code

45. GAPIC Project Architecture (hosted at github.com/googleapis)
gapic-generator-python
WANTED: ruby, swift, rust, elixir...
gapic-generator
(java, php, python, go,
csharp, node, ruby)
gapic-generator-kotlin
gapic-generator-go
gapic-generator-csharp
gapic-generator-typescript
One API
description
(Annotated
.proto files)
Many
language-
specific
client
libraries

46. “Showcase” API verification

47. Kotlin Kiosk (Android Things)
https://shop.technexion.com/pico-pi-imx7-startkit-rainbow-hat.html

48. Generate Kotlin Client
$ mkdir kotlin
$ docker run --rm
--mount type=bind,source="$(pwd)"/protos,target=/proto
--mount type=bind,source="$(pwd)"/kotlin,target=/generated
gcr.io/kotlin-gapic/kgen

50. Manage Your Service With Endpoints
Develop, deploy and manage APIs on any Google Cloud Platform backend.
● User Authentication via JSON Web Token validation
● Logging and Monitoring
● API Keys
● Easy integration + deployment

51. Endpoints Architecture
GCE, GKE, Kubernetes or App Engine
Flexible
Environment Instance
GCE, GKE, Kubernetes or App Engine Flexible
Environment Instance (or off-GCP)
Extensible Service
Proxy Container
API Container
Google Service
Management API
User
Code
api
calls
gcloud
Config
deployment
Google Cloud
Console
Google Service
Control API
config Runtime
check & report
Load
balanci
ng
Stackdriver
Metrics &
logs
Metrics &
logs
visualized

52. Generate FileDescriptorSet
protoc kiosk.proto
-I api-common-protos
-I .
--include_imports
--include_source_info
--descriptor_set_out=generated/kiosk_descriptor.pb
--go_out=plugins=grpc:generated

53. Describing the Service
type: google.api.Service
config_version: 3
name: my-api.endpoints.my-gcp-project.cloud.goog
title: Kiosk gRPC API
apis:
- name: kiosk.Display

54. Starting Endpoints Proxy in Front of Application
gcloud endpoints services deploy service.yaml kiosk_descriptor.pb
./endpoints/START_ENDPOINTS_PROXY.sh my-gcp-project my-api <keyfile>

55. $ export KIOSK_PORT=8083
$ k list kiosks
FROM localhost:8083 LIST KIOSKS
rpc error: code = Unauthenticated
desc = Method doesn't allow
unregistered callers (callers
without established identity).
Please use API Key or other form of
API consumer identity to call this
API.
Starting Endpoints Proxy in Front of Application

56. $ export KIOSK_APIKEY=AIzaSy[...]bBo
$ k list kiosks
FROM localhost:8083 2018/09/21 19:08:25 Using API key: AIzaSy[...]bBo
LIST KIOSKS
id:1 name:"HTTP Kiosk" size:<width:1080 height:720 >
id:2 name:"HTTP Kiosk" size:<width:1080 height:720 >
Getting an API Key

57. Build a gRPC service, get a free REST API!

58. service Display {
rpc CreateKiosk(Kiosk) returns (Kiosk) {
option (google.api.http) = {post:"/v1/kiosks"};
}
rpc GetKiosk(GetKioskRequest) returns (Kiosk) {
option (google.api.http) =
{get: "/v1/kiosks/{id}" };
}
rpc DeleteKiosk(DeleteKioskRequest)
returns (google.protobuf.Empty) {
option (google.api.http) =
{ delete: "/v1/signs/{id}" };
}
[...]
}
Add REST annotations to your service
gRPC definitions with HTTP annotations
service Display {
rpc CreateKiosk(Kiosk) returns (Kiosk) {
}
rpc GetKiosk(GetKioskRequest) returns(Kiosk) {
}
rpc DeleteKiosk(DeleteKioskRequest)
returns (google.protobuf.Empty) {
}
[...]
}
gRPC definitions

59. # Add a kiosk
$ curl -d '{"name":"HTTP Kiosk", "size": { width: 1080, height: 720 } }'
localhost:8082/v1/kiosks?key=AIzaSy[...]bBo
# Get kiosk
$ curl localhost:8082/v1/kiosks/1?key=AIzaSy[...]bBo
Now you can use HTTP+JSON!

60. Use Endpoints, get a free developer portal!