Implementing OpenAPI and GraphQL services with gRPC

Implementing OpenAPI and
GraphQL Services with gRPC
Tim Burks (timburks@google.com, @timburks)

1. API technologies: OpenAPI, GraphQL, gRPC
2. Why implement your API with gRPC?
3. Implementing OpenAPI services with gRPC
4. Implementing GraphQL services with gRPC
Topics

timburks@google.com
@timburks
Electronic Design Automation
Mobile Apps
APIs
Hello!

Projects at Google
gnostic
(OpenAPI Compiler)
gRPC-Swift
GAPICs for
Google APIs

1
API Technologies:
OpenAPI, GraphQL,
gRPC

APIs are Interfaces
for Application
Programmers.

Five Questions about API Systems
1. What problem is this API system addressing?
2. How are APIs called?
3. How are APIs built?
4. What’s great about this?
5. What’s challenging about this?

What problem is OpenAPI addressing?
“The OpenAPI Specification, originally known
as the Swagger Specification, is a
specification for machine-readable interface
files for describing, producing, consuming, and
visualizing RESTful web services.”
Wikipedia

How an OpenAPI service is called:
You might call it from your browser.
In your applications, you call it from code.
openapi-generator
autorest
APIMatic
SwagGen (Swift)
Paperclip (Rust)

How an OpenAPI service is built:
🤞 Code-first
Handwritten implementations.
Handwritten API descriptions.
⚙ Code-first (better)
Handwritten implementations.
API descriptions generated
from code.
📐 Spec-first
Handwritten API descriptions.
Implementations generated
from API descriptions.
?🤖

What’s great about OpenAPI:
● Standards and tools make it easy to get
started.
○ HTTP
○ JSON/YAML
● Very popular - large community.

What’s challenging about OpenAPI:
REST?
“Unfortunately, many of the APIs that claim to be RESTful
layer a lot of proprietary concepts on top of HTTP.
Essentially, they invent their own API and use HTTP as a
lower-level transport layer, rather than using HTTP directly as
it was designed. In fact, there’s so much variability in the way
that people use the term REST in the context of APIs that it’s
difficult to know what they mean by it unless you know them
well.”
Martin Nally

What problem is GraphQL addressing?
“GraphQL is a query language for APIs and a
runtime for fulfilling those queries with your
existing data. GraphQL provides a complete
and understandable description of the data in
your API, gives clients the power to ask for
exactly what they need and nothing more,
makes it easier to evolve APIs over time, and
enables powerful developer tools.”
graphql.org

How a GraphQL service is called:
“With express-graphql, you can just send an
HTTP POST request to the endpoint you
mounted your GraphQL server on, passing the
GraphQL query as the query field in a JSON
payload.”
https://graphql.org/graphql-js/graphql-clients/
Also GraphiQL!

How a GraphQL service is built:
● Code-first, schema extracted from code
● Schema-first - code generated from SDL
● Generated from underlying APIs
○ Federated from other GraphQL APIs
○ OpenAPI-to-GraphQL
○ rejoiner

What’s great about GraphQL:
● Simple client-side interface
● Optimizes for client performance
● Layered on familiar technology
● Sharable BFF

What’s challenging about GraphQL:
● Complex implementation
● Performance challenges
● Versioning?
● Schema design
Public GraphQL APIs are rare.

What problem is gRPC addressing?
“gRPC is a modern open source high
performance RPC framework that can run in
any environment. It can efficiently connect
services in and across data centers with
pluggable support for load balancing, tracing,
health checking and authentication. It is also
applicable in last mile of distributed computing
to connect devices, mobile applications and
browsers to backend services.”
grpc.io

How a gRPC service is called:
ALWAYS specification-first.
Clients are generated from API
descriptions written in the Protocol Buffer
language.

How a gRPC service is built:
ALWAYS specification-first.
Service code is generated. Typically this
generated code declares an interface that is
implemented in language-native data
structures and patterns.

What’s great about gRPC:
So much is built-in.
High performance.
Cross-language compatibility.
Potential for very nice client integration.

What’s challenging about gRPC:
Some learning and tooling is required!

2
Why implement
your API with
gRPC?

APIs have a native language.
Protocol Buffers

Protocol Buffers is a language-neutral, platform-neutral, extensible
mechanism for serializing structured data.

“Protocol Buffers” means several things
1. A serialization mechanism
2. An interface description language
3. A methodology

Protocol Buffer Serialization
A message is just a stream of bytes
[field_number<<3 + wire_type] [length if necessary][data]...
$ hexdump /tmp/request.bin
0000000 0a 05 68 65 6c 6c 6f
0a is “0000 1010”, so
field_number = 1 and wire_type = 2

Protocol Buffer IDL
message EchoMessage {
string text = 1;
// other fields...
}

Protocol Buffer Methodology
% protoc echo.proto --go_out=.
# This runs the Protocol Buffer Compiler
# (https://github.com/protocolbuffers/protobuf/releases)
#
# with a plugin called protoc-gen-go
#
# The plugin generates a Go source file that implements
# the data structures defined in echo.proto and code
# for reading and writing them as serialized bytes.

Try it!
package main
import (
"fmt"
"github.com/golang/protobuf/proto"
echo "./generated"
)
func main() {
document := echo.EchoMessage{Text:"hello"}
bytes, _ := proto.Marshal(&document)
fmt.Printf("%+vn", bytes)
}

gRPC scales to multiple languages.
Java
Service
Python
Service
Go
Service
C++
Service
gRPC
Server
gRPC
Stub
gRPC
Stub
gRPC
Stub
gRPC
Stub
gRPC
Server
gRPC
Server
gRPC
Server
gRPC
Stub

gRPC scales to distributed systems.

Load
Balancer
Users
SQL Database
Cluster
NoSQL Database
Cluster
ETL Pipeline
Front End Business
Logic
Back End Business
Logic
Data Warehouse
Invoicing
Transcoding
Illustration: Sandeep Dinesh, Google, Developer Advocate

Load Balancer
Users
SQL Database Cluster
NoSQL Database Cluster
ETL Pipeline
Front End Business Logic
Back End Business Logic
Data Warehouse
Load Balancer
Users
ETL Pipeline
Data Warehouse
Load Balancer
Users
ETL Pipeline
Data Warehouse
Load Balancer
Users
ETL Pipeline
Data Warehouse
Illustration: Sandeep Dinesh, Google, Developer Advocate

Load Balancer
Users
ETL Pipeline
Data Warehouse
Load Balancer
Users
ETL Pipeline
Data Warehouse
Load Balancer
Users
ETL Pipeline
Data Warehouse
Load Balancer
Users
ETL Pipeline
Data Warehouse
API
API

Generated API Clients
What do you get?

Trivial Getting-Started Experience
GAPIC client:
Channel channel =
NettyChannelBuilder.forAddress(API_ENTRY, API_PORT).build();
List<ClientInterceptor> interceptors = new ArrayList<>();
GoogleCredentials credentials =
GoogleCredentials.getApplicationDefault();
interceptors.add(
ChannelFactory.credentialInterceptor(credentials));
LoggingService stub =
LoggingServiceGrpc.newBlockingStub(channel, interceptors);
gRPC:
LoggingClient client = LoggingClient.create();

Flattening, Automatic Retry
GAPIC client:
LoggingService stub = ...;
// Note: this only does a simple retry, exponential backoff
// is more complex
DeleteLogRequest request = DeleteLogRequest.newBuilder()
.setLogName(LOG_NAME)
.build();
for (int i = 0; i < MAX_RETRY; i++) {
try {
stub.deleteLog(request);
} catch (RpcException e) {
if (i == MAX_RETRY) throw e;
}
}
gRPC:
client.deleteLog(LOG_NAME);

Natural Pagination
GAPIC client:
LoggingService stub = ...;
ListLogsRequest request = ListLogsRequest.newBuilder()
.setParentName("projects/" + PROJECT_ID)
.build();
while (true) {
ListLogsResponse response = stub.listLogs(request);
for (Log log : response.getLogsList()) {
System.out.printf("Log: %s%n", log.getDisplayName());
}
String nextPageToken = response.getNextPageToken();
if (nextPageToken != null) {
request = ListLogsRequest.newBuilder()
.setPageToken(nextPageToken).build();
} else {
break;
}
}
gRPC:
ParentNameOneOf parent =
ParentNameOneOf.from(ProjectName.create(PROJECT_ID));
for (Log log : client.listLogs(PARENT).iterateAll()) {
System.out.printf("Log: %s%n", log.getDisplayName());
}

Long-Running Operations
SpeechClient speechClient = SpeechClient.create();
OperationFuture<LongRunningRecognizeResponse> recognizeOperation =
speechClient.longRunningRecognizeAsync(config, audio);
…
LongRunningRecognizeResponse response = recognizeOperation.get();
● Java: OperationFuture
○ Polls the service until the LRO is done
○ Provides metadata as the LRO is in progress
○ Provides the final result

Resource Name Types
GAPIC client:
PublisherService stub = ...;
CreateTopicRequest request = CreateTopicRequest.newBuilder()
.setTopic("projects/my-project/topics/my-topic")
.build();
Topic response = stub.createTopic(request);
gRPC:
TopicAdminClient topicAdminClient = TopicAdminClient.create();
TopicName name = TopicName.create("my-project", "my-topic");
Topic response = topicAdminClient.createTopic(name);

GAPIC Project Architecture (at github.com/googleapis)
gapic-generator-python
Wanted: Swift, Rust, Dart...
gapic-generator
(java, php, python, go,
csharp, node, ruby)
gapic-generator-ruby
gapic-generator-go
gapic-generator-typescript
gapic-generator-csharp
One API
description
(Annotated
.proto ﬁles)
Many
language-
speciﬁc
client
libraries

“Showcase” API verification

protoc-gen-go_cli
A protoc plugin that generates
CLIs in Go for APIs described
with protocol buffers.

gRPC Fallback
● Go proxy
● Description and handwritten examples
● Node client (gax-nodejs)

gRPC Reflection
● Protocol Discussion
● Go implementation (example)
● Specification (protos)

Manage gRPC Services with Cloud Endpoints

Cloud Endpoints Architecture
GCE, GKE, Kubernetes or App Engine
Flexible
Environment Instance
GCE, GKE, Kubernetes or App Engine Flexible
Environment Instance (or off-GCP)
Extensible Service
Proxy Container
API Container
Google Service
Management API
User
Code
api
calls
gcloud
Config
deployment
Google Cloud
Console
Google Service
Control API
config Runtime
check & report
Load
balanci
ng
Stackdriver
Metrics &
logs
Metrics &
logs
visualized

3
Implementing
OpenAPI Services
with gRPC

Build a gRPC service, get a free REST API!
Originally defined in google/api/http.proto
Now documented in AIP-127: HTTP and gRPC Transcoding

service Display {
rpc CreateKiosk(Kiosk) returns (Kiosk) {
option (google.api.http) = {post:"/v1/kiosks"};
}
rpc GetKiosk(GetKioskRequest) returns (Kiosk) {
option (google.api.http) =
{get: "/v1/kiosks/{id}" };
}
rpc DeleteKiosk(DeleteKioskRequest)
returns (google.protobuf.Empty) {
option (google.api.http) =
{ delete: "/v1/signs/{id}" };
}
[...]
}
Add REST annotations to your service
gRPC definitions with HTTP annotations
service Display {
rpc CreateKiosk(Kiosk) returns (Kiosk) {
}
rpc GetKiosk(GetKioskRequest) returns(Kiosk) {
}
rpc DeleteKiosk(DeleteKioskRequest)
returns (google.protobuf.Empty) {
}
[...]
}
gRPC definitions

# Call the CreateKiosk method
$ curl -d '{"name":"HTTP Kiosk", "size": { width: 1080, height: 720 } }'
localhost:8082/v1/kiosks?key=AIzaSy[...]bBo
# Call the GetKiosk method
$ curl localhost:8082/v1/kiosks/1?key=AIzaSy[...]bBo
Now you can use HTTP+JSON!

Schema-first OpenAPI
gnostic-grpc

gnostic-grpc
● 6 useful things I learned from GSoC
● How to build a REST API with gRPC and get
the best of two worlds
● gnostic-grpc (end-to-end example)
● Envoy gRPC-JSON transcoding

4
Implementing
GraphQL Services
with gRPC

Code-first or Schema-first?
● Evolving the Graph (Jon Wong, Coursera)
● Hard-Learned GraphQL Lessons: Based on
a True Story (Natalie Qabazard & Aditi
Garg, Trulia)
● The Problems of "Schema-First" GraphQL
Server Development (Nikolas Burk, Prisma)

Code-first GraphQL with Rejoiner

Medallion (Rejoiner demo by Danny Baggett)

Sample query (Rejoiner + Medallion)
{
events {
id
name
olympics {
year
season
}
olympians {
id
gender
last
first
}
}
}

Sample query (Rejoiner + Medallion)
{
getEvents(input: {year: 2018, season: "WINTER"}) {
id
name
}
}

Let’s hand-wrap a gRPC service (graphql-showcase)

Impressions / Project Ideas
gRPC has a lot of API information in the .proto files and we
can extend that with annotations.
There are some gaps between GraphQL assumptions and
gRPC practices that we would want to smooth with
configuration or conventions.
We could generate a GraphQL interface directly from gRPC
reflection.
Can we define a standard for gRPC-GraphQL transcoding?

resolver
in-mem
response
schema
GraphQL server
on Apigee
Hosted Target
Google Apigee and GraphQL
app
Edge
● AuthZ
● security
● aggregate analytics
Integrated Dev Portal
● GraphQL playgroundapp developer
AX plug-in
batch AX
app
GQL query
response
GQL query
response
Stackdriver
● query-level analytics
on-board,
use APIs

Implementing OpenAPI and GraphQL services with gRPC

Implementing OpenAPI and GraphQL services with gRPC

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Implementing OpenAPI and GraphQL services with gRPC

Similar to Implementing OpenAPI and GraphQL services with gRPC (20)

More from Tim Burks

More from Tim Burks (11)

Recently uploaded

Recently uploaded (20)

Implementing OpenAPI and GraphQL services with gRPC