Slides of the Devoxx Belgium 2022 talk "gRPC Cornerstone: HTTP/2… or HTTP/3?"

1. gRPC Cornerstone:
HTTP/2… or HTTP/3?
Oct 13, 2022
Devoxx Belgium
Alex BORYSOV
Mykyta PROTSENKO

2. @aiborisov
@mykyta_p
Manage Profiles
Mykyta Alex
Who’s presenting?

3. @aiborisov
@mykyta_p
Manage Profiles
Mykyta Alex
Who’s presenting?

4. @aiborisov
@mykyta_p
gRPC?
The image is property of Netflix Inc. and cannot be reused

5. @aiborisov
@mykyta_p
schemes:
- "https"
- "http"
paths:
/currency:
post:
summary: "Create new currency"
description: "Create a new currency to be used in minting"
operationId: "createCurrency"
consumes:
- "application/json"
produces:
- "application/json"
responses:
"200":
description: "Success"
schema:
$ref: "#/definitions/Currency"
"400":
description: "Invalid input"
get:
summary: "List existing currencies"
description: "List existing currencies sorted by most minted coins"
operationId: "getTopMintedCurrencies"
produces:
- "application/json"
responses:
"200":
description: "successful operation"
schema:
type: "array"
items:
$ref: "#/definitions/Currency"
"500":
description: "Internal server error"
/currency/{currency}:
put:
parameters:
- name: "currency"
in: "path"
required: true
description: "Currency object to be created"
type: "string"
summary: "Mint currency"
description: "Create more coins for a given currency"
operationId: "mint"
consumes:
- "application/json"
produces:
- "application/json"
responses:
"200":
description: "Success"
schema:
$ref: "#/definitions/Currency"
"400":
description: "Invalid input"
REST API DEFINITIONS (DON’T READ!)

6. @aiborisov
@mykyta_p
syntax = "proto3";
service MintService {
rpc CreateCurrency (CreateCurrencyRequest)
returns (CreateCurrencyResponse);
rpc Mint (MintRequest) returns (MintResponse);
}
gRPC API DEFINITIONS

7. @aiborisov
@mykyta_p
MINT SERVICE
gRPC
Stub
gRPC
Service
MINT.PROTO
gRPC over
HTTP/2

8. @aiborisov
@mykyta_p
gRPC: TROUBLESHOOTING and TUNING?
The image is property of Netflix Inc. and cannot be reused

9. @aiborisov
@mykyta_p
HTTP over TCP
SERVER
TCP
TCP
TCP
TCP
TCP
HTTP/1

10. @aiborisov
@mykyta_p
HTTP/2
HTTP/2 STREAMS
…

11. @aiborisov
@mykyta_p
HTTP/2
HTTP/2 STREAMS
…
TCP

12. @aiborisov
@mykyta_p
HTTP/2 FRAME
LENGTH (24)
FLAGS (8)
TYPE (8)
STREAM ID (31)
FRAME PAYLOAD
(0 … )

13. HTTP/2
TCP
@aiborisov
@mykyta_p

14. HTTP/2
TCP
SID 1
@aiborisov
@mykyta_p

15. HTTP/2
TCP
SID 1
SID 3
@aiborisov
@mykyta_p

16. HTTP/2
…
TCP
SID 1
SID 3
SID X
@aiborisov
@mykyta_p

17. HTTP/2
…
TCP
SID 1 SID 1
@aiborisov
@mykyta_p

18. HTTP/1.1
TCP
TCP
TCP
TCP
TCP
@aiborisov
@mykyta_p
Icons https://commons.wikimedia.org/wiki/File:Feather-core-shopping-cart.svg
MIT License

19. HTTP/1.1: HEAD-OF-LINE BLOCKING
TCP
TCP
TCP
TCP
TCP
???
@aiborisov
@mykyta_p
Icons https://commons.wikimedia.org/wiki/File:Feather-core-shopping-cart.svg
MIT License

20. HTTP/2
SID 1 SID 1
@aiborisov
@mykyta_p
Icons used: https://www.flaticon.com/free-icon/milk_1626757, https://www.flaticon.com/free-icon/beet_7179853,
https://www.flaticon.com/premium-icon/bread_750878, https://www.flaticon.com/free-icon/strawberry_7189510

21. HTTP/2
SID 1
SID 3
SID 1
SID 3
@aiborisov
@mykyta_p
Icons used: https://www.flaticon.com/free-icon/milk_1626757, https://www.flaticon.com/free-icon/beet_7179853,
https://www.flaticon.com/premium-icon/bread_750878, https://www.flaticon.com/free-icon/strawberry_7189510

22. https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
gRPC over HTTP/2
@aiborisov
@mykyta_p

23. https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
gRPC over HTTP/2
HEADERS
REQUEST
@aiborisov
@mykyta_p

24. https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
gRPC over HTTP/2
HEADERS
REQUEST
DATA
DATA
@aiborisov
@mykyta_p

25. https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
gRPC over HTTP/2
HEADERS
RESPONSE
@aiborisov
@mykyta_p

26. https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
gRPC over HTTP/2
HEADERS
RESPONSE
DATA
DATA
@aiborisov
@mykyta_p

27. https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
gRPC over HTTP/2
HEADERS HEADERS
RESPONSE
DATA
DATA
@aiborisov
@mykyta_p

28. https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
gRPC over HTTP/2
HEADERS
REQUEST
HEADERS HEADERS
RESPONSE
DATA
DATA
DATA
DATA
@aiborisov
@mykyta_p

29. STREAMS? CONNECTIONS?
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

30. syntax = "proto3";
service MintService {
rpc CreateCurrency (CreateCurrencyRequest)
returns (CreateCurrencyResponse);
rpc Mint (MintRequest) returns (MintResponse);
}
gRPC API
@aiborisov
@mykyta_p

31. val channel =
ManagedChannelBuilder
.forTarget(“mint:8089”)
.build();
val mintClient =
MintServiceGrpc
.newStub(channel);
gRPC Channel
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

32. val channel =
ManagedChannelBuilder
.forTarget(“mint:8089”)
.build();
val mintClient =
MintServiceGrpc
.newStub(channel);
gRPC Channel
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

33. val channel =
ManagedChannelBuilder
.forTarget(“mint:8089”)
.build();
val mintClient =
MintServiceGrpc
.newStub(channel);
mintClient.mint(req, ...);
gRPC Channel
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

34. MINT
SERVICE
CHANNEL
MINT
SERVICE
MINT
SERVICE
gRPC
@aiborisov
@mykyta_p

35. MINT
SERVICE
CHANNEL
MINT
SERVICE
MINT
SERVICE
gRPC
@aiborisov
@mykyta_p

36. MINT
SERVICE
CHANNEL
MINT
SERVICE
MINT
SERVICE
gRPC
@aiborisov
@mykyta_p

37. MINT
SERVICE
CHANNEL
MINT
SERVICE
MINT
SERVICE
gRPC
@aiborisov
@mykyta_p

38. Mint Request
HEADERS: streamId=35 endStream=false
:path: /MintService/Mint
:method: POST
:scheme: http
grpc-timeout: 1S
x-user-agent: grpc-web-javascript/0.1
content-type: application/grpc
@aiborisov
@mykyta_p

39. Mint Request
HEADERS: streamId=35 endStream=false
:path: /MintService/Mint
:method: POST
:scheme: http
grpc-timeout: 1S
x-user-agent: grpc-web-javascript/0.1
content-type: application/grpc
@aiborisov
@mykyta_p

40. Mint Request
HEADERS: streamId=35 endStream=false
:path: /MintService/Mint
:method: POST
:scheme: http
grpc-timeout: 1S
x-user-agent: grpc-web-javascript/0.1
content-type: application/grpc
DATA: streamId=35 endStream=true length=25
bytes=00000000140a126469706c6f6d61746963436f696e39383033
@aiborisov
@mykyta_p

41. Mint Request
HEADERS: streamId=35 endStream=false
:path: /MintService/Mint
:method: POST
:scheme: http
grpc-timeout: 1S
x-user-agent: grpc-web-javascript/0.1
content-type: application/grpc
DATA: streamId=35 endStream=true length=25
bytes=00000000140a126469706c6f6d61746963436f696e39383033
@aiborisov
@mykyta_p

42. Mint Response
HEADERS: streamId=35 endStream=false
:status: 200
content-type: application/grpc
grpc-accept-encoding: gzip
@aiborisov
@mykyta_p

43. Mint Response
HEADERS: streamId=35 endStream=false
:status: 200
content-type: application/grpc
grpc-accept-encoding: gzip
DATA: streamId=35 padding=0 endStream=false length=8
bytes=000000000308e402
@aiborisov
@mykyta_p

44. Mint Response
HEADERS: streamId=35 endStream=false
:status: 200
content-type: application/grpc
grpc-accept-encoding: gzip
DATA: streamId=35 padding=0 endStream=false length=8
bytes=000000000308e402
HEADERS: streamId=35 endStream=true
grpc-status: 0
@aiborisov
@mykyta_p

45. Mint Response
HEADERS: streamId=35 endStream=false
:status: 200
content-type: application/grpc
grpc-accept-encoding: gzip
DATA: streamId=35 padding=0 endStream=false length=8
bytes=000000000308e402
HEADERS: streamId=35 endStream=true
grpc-status: 0
@aiborisov
@mykyta_p Icons used: https://www.flaticon.com/free-icon/beet_7179853, https://www.flaticon.com/premium-icon/bread_750878

46. service MintService {
rpc Mint (stream MintRequest)
returns (stream MintResponse);
}
gRPC STREAMING
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

47. gRPC STREAMING
DATA
HEADERS DATA DATA
[EOS]
…
REQUEST
STREAM
@aiborisov
@mykyta_p

48. gRPC STREAMING
REQUEST
STREAM
RESPONSE
STREAM
DATA
HEADERS DATA DATA
[EOS]
…
DATA
HEADERS
[EOS]
DATA HEADERS
…
@aiborisov
@mykyta_p

49. MINT
SERVICE
MINT
SERVICE
MINT
SERVICE
CONNECTION MANAGEMENT
@aiborisov
@mykyta_p

50. MINT
SERVICE
MINT
SERVICE
MINT
SERVICE
CONNECTION MANAGEMENT
GOAWAY
I’M SHUTTING
DOWN
@aiborisov
@mykyta_p

51. MINT
SERVICE
MINT
SERVICE
CONNECTION MANAGEMENT
@aiborisov
@mykyta_p

52. MINT
SERVICE
MINT
SERVICE
CONNECTION MANAGEMENT
@aiborisov
@mykyta_p

53. MINT
SERVICE
MINT
SERVICE
MINT
SERVICE
CONNECTION MANAGEMENT
@aiborisov
@mykyta_p

54. MINT
SERVICE
MINT
SERVICE
CONNECTION MANAGEMENT
NO MINT
SERVICE
@aiborisov
@mykyta_p

55. MINT
SERVICE
MINT
SERVICE
CONNECTION MANAGEMENT
NO MINT
SERVICE
PING
STILL HERE?
@aiborisov
@mykyta_p

56. MINT
SERVICE
MINT
SERVICE
CONNECTION MANAGEMENT
NO MINT
SERVICE
I DIDN’T
HEAR BACK
@aiborisov
@mykyta_p

57. MINT
SERVICE
MINT
SERVICE
CONNECTION MANAGEMENT
@aiborisov
@mykyta_p

58. FAILED CALLS
DEADLINE_EXCEEDED ?
CANCELED ?
UNAVAILABLE ?
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

59. GRACEFUL SHUTDOWN
GOAWAY: lastStreamId=39
errorCode=0
length=13
bytes=...
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

60. STREAM TERMINATION
RST_STREAM: streamId=37
errorCode=8
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

61. PING
PING: ack=false
PING: ack=true
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

62. READ BINARY FRAMES
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

63. logging.properties
handlers=java.util.logging.ConsoleHandler
io.grpc.netty.level=FINE
java.util.logging.ConsoleHandler.level=FINE
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
ENABLE NETTY FRAME LOGGING
@aiborisov
@mykyta_p

64. logging.properties
handlers=java.util.logging.ConsoleHandler
io.grpc.netty.level=FINE
java.util.logging.ConsoleHandler.level=FINE
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
ENABLE NETTY FRAME LOGGING
@aiborisov
@mykyta_p

65. BACKEND ARCHITECTURE
GATEWAY MINT LEDGER
@aiborisov
@mykyta_p

66. BACKEND ARCHITECTURE
GATEWAY MINT LEDGER
MINT
@aiborisov
@mykyta_p

67. BACKEND ARCHITECTURE
GATEWAY MINT LEDGER
MINT
@aiborisov
@mykyta_p

68. BACKEND ARCHITECTURE
GATEWAY MINT LEDGER
MINT
@aiborisov
@mykyta_p

69. HOW TO CORRELATE STREAMS ?
GATEWAY MINT LEDGER
@aiborisov
@mykyta_p
STREAM ID 1 STREAM ID 43 STREAM ID 91

70. // Example with Brave
var grpcTracing = GrpcTracing.create(rpcTracing);
val channel = ManagedChannelBuilder
.forTarget(“mint:8089”)
.intercept(grpcTracing.newClientInterceptor())
.build();
ADD TRACING OF YOUR CHOICE
@aiborisov
@mykyta_p

71. FINE: [id: 0xed04e7b3, L:/10.92.128.67:8090 - R:/10.92.128.2:36350]
INBOUND HEADERS: streamId=77 headers=GrpcHttp2RequestHeaders[:path:
/GatewayService/Mint, ..., x-b3-traceid: e5dde89783d20c01] padding=0
endStream=false
FINE: [id: 0xed04e7b3, L:/10.92.128.67:8090 - R:/10.92.128.2:36350]
INBOUND DATA: streamId=77 padding=0 endStream=false length=22
bytes=00000000110a0f44796e616d6963436f696e34373831
FINE: [id: 0xed04e7b3, L:/10.92.128.67:8090 - R:/10.92.128.2:36350]
INBOUND DATA: streamId=77 padding=0 endStream=true length=0 bytes=
BINARY FRAME LOGS: EXAMPLE
@aiborisov
@mykyta_p

72. FINE: [id: 0xed04e7b3, L:/10.92.128.67:8090 - R:/10.92.128.2:36350]
INBOUND HEADERS: streamId=77 headers=GrpcHttp2RequestHeaders[:path:
/GatewayService/Mint, ..., x-b3-traceid: e5dde89783d20c01] padding=0
endStream=false
FINE: [id: 0xed04e7b3, L:/10.92.128.67:8090 - R:/10.92.128.2:36350]
INBOUND DATA: streamId=77 padding=0 endStream=false length=22
bytes=00000000110a0f44796e616d6963436f696e34373831
FINE: [id: 0xed04e7b3, L:/10.92.128.67:8090 - R:/10.92.128.2:36350]
INBOUND DATA: streamId=77 padding=0 endStream=true length=0 bytes=
BINARY FRAME LOGS: EXAMPLE
@aiborisov
@mykyta_p

73. $ grep "x-b3-traceid: e5dde89783d20c01"
FINE: [id: 0xed04e7b3, L:/10.92.128.67:8090 - R:/10.92.128.2:36350]
INBOUND HEADERS: streamId=77 headers=GrpcHttp2RequestHeaders[:path:
/GatewayService/Mint, ..., x-b3-traceid: e5dde89783d20c01] padding=0
endStream=false
FINE: [id: 0xdc6c37f1, L:/10.92.128.67:42272 - R:mint/10.93.0.113:8091]
OUTBOUND HEADERS: streamId=43 headers=GrpcHttp2OutboundHeaders[path:
/MintService/Mint, ..., x-b3-traceid: e5dde89783d20c01] streamDependency=0
weight=16 exclusive=false padding=0 endStream=false
BINARY FRAME LOGS: CORRELATE
@aiborisov
@mykyta_p

74. CORRELATE STREAMS
GATEWAY MINT
@aiborisov
@mykyta_p
STREAM
ID 77
STREAM
ID 43
TRACE ID
e5dde89783d20c01

75. io.grpc.StatusRuntimeException: CANCELLED: Call cancelled
at io.grpc.Status.asRuntimeException(Status.java:535)
CANCELLED STATUS
GATEWAY
@aiborisov
@mykyta_p

76. Oct 13, 2022 7:44:30 PM logRstStream
FINE: [id: 0x81f2d1b4, L:/10.86.128.132:8090 - R:/10.86.129.131:43730]
INBOUND RST_STREAM: streamId=21 errorCode=8
Oct 13, 2022 7:44:30 PM logRstStream
FINE: [id: 0xa0d0f73c, L:/10.86.128.132:41210 - R:mint/10.87.3.240:8091]
OUTBOUND RST_STREAM: streamId=37 errorCode=8
LOOK FOR RST_STREAM IN GATEWAY LOGS
@aiborisov
@mykyta_p

77. Oct 13, 2022 7:44:30 PM logHeaders FINE: [...]
INBOUND HEADERS: streamId=21 headers=GrpcHttp2RequestHeaders[:path:
/GatewayService/Mint, …, x-b3-traceid: 7845c28350959cc0] …
Oct 13, 2022 7:44:30 PM logData FINE: [id: 0x81f2d1b4, …]
INBOUND DATA: streamId=21 padding=0 endStream=true length=16 bytes=...
Oct 13, 2022 7:44:30 PM logRstStream
FINE: [id: 0x81f2d1b4, ...] INBOUND RST_STREAM: streamId=21 errorCode=8
FIND FRAMES BY STREAM BY ID (21)
@aiborisov
@mykyta_p

78. Oct 13, 2022 7:44:30 PM logHeaders FINE: [...]
INBOUND HEADERS: streamId=21 headers=GrpcHttp2RequestHeaders[:path:
/GatewayService/Mint, …, x-b3-traceid: 7845c28350959cc0] …
Oct 13, 2022 7:44:30 PM logData FINE: [id: 0x81f2d1b4, …]
INBOUND DATA: streamId=21 padding=0 endStream=true length=16 bytes=...
Oct 13, 2022 7:44:30 PM logRstStream
FINE: [id: 0x81f2d1b4, ...] INBOUND RST_STREAM: streamId=21 errorCode=8
NOTE TRACE ID
@aiborisov
@mykyta_p

79. Oct 13, 2022 7:44:30 PM logHeaders FINE: [...]
INBOUND HEADERS: streamId=21 headers=GrpcHttp2RequestHeaders[:path:
/GatewayService/Mint, …, x-b3-traceid: 7845c28350959cc0] …
Oct 13, 2022 7:44:30 PM logHeaders
FINE: [id: 0xa0d0f73c, L:/10.86.128.132:41210 - R:mint/10.87.3.240:8091]
OUTBOUND HEADERS: streamId=37 headers=GrpcHttp2OutboundHeaders[:path:
/MintService/Mint, ..., x-b3-traceid: 7845c28350959cc0] padding=0
endStream=false
FIND CORRELATED STREAMS BY TRACE ID
@aiborisov
@mykyta_p

80. Oct 13, 2022 7:44:30 PM logHeaders FINE: [id: 0xa0d0f73c, ...]
OUTBOUND HEADERS: streamId=37 headers=GrpcHttp2OutboundHeaders[:path:
/MintService/Mint, ..., x-b3-traceid: 7845c28350959cc0] padding=0
endStream=false
Oct 13, 2022 7:44:30 PM logData FINE: [id: 0xa0d0f73c, ...]
OUTBOUND DATA: streamId=37 padding=0 endStream=true length=16 bytes=...
Apr 13, 2022 7:44:30 PM logRstStream FINE: [id: 0xa0d0f73c, ...]
OUTBOUND RST_STREAM: streamId=37 errorCode=8
GATEWAY LOGS: OUTBOUND STREAM 37
@aiborisov
@mykyta_p

81. Apr 13, 2022 7:44:30 PM logRstStream
FINE: [id: 0xb2a8c95d, L:/10.86.129.2:8091 - R:/10.86.128.132:55176]
INBOUND RST_STREAM: streamId=37 errorCode=8
Apr 13, 2022 7:44:30 PM logRstStream
FINE: [id: 0xe7c5ca6d, L:/10.86.129.2:48758 - R:ledger/10.87.3.172:8092]
OUTBOUND RST_STREAM: streamId=43 errorCode=8
MINT SERVICE LOGS
@aiborisov
@mykyta_p

82. CANCELLED STATUS: CLIENT CANCELLATION
GATEWAY MINT LEDGER
@aiborisov
@mykyta_p

83. io.grpc.StatusRuntimeException: CANCELLED: Call cancelled
at io.grpc.Status.asRuntimeException(Status.java:535)
CANCELLED STATUS (AGAIN)
GATEWAY
@aiborisov
@mykyta_p

84. Oct 13, 2022 5:28:23 PM logHeaders FINE: [id: 0xfc1722f6, …]
OUTBOUND HEADERS: streamId=17 headers=GrpcHttp2OutboundHeaders[:path:
/MintService/Mint, ...] padding=0 endStream=false
Oct 13, 2022 5:28:23 PM logData FINE: [id: 0xfc1722f6, …]
OUTBOUND DATA: streamId=17 padding=0 endStream=true length=16 bytes=...
Oct 13, 2022 5:28:23 PM logHeaders
FINE: [id: 0xfc1722f6, L:/10.86.128.132:41210 - R:mint/10.87.3.240:8091]
INBOUND HEADERS: streamId=17 headers=GrpcHttp2ResponseHeaders[:status:
200, content-type: application/grpc, grpc-status: 1, grpc-message: Call cancelled]
padding=0 endStream=true
GATEWAY LOGS: NO INBOUND RST_STREAM
@aiborisov
@mykyta_p

85. Oct 13, 2022 5:28:23 PM logHeaders FINE: [id: 0xfc1722f6, …]
OUTBOUND HEADERS: streamId=17 headers=GrpcHttp2OutboundHeaders[:path:
/MintService/Mint, ...] padding=0 endStream=false
Oct 13, 2022 5:28:23 PM logData FINE: [id: 0xfc1722f6, …]
OUTBOUND DATA: streamId=17 padding=0 endStream=true length=16 bytes=...
Oct 13, 2022 5:28:23 PM logHeaders
FINE: [id: 0xfc1722f6, L:/10.86.128.132:41210 - R:mint/10.87.3.240:8091]
INBOUND HEADERS: streamId=17 headers=GrpcHttp2ResponseHeaders[:status:
200, content-type: application/grpc, grpc-status: 1, grpc-message: Call cancelled]
padding=0 endStream=true
GATEWAY LOGS: CANCELLATION FROM MINT
@aiborisov
@mykyta_p

86. MINT SERVICE LOGS
Oct 13, 2022 5:28:23 PM logHeaders FINE: [id: 0x5ca89c01, …]
INBOUND HEADERS: streamId=17 headers=GrpcHttp2RequestHeaders[:path:
/MintService/Mint, :authority: mint:8091, …, x-b3-traceid: 0b9db47309315af2]
endStream=false
Oct 13, 2022 5:28:23 PM logData FINE: [id: 0x5ca89c01, …]
INBOUND DATA: streamId=17 padding=0 endStream=true length=16 bytes=...
Oct 13, 2022 5:28:23 PM logHeaders
FINE: [id: 0x5ca89c01, L:/10.86.129.2:8091 - R:/10.86.128.132:44564]
OUTBOUND HEADERS: streamId=17 headers=GrpcHttp2OutboundHeaders[:status:
200, grpc-status: 1, grpc-message: Call cancelled] padding=0 endStream=true
@aiborisov
@mykyta_p

87. MINT SERVICE LOGS: TRACE ID
Oct 13, 2022 5:28:23 PM logHeaders FINE: [id: 0x5ca89c01, …]
INBOUND HEADERS: streamId=17 headers=GrpcHttp2RequestHeaders[:path:
/MintService/Mint, :authority: mint:8091, …, x-b3-traceid: 0b9db47309315af2]
endStream=false
Oct 13, 2022 5:28:23 PM logData FINE: [id: 0x5ca89c01, …]
INBOUND DATA: streamId=17 padding=0 endStream=true length=16 bytes=...
Oct 13, 2022 5:28:23 PM logHeaders
FINE: [id: 0x5ca89c01, L:/10.86.129.2:8091 - R:/10.86.128.132:44564]
OUTBOUND HEADERS: streamId=17 headers=GrpcHttp2OutboundHeaders[:status:
200, grpc-status: 1, grpc-message: Call cancelled] padding=0 endStream=true
@aiborisov
@mykyta_p

88. MINT SERVICE LOGS: BY TRACE ID
Oct 13, 2022 5:28:23 PM logHeaders FINE: [id: 0x5ca89c01, …]
INBOUND HEADERS: streamId=17 headers=GrpcHttp2RequestHeaders[:path:
/MintService/Mint, :authority: mint:8091, …, x-b3-traceid: 0b9db47309315af2]
endStream=false
Oct 13, 2022 5:28:23 PM logHeaders
FINE: [id: 0xeab70334, L:/10.86.129.2:52928 - R:ledger/10.87.3.172:8092]
OUTBOUND HEADERS: streamId=55
headers=GrpcHttp2OutboundHeaders[::authority: ledger:8092, :path:
/LedgerService/RecordMinting, …, x-b3-traceid: 0b9db47309315af2]
endStream=false
@aiborisov
@mykyta_p

89. MINT SERVICE LOGS: OUTBOUND STREAM 55
Oct 13, 2022 5:28:23 PM logHeaders
FINE: [id: 0xeab70334, L:/10.86.129.2:52928 - R:ledger/10.87.3.172:8092]
OUTBOUND HEADERS: streamId=55
headers=GrpcHttp2OutboundHeaders[::authority: ledger:8092, :path:
/LedgerService/RecordMinting, …, x-b3-traceid: 0b9db47309315af2]
endStream=false
Oct 13, 2022 5:28:23 PM logData FINE: [id: 0xeab70334, . . .]
OUTBOUND DATA: streamId=55 padding=0 endStream=true length=16 bytes=...
Oct 13, 2022 5:28:23 PM logRstStream
FINE: [id: 0xeab70334, L:/10.86.129.2:52928 - R:ledger/10.87.3.172:8092]
OUTBOUND RST_STREAM: streamId=55 errorCode=8
@aiborisov
@mykyta_p

90. CANCELLED STATUS: SERVER CANCELLATION
GATEWAY LEDGER
@aiborisov
@mykyta_p

91. FAILED CALLS ?
Inspect:
Metrics + Status Codes
Exceptions
Frame Logs
Binary Logs
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

92. FRAME LOGS
Types:
HEADERS
DATA
RST_STREAM
GOAWAY
PING
https://github.com/grpc/grpc/blob/
master/doc/PROTOCOL-HTTP2.md
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

93. BINARY LOGS
Machine-readable
var binaryLog =
BinaryLogs.createBinaryLog();
var channelWithBinaryLogs =
binaryLog.wrapChannel(channel);
var server = ServerBuilder
.forPort(port)
.addService(...)
.setBinaryLog(binaryLog)
.build();
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

94. gRPC PERFORMANCE
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

95. LIMITATION: MAX CONCURRENT STREAMS
HTTP/2 STREAMS
TCP
@aiborisov
@mykyta_p

96. val regularChannel =
ManagedChannelBuilder
.forTarget(“mint:8089”)
.build();
val highLoadChannel =
ManagedChannelBuilder
.forTarget(“mint:8089”)
.build();
DEDICATED CHANNEL
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

97. Use GAX Java ChannelPool
or
pool yourself
POOL OF CHANNELS
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

98. gRPC STREAMING
REQUEST
STREAM
RESPONSE
STREAM
DATA
HEADERS DATA DATA
[EOS]
…
DATA
HEADERS
[EOS]
DATA HEADERS
…
@aiborisov
@mykyta_p

99. gRPC STREAMING: PROXIES
MINT
GATEWAY
gRPC stream over HTTP/2
@aiborisov
@mykyta_p

100. gRPC STREAMING: PROXIES
MINT
GATEWAY
@aiborisov
@mykyta_p

101. gRPC STREAMING: PROXIES
MINT
GATEWAY
I’M KILLING
IDLE
CONNECTIONS
@aiborisov
@mykyta_p

102. gRPC STREAMING:
KEEP-ALIVE
val channel =
ManagedChannelBuilder
.forTarget(“mint:8089”)
.idleTimeout(1, MINUTES)
.build();
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

103. gRPC STREAMING:
CAVEATS
IDLE CONNECTIONS
LOAD-BALANCING
DEADLINES
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

104. gRPC STREAMING
DON’T USE TO OPTIMIZE
PERFORMANCE
USE TO OPTIMIZE YOUR APP
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

105. MINT.GRPCTALK.COM
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

106. AN IETF ORIGINAL SERIES
HTTP
PART 3
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

107. HTTP/2 FRAMES
TCP
@aiborisov
@mykyta_p
Icons used: https://www.flaticon.com/free-icon/milk_1626757, https://www.flaticon.com/free-icon/beet_7179853,
https://www.flaticon.com/premium-icon/bread_750878, https://www.flaticon.com/free-icon/strawberry_7189510

108. TCP PACKETS
@aiborisov
@mykyta_p
Icons used: https://www.flaticon.com/free-icon/package_1007988

109. TCP PACKET LOSS
@aiborisov
@mykyta_p
Icons used: https://www.flaticon.com/free-icon/package_1007988, https://www.flaticon.com/free-icon/package_1007989

110. TCP PACKET LOSS
@aiborisov
@mykyta_p
Icons used: https://www.flaticon.com/free-icon/package_1007988, https://www.flaticon.com/free-icon/package_1007989

111. RETRANSMIT LOST PACKET(S)
@aiborisov
@mykyta_p
Icons used: https://www.flaticon.com/free-icon/package_1007988

112. RETRANSMIT LOST PACKET(S)
@aiborisov
@mykyta_p
Icons used: https://www.flaticon.com/free-icon/package_1007988

113. TCP HEAD-OF-LINE BLOCKING
???
@aiborisov
@mykyta_p
Icons used: https://www.flaticon.com/free-icon/package_1007988, https://www.flaticon.com/free-icon/package_1007989

114. TCP OR NOT TCP ?
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

115. QUIC
@aiborisov
@mykyta_p
ON TOP OF UDP
RE-IMPLEMENTS TCP-LIKE
PROPERTIES
The image is property of Netflix Inc. and cannot be reused

116. HTTP/2 VS HTTP/3
@aiborisov
@mykyta_p
CONNECTIONS
FRAMES
CONNECTIONS
STREAMS
FRAMES
LOSS RECOVERY
FLOW CONTROL
PACKETS
PACKETS
IP
QUIC
HTTP/3
CONNECTIONS
STREAMS
FRAMES
LOSS RECOVERY
FLOW CONTROL
PACKETS
TCP
HTTP/2
UDP

117. QUIC
@aiborisov
@mykyta_p
CONNECTIONS
FRAMES
CONNECTIONS
STREAMS
FRAMES
LOSS RECOVERY
FLOW CONTROL
PACKETS
PACKETS
HTTP/3
CONNECTIONS
STREAMS
FRAMES
LOSS RECOVERY
FLOW CONTROL
PACKETS
TCP
HTTP/2
UDP
IP
QUIC

118. HTTP/2
@aiborisov
@mykyta_p
CONNECTIONS
FRAMES
CONNECTIONS
STREAMS
FRAMES
LOSS RECOVERY
FLOW CONTROL
PACKETS
PACKETS
QUIC
HTTP/3
CONNECTIONS
STREAMS
FRAMES
LOSS RECOVERY
FLOW CONTROL
PACKETS
TCP
HTTP/2
UDP
IP

119. UDP
QUIC: NO HEAD-OF-LINE BLOCKING
@aiborisov
@mykyta_p
CONNECTIONS
FRAMES
CONNECTIONS
STREAMS
FRAMES
LOSS RECOVERY
FLOW CONTROL
PACKETS
PACKETS
IP
HTTP/3
CONNECTIONS
STREAMS
FRAMES
LOSS RECOVERY
FLOW CONTROL
PACKETS
TCP
HTTP/2
QUIC

120. TLS 1.2 VS 0-RTT (TLS 1.3, QUIC)
@aiborisov
@mykyta_p
QUIC
+
TLS
+
HTTP
TCP
TLS 1.3
+
HTTP
TCP
TLS
1.2
HTTP

121. CONNECTION MIGRATION
@aiborisov
@mykyta_p
HTTP:
IPs + Ports Quadruplet:
Source: 10.86.128.132:41210
Destination: 10.87.3.240:8091

122. CONNECTION MIGRATION
@aiborisov
@mykyta_p
HTTP:
IPs + Ports Quadruplet:
Source: 10.86.128.132:41210
Destination: 10.87.3.240:8091
QUIC:
Connection ID:
bbf4f6198293…

123. CONNECTION MIGRATION
@aiborisov
@mykyta_p
GATEWAY
Connection ID: bbf4f…
Icons used: https://www.flaticon.com/premium-icon/wifi_3876106

124. CONNECTION MIGRATION
@aiborisov
@mykyta_p
GATEWAY
Connection ID: bbf4f…
Icons used: https://www.flaticon.com/free-icon/phone_39934, https://www.flaticon.com/premium-icon/wifi_3876106

125. HTTP/3
@aiborisov
@mykyta_p
AN INTERNET STANDARD
(AS OF JUNE 2022)
RFC 9114
datatracker.ietf.org/doc/html/
rfc9114
The image is property of Netflix Inc. and cannot be reused

126. HTTP/3
@aiborisov
@mykyta_p
PRESERVES HTTP* SEMANTICS
The image is property of Netflix Inc. and cannot be reused

127. HTTP/3
@aiborisov
@mykyta_p
PRESERVES HTTP* SEMANTICS
DELEGATES TO QUIC
STREAM LIFECYCLE
MULTIPLEXING
FLOW CONTROL
The image is property of Netflix Inc. and cannot be reused

128. @aiborisov
@mykyta_p
HTTP/3 FRAME FORMAT
TYPE (I)
FRAME PAYLOAD
(0 … )
LENGTH (I)

129. @aiborisov
@mykyta_p
FRAME FORMAT
LENGTH (24)
FLAGS (8)
TYPE (8)
STREAM ID (31)
FRAME PAYLOAD
(0 … )
TYPE (I)
FRAME PAYLOAD
(0 … )
LENGTH (I)
HTTP/3 HTTP/2

130. HTTP/3 FRAMES TYPES
@aiborisov
@mykyta_p
DATA
HEADERS
SETTINGS
GOAWAY
CANCEL_PUSH
PUSH_PROMISE
MAX_PUSH_ID
Reserved
The image is property of Netflix Inc. and cannot be reused

131. @aiborisov
@mykyta_p
gRPC OVER HTTP/3
https://github.com/grpc/proposal/blob/master/G2-http3-protocol.md

132. @aiborisov
@mykyta_p
gRPC OVER HTTP/3
HTTP/3 FRAMES HTTP/2 FRAMES
https://github.com/grpc/proposal/blob/master/G2-http3-protocol.md
HTTP/3 DATA
HTTP/3 HEADERS
QUIC RESET_STREAM
QUIC GOAWAY
QUIC PING
HTTP/2 DATA
HTTP/2 HEADERS
HTTP/2 RST_STREAM
HTTP/2 GOAWAY
HTTP/2 PING

133. gRPC OVER HTTP/3
@aiborisov
@mykyta_p
The image is property of Netflix Inc. and cannot be reused

134. gRPC OVER HTTP/3
@aiborisov
@mykyta_p
FASTER CONNECTION
NEGOTIATION
NO HEAD-OF-LINE BLOCKING
CONNECTION TRANSITION
BETWEEN NETWORKS
The image is property of Netflix Inc. and cannot be reused

135. UDP TRADEOFFS
@aiborisov
@mykyta_p
FIREWALLS
ROUTING
REPLAY ATTACKS
The image is property of Netflix Inc. and cannot be reused

136. gRPC OVER HTTP/*
@aiborisov
@mykyta_p
BINARY FRAMES
OPTIMIZATIONS AND
TRADEOFFS
HTTP/3 IS HERE
The image is property of Netflix Inc. and cannot be reused

137. gRPC OVER HTTP/2 SPEC:
https://github.com/grpc/grpc/blob/master/doc/
PROTOCOL-HTTP2.md
QUIC RFC:
https://datatracker.ietf.org/doc/html/rfc9000
gRPC OVER HTTP/3 PROPOSAL:
https://github.com/grpc/proposal/blob/master/
G2-http3-protocol.md
LINKS
@aiborisov
@mykyta_p

138. Thank You
@aiborisov
@mykyta_p