Armeriaworkshop2019 openchat julie

Openchat Journey
with Armeria
> Julie Kim@LINE Openchat Dev

Outline
> Openchat Introduction
> Architecture
> User Journey

Make new friends based on your interests
(and please more chat)

Aug. 2019
THAILAND
INDONESIA
JAPAN

Technical Challenges
> Asynchronous server application
> Armeria in Production
> Pre-sharded MySql databases
> Server Push based on Subscriptions
> Dynamic Configuration
> …

Naive Approach
Server
icons made by smashicons
Client Storage

Openchat Internal
SPDY/thrift
Reactive
X
(RxJava)
LEGY
auth
…

Openchat Internal
SPDY/thrift
Reactive
X
(RxJava)
LEGY
auth
…
thrift
/AsyncIface

Openchat Internal
SPDY/thrift
Reactive
X
(RxJava)
LEGY
auth
…
thrift
/AsyncIface
External
API Invocation
rpc/thrift
http 1.1
http 2

ThttpService
> Service Registration
• ServerBuilder
• ThriftServiceRegistrationBean

ThttpService
• ServerBuilder
• ThriftServiceRegistrationBean
@Bean
public ThriftServiceRegistrationBean service(ServiceHandler handler) {
Service<HttpRequest, HttpResponse> service = ThriftCallService.of(handler);
return new ThriftServiceRegistrationBean()
.setPath(”/path/to/open/chat)
.setService(service)
.setServiceName(”Openchat");
}

ThttpService
> Interop w/rxJava2
@Override
public void sendMessage(SendMessageRequest request, AsyncMethodCallback resultHandler)
throws TException {
AuthenticationContext context = authenticationManager.getContext();
Single<SendMessageResponse> delegate = Single.defer(
() -> messageService.sendMessage(context, request));
setAsyncResult(resultHandler, () -> delegate);
}

ThttpService
> Interop w/rxJava2
@Override
public void sendMessage(SendMessageRequest request, AsyncMethodCallback resultHandler)
throws TException {
AuthenticationContext context = authenticationManager.getContext();
Single<SendMessageResponse> delegate = Single.defer(
() -> messageService.sendMessage(context, request));
setAsyncResult(resultHandler, () -> delegate);
}
o Subscribes `Single`
o Hands over a result or throws an error
o Translations the `Thrown`

AnnotatedHttpService
• ServerBuilder
• AnnotatedServiceRegistrationBean

• ServerBuilder
• AnnotatedServiceRegistrationBean
@Bean
public AnnotatedServiceRegistrationBean newAnnotatedHttpService(
String urlPrefix,
BaseHttpRequestHandler handler) {
return new AnnotatedServiceRegistrationBean().setPathPrefix(urlPrefix)
.setServiceName("httpService")
.setService(handler);
}

> AnnotatedServiceRegistrationBean
@Bean
public AnnotatedServiceRegistrationBean newAnnotatedHttpService(
String urlPrefix,
BaseHttpRequestHandler handler) {
return new AnnotatedServiceRegistrationBean().setPathPrefix(urlPrefix)
.setServiceName("httpService")
.setService(handler);
}
@Get("/openchats")
public CompletableFuture<OpenchatsResponse> getOpenchats(@Param("limit") int limit,
@Param("offset") int offset) {
return response(() -> service.getOpenchats(getContext(), limit, offset));
}

Http Client
thrift/RPC client
http client
Retrofit/http client
build(Service.AsyncIface.class)
build(HttpClient.class)
RxJava2CallAdapterFactory
ArmeriaRetrofitBuilder
ClientFactory
ClientBuilder
@GET("/path/to/get")

Decorators
A ‘decorating service/client’ (or a ‘decorator’) is a Service/Client that wraps another Service/Client
to intercept an incoming request or an outgoing response.
As its name says, it is an implementation of the decorator pattern.

Decorators
https://flic.kr/p/RX8b5d
@Logging
@Monitoring
@Authenticate
MyService
Spring Annotations
https://qph.fs.quoracdn.net/main-qimg-47591f98c0246fba530cf90018a0c1aa
Decorators
MyService
Logging
Service
Monitoring
Service
Auth
Service

Logging
> KafkaAccessLogWriter
> Access/Client Logging
logback.xml
Files
>_
Console
Log Search/Monitoring

Logging
> KafkaAccessLogWriter
>Configuration ONLY
> Access/Client Logging
logback.xml
Files
>_
Console
Log Search/Monitoring

Customized Decorators
> AuthenticationService
> RpcLoggingService
> RequestIdService
> MaintenanceService
> …

RequestIdService
>Recommended & Required
>Similar to Logback’s MDC (Mapped Dignostic Context)

@Override
public O serve(ServiceRequestContext ctx, I req) throws Exception {
// For example,
long requestId = System.currentTimeMillis();
ctx.attr(ATTRIBUTE_KEY).set(requestId);
// Copy to a child
ctx.onChild((curCtx, newCtx) -> {
if (curCtx.hasAttr(ATTRIBUTE_KEY)) {
// Set the id to newCtx's attr
}
});
return delegate().serve(ctx, req);
}
RequestIdService

<property name="RCEA_EXPORTS"
value="attrs.reqid:com.linecorp..RequestId#ATTRIBUTE_KEY" />
…
<appender name="RCEA_SERVICE”
class="com.linecorp.armeria.common.logback.RequestContextExportingAppender">
<appender-ref ref="ASYNC_SERVICE" />
<exports>${RCEA_EXPORTS}</exports>
</appender>
RequestIdService

2019-11-15 17:00:00.221 [15738048002200] [armeria-common-worker-epoll-2-1]
armeria.services...
2019-11-15 17:00:00.228 [15738048002200] [talk-server-1] c.l.s.s.auth....
c.l.s.c.a.c.logging...
2019-11-15 17:00:00.230 [15738048002200] [lettuce-epollEventLoop-21-2] c.l.s.service.sync..
2019-11-15 17:00:00.230 [15738048002200] [lettuce-epollEventLoop-21-2] c.l.s.db.redis...
RequestIdService

Metrics
> MetricCollectingService
• Supported by ArmeriaAutoConfiguration
• Backed by ArmeriaSettings
> MetricCollectingClient
ClientDecorationBuilder builder = new ClientDecorationBuilder()
.add(MetricCollectingClient
.newDecorator(MeterIdPrefixFunctionFactory.DEFAULT
.get("client", serviceName)))
.build();

Client Content Preview
>(note) RPC Client
>Without ContentPreview
• Similar to RPC but only headers
c.l.s.c.a.c.logging.ClientLoggingClient : [id: 0x3b958e6d] Response: {startTime=2019-11-
15T08:00:00.228Z(1573804800228000), length=275B, duration=108µs(108519ns), headers=[:status=200,
server=legy, content-type=application/x-thrift; protocol=TBINARY, content-length=275],
content=DefaultRpcResponse{LegyResponse(...)}} seqid: 177069

Client Content Preview
>(note) RPC Client
>Without ContentPreview
• Similar to RPC but only headers
>With ContentPreview
c.l.s.c.a.c.logging.ClientLoggingClient : [id: 0x3b958e6d] Response: {startTime=2019-11-
15T08:00:00.228Z(1573804800228000), length=275B, duration=108µs(108519ns), headers=[:status=200,
server=legy, content-type=application/x-thrift; protocol=TBINARY, content-length=275],
content=DefaultRpcResponse{LegyResponse(...)}} seqid: 177069
2019-10-24 15:34:16.711 [-worker-nio-2-3] c.l.s.c.a.c.logging.HttpLoggingClient :
serviceName=ObsClient, request={startTime=2019-10-24T06:34:16.675Z(1571898856675000), length=94B,
duration=30028µs(30028897ns), scheme=none+h1c, headers=[:method=POST, :path=/for/somewhere,
:authority=upstream.line.me:80, :scheme=http, custom-header=value, content-type=application/json,
content-length=94, user-agent=armeria/0.89.0], contentPreview={...}}

CircuitBreaker
Half Open
Closed Open

CircuitBreaker
Half Open
Closed Open
>CircuitBreakerStrategy
• HTTP, RPC, etc.
Rejected request count (1m)
Transition(Open) count (1m)

RequestTrackingClient
>Cooperate with RequestIdService
@Override
public HttpResponse execute(ClientRequestContext ctx, HttpRequest req) throws Exception {
long reqId = ctx.attr(ATTRIBUTE_KEY).get();
RequestHeadersBuilder builder = req.headers().toBuilder();
builder.add(AsciiString.of(“request-tracking-header”), reqId);
final HttpRequest newReq = HttpRequest.of(req, builder.build());
ctx.updateRequest(newReq);
return delegate().execute(ctx, newReq);
}

RequestTrackingClient
>Cooperate with RequestIdService
2019-11-15 [15738048002200] [armeria-common-worker-epoll-2-1] armeria.services...
2019-11-15 [15738048002200] [talk-server-1] c.l.s.s.auth....
2019-11-15 [15738048000810] [armeria-common-worker-epoll-2-6]
c.l.s.c.a.c.logging.ClientLoggingClient : [id: 0x03761f46, ...][h1c://.../path/to/api#POST]
{..., headers=[:method=POST, :path=..., ... request-tracking-header=15738048000810, ...
2019-11-15 [15738048002200] [lettuce-epollEventLoop-21-2] c.l.s.db.redis...

Ep 0. Learning Curve
https://www.psywww.com/intropsych/ch07-cognition/motor-activity.html

https://www.psywww.com/intropsych/ch07-cognition/motor-activity.html
>Armeria?!
>How to make the API?
>Logging?!
>Metrics?
>Bug Investigation..?!
>Decorators?!!

> ServerBuilder vs ThriftServiceRegistrationBean
> ServerBuilder vs AnnotatedServiceRegistrationBean
> HttpClient vs RetrofitArmeriaClient
> Custom ExceptionHandling vs ExceptionHandler interface
> AOP vs Decorators

https://i.imgur.com/zGHwpAb.png

Ep 1. Boot Up
Spring Boot makes it easy to create stand-alone,
production-grade Spring based Applications that you can "just run".

Ep 1. Boot Up
AutoConfigration
ArmeriaSettings

Ep 1. Boot Up
AutoConfigration
ArmeriaSettings
ServiceRegistrationBeans

Ep 1. Boot Up
AutoConfigration
ArmeriaSettings
ServiceDecorators
ServiceRegistrationBeans

Ep 2. Efficiency
> n(connection) = n(core)*2
• Under the most frequently invoked cases,
• Server requests ~10K/min
• Client requests ~100K/min

Ep 3. Outage
> Upstream Outage
• Notification Server Failure
notification
response
timeout

Ep 3. Outage
> Upstream Outage
• Notification Server Failure
> Impact Scope
• Partial notification failure
• No big problem to serve
the other APIs
Async Client
notification
response
timeout

Ep 3. Outage
request
Reactive
X
(RxJava)

Ep 3. Outage
request
response
Reactive
X
(RxJava)
?

Ep 3. Outage
>Thread Blocking
request
armeria
io-thread
working
thread
< long time processing >

Ep 3. Outage
>Thread Blocking
request
armeria
io-thread
armeria
io-thread
working
thread
working
thread
request

Ep 3. Outage
>Thread Blocking
request
armeria
io-thread
armeria
io-thread
working
thread
working
thread
response
armeria
io-thread

Ep 3. Outage
>Thread Blocking
request
armeria
io-thread
armeria
io-thread
working
thread
working
thread
response
armeria
io-thread
armeria
io-thread

Ep 3. Outage
>Should avoid blocking armeria io threads
request
armeria
io-thread
armeria
io-thread
working
thread
working
thread
response
armeria
io-thread
working
thread
working
thread
armeria
io-thread

Thank You
Julie Kim@LINE Openchat Dev

Armeriaworkshop2019 openchat julie

More Related Content

Similar to Armeriaworkshop2019 openchat julie

More from LINE+

Recently uploaded

Armeriaworkshop2019 openchat julie