Walkthrough a LINE OpenChat project working with Armeria. Introduces benefits, and shares an episode while using Armeria.

1. Openchat Journey
with Armeria
> Julie Kim@LINE Openchat Dev

2. Outline
> Openchat Introduction
> Architecture
> User Journey

4. Make new friends based on your interests
(and please more chat)

5. Aug. 2019
THAILAND
INDONESIA
JAPAN

6. Technical Challenges
> Asynchronous server application
> Armeria in Production
> Pre-sharded MySql databases
> Server Push based on Subscriptions
> Dynamic Configuration
> …

7. Technical Challenges
> Asynchronous server application
> Armeria in Production
> Pre-sharded MySql databases
> Server Push based on Subscriptions
> Dynamic Configuration
> …

8. Architecture

9. Naive Approach
Server
icons made by smashicons
Client Storage

10. Openchat Internal
SPDY/thrift
Reactive
X
(RxJava)
LEGY
auth
…

11. Openchat Internal
SPDY/thrift
Reactive
X
(RxJava)
LEGY
auth
…
thrift
/AsyncIface

12. Openchat Internal
SPDY/thrift
Reactive
X
(RxJava)
LEGY
auth
…
thrift
/AsyncIface
External
API Invocation
rpc/thrift
http 1.1
http 2

13. ThttpService
> Service Registration
• ServerBuilder
• ThriftServiceRegistrationBean

14. ThttpService
> Service Registration
• ServerBuilder
• ThriftServiceRegistrationBean
@Bean
public ThriftServiceRegistrationBean service(ServiceHandler handler) {
Service<HttpRequest, HttpResponse> service = ThriftCallService.of(handler);
return new ThriftServiceRegistrationBean()
.setPath(”/path/to/open/chat)
.setService(service)
.setServiceName(”Openchat");
}

15. ThttpService
> Interop w/rxJava2
@Override
public void sendMessage(SendMessageRequest request, AsyncMethodCallback resultHandler)
throws TException {
AuthenticationContext context = authenticationManager.getContext();
Single<SendMessageResponse> delegate = Single.defer(
() -> messageService.sendMessage(context, request));
setAsyncResult(resultHandler, () -> delegate);
}

16. ThttpService
> Interop w/rxJava2
@Override
public void sendMessage(SendMessageRequest request, AsyncMethodCallback resultHandler)
throws TException {
AuthenticationContext context = authenticationManager.getContext();
Single<SendMessageResponse> delegate = Single.defer(
() -> messageService.sendMessage(context, request));
setAsyncResult(resultHandler, () -> delegate);
}
o Subscribes `Single`
o Hands over a result or throws an error
o Translations the `Thrown`

17. AnnotatedHttpService
> Service Registration
• ServerBuilder
• AnnotatedServiceRegistrationBean

18. AnnotatedHttpService
> Service Registration
• ServerBuilder
• AnnotatedServiceRegistrationBean
@Bean
public AnnotatedServiceRegistrationBean newAnnotatedHttpService(
String urlPrefix,
BaseHttpRequestHandler handler) {
return new AnnotatedServiceRegistrationBean().setPathPrefix(urlPrefix)
.setServiceName("httpService")
.setService(handler);
}

19. AnnotatedHttpService
> AnnotatedServiceRegistrationBean
@Bean
public AnnotatedServiceRegistrationBean newAnnotatedHttpService(
String urlPrefix,
BaseHttpRequestHandler handler) {
return new AnnotatedServiceRegistrationBean().setPathPrefix(urlPrefix)
.setServiceName("httpService")
.setService(handler);
}
@Get("/openchats")
public CompletableFuture<OpenchatsResponse> getOpenchats(@Param("limit") int limit,
@Param("offset") int offset) {
return response(() -> service.getOpenchats(getContext(), limit, offset));
}

20. Http Client
thrift/RPC client
http client
Retrofit/http client
build(Service.AsyncIface.class)
build(HttpClient.class)
RxJava2CallAdapterFactory
ArmeriaRetrofitBuilder
ClientFactory
ClientBuilder
@GET("/path/to/get")

21. Decorators

22. Decorators
A ‘decorating service/client’ (or a ‘decorator’) is a Service/Client that wraps another Service/Client
to intercept an incoming request or an outgoing response.
As its name says, it is an implementation of the decorator pattern.

23. Decorators
https://flic.kr/p/RX8b5d
@Logging
@Monitoring
@Authenticate
MyService
Spring Annotations
https://qph.fs.quoracdn.net/main-qimg-47591f98c0246fba530cf90018a0c1aa
Decorators
MyService
Logging
Service
Monitoring
Service
Auth
Service

24. Logging
> KafkaAccessLogWriter
> Access/Client Logging
logback.xml
Files
>_
Console
Log Search/Monitoring

25. Logging
> KafkaAccessLogWriter
>Configuration ONLY
> Access/Client Logging
logback.xml
Files
>_
Console
Log Search/Monitoring

26. Customized Decorators
> AuthenticationService
> RpcLoggingService
> RequestIdService
> MaintenanceService
> …

27. RequestIdService
>Recommended & Required
>Similar to Logback’s MDC (Mapped Dignostic Context)

28. @Override
public O serve(ServiceRequestContext ctx, I req) throws Exception {
// For example,
long requestId = System.currentTimeMillis();
ctx.attr(ATTRIBUTE_KEY).set(requestId);
// Copy to a child
ctx.onChild((curCtx, newCtx) -> {
if (curCtx.hasAttr(ATTRIBUTE_KEY)) {
// Set the id to newCtx's attr
}
});
return delegate().serve(ctx, req);
}
RequestIdService
>Recommended & Required
>Similar to Logback’s MDC (Mapped Dignostic Context)

29. <property name="RCEA_EXPORTS"
value="attrs.reqid:com.linecorp..RequestId#ATTRIBUTE_KEY" />
…
<appender name="RCEA_SERVICE”
class="com.linecorp.armeria.common.logback.RequestContextExportingAppender">
<appender-ref ref="ASYNC_SERVICE" />
<exports>${RCEA_EXPORTS}</exports>
</appender>
RequestIdService
>Recommended & Required
>Similar to Logback’s MDC (Mapped Dignostic Context)

30. 2019-11-15 17:00:00.221 [15738048002200] [armeria-common-worker-epoll-2-1]
armeria.services...
2019-11-15 17:00:00.228 [15738048002200] [talk-server-1] c.l.s.s.auth....
2019-11-15 17:00:00.228 [15738048002200] [armeria-common-worker-epoll-2-3]
c.l.s.c.a.c.logging...
2019-11-15 17:00:00.230 [15738048002200] [lettuce-epollEventLoop-21-2] c.l.s.service.sync..
2019-11-15 17:00:00.230 [15738048002200] [lettuce-epollEventLoop-21-2] c.l.s.db.redis...
RequestIdService
>Recommended & Required
>Similar to Logback’s MDC (Mapped Dignostic Context)

31. Metrics
> MetricCollectingService
• Supported by ArmeriaAutoConfiguration
• Backed by ArmeriaSettings
> MetricCollectingClient
ClientDecorationBuilder builder = new ClientDecorationBuilder()
.add(MetricCollectingClient
.newDecorator(MeterIdPrefixFunctionFactory.DEFAULT
.get("client", serviceName)))
.build();

32. Metrics
> MetricCollectingService
• Supported by ArmeriaAutoConfiguration
• Backed by ArmeriaSettings
> MetricCollectingClient
ClientDecorationBuilder builder = new ClientDecorationBuilder()
.add(MetricCollectingClient
.newDecorator(MeterIdPrefixFunctionFactory.DEFAULT
.get("client", serviceName)))
.build();

33. Client Content Preview
>(note) RPC Client
>Without ContentPreview
• Similar to RPC but only headers
2019-11-15 17:00:00.228 [15738048002200] [armeria-common-worker-epoll-2-3]
c.l.s.c.a.c.logging.ClientLoggingClient : [id: 0x3b958e6d] Response: {startTime=2019-11-
15T08:00:00.228Z(1573804800228000), length=275B, duration=108µs(108519ns), headers=[:status=200,
server=legy, content-type=application/x-thrift; protocol=TBINARY, content-length=275],
content=DefaultRpcResponse{LegyResponse(...)}} seqid: 177069

34. Client Content Preview
>(note) RPC Client
>Without ContentPreview
• Similar to RPC but only headers
>With ContentPreview
2019-11-15 17:00:00.228 [15738048002200] [armeria-common-worker-epoll-2-3]
c.l.s.c.a.c.logging.ClientLoggingClient : [id: 0x3b958e6d] Response: {startTime=2019-11-
15T08:00:00.228Z(1573804800228000), length=275B, duration=108µs(108519ns), headers=[:status=200,
server=legy, content-type=application/x-thrift; protocol=TBINARY, content-length=275],
content=DefaultRpcResponse{LegyResponse(...)}} seqid: 177069
2019-10-24 15:34:16.711 [-worker-nio-2-3] c.l.s.c.a.c.logging.HttpLoggingClient :
serviceName=ObsClient, request={startTime=2019-10-24T06:34:16.675Z(1571898856675000), length=94B,
duration=30028µs(30028897ns), scheme=none+h1c, headers=[:method=POST, :path=/for/somewhere,
:authority=upstream.line.me:80, :scheme=http, custom-header=value, content-type=application/json,
content-length=94, user-agent=armeria/0.89.0], contentPreview={...}}

35. CircuitBreaker
Half Open
Closed Open

36. CircuitBreaker
Half Open
Closed Open
>CircuitBreakerStrategy
• HTTP, RPC, etc.
Rejected request count (1m)
Transition(Open) count (1m)

37. RequestTrackingClient
>Cooperate with RequestIdService
@Override
public HttpResponse execute(ClientRequestContext ctx, HttpRequest req) throws Exception {
long reqId = ctx.attr(ATTRIBUTE_KEY).get();
RequestHeadersBuilder builder = req.headers().toBuilder();
builder.add(AsciiString.of(“request-tracking-header”), reqId);
final HttpRequest newReq = HttpRequest.of(req, builder.build());
ctx.updateRequest(newReq);
return delegate().execute(ctx, newReq);
}

38. RequestTrackingClient
>Cooperate with RequestIdService
2019-11-15 [15738048002200] [armeria-common-worker-epoll-2-1] armeria.services...
2019-11-15 [15738048002200] [talk-server-1] c.l.s.s.auth....
2019-11-15 [15738048000810] [armeria-common-worker-epoll-2-6]
c.l.s.c.a.c.logging.ClientLoggingClient : [id: 0x03761f46, ...][h1c://.../path/to/api#POST]
{..., headers=[:method=POST, :path=..., ... request-tracking-header=15738048000810, ...
2019-11-15 [15738048002200] [lettuce-epollEventLoop-21-2] c.l.s.db.redis...

39. Armeria User Journey

40. Ep 0. Learning Curve
https://www.psywww.com/intropsych/ch07-cognition/motor-activity.html

41. Ep 0. Learning Curve
https://www.psywww.com/intropsych/ch07-cognition/motor-activity.html
>Armeria?!
>How to make the API?
>Logging?!
>Metrics?
>Bug Investigation..?!
>Decorators?!!

42. Ep 0. Learning Curve
> ServerBuilder vs ThriftServiceRegistrationBean
> ServerBuilder vs AnnotatedServiceRegistrationBean
> HttpClient vs RetrofitArmeriaClient
> Custom ExceptionHandling vs ExceptionHandler interface
> AOP vs Decorators

43. > ServerBuilder vs ThriftServiceRegistrationBean
> ServerBuilder vs AnnotatedServiceRegistrationBean
> HttpClient vs RetrofitArmeriaClient
> Custom ExceptionHandling vs ExceptionHandler interface
> AOP vs Decorators
Ep 0. Learning Curve

44. https://i.imgur.com/zGHwpAb.png

45. Ep 1. Boot Up
Spring Boot makes it easy to create stand-alone,
production-grade Spring based Applications that you can "just run".

46. Ep 1. Boot Up
AutoConfigration
ArmeriaSettings

47. Ep 1. Boot Up

48. Ep 1. Boot Up
AutoConfigration
ArmeriaSettings
ServiceRegistrationBeans

49. Ep 1. Boot Up
AutoConfigration
ArmeriaSettings
ServiceDecorators
ServiceRegistrationBeans

50. Ep 2. Efficiency
> n(connection) = n(core)*2
• Under the most frequently invoked cases,
• Server requests ~10K/min
• Client requests ~100K/min

51. Ep 3. Outage
> Upstream Outage
• Notification Server Failure
notification
response
timeout

52. Ep 3. Outage
> Upstream Outage
• Notification Server Failure
> Impact Scope
• Partial notification failure
• No big problem to serve
the other APIs
Async Client
notification
response
timeout

53. Ep 3. Outage
request
Reactive
X
(RxJava)

54. Ep 3. Outage
request
response
Reactive
X
(RxJava)
?

55. Ep 3. Outage
>Thread Blocking
request
armeria
io-thread
working
thread
< long time processing >

56. Ep 3. Outage
>Thread Blocking
request
armeria
io-thread
armeria
io-thread
working
thread
< long time processing >
working
thread
request

57. Ep 3. Outage
>Thread Blocking
request
armeria
io-thread
armeria
io-thread
working
thread
working
thread
response
armeria
io-thread
< long time processing >

58. Ep 3. Outage
>Thread Blocking
request
armeria
io-thread
armeria
io-thread
working
thread
< long time processing >
working
thread
response
armeria
io-thread
armeria
io-thread

59. Ep 3. Outage
>Thread Blocking
request
armeria
io-thread
armeria
io-thread
working
thread
< long time processing >
working
thread
response
armeria
io-thread
armeria
io-thread

60. Ep 3. Outage
>Should avoid blocking armeria io threads
request
armeria
io-thread
armeria
io-thread
working
thread
< long time processing >
working
thread
response
armeria
io-thread
working
thread
working
thread
armeria
io-thread

61. Wrap Up

62. Openchat Internal
SPDY/thrift
Reactive
X
(RxJava)
LEGY
auth
…
thrift
/AsyncIface
External
API Invocation
rpc/thrift
http 1.1
http 2

63. Decorators
https://flic.kr/p/RX8b5d
@Logging
@Monitoring
@Authenticate
MyService
Spring Annotations
https://qph.fs.quoracdn.net/main-qimg-47591f98c0246fba530cf90018a0c1aa
Decorators
MyService
Logging
Service
Monitoring
Service
Auth
Service

64. Ep 0. Learning Curve
https://www.psywww.com/intropsych/ch07-cognition/motor-activity.html
>Armeria?!
>How to make the API?
>Logging?!
>Metrics?
>Bug Investigation..?!
>Decorators?!!

65. Thank You
Julie Kim@LINE Openchat Dev