1.
2017年の
LINEのマイクロサービスを
⽀えるSpring
Masahiro Ide, LINE Corporation

2.
@imasahiro
https://github.com/imasahiro
Self Intro
l Masahiro Ide
l Server side engineer at LINE
l Mainly worked on スタンプ/着せかえ ショップ
l Contribute to Armeria - Asynchronous RPC/REST library
l https://github.com/line/armeria

3.
• Where is Spring used at LINE?
• Useful/Customized Spring features
• Monitoring
Agenda

4.
LINE

5.
LINE System
l LINE has hundreds of applications
l Recent apps are Java/Spring applications
l Mainly communicates with Thrift, and REST
l Uses same in-house deployment, monitoring system

6.
今⽇お話しするLINEのシステム
lスタンプショップ

7.
スタンプショップ
l Spring-boot 1.5.8, Spring-MVC
l Mostly Asynchronous Http/2 Thrift/REST service
l Java8, RxJava2, (Completable|Listenable)Future
l Storage
l Redis, MySQL, Mongo, ElasticSearch
l Monitoring
l Micrometer, Dropwizard Metrics, Prometheus, Zipkin

8.
Sticker server Mongo
MySQL
ElasticSearch
Redis
LINE Client
Theme server Search server
STORE Server
スタンプショップ 構成

9.
スタンプショップ
l Spring-boot 1.5.8, Spring-MVC
l Mostly Asynchronous Http/2 Thrift/REST service
l Java8, RxJava2, (Completable|Listenable)Future
l Storage
l Redis, MySQL, Mongo, ElasticSearch
l Monitoring
l Micrometer, Dropwizard Metrics, Prometheus, Zipkin

10.
l Easy to setup
l Have many integrations with other libraries
l Mybatis, Thymeleaf, Tomcat, etc.
Why spring, why spring-boot?
@SpringBootApplication
public class SampleApp {
@Configuration
class MvcConf extends WebMvcConfigurerAdapter {…}
public static void main(String[] args) {
SpringApplication.run(SampleApp.class, args);
}
}

11.
Simple & Useful Spring features
1. Spring cache abstraction
1. (Spring) AOP
2. Dependency management

12.
1. Spring Cache Abstraction
l SpringにおけるCacheの抽象化
l Can choose cache implementation
l Redis, Caffeine, Guava, etc.
l …But need to use carefully on async service
@Cacheable(cacheNames = "listCache")
List<Item> selectByLang(String lang) {
return sqlSession.selectList(…);
}

13.
Async support in cache abstraction
lNaive (and wrong) solution
l This just caches a Future…
l But donʼt cache value in Future
l See https://jira.spring.io/browse/SPR-12967
@Cacheable(cacheNames = "listCache")
CompletableFuture<List<Item>> selectByLang(String lang) {
return CompletableFuture.supplyAsync(
() -> sqlSession.selectList(…), executor);
}

14.
Async support in cache abstraction
lMade similar cache abstraction
l with Caffeine AsyncLoadingCache and
Lettuce (Async redis client)
@AsyncCacheable(cacheNames = "listCache")
CompletableFuture<List<Item>> selectByLang(String lang) {
return CompletableFuture.supplyAsync(
() -> sqlSession.selectList(…), executor);
}

15.
Simple & Useful Spring features
1. Spring cache abstraction
1. (Spring) AOP
2. Dependency management

16.
2. (Spring) AOP
lAOP - Aspect Oriented Programming
l Use case: logging, perf instrumentation
@Loggable @GetMapping(“/”)
ModelAndView home(…) {…}
@Around("@annotation(Loggable)")
public Object logging(ProceedingJoinPoint pjp) {
Timer timer = newTimer()
try {
return pjp.proceed();
} finally {
long latency = timer.stop();
logger.info(“request path: {}, args:{}, latency:{}”, …, latency));
}
}

17.
Client side MySQL sharding using AOP
ApplicationApplication
MySQL Slave
MySQL Master
Shard01
MySQL Slave
MySQL Master
Shard01
MySQL Slave
MySQL Master
Shard01
@Shardable
List<Item> selectListByUserId(int id) {…}
List<DataSource> dataSources = …;
@Around("@annotation(Shardable)")
Object sharding(ProceedingJoinPoint pjp) {
int id = getArguments(pjp)[0];
source = dataSrouces.get(id % 12);
useDataSource(source);
return pjp.proceed();
}

18.
3. Dependency management
lBOM and Dependency management plugin
lSpringがlibrary versionを管理してくれる
dependencyManagement {
imports {
mavenBom 'io.spring.platform:platform-bom:Brussels-SR5'
}
dependencies {
dependencySet(group: 'com.linecorp.armeria', version: '0.55.0') {
entry 'armeria-tomcat-shaded’
entry 'armeria-thrift0.9-shaded'
}
dependency 'biz.paluch.redis:lettuce:4.4.1.Final’
…
}}

19.
スタンプショップ
l Spring-boot 1.5.8, Spring-MVC
l Mostly Asynchronous Http/2 Thrift/REST service
l Java8, RxJava2, (Completable|Listenable)Future
l Storage
l Redis, MySQL, Mongo, ElasticSearch
l Monitoring
l Micrometer, Dropwizard Metrics, Prometheus, Zipkin

20.
How to make a service asynchronous?
l元々はTomcat+Spring MVC or thrift+DBなシステム
@Repository public class ShopStorage {
List<Item> selectByLang(String lang) {
return sqlSession.selectList(
“shop.selectByLang”, lang);
}}
@Controller public class ShopController {
@ResponseBody List<Item> home() {
return storage.getList();
}}

21.
How to make a service asynchronous?
lSync accessを１つずつAsyncに書き直す
@Repository public class ShopStorage {
List<Item> selectByLang(String lang) {
return sqlSession.selectList(
“shop.selectByLang”, lang);
}}
@Controller public class ShopController {
@ResponseBody List<Item> home() {
return storage.getList();
}}
@Repository public class AsyncShopStorage {
CompletableFuture<List<Item>>
selectByLang(String lang) {
return CompletableFuture.supplyAsync(
() -> sqlSession.selectList(
“shop.selectByLang”, lang),
executor);
}}
@Controller public class AsyncShopController {
@ResponseBody
DeferredResult<List<Item>> home() {
result = new DeferredResult<>();
storage.getList()
.thenApply(result::setResult);
return result;
}}

22.
• Where is Spring used at LINE?
• Useful/Customized Spring features
• Monitoring
Agenda

23.
How to observe our services?
lLog
l その瞬間に何が起こったかを確認する
lMetric
l 現在起きている現象のトレンドを確認する
lTrace
l システム間で何が何が起きているかを確認する

24.
How to observe our services?
lLog
l Slf4j + Kibana
lMetric
l Micormeter, Prometheus, Dropwizard metrics
l Grafana
lTrace
l Zipkin
l io.zipkin.brave:brave-instrumentation-mysql
l io.zipkin.brave:brave-instrumentation-spring-webmvc
l And elasticsearch, mongo, redis integrations, etc.

25.
Micrometer
l Supports multiple monitoring tools
l Dropwizard Metrics, Prometheus, etc.
l Will be provide as a Spring Boot 2.0 metrics collection
l Already used in production servers
l Found some concurrency issues but now looks stable
l https://github.com/micrometer-metrics/micrometer/issues/139
l https://github.com/micrometer-metrics/micrometer/issues/172
l https://github.com/micrometer-metrics/micrometer/issues/208

26.
Prometheus
lSimple TSDB, Pull型
lSupports querying, alerting
Application
# HELP armeria_client_response_duration_seconds
# TYPE armeria_client_response_duration_seconds summary
armeria_client_response_duration_seconds{method="sendMessage",
service=”Service",quantile="0.5",} 8.194E-4
armeria_client_response_duration_seconds{method="sendMessage",
service=”Service",quantile="0.75",} 9.494E-4
armeria_client_response_duration_seconds{method="sendMessage",
service=”Service",quantile="0.90",} 9.994E-4
…
/internal/metrics
Node exporterPrometheus
pull ApplicationNode exporter

27.
Micrometer+Prometheus+Grafana

28.
Zipkin

29.
Put all together
lArmeria - Spring, micrometer, zipkin integration
@Bean
ThriftServiceRegistrationBean service(MyService.AsyncIface myService,
Tracing tracing) {
return new ThriftServiceRegistrationBean()
.setPath("/thrift/")
.setService(
THttpService.of(shopCapabilityService)
.decorate(LoggingService.newDecorator())
.decorate(MetricCollectingService.newDecorator(
MeterIdFunction.ofDefault("LineShopService")))
.decorate(HttpTracingService.newDecorator(tracing))));
}

30.
Thank you