Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LINE スタンプショップにおける Zipkin 利用事例

1,313 views

Published on

林康司 (LINE Fukuoka) / 川田裕貴 (LINE)
LINE スタンプショップにおける Zipkin 利用事例
「LINE Developer Meetup #35 - Zipkin -」2018/05/23 の発表資料です

Published in: Technology
  • Be the first to comment

  • Be the first to like this

LINE スタンプショップにおける Zipkin 利用事例

  1. 1. 7592 
 in LSNLQ
  2. 2. 7592 in LSNLQ E E L A C NIK H 1 1 1 E - -
  3. 3. 7592 in LSNLQ 7592 RS a a u h c c . 5 i yn m .W R WJJHU 7592 : 2 7592 ADOOH s c
  4. 4. 7592 in LSNLQ 7592 RS c ck o 5Q .SSa : 2 i t x . 5 a a c c a d c cp a t . 5 a v c c t j gj t . 5 v a .W R UWJJHU x t . 5 a
  5. 5. 7592 in LSNLQ RPSRQHQ U 8L TRUHT L HU TLI .TPHTLD .U Q TRQRWU 5 : D D Z D D
  6. 6. 7592 in LSNLQ 8L TRUHT L HU LQ 7592 RS c f t c c f o• c c u g . 5 l u ha f
  7. 7. 7592 in LSNLQ 8L TRUHT L HU LQ 7592 RS .TPHTLD rx TLI .TPHTLD, 7592 : SU, OLQH JL WE LR DTPHTLD 7592 TLI x r f :9 c f f g f•f• c f a c c , 2ODU L UHDT , 8 7 u ha , 8RQJR1/ , HGLU
  8. 8. 7592 in LSNLQ 9RQ EOR NLQJ j .TPHTLD .U Q TRQRWU TLI HT HT OLHQ k 9RQ EOR NLQJ Z D D c c Z D D rx HD L H THDPU k c Z D D rx xz rx d rm Z D D .TPHTLD c r “x_
 SU, HQJLQHHTLQJ OLQH RTS RP MD EORJ GH DLO
  9. 9. 7592 in LSNLQ c “ j . 5 rk . 5 rk ok j k f j m j f k f . 5 ke k o” f j j f c cyn oj f ske c . 5 k oj j rxf . 5 c f k oj x j j f c xk 5 : k j r mf
  10. 10. 7592 in LSNLQ 9RQ EOR NLQJ .U Q TRQRWU “ u jm rx yj f f sk xf f jb ” . 5 k f j j fa l ke j rjrx o r f j b c TRGW LRQ t
  11. 11. 7592 in LSNLQ vo BLSNLQ g d . 5 j c t c “xf k f j f oj x j ” f j c PLUU こんな問題を解決できる
  12. 12. 7592 in LSNLQ BLSNLQ “ r SDQ r h 7592 RS .TPHTLD k TLI t p SDQ HDGHT r c c v w c c x SDQ TD H Server A Server B Server C Server D traceId: 5 getB() traceId: 5 getD() traceId: 5 getC() TracegetA()
  13. 13. 7592 in LSNLQ Server A Server B Server C Server D traceId: 5 getA() traceId: 5 getD() traceId: 5 getC() Zipkin Storage Span 1 Span 2 Span 3 Span 4 Zipkin UI BLSNLQ RTDJH x LQ PHPRT SDQ l ” BLSNLQ 5 TD H
  14. 14. 7592 in LSNLQ
  15. 15. 7592 in LSNLQ ユーザー token 認証 Product 情報取得 Ownership 情報取得 JH TRGW . 5
  16. 16. 7592 in LSNLQ SDQ k f . 5
  17. 17. 7592 in LSNLQ SDQ k f . 5 ちゃんと非同期で動いてるね。
  18. 18. 7592 in LSNLQ BLSNLQ g D D, /TD H BLSNLQ D D STLQJ HE .SD H S OLHQ 8 7 k f SU, JL WE RP RSHQ LSNLQ ETD H v c c e SU, LSNLQ LR SDJHU HZLU LQJCLQU TWPHQ D LRQU PO c SDQ u
  19. 19. 7592 in LSNLQ BLSNLQ TD LQJ LQ 7592 RS TLI .TPHTLD .TPHTLD TH TRIL SU, OLQH JL WE LR DTPHTLD DG DQ HG LSNLQ PO 1D DEDUH RTDJH 8 7 /TD H 8RQJR1/ HGLU 7H W H 2ODU L UHDT BLSNLQ SDQ c
  20. 20. 7592 in LSNLQ BLSNLQ TD LQJ LQ 7592 RS TRGW LRQ TD H r t TD H t c k d f TD H k e f DJ n f 7 D H PLUU RT L RU 5 D WU RGHd
  21. 21. 7592 in LSNLQ BLSNLQ RTDJH LQ 7592 RS x TLI . 5 r f xyr D ORDG :9 .TPHTLD x 9RQ EOR NLQJ SDQ rxj x RTDJH 2ODU L UHDT BLSNLQ 5 .TPHTLD u r f Server A Custom Zipkin ServerServer B Thrift Spans Thrift Spans Elasticsearch Zipkin UI
  22. 22. 7592 in LSNLQ 8H TL U 8RQL RTLQJ “ u oo BLSNLQ ff o• r lxn d . 5 t BLSNLQ em x c r BLSNLQ yn 8H TL U PRQL RTLQJ f 7592 RS 8L TRPH HT 8H TL TRPH HWU 3TDIDQD 1DU ERDTG
  23. 23. 7592 in LSNLQ 8H TL U 8RQL RTLQJ BLSNLQ BLSNLQ k . 5 7D HQ k S)( m fk S ( f HT HT OLHQ PH TL U z ” f f l rf BLSNLQ ke f j r tf . 5 7D HQ k fk k e z k j xf /H D l fd BLSNLQ ke TRGW LRQ TD LQJ r j
  24. 24. 7592 in LSNLQ Zipkin を運用する時に苦労した話は後半で
  25. 25. LINE スタンプショップにおける zipkin 利用事例 林 康司 LINE Fukuoka Corporation 開発一室 KOJI LIN
  26. 26. LINE スタンプショップにおける zipkin 利用事例 About Me • Software engineer at LINE Fukuoka • Sticker/Theme Shop/Wallet • Mainly working on Server Side • Taiwan Java User Group Organizer
  27. 27. LINE スタンプショップにおける zipkin 利用事例 Shop is fully non-blocking/asynchronous • Synchronous call don’t scale, especially in micro service architecture • Max concurrent requests <= max threads • Threads are doing nothing but just waiting for the backend • But asynchronous call is hard to read/write • Callback ? Reactive Streams ? Future ? • Which thread is running this code ? How to get Context ?
  28. 28. LINE スタンプショップにおける zipkin 利用事例 Shop is fully non-blocking/asynchronous • Armeria • LINE’s open source RPC/HTTP client/server Library • Controller uses callback/CompletableFuture/Deferred type to sending response • RxJava 2 • Service/Dao/External clients are using RxJava2
  29. 29. LINE スタンプショップにおける zipkin 利用事例 ServerBuilder sb = new ServerBuilder(); new ServerBuilder().service("/greet/{name}", new AbstractHttpService() { @Override protected HttpResponse doGet(ServiceRequestContext ctx, HttpRequest req) return HttpResponse.from(CompletableFuture.supplyAsync(() -> { // … return HttpResponse.of("Hello, %s!", ctx.pathParam("name")); })); } });
  30. 30. LINE スタンプショップにおける zipkin 利用事例 Advantage of using RxJava2 • Easier to compose async tasks • Tasks can run concurrently • zip • Tasks have dependency • map • flatMap • There is CompletableFuture/Future <-> RxJava2 converters
  31. 31. LINE スタンプショップにおける zipkin 利用事例
  32. 32. LINE スタンプショップにおける zipkin 利用事例 getAvailableMenu(ctx) .observeOn(Schedulers.from( currentExecutorService.get())) .flatMap(availableMenuItem -> Single.zip( getEvents(ctx), getSummaries(ctx), getGames(ctx, availableMenuItem), getAggregatedHome(ctx, availableMenuItem), getOfficials(ctx, availableMenuItem), banner(ctx), getOthers(ctx), AggregateResponse::new));
  33. 33. LINE スタンプショップにおける zipkin 利用事例
  34. 34. LINE スタンプショップにおける zipkin 利用事例 Single.zip( findCategory(categoryName) .observeOn(Schedulers.from( currentExecutorService.get())) .flatMapSingleElement(category -> Single.zip( listFromStorage(category), listFromApi(category), this::mergePromotions)) .flattenAsFlowable(r -> r) .skip(offset) .take(limit) .toList(), promotionService.listCategories(), Response::new);
  35. 35. LINE スタンプショップにおける zipkin 利用事例 Where/How do we add zipkin trace? • All servers • Redis • MySQL • MongoDB • Retrofit • Thrift Client
  36. 36. LINE スタンプショップにおける zipkin 利用事例 Armeria • Provides zipkin module • Automatically inject/extract zipkin related B3 attribute to/from http header • When using decorator, Armeria create/close span scope with RequestContext’s push/close event .setService(…) .setDecorator(HttpTracingService.newDecorator(tracing) …
  37. 37. LINE スタンプショップにおける zipkin 利用事例 Armeria public final <T, R> Function<T, R> makeContextAware(Function<T, R> fn) { return t -> { try (SafeCloseable ignored = propagateContextIfNotPresent()) { return function.apply(t); } }; } public <U> CompletableFuture<U> thenApply(Function<? super T,? extends U> fn) { return ctx.makeContextAware(super.thenApply(ctx.makeContextAware(fn))); } ctx.makeContextAware(future).handle(…) public final <T> CompletionStage<T> makeContextAware(CompletionStage<T> stage) { final CompletableFuture<T> future = new RequestContextAwareCompletableFuture<>(this); … return future; }
  38. 38. LINE スタンプショップにおける zipkin 利用事例 MySQL • JDBC API is blocking, so using thread pool to make it Async • Brave already provides TracingStatementInterceptor • Just append `?statementInterceptors=brave.mysql.TracingStatementInterceptor` to the connection url ListeningExecutorService asyncExecutor = …; asyncExecutor.submit(() -> delegate.selectList(statement, parameter));
  39. 39. LINE スタンプショップにおける zipkin 利用事例 Redis • Lettuce • Provides sync/async and reactive usage • Has Future/CompletableFuture API • So we can easily wrap command with span start/finish
  40. 40. LINE スタンプショップにおける zipkin 利用事例 Redis Span span = tracer.nextSpan(); span.kind(Kind.CLIENT) .remoteEndpoint(endpoint); maybe.doOnSubscribe(disposable -> span.start()) .doOnEvent((result, thrown) -> span.finish());
  41. 41. LINE スタンプショップにおける zipkin 利用事例 Redis • Lettuce • Provides sync/async and reactive usage • Has Future/CompletableFuture API • We can wrap command with span start/finish • The drawback is it’s not just before/after command send/receive
  42. 42. LINE スタンプショップにおける zipkin 利用事例 MongoDB • MongoDB Java driver doesn’t have interceptor like MySQL driver • We customize AsyncOperationExecutor to make it support tracing
  43. 43. LINE スタンプショップにおける zipkin 利用事例 MongoDB @Override public <T> void execute(AsyncReadOperation<T> operation, ReadPreference readPreference, SingleResultCallback<T> callback) { final SingleResultCallback<T> wrappedCallback = errorHandlingCallback(callback, logger); final AsyncReadBinding binding = getReadWriteBinding(readPreference); Tracer tracer = Tracing.currentTracer(); final Span span = tracer.nextSpan().kind(Kind.CLIENT).name(“read").start(); operation.executeAsync(binding, (result, t) -> { closeTrace(span, result, t); try { wrappedCallback.onResult(result, t); } finally { binding.release(); } }); }
  44. 44. LINE スタンプショップにおける zipkin 利用事例 MongoDB • MongoDB Java driver doesn’t have interceptor like MySQL driver • We customize AsyncOperationExecutor to make it support tracing • The drawback is we need to check/re-test when upgrading driver
  45. 45. LINE スタンプショップにおける zipkin 利用事例 Retrofit and Thrift Client • Both are on top of Armeria • Armeria provides retrofit module • Using RxJava2CallAdapter
  46. 46. LINE スタンプショップにおける zipkin 利用事例 The problem we met • Unfinished trace • Using CurrentTraceContext with INHERITABLE ThreadLocal • Thread pool keeps it’s own scope forever, so we have super long running trace
  47. 47. LINE スタンプショップにおける zipkin 利用事例 The problem we met • Unfinished trace • Using CurrentTraceContext with INHERITABLE ThreadLocal • Thread pool keeps it’s own scope forever, so we have super long running trace • Fix it by using ThreadLocal instead of InheritableThreadLocal • CurrentTraceContext.Default#create
  48. 48. LINE スタンプショップにおける zipkin 利用事例 The problem we met • Unbalanced span • Some of external client/server requests not traced correctly when client/server are all run in the same event-loop • Preserve/remove SpanScope with request context’s lifecycle • Using same thread and ThreadLocal instance for client and server contexts have intersections
  49. 49. LINE スタンプショップにおける zipkin 利用事例 Unbalanced span Event loop thread server context client context client context client context
  50. 50. LINE スタンプショップにおける zipkin 利用事例 Unbalanced span public static void setupContext(FastThreadLocal<SpanInScope> threadLocalSpan, 
 RequestContext ctx, Span span, Tracer tracer) { ctx.onEnter(unused -> threadLocalSpan.set(tracer.withSpanInScope(span))); ctx.onExit(unused -> { SpanInScope spanInScope = threadLocalSpan.get(); if (spanInScope != null) { spanInScope.close(); threadLocalSpan.remove(); } }); }
  51. 51. LINE スタンプショップにおける zipkin 利用事例 Unbalanced span Event loop thread server context client context client context client context onEnter onEnter onExit
  52. 52. LINE スタンプショップにおける zipkin 利用事例 The problem we met • Unbalanced span • Some of external client/server requests not traced correctly when client/server are all run in the same event-loop • Preserve/remove SpanScope with request context’s lifecycle • Using same thread/ThreadLocal instance for client/server contexts have intersections • Fix it by using different ThreadLocal for client/server contexts
  53. 53. LINE スタンプショップにおける zipkin 利用事例 The challenge of using RxJava2 • We need to keep context during the chain of operators • Armeria’s RequestContext • Zipkin • All use ThreadLocal to preserve the context • So we don’t need to add RequestContext and Tracer in every method’s parameter • We need to jump back to context aware thread or wrapping it with scope • Add `observeOn` before every operator chains next RxJava2 call
  54. 54. LINE スタンプショップにおける zipkin 利用事例 Single.zip( findCategory(categoryName) .observeOn(Schedulers.from( currentExecutorService.get())) .flatMapSingleElement(category -> Single.zip( listFromStorage(category), listFromApi(category), this::mergePromotions)) .flattenAsFlowable(r -> r) .skip(offset) .take(limit) .toList(), promotionService.listCategories(), Response::new);
  55. 55. LINE スタンプショップにおける zipkin 利用事例 Add observeOn for each flatMap is verbose • Using RxJavaPlugins to add CurrentTraceContextAssemblyTracking • https://github.com/openzipkin/brave/pull/665 • We are using Armeria, so we can add RequestContext’s one similar to above PR • Using RxJavaPlugins to add observeOn hook • Always jump back to Armeria’s event loop • Using kotlinx’s coroutines-rx2/coroutines-jdk8
  56. 56. LINE スタンプショップにおける zipkin 利用事例 Using kotlin’s coroutines-rx2 val context = currentExecutorService.get().asCoroutineDispatcher() return future(context) { return@future Response.of(…) }
  57. 57. Q&A

×