Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Architecture Sustaining LINE Sticker services

480 views

Published on

Masahiro Ide (LINE Corporation)
LINE Vietnam Opening Day, March 31st 2018

Published in: Technology
  • Be the first to comment

Architecture Sustaining LINE Sticker services

  1. 1. Architecture Sustaining LINE Sticker Services Masahiro IDE, LINE CORP
  2. 2. About me • Masahiro Ide • Software Engineer of LINE server development • Especially StickerShop backend server • Joined: Oct, 2015 • Contribute to Armeria • Our open sourced asynchronous RPC/REST library • https://github.com/line/armeria
  3. 3. Agenda • Sticker Shop Server • History • Our Architecture • Implementations
  4. 4. What is Sticker Shop?
  5. 5. Sticker Shop … in 2015 One of the biggest monolithic Java application in LINE • Purchasing, Serving, Ranking stickers in 1 app • Many pains, troubles • Takes more than 3 hours to restart app
  6. 6. Sticker Shop in 2018 Now shift to microservice way for scalability • 1 million stickers • 30 services • Purchasing, Serving, Ranking stickers, etc. • 70K requests/sec in peek time
  7. 7. StickerShop Architecture Overview (Simplified) Central Dogma – Configuration Repository Service Talk server ElasticSearch Client ReverseProxy Thrift/HTTP2 Thrift/HTTP2 Redis MySQL Sticker capability server Shop server Search FE Thrift/HTTP2
  8. 8. Implementation • What we need is… • Performance • Prepared to faults • Observability • Stable feature rollout
  9. 9. Central Dogma • Repository service for textual configurations • JSON, YAML, XML … • Highly available • multi-master, eventually consistency • Version controlled • https://line.github.io/centraldogma/
  10. 10. Armeria - Our RPC layer • Asynchronous RPC/REST library • built on top of Java 8, Netty, HTTP/2, Thrift and gRPC • Take care common functionality for microservice 1. Load balancing 2. Circuit Breaker / Retry / Throttling 3. Tracing (Zipkin) / Monitoring integration • etc. https://line.github.io/armeria/
  11. 11. 1. Client-side load balancing • No proxy load balancers • Proxy would be inefficient when considering request heavy services • CentralDogma as a service discovery • All services register a name and a list of machines in CentralDogma • All clients monitor CentralDogma and balance requests across machines LB Client Server Client Server Server Proxy load balancing
  12. 12. 1. Client-side load balancing • No proxy load balancers • Proxy would be inefficient when considering request heavy services • CentralDogma as a service discovery • All services register a name and a list of machines in CentralDogma • All clients monitor CentralDogma and balance requests across machines Central Dogma Client Server Client Server Server
  13. 13. 2. Circuit Breaker - Avoid cascading failure • Server die suddenly • e.g. Hardware failure • Upstream cannot proceed part of requests, in worst case • Blocks all of application thread • LINE die Service A Service C Service B
  14. 14. 2. Circuit Breaker - Avoid cascading failure • Circuit Breaker pattern* 1. Monitors failures 2. Return error immediately without RPC once the failures reach a threshold Service A Service C Service B CircuitBreaker CircuitBreaker * https://martinfowler.com/bliki/CircuitBreaker.html
  15. 15. 3. Tracing • Hard to point out the slowness/bug in microservice • No one provides complete picture of system performance Talk server ElasticSearch Reverse Proxy Redis MySQL Shop server Search FE T=start T=end
  16. 16. 3. Tracing • Solution: Distributed tracing with Zipkin
  17. 17. Rollout features using CentralDogma (CD) • Rollout to production within 10 minutes of being changed • Deploy a YAML file to all our production servers via CD • Record changes in CD commit log
  18. 18. Rollout features using CentralDogma • Rollout to production within 10 minutes of being changed • Deploy a YAML file to all our production servers via CD • Record changes in CD commit log 30%CacheV2 rollout: Central Dogma Service Service Service commit YAMLPull & Reload config Developer
  19. 19. Conclusion • Our system is built on top of our OSSs • Armeria, CentralDogma, and more • … are still evolving • Trouble may happen • Make a well trouble-prepared system • Let's do a smooth release
  20. 20. Q & A

×