Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
SmartNews's
journey into microservices
Takumi Sakamoto
Livesense TechNight
2015.09.29
Takumi Sakamoto
@takus
News Delivery Pipelines
36 months ago
The first system is developed
Crawler
Analyzer
Indexer
On memory DB
API
Monolithic App
Social
Media
Article
L...
12 months ago
Split crawlers
Crawler
Queue
Monolithic App
Analyzer
Indexer
On memory DB
API
https://medium.com/@takus/velocity-conference-2015-wrap-up-velocityconf-f3811e05f1e8
4 months ago
At Velocity Conference
2 months ago
Dive into the microservices world
Crawler
Kinesis
IndexerAnalyzer
Kinesis CloudSearch
Why microservices?
• A lot of different types of services are required for us
• crawler, image analyzer, content extractor...
A part of microservices
Source A
Analyzer Indexer Search API
A
PhotoqualCaffe
Reporting
API
Gateway
Realtime
feedback
Imag...
References (Japanese only)
Stream 処理と Offline 処理の統合
http://bit.ly/1VnQQao
スマートニュースの世界進出を支えるログ解析基盤
http://bit.ly/1MCOyBX
Sm...
Journey into microservices
When you use microservices you have to work on
automated deployment,
monitoring,
dealing with f...
Automated Deployment
with AWS CodeDeploy
Today's topic
Deployment is not easy
• Manage complexity
• multiple roles
• multiple environments
• auto scaling
• Avoid downtime
• JVM ...
AWS CodeDeploy
http://www.slideshare.net/AmazonWebServices/aws-codedeploy-manage-deployment-complexity
2W1H configurations
http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-steps.html
Where?How?What?
Deployment Flow
http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-steps.html
What?
How?
Where?
What to Deploy?
• Prepare your "revision" archive
• appspec.yml
• application (WAR/JAR etc...)
• (optional) scripts
• regi...
App Spec
How to Deploy?
Where to Deploy?
EC2 Tag / Auto Scaling Group
My favorites
• Simple, Fast, Flexible, Free
• Require small changes for current system
• Integration w/ AWS Auto Scaling
•...
Pitfall
Exclusive lock is required :P
Deployment is skipped during scale-out
Automated Deployment
at SmartNews
Deployment pipelines
Slack notification
Role / Env / Version
Deployment status
Commits
Datadog monitoring
Datadog alerts
Manage alerts by DSL
Most of alerts are created automatically
Powered by airbnb/interferon
Docker for C++ apps
Towards future...
We’re hiring!!!
Join our journey into microservices
Site reliability engineer
Productivity engineer
Web application engine...
Wait for you @ SmartKitchen
SmartNews's journey into microservices
Upcoming SlideShare
Loading in …5
×

SmartNews's journey into microservices

8,177 views

Published on

SmartNews's journey into microservices
Livesnse TechNight
http://made.livesense.co.jp/blogs/431

Takumi Sakamoto @ SmartNews

Published in: Technology

SmartNews's journey into microservices

  1. 1. SmartNews's journey into microservices Takumi Sakamoto Livesense TechNight 2015.09.29
  2. 2. Takumi Sakamoto @takus
  3. 3. News Delivery Pipelines
  4. 4. 36 months ago The first system is developed Crawler Analyzer Indexer On memory DB API Monolithic App Social Media Article Lists
  5. 5. 12 months ago Split crawlers Crawler Queue Monolithic App Analyzer Indexer On memory DB API
  6. 6. https://medium.com/@takus/velocity-conference-2015-wrap-up-velocityconf-f3811e05f1e8 4 months ago At Velocity Conference
  7. 7. 2 months ago Dive into the microservices world Crawler Kinesis IndexerAnalyzer Kinesis CloudSearch
  8. 8. Why microservices? • A lot of different types of services are required for us • crawler, image analyzer, content extractor ... • Adapt to technical progress / market change • every system should be replaceable • Make developers happy :) • avoid to maintain "legacy" "large" "monolithic" • enable us to develop any languages
  9. 9. A part of microservices Source A Analyzer Indexer Search API A PhotoqualCaffe Reporting API Gateway Realtime feedback Image Analysis Extractor Content extraction Source B Search API B Ranking algorithm A/B test Language detecter Crawling Language detection Spark Streaming KPI Dashboard Admin Operation
  10. 10. References (Japanese only) Stream 処理と Offline 処理の統合 http://bit.ly/1VnQQao スマートニュースの世界進出を支えるログ解析基盤 http://bit.ly/1MCOyBX SmartNews の Webmining を支えるプラットフォーム http://bit.ly/1RcVGqk
  11. 11. Journey into microservices When you use microservices you have to work on automated deployment, monitoring, dealing with failure, eventual consistency, and other factors that a distributed system introduces. http://martinfowler.com/bliki/MicroservicePremium.html
  12. 12. Automated Deployment with AWS CodeDeploy Today's topic
  13. 13. Deployment is not easy • Manage complexity • multiple roles • multiple environments • auto scaling • Avoid downtime • JVM based applications doesn't support hot deploy • Monitoring • stop deployment if something wrong • which instances are succeeded or failed
  14. 14. AWS CodeDeploy http://www.slideshare.net/AmazonWebServices/aws-codedeploy-manage-deployment-complexity
  15. 15. 2W1H configurations http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-steps.html Where?How?What?
  16. 16. Deployment Flow http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-steps.html What? How? Where?
  17. 17. What to Deploy? • Prepare your "revision" archive • appspec.yml • application (WAR/JAR etc...) • (optional) scripts • register to ELB / deregister from ELB • pre-compile assets • waiting cache warmup
  18. 18. App Spec
  19. 19. How to Deploy?
  20. 20. Where to Deploy? EC2 Tag / Auto Scaling Group
  21. 21. My favorites • Simple, Fast, Flexible, Free • Require small changes for current system • Integration w/ AWS Auto Scaling • deploy last successful revision to a new instance • wait for registering ELB until finishing deployment
  22. 22. Pitfall Exclusive lock is required :P Deployment is skipped during scale-out
  23. 23. Automated Deployment at SmartNews
  24. 24. Deployment pipelines
  25. 25. Slack notification Role / Env / Version Deployment status Commits
  26. 26. Datadog monitoring
  27. 27. Datadog alerts
  28. 28. Manage alerts by DSL Most of alerts are created automatically Powered by airbnb/interferon
  29. 29. Docker for C++ apps
  30. 30. Towards future...
  31. 31. We’re hiring!!! Join our journey into microservices Site reliability engineer Productivity engineer Web application engineer iOS/Android engineer ML/NLP engineer Ad engineer http://about.smartnews.com/en/careers/
  32. 32. Wait for you @ SmartKitchen

×