Successfully reported this slideshow.

SmartNews's journey into microservices

14

Share

Loading in …3
×
1 of 33
1 of 33

SmartNews's journey into microservices

14

Share

Download to read offline

Description

SmartNews's journey into microservices
Livesnse TechNight
http://made.livesense.co.jp/blogs/431

Takumi Sakamoto @ SmartNews

Transcript

  1. 1. SmartNews's journey into microservices Takumi Sakamoto Livesense TechNight 2015.09.29
  2. 2. Takumi Sakamoto @takus
  3. 3. News Delivery Pipelines
  4. 4. 36 months ago The first system is developed Crawler Analyzer Indexer On memory DB API Monolithic App Social Media Article Lists
  5. 5. 12 months ago Split crawlers Crawler Queue Monolithic App Analyzer Indexer On memory DB API
  6. 6. https://medium.com/@takus/velocity-conference-2015-wrap-up-velocityconf-f3811e05f1e8 4 months ago At Velocity Conference
  7. 7. 2 months ago Dive into the microservices world Crawler Kinesis IndexerAnalyzer Kinesis CloudSearch
  8. 8. Why microservices? • A lot of different types of services are required for us • crawler, image analyzer, content extractor ... • Adapt to technical progress / market change • every system should be replaceable • Make developers happy :) • avoid to maintain "legacy" "large" "monolithic" • enable us to develop any languages
  9. 9. A part of microservices Source A Analyzer Indexer Search API A PhotoqualCaffe Reporting API Gateway Realtime feedback Image Analysis Extractor Content extraction Source B Search API B Ranking algorithm A/B test Language detecter Crawling Language detection Spark Streaming KPI Dashboard Admin Operation
  10. 10. References (Japanese only) Stream 処理と Offline 処理の統合 http://bit.ly/1VnQQao スマートニュースの世界進出を支えるログ解析基盤 http://bit.ly/1MCOyBX SmartNews の Webmining を支えるプラットフォーム http://bit.ly/1RcVGqk
  11. 11. Journey into microservices When you use microservices you have to work on automated deployment, monitoring, dealing with failure, eventual consistency, and other factors that a distributed system introduces. http://martinfowler.com/bliki/MicroservicePremium.html
  12. 12. Automated Deployment with AWS CodeDeploy Today's topic
  13. 13. Deployment is not easy • Manage complexity • multiple roles • multiple environments • auto scaling • Avoid downtime • JVM based applications doesn't support hot deploy • Monitoring • stop deployment if something wrong • which instances are succeeded or failed
  14. 14. AWS CodeDeploy http://www.slideshare.net/AmazonWebServices/aws-codedeploy-manage-deployment-complexity
  15. 15. 2W1H configurations http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-steps.html Where?How?What?
  16. 16. Deployment Flow http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-steps.html What? How? Where?
  17. 17. What to Deploy? • Prepare your "revision" archive • appspec.yml • application (WAR/JAR etc...) • (optional) scripts • register to ELB / deregister from ELB • pre-compile assets • waiting cache warmup
  18. 18. App Spec
  19. 19. How to Deploy?
  20. 20. Where to Deploy? EC2 Tag / Auto Scaling Group
  21. 21. My favorites • Simple, Fast, Flexible, Free • Require small changes for current system • Integration w/ AWS Auto Scaling • deploy last successful revision to a new instance • wait for registering ELB until finishing deployment
  22. 22. Pitfall Exclusive lock is required :P Deployment is skipped during scale-out
  23. 23. Automated Deployment at SmartNews
  24. 24. Deployment pipelines
  25. 25. Slack notification Role / Env / Version Deployment status Commits
  26. 26. Datadog monitoring
  27. 27. Datadog alerts
  28. 28. Manage alerts by DSL Most of alerts are created automatically Powered by airbnb/interferon
  29. 29. Docker for C++ apps
  30. 30. Towards future...
  31. 31. We’re hiring!!! Join our journey into microservices Site reliability engineer Productivity engineer Web application engineer iOS/Android engineer ML/NLP engineer Ad engineer http://about.smartnews.com/en/careers/
  32. 32. Wait for you @ SmartKitchen

Description

SmartNews's journey into microservices
Livesnse TechNight
http://made.livesense.co.jp/blogs/431

Takumi Sakamoto @ SmartNews

Transcript

  1. 1. SmartNews's journey into microservices Takumi Sakamoto Livesense TechNight 2015.09.29
  2. 2. Takumi Sakamoto @takus
  3. 3. News Delivery Pipelines
  4. 4. 36 months ago The first system is developed Crawler Analyzer Indexer On memory DB API Monolithic App Social Media Article Lists
  5. 5. 12 months ago Split crawlers Crawler Queue Monolithic App Analyzer Indexer On memory DB API
  6. 6. https://medium.com/@takus/velocity-conference-2015-wrap-up-velocityconf-f3811e05f1e8 4 months ago At Velocity Conference
  7. 7. 2 months ago Dive into the microservices world Crawler Kinesis IndexerAnalyzer Kinesis CloudSearch
  8. 8. Why microservices? • A lot of different types of services are required for us • crawler, image analyzer, content extractor ... • Adapt to technical progress / market change • every system should be replaceable • Make developers happy :) • avoid to maintain "legacy" "large" "monolithic" • enable us to develop any languages
  9. 9. A part of microservices Source A Analyzer Indexer Search API A PhotoqualCaffe Reporting API Gateway Realtime feedback Image Analysis Extractor Content extraction Source B Search API B Ranking algorithm A/B test Language detecter Crawling Language detection Spark Streaming KPI Dashboard Admin Operation
  10. 10. References (Japanese only) Stream 処理と Offline 処理の統合 http://bit.ly/1VnQQao スマートニュースの世界進出を支えるログ解析基盤 http://bit.ly/1MCOyBX SmartNews の Webmining を支えるプラットフォーム http://bit.ly/1RcVGqk
  11. 11. Journey into microservices When you use microservices you have to work on automated deployment, monitoring, dealing with failure, eventual consistency, and other factors that a distributed system introduces. http://martinfowler.com/bliki/MicroservicePremium.html
  12. 12. Automated Deployment with AWS CodeDeploy Today's topic
  13. 13. Deployment is not easy • Manage complexity • multiple roles • multiple environments • auto scaling • Avoid downtime • JVM based applications doesn't support hot deploy • Monitoring • stop deployment if something wrong • which instances are succeeded or failed
  14. 14. AWS CodeDeploy http://www.slideshare.net/AmazonWebServices/aws-codedeploy-manage-deployment-complexity
  15. 15. 2W1H configurations http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-steps.html Where?How?What?
  16. 16. Deployment Flow http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-steps.html What? How? Where?
  17. 17. What to Deploy? • Prepare your "revision" archive • appspec.yml • application (WAR/JAR etc...) • (optional) scripts • register to ELB / deregister from ELB • pre-compile assets • waiting cache warmup
  18. 18. App Spec
  19. 19. How to Deploy?
  20. 20. Where to Deploy? EC2 Tag / Auto Scaling Group
  21. 21. My favorites • Simple, Fast, Flexible, Free • Require small changes for current system • Integration w/ AWS Auto Scaling • deploy last successful revision to a new instance • wait for registering ELB until finishing deployment
  22. 22. Pitfall Exclusive lock is required :P Deployment is skipped during scale-out
  23. 23. Automated Deployment at SmartNews
  24. 24. Deployment pipelines
  25. 25. Slack notification Role / Env / Version Deployment status Commits
  26. 26. Datadog monitoring
  27. 27. Datadog alerts
  28. 28. Manage alerts by DSL Most of alerts are created automatically Powered by airbnb/interferon
  29. 29. Docker for C++ apps
  30. 30. Towards future...
  31. 31. We’re hiring!!! Join our journey into microservices Site reliability engineer Productivity engineer Web application engineer iOS/Android engineer ML/NLP engineer Ad engineer http://about.smartnews.com/en/careers/
  32. 32. Wait for you @ SmartKitchen

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

×