Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWSの進化とSmartNewsの裏側

10,854 views

Published on

スマートニュース株式会社 マネージャ Site Reliability Engineering担当 尾形 暢俊
次々に出てくるAWSの新サービスを使って、SmartNewsのバックエンドがどうように変わっていったかをご紹介します。

Published in: Technology

AWSの進化とSmartNewsの裏側

  1. 1. Evolution of AWS Backend of SmartNews 2016/01/27 Classmethod x SmartNews AWS Night
  2. 2. Self Introduction • Nobutoshi Ogata • Manager, Site Reliability Engineering • @nobu666 • ❤ Whiskey, Cat, Heavy Metal • Entrusted dev.(10y) ➡ Game platform infrastructure(3y) ➡ Some startup(1y) ➡ SmartNews(2015/05-)
  3. 3. SmartNews
  4. 4. 15,000,000+ downloads worldwide
  5. 5. SmartNews 3.0
  6. 6. LINE Stamp shipped today
  7. 7. ! Channel!
  8. 8. ! Channel!
  9. 9. Questionnaire
  10. 10. Agenda • Scale • Services • Changes in the configuration • Configuration management • Future plan
  11. 11. Scale
  12. 12. Scale • 300+ EC2 instances • 80 ELBs • 70 Auto Scaling Groups • 80+ S3 buckets (300+TB) • 40 RDS instances (MySQL & Aurora) • 100 Kinesis shards • 10 EMR clusters (Hive+Spark)
  13. 13. Scale • 50+ ElastiCache clusters (Redis + Memcached) • 30+ CodeDeploy applications • 70,000,000 CloudSearch seachable documents • 8 Redshift nodes • 10+ Lambda functions • 80+ SQS queues • 20+ SNS topics
  14. 14. Services
  15. 15. Changes in the configuration
  16. 16. Stone Age Analyzer Crawler Indexer On-memory DB API
  17. 17. Monolithic Application
  18. 18. Bronze Age Analyzer Indexer On-memory DB API Crawler Crawler Crawler Queue
  19. 19. Split Crawler
  20. 20. Iron Age API Indexer Analyzer Crawler
  21. 21. Split more
  22. 22. Present day Crawler Nationality detector Language detector Analyzer Facial recognition Indexer Ranking Search Recommend API Gateway Admin tool Validator Proxy Live Push Sexy detector Auth Proxy
  23. 23. Micro Services?
  24. 24. • SmartNews's journey into microservices • インフラ専任エンジニアが一人もいないSmartNewsにおける クラウド活用法 • SmartNews Ads • Next session about data platform
  25. 25. Configuration management
  26. 26. Pull style deploy
  27. 27. ➡ ➡ • CodeDeploy was used to automate deployment tasks • Operating an AWS ELB • Fetching a new revision • Restarting services with our custom program • e.g. downloading machine learning model data, precompiling assets and deploying Docker containers
  28. 28. Auto Scaling Group • There is an advantage even if you don't want to autoscaling • Easy to dispose • replace kernel, switch to other OS, ... • Easy A/B testing • change config only specific server • tag:Attributes • Instance Protection
  29. 29. ASG ! InstanceName • We were stop using InstanceName • Using tag:Group and tag:Env
  30. 30. Infrastructure as Code • Route53 ➡ winebarrel/roadworker • Security Group ➡ winebarrel/piculet • Elastic IP Addresses ➡ winebarrel/eipmap • Provisioning ➡ itamae-kitchen/itamae • Datadog ➡ airbnb/interferon
  31. 31. DSL management is not perfect • Complicated automation • ChatOps ➡ change code ➡ git add && git commit && git push • Too much hassle... !
  32. 32. Run Command • Secure • IAM, CloudTrail • Easy CI integration w/o SSH • be able to use EMR also
  33. 33. Example aws ssm send-command --document-name "AWS-RunShellScript" --instance-ids $(aws emr list-instances --cluster-id $(aws emr list-clusters --active | jq -r '.Clusters[] | select (.Name=="FOOBAR") | .Id') | jq -r '.Instances[].Ec2InstanceId') --parameters '{"commands":["ls"],"executionTimeout":["10"]}' --timeout-seconds 10 --output-s3-bucket-name "ssm-logs" --output-s3-key-prefix "prefix-" --region ap-northeast-1
  34. 34. Future plan
  35. 35. ECS Spot Fleet Contaniner Registry ConsulAWS Blackbelt 2015 ECS p.98
  36. 36. Containerize all services
  37. 37. Scalable Safety Efficient Portable Measurable
  38. 38. Aggregates events
  39. 39. Events aggregator • Events • Deploy/Provision/Restart/Terminate etc. • All events goes to SNS or Kinesis. Users will describe and hook scripts • Visualize via Datadog • Notice to Slack • Auto healing
  40. 40. Summary
  41. 41. Summary• AWS release new service continuously • We ride on the shoulders of giants • We will never forget that OSS helped us
  42. 42. We're hiring! Only two people on Site Reliability Engineering Team http://about.smartnews.com/en/careers/
  43. 43. Credit https://flic.kr/p/hAcPRk taken by Niuton may https://flic.kr/p/9UnRWS taken by Alberto G. https://flic.kr/p/ydnEvw taken by ep_jhu https://flic.kr/p/pyfnTV taken by Kevin Dooley https://flic.kr/p/mtLpc4 taken by L1mey https://flic.kr/p/fgG226 taken by Linux Screenshots https://flic.kr/p/7DsuLz taken by Gloson Teh https://flic.kr/p/jfGpn7 taken by julochka

×