![Example
aws ssm send-command
--document-name "AWS-RunShellScript"
--instance-ids
$(aws emr list-instances
--cluster-id
$(aws emr list-clusters --active |
jq -r '.Clusters[] | select (.Name=="FOOBAR") | .Id') |
jq -r '.Instances[].Ec2InstanceId')
--parameters '{"commands":["ls"],"executionTimeout":["10"]}'
--timeout-seconds 10
--output-s3-bucket-name "ssm-logs"
--output-s3-key-prefix "prefix-"
--region ap-northeast-1](https://image.slidesharecdn.com/20160127awsnightogatav4-160127101618/75/awssmartnews-40-2048.jpg?cb=1667511156)
AWSの進化とSmartNewsの裏側
•38 likes•12,773 views
Download to read offline
Report
スマートニュース株式会社 マネージャ Site Reliability Engineering担当 尾形 暢俊 次々に出てくるAWSの新サービスを使って、SmartNewsのバックエンドがどうように変わっていったかをご紹介します。
Recommended
More Related Content
What's hot
What's hot(20)
Viewers also liked
Viewers also liked(20)
Similar to AWSの進化とSmartNewsの裏側
Similar to AWSの進化とSmartNewsの裏側(20)
Recently uploaded
Recently uploaded(20)
AWSの進化とSmartNewsの裏側
- 1. Evolution of AWS Backend of SmartNews 2016/01/27 Classmethod x SmartNews AWS Night
- 2. Self Introduction • Nobutoshi Ogata • Manager, Site Reliability Engineering • @nobu666 • ❤ Whiskey, Cat, Heavy Metal • Entrusted dev.(10y) ➡ Game platform infrastructure(3y) ➡ Some startup(1y) ➡ SmartNews(2015/05-)
- 4. SmartNews
- 8. ! Channel!
- 10. ! Channel!
- 12. Questionnaire
- 13. Agenda • Scale • Services • Changes in the configuration • Configuration management • Future plan
- 14. Scale
- 15. Scale • 300+ EC2 instances • 80 ELBs • 70 Auto Scaling Groups • 80+ S3 buckets (300+TB) • 40 RDS instances (MySQL & Aurora) • 100 Kinesis shards • 10 EMR clusters (Hive+Spark)
- 16. Scale • 50+ ElastiCache clusters (Redis + Memcached) • 30+ CodeDeploy applications • 70,000,000 CloudSearch seachable documents • 8 Redshift nodes • 10+ Lambda functions • 80+ SQS queues • 20+ SNS topics
- 19. Services
- 25. Split Crawler
- 26. Iron Age API Indexer Analyzer Crawler
- 27. Split more
- 28. Present day Crawler Nationality detector Language detector Analyzer Facial recognition Indexer Ranking Search Recommend API Gateway Admin tool Validator Proxy Live Push Sexy detector Auth Proxy
- 29. Micro Services?
- 30. • SmartNews's journey into microservices • インフラ専任エンジニアが一人もいないSmartNewsにおける クラウド活用法 • SmartNews Ads • Next session about data platform
- 33. ➡ ➡ • CodeDeploy was used to automate deployment tasks • Operating an AWS ELB • Fetching a new revision • Restarting services with our custom program • e.g. downloading machine learning model data, precompiling assets and deploying Docker containers
- 35. Auto Scaling Group • There is an advantage even if you don't want to autoscaling • Easy to dispose • replace kernel, switch to other OS, ... • Easy A/B testing • change config only specific server • tag:Attributes • Instance Protection
- 36. ASG ! InstanceName • We were stop using InstanceName • Using tag:Group and tag:Env
- 37. Infrastructure as Code • Route53 ➡ winebarrel/roadworker • Security Group ➡ winebarrel/piculet • Elastic IP Addresses ➡ winebarrel/eipmap • Provisioning ➡ itamae-kitchen/itamae • Datadog ➡ airbnb/interferon
- 38. DSL management is not perfect • Complicated automation • ChatOps ➡ change code ➡ git add && git commit && git push • Too much hassle... !
- 39. Run Command • Secure • IAM, CloudTrail • Easy CI integration w/o SSH • be able to use EMR also
- 40. Example aws ssm send-command --document-name "AWS-RunShellScript" --instance-ids $(aws emr list-instances --cluster-id $(aws emr list-clusters --active | jq -r '.Clusters[] | select (.Name=="FOOBAR") | .Id') | jq -r '.Instances[].Ec2InstanceId') --parameters '{"commands":["ls"],"executionTimeout":["10"]}' --timeout-seconds 10 --output-s3-bucket-name "ssm-logs" --output-s3-key-prefix "prefix-" --region ap-northeast-1
- 41. Future plan
- 42. ECS Spot Fleet Contaniner Registry ConsulAWS Blackbelt 2015 ECS p.98
- 46. Events aggregator • Events • Deploy/Provision/Restart/Terminate etc. • All events goes to SNS or Kinesis. Users will describe and hook scripts • Visualize via Datadog • Notice to Slack • Auto healing
- 47. Summary
- 48. Summary• AWS release new service continuously • We ride on the shoulders of giants • We will never forget that OSS helped us
- 49. We're hiring! Only two people on Site Reliability Engineering Team http://about.smartnews.com/en/careers/
- 51. Credit https://flic.kr/p/hAcPRk taken by Niuton may https://flic.kr/p/9UnRWS taken by Alberto G. https://flic.kr/p/ydnEvw taken by ep_jhu https://flic.kr/p/pyfnTV taken by Kevin Dooley https://flic.kr/p/mtLpc4 taken by L1mey https://flic.kr/p/fgG226 taken by Linux Screenshots https://flic.kr/p/7DsuLz taken by Gloson Teh https://flic.kr/p/jfGpn7 taken by julochka