SmartNews has evolved its use of AWS over time from a monolithic application to microservices as its scale increased. It now uses over 300 EC2 instances, 80 ELBs, and many other AWS services. Configuration management has moved from pull-style deploys to using tools like CodeDeploy, Auto Scaling Groups, and infrastructure as code. Future plans include further containerization and event aggregation to improve scalability, safety, and measureability across services.

1. Evolution of AWS
Backend of SmartNews
2016/01/27 Classmethod x SmartNews AWS Night

2. Self Introduction
• Nobutoshi Ogata
• Manager, Site Reliability Engineering
• @nobu666
• ❤ Whiskey, Cat, Heavy Metal
• Entrusted dev.(10y) ➡ Game
platform infrastructure(3y) ➡ Some
startup(1y) ➡ SmartNews(2015/05-)

4. SmartNews

5. 15,000,000+
downloads
worldwide

6. SmartNews 3.0

7. LINE Stamp
shipped today

8. ! Channel!

10. ! Channel!

12. Questionnaire

13. Agenda
• Scale
• Services
• Changes in the configuration
• Configuration management
• Future plan

14. Scale

15. Scale
• 300+ EC2 instances
• 80 ELBs
• 70 Auto Scaling Groups
• 80+ S3 buckets (300+TB)
• 40 RDS instances (MySQL & Aurora)
• 100 Kinesis shards
• 10 EMR clusters (Hive+Spark)

16. Scale
• 50+ ElastiCache clusters (Redis + Memcached)
• 30+ CodeDeploy applications
• 70,000,000 CloudSearch seachable documents
• 8 Redshift nodes
• 10+ Lambda functions
• 80+ SQS queues
• 20+ SNS topics

19. Services

21. Changes in the
configuration

22. Stone Age
Analyzer
Crawler
Indexer
On-memory DB
API

23. Monolithic Application

24. Bronze Age
Analyzer
Indexer
On-memory DB
API
Crawler
Crawler
Crawler
Queue

25. Split Crawler

26. Iron Age
API Indexer Analyzer Crawler

27. Split more

28. Present day
Crawler
Nationality detector
Language detector
Analyzer
Facial recognition
Indexer
Ranking
Search
Recommend
API Gateway
Admin tool Validator
Proxy
Live
Push
Sexy detector
Auth Proxy

29. Micro Services?

30. • SmartNews's journey into microservices
• インフラ専任エンジニアが一人もいないSmartNewsにおける
クラウド活用法
• SmartNews Ads
• Next session about data platform

31. Configuration
management

32. Pull style deploy

33. ➡ ➡
• CodeDeploy was used to automate deployment tasks
• Operating an AWS ELB
• Fetching a new revision
• Restarting services with our custom program
• e.g. downloading machine learning model data,
precompiling assets and deploying Docker containers

35. Auto Scaling Group
• There is an advantage even if you don't want to autoscaling
• Easy to dispose
• replace kernel, switch to other OS, ...
• Easy A/B testing
• change config only specific server
• tag:Attributes
• Instance Protection

36. ASG ! InstanceName
• We were stop using InstanceName
• Using tag:Group and tag:Env

37. Infrastructure as Code
• Route53 ➡ winebarrel/roadworker
• Security Group ➡ winebarrel/piculet
• Elastic IP Addresses ➡ winebarrel/eipmap
• Provisioning ➡ itamae-kitchen/itamae
• Datadog ➡ airbnb/interferon

38. DSL management
is not perfect
• Complicated automation
• ChatOps ➡ change code ➡ git add && git commit && git
push
• Too much hassle... !

39. Run Command
• Secure
• IAM, CloudTrail
• Easy CI integration w/o SSH
• be able to use EMR also

40. Example
aws ssm send-command
--document-name "AWS-RunShellScript"
--instance-ids
$(aws emr list-instances
--cluster-id
$(aws emr list-clusters --active |
jq -r '.Clusters[] | select (.Name=="FOOBAR") | .Id') |
jq -r '.Instances[].Ec2InstanceId')
--parameters '{"commands":["ls"],"executionTimeout":["10"]}'
--timeout-seconds 10
--output-s3-bucket-name "ssm-logs"
--output-s3-key-prefix "prefix-"
--region ap-northeast-1

41. Future plan

42. ECS
Spot Fleet
Contaniner Registry
ConsulAWS Blackbelt 2015 ECS p.98

43. Containerize
all services

44. Scalable
Safety
Efficient
Portable
Measurable

45. Aggregates events

46. Events aggregator
• Events
• Deploy/Provision/Restart/Terminate etc.
• All events goes to SNS or Kinesis. Users will describe and
hook scripts
• Visualize via Datadog
• Notice to Slack
• Auto healing

47. Summary

48. Summary• AWS release new service continuously
• We ride on the shoulders of giants
• We will never forget that OSS helped us

49. We're hiring!
Only two people on
Site Reliability Engineering Team
http://about.smartnews.com/en/careers/

51. Credit
https://flic.kr/p/hAcPRk taken by Niuton may
https://flic.kr/p/9UnRWS taken by Alberto G.
https://flic.kr/p/ydnEvw taken by ep_jhu
https://flic.kr/p/pyfnTV taken by Kevin Dooley
https://flic.kr/p/mtLpc4 taken by L1mey
https://flic.kr/p/fgG226 taken by Linux Screenshots
https://flic.kr/p/7DsuLz taken by Gloson Teh
https://flic.kr/p/jfGpn7 taken by julochka