This document discusses the good and hard things about developing and operating a SaaS platform. It describes how the backend team at Treasure Data owns and manages various components of their distributed platform. It also discusses how they have modernized their deployment process from a periodic Chef-based approach to using CodeDeploy for more frequent deployments. This allows them to move faster by doing many small releases that minimize the number of affected components and customers.

1. © 2019 Arm Limited
Satoshi Tagomori (@tagomoris)
Principal Software Engineer, Arm Treasure Data
クラウド活用のアーキテクチャとDevOps事例勉強会
Good Things and Hard Things
of SaaS
Development/Operations

2. © 2019 Arm Limited2
Satoshi Tagomori (@tagomoris)
• Treasure Data: 2015~
• Arm Treasure Data (2018~)
• Current: Backend Team
• OSS: Fluentd, MessagePack-Ruby, Woothee, Norikra, ...
• ISUCON
• 2011~ (livedoor, NHN Japan, LINE)

3. © 2019 Arm Limited3

4. © 2019 Arm Limited4
What's DevOps?
• Collaboration between Devs and Ops
• "Development" and "System Operations"
• For faster development, deployment, release cycles
• For quicker improvements
• Performance
• Business Values

5. © 2019 Arm Limited5
Treasure Data Platform
• Distributed Systems
• Distributed Database Systems
• Distributed Data Processing Systems
• Job Queue & Workers
• API Endpoints
• Data Transferring, Conversion
• ...
• CDP (Customer Data Platform) Application
• on the top of the platform
• our application, as a Marketing platform for customers
Treasure Data Platform
Treasure Data CDP
Customers' Marketing
Businesses, Apps
Customers'
Data
Analytics
Workloads

6. © 2019 Arm Limited6
Engineering Teams: Own, Build and Operate Systems
Teams own components
• Design
• Development
• Configuration
• Testing
• Deployment
• Release
• Monitoring
• Alert
• Operation
• On-call!
SRE provides common features
• OS base images
• System-wide configurations
• Shared tools (chatbot, etc)
• ...

7. © 2019 Arm Limited7
DevOps... ?
A team own everything around a component
including Monitoring, Operations, On-call, ...
🤔

8. © 2019 Arm Limited8
Developers do Operations
Nothing like "Devs vs Ops"!
😆

9. © 2019 Arm Limited9
THE END ... ?
No, No, Wait, WE HAVE PROBLEMS 👻

10. © 2019 Arm Limited10
Distributed Systems by Many Teams
Complexity over Teams
Component Dependencies
• Components depend on each other
• Query engine <-> Database
• Data ingestion <-> Database
• Worker <-> Query engine
• API <-> Worker
• ...
• Feature dependencies between
components
Data Compatibilities
• Write data, Read data *later*
• Data ingestion write data
• Database subsystem read data
• Query engine read data
• ...
• Incompatible data cause crashes *later*

11. © 2019 Arm Limited11
Various Backend Team Components
Treasure Data Platform
Backend Team Components
Plazma
(Distributed Database)
Data Ingestion
Workers
Workflow
API
Hadoop
Presto
Data Connector
CDP
Frontend
Our Customers

12. © 2019 Arm Limited
• Historic components have
stories...
• Chef-based deployments
• Old-style configurations
• Old JVMs
• Unorganized AWS
components
• Unorganized monitoring/
alerts
Long History
12
Hard Things: Backend Components
with Long History, Many Components and Severe Uptime Requirement
• Backend was "not Frontend"
in past
• Different components
• Distributed database
• Data ingestion APIs
• Data ingestion workers
• Job workers
• Workflow manager
• Many ways to do things
Many Components
• Database, Workers
• Referred by all other
components
• Downtime means entire
service downtime
• Data Ingestions
• Downtime means user-side
data loss
Uptime Requirement

13. © 2019 Arm Limited13
The Thing Makes Us Slower
Outdated Systems
Poor System
Improvements
Bad Stability
High Operation Cost
Low Business Income
Low QoL
😱
😨 😰

14. © 2019 Arm Limited
Quick
Delivery
Feedbacks
From Customers
& Teams
Frequent
Deployment
14
Move Faster
Modernized
Deployment
😀

15. © 2019 Arm Limited15
Modernizing Deployment
Old-style (Chef) vs Modern-style (CodeDeploy)
Chef Deployed Systems Periodically
• Developer pushed "production" branch
• Release: Merge into "production"
• Chef pulls "production" branch
• On EC2 instances
• Once per 30 minutes
• Then, chef recipe does:
• Build, Install dependencies
• Restart processes (if needed)
Developers Kick CodeDeploy
• Service repo kicks CircleCI
• CI creates CodeDeploy packages on S3
• Developers kick "deploy" or "release"
• On Slack
• To trigger CodeDeploy
• CodeDeploy fetches packages from S3
• Build, Install, Restarts, etc
👤 👤

16. © 2019 Arm Limited16
Many, Quick, Frequent Deployments & Releases
Split Release into Small Releases
Minimize Affected Components
• Giant release affects many components
• Hard to say:
"it's safe for all components"
• Many small releases
• Easy to say:
"it's safe for THIS component!"
Minimize Affected Customers
• Many customers do various things on our
platform
• Release MAY affect their workloads
• Query compatibility
• Query performance
• Invalid data handling
• Data ingestion request patterns
• "A customer says something goes wrong"
• "What did happen in this 1 week?"
• Make things clear for support operations

17. © 2019 Arm Limited17
Many, Quick, Frequent Deployments & Releases
Split Release into Small Releases
Minimize Affected Components
• Giant release affects many components
• Hard to say:
"it's safe for all components"
• Many small releases
• Easy to say:
"it's safe for THIS component!"
Minimize Affected Customers
• Many customers do various things on our
platform
• Release MAY affect their workloads
• Query compatibility
• Query performance
• Invalid data handling
• Data ingestion request patterns
• "A customer says something goes wrong"
• "What did happen in this 1 week?"
• Make things clear for support operations

18. © 2019 Arm Limited18
Revisiting: What's DevOps?
• Collaboration between Devs and Ops
• "Development" and "System Operations"
• Is It Only for System Operations?
• "Operation" means not only system operations in many cases
(For example, Chief "Operation" Officer)
• We need to support many type of operations:
• Support Operations
• Sales Operations
• Audit Operations

19. © 2019 Arm Limited19
Audit
• Standards Compliance
• ISO/IEC 27001:2013
• SOC-2 (Service Organization Controls) type 2
https://www.treasuredata.co.jp/security/

20. © 2019 Arm Limited20
Executing Infra Operations in Explicit Way
Terraform Enterprise
• AWS infra
by code
• Run code on
Terraform
Enterprise
workspaces
with history

21. © 2019 Arm Limited21
Executing Release Operations in Explicit Way
Slack, Jira, Github and Automation
• Trying release
automation
• Communication
Central - Slack
• History Central -
GitHub
• Release Central
- Jira

22. © 2019 Arm Limited22
Own Your Service Only, Use Cloud Services
Your service is heavy enough
Own Your Service Only
• Many things to do about your service
• Design
• Development
• Configuration
• Testing
• Deployment
• Release
• Monitoring
• Alert
• Operation
• On-call
Use Cloud Services
• No additional spaces
to own additional things
• Cloud services are to help your work
• Be Agile, Move Faster!

23. Thank You
Danke
Merci
谢谢
ありがとう
Gracias
Kiitos
감사합니다
धन्यवाद
‫ا‬ً‫شكر‬
‫תודה‬© 2019 Arm Limited