1. Service Platform Architect
Brandon Kang
sangjinn@gmail.com
https://tech.brandonkang.net
May 2020
Scalability Strategies for
Cloud based System Architecture

2. Agenda
• Scalability & Availability for the Global Markets
• Global scaled Scalability, Availability and Security
• Architecture for 100, 1K, 100K, 500K, 1M and 10M global users
• Auto-Scaling
• Understand Cloud Services
• Cloud Demo(AWS, GCP, Azure and Cloudflare)
• Wrap-Up

3. - Scalability -

4. Scalability
• Scalability = capability of a system to handle a growing work
• Vertical : Scale Up or Down
ü Add or Remove Resources
ü CPU
ü Memory
ü Storage
• Horizontal: Scale Out or In
ü Add or Remove Systems
ü Instance Scale OutScale In
VM VM VM VMVM VMVM VM
Scale Down
Scale Up
VMVM

5. Scalability ≠ Availability

6. Scalability vs. Availability
• Need 4 * VMs to provide services
Scalability: (2* VMs in a region) + (2* VMs in another region)
Availability: (4* VMs in a region) + (4* VMs in another region) for HA

7. 1 User
Network
Fixed IP
Application
Database

8. Users < 100
Network
Fixed IP
Application
Database

9. Users > 1,000
Master
Slave
Load Balancer
Zone A
Zone B
Write
Write
Read
Replication
Region

10. Users > 100,000 M
R
R
S
R
R
Active/
Write
Read
Replica
Read
Replica
Read
Replica
Read
Replica
Stand-by/
Write

11. Users > 100,000 M
R
R
S
R
R
Object
Storage
Object
Storage
www.example.com
api.example.com
…
CDN
static.example.com
image.example.com
…

12. Users > 500,000
M
R
R
S
R
R
Object
Storage
Object
Storage
CDN
static.example.com
image.example.com
www.example.com
… DB
Caching
DB
Caching
API
Gateway
Service
Micro-Services
Architecture
api.example.com

13. Users > 500,000
• MSA(Microservices Architecture)
ü Every functions move to Microservices
ü Independent and loosely coupled
• API Gateway
ü API Routing
ü API Security
ü Authentication
ü Authorization
ü API Caching
ü Hits Rate Limit
ü Hits Throttling
ü Traffic Monitoring

14. Users > 1M
M
R
R
S
R
R
Object
Storage
Object
Storage
CDN
static.example.com
image.example.com
www.example.com
… DB
Caching
DB
Caching
API
Gateway
Service
api.example.com
Global
Queue
No
SQL
No
SQL
Auto
Scaling

15. Actions for 1M Users
• What data should be move to NoSQL from RDBMS?
ü Temporary but massive data e.g.) click information, log data, session data
ü Hot tables (tables who have very frequent updates)
ü Meta data store and search e.g.) object information from storage
ü Dynamic schema
ü Time-series table e.g.) monitoring logs

16. Users > 10M
M
R
R
S
R
R
Object
Storag
e
Object
Storag
e
CDN
static.example.com
image.example.com
www.example.com
…
DB
Cachin
g
DB
Cachin
g
API
Gateway
Service
api.example.com
No
SQL
No
SQL
Auto
Scaling
DB
Federation
+
DB Sharding
M
M
M
User
Message
Forum
Id=1~3
Id=4~6
Id=7~9
Auto
Scaling
Auto
Scaling
Global
Queue
API
Gateway
API
Gateway
CDN
Serverless
Functions
y = f(x) + α

17. Actions for 10M Users
• Multi Zone architecture in every region
• Consider expansion for all components e.g.) Multi-Master DB
• Maximize caching utilization from architecture and network
• Make autonomous systems
• Build auto-scaled systems using Cloud or Kubernetes
ü Virtual machine
ü Load balancer
ü Object storage
ü Notification system
ü Queuing system,
ü Workflows like Emailing, Alarming, etc.

18. Actions for 10M Users
• Entire Architecture tuning
• Application level tuning
• Globally distributed systems
• Global HA architecture (Multi Zone -> Multi Region)
• DevOps based application deployment
• CI(Continuous Integration)& CD(Continuous Development)

19. - Auto Scaling -

20. Actions for 10M Users
• What is Autoscaling?

21. Actions for 10M Users
• Autoscaling in On-Premise
desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]
* RC
Replication Controller
* Pod
A Group of containers

22. Actions for 10M Users
• Autoscaling in Cloud

23. Actions for 10M Users
• Autoscaling in Cloud

24. - Cloud -

25. Cloud Pros
• Fully Managed Services
• Flexible Scale Up/Down and Scale Out/In
• Auto Scaling
• HA using Multi Availability Zone and Multi Regions
• Performance
• Unlimited Capacity
• Serverless Architecture
• KMS(Key Management System)
• Computing, Network Big Data, IoT, AI, Machine Learning, etc.
• Low CAPEX/OPEX
• Compliance and Regulation e.g.) GDPR, CCPA

26. Cloud Cons
• Not Cheap and Unexpected Cost
• Not easy to migrate from On-Premise
• Sensitive Security
• No regions in Vietnam yet
• Vendor Stickiness?
• 99.999999999% Availability e.g.)AWS S3
• Support from outside vendors
• Need Experience and Training

27. AWS(Amazon Web Services) Regions

28. AWS Edge Locations ­ CDN, Serverless

29. AWS Services

30. AWS(Amazon Web Services)

31. AWS Basic Networking
a Internet
Gateway
Virtual
Private
Gateway
Router
Route
Table
Route
Table
Network
ACL
Network
ACL
Public Subnet (10.0.1.0/24)
Security GroupInstance
Private Subnet (10.0.2.0/24)
Security GroupInstance
VPC (MY_VPC_NAME) ­ 10.0.0.0/16
REGION (us-east-1)
NAT
Gateway

32. GCP(Google Cloud Platform)

33. Microsoft Azure

34. Cloudflare (CDN, WAF, DDoS, Bot, VPN, DNS)

35. - Cloud Demo ­
Demo for 20minutes with these implementation
1. Video Conference(WebRTC) platform using GCP
2. Testing PostgreSQL Multi-Master using Azure
3. AWS Chatbot usages for CloudWatch and Serverless Lambda
4. CDN, DNS and WAF integration using Cloudflare

36. - Wrap Up -

37. Best Practices for Global Markets
• Multi Regional Data Center (or Cloud)
• CDN(Content Delivery Network)
• Prepare Attacks -> WAF, DDoS Scrubbing, Bot Managing
• Server -> Instance in Cloud -> Serverless/Kubernetes
• Maximize Queuing
• Authentication out of Server
• DB -> Master/Slave, Primary/Secondary
• N/W Caching, DB Caching, Client Caching for Performance!
• Manual and routine jobs -> Automatic Workflows
• Microservices Architecture, DevOps and CI/CD
• Global Regulations Awareness
• Smart small, Grow Bigger!

38. Best Practices for Global Markets
Auto-Scale Everything,
Cache Everything,
And Protect Everything!

39. - Thank You. -
Service Platform Architect
Brandon Kang
sangjinn@gmail.com
https://tech.brandonkang.net