ID Platform Product can be used by every Rakuten Group Companies and can easily serve millions of users. Multi-Region product challenges are many, example: - Ensure 4 9’s availability - Management across each region - Alerting and Monitoring across each region - Auto scaling (Scale up and Scale down) across each region - Performance (vertical scale up) - Cost - DB Consistency Across Multiple Regions - Resiliency At Ecosystem Platform Layer for Rakuten, we handle each of these and this presentation is about how we handle these challenging scenarios.

1. October 27, 2018
Vinaya Thimmappa
Eco-System Services
Department.
Rakuten, Inc.

2. 2

3. 3
• Businesses in single region
• Single Data Center
• Application running on VM’s
• Single Enterprise Database
• No multi-tenancy
Picture Credit : http://capitalsource.us

4. 4
• Businesses multiple region
• Multiple & Hybrid Data Center
• Dockerized application
• Database in multiregion
• Multi-tenancy

5. 5
Cloud Native
• High Availability (99.99%)
• Dynamically managed
• Containerized
• Platform agnostic
• Write once and deploy across regions
• Reduce operating cost
• Easy maintenance.
• Multi-tenancy
• Elastic scaling
Cloud Enabled
• Originally developed for deployment in a
traditional data center
• An application that was moved to cloud,
• Some characteristics of the application had
to be changed or customized for the cloud

6. 6
Latency of services
The picture can't be displayed.
Multi region deployment [
CI/CD ]
The picture can't be displayed.
Database Synchronization
The picture can't be displayed.
Observability
The picture can't be displayed.
Resiliency

7. 7
Challenges Opportunities

8. 8
Cloud service latency is the delay between a client request and a cloud service provider’s
response.
• Higher application response time
• Application SLA gets impacted
• Delay in synchronization b/w data
center
• Delay in arrival of help from other
services
Challenges
• Go closer to customer
• Install application in datacenter with
less latency b/w regions
• Avoid multiple round trips to
services in application
• Define clear SLA from various
services and its support
Solutions

9. 9
Automated mechanisms for deployment and rollback
The
pictu
re
can't
be
displ
aye…
The
pictu
re
can't
be
displ
aye…
The
pictur
e
can't
be
displa
yed.
Challenges
• Deploying in all region at the same
time
• Maintenance of code for each region
is tedious
• One release version for all regions
• Understand region specific business
peak hours
• Build region specific deployment
pipeline
• Have region specific flags for region
specific functionality
Solutions

10. 10
High performance, Active-active data replication, High availability of data, Cost-
effective , Consistency Challenges
• Muti-Region Sync brings down
QPS
• One region DB alter may have
impact other regions
• Consistentence across region
result have performance impact
• Tweak database parameters for
better performance
• Eventual Consistency for multi-
region datacenter
• Consistent in the same datacenter
Solutions

11. 11
Ability of a system to recover from failures and continue to function and
responding to failures in a way that avoids downtime or data loss
Challenges
• Transient failures
• Application Issues
• Catastrophic failures
• Retrying for idempotent operations
• Circuit breaker pattern
• Handle failures gracefully
• Prevent Cascading failures
• Global Load Balancers
• Graceful degradation
Solutions

12. 12
Monitoring, Alerting ,Logging , Analytics and Metrics collection
Monitoring
• Infrastructure monitoring
• Application monitoring
• Real-user monitoring
Metrics & Alerting
• Host-Based Metrics
• Application Metrics
• Network and Connectivity
Metrics
• Server Pool Metrics
• External Dependency Metrics
• Able to Correlate Factors from
Different Sources

13. 13
LATENCY
of services
Closer
App
Installation
Multi Region
CI/CD
Assess
Build
Database
Sync
Control
Optimize
Observability
Add
Audit
Resiliency
Identify
Configure
PlanAvoid
The picture can't be displayed.
Support
Ensure
Learn

14. Vinaya Thimmappa
Rakuten India Enterprise Private Limited, Bangalore, India
Vinaya.thimmappa@rakuten.com