This is a slide from Pan Juan, ShardingSphere PMC which presented in ApacheCon.

1. Another backend
storage solution
for the APM system
Juan Pan, JD.com,
Apache ShardingSphere PMC
APACHECON @HOME
Spt, 29th – Oct. 1st 2020

2. As a senior DBA worked at JD.com, the responsibility is to develop the distributed
database and middleware, and the automated management platform for database
clusters.
As a PMC of Apache ShardingSphere, I am willing to contribute to the OS community and
explore the area of distributed databases and NewSQL.
Juan Pan
Senior DBA and PMC of Apache ShardingSphere
About me

3. APM Overview1
2 Backend Storage
目录
CONTENTS 3 Apache ShardingSphere
4 Solution Detail

4. APM Overview
App1 App2 AppN
Application hosts
…
Agent1 Agent2 AgentN
Collector
DB2
DB
N
DB1
Storage
UI
Aggregator
Server
…

5. Backend Storage
Solutions
Elastic Search
Cassandra
HBase
MySQL
Memory/H2
Requirements
Scalability
Simple Query
Like SQL
Reliability
TTL

6. Requirements
Requirements
Scalability
Simple Query
Like SQL
Performance
Reliability
TTL
New
SQL
Scalable
SQL
Open
Source

7. New SQL
A single logical
DB
New
Architecture
Transparent
Sharding
Database as a
Service

8. Apache ShardingSphere
Website: https://shardingsphere.apache.org
GitHub: https://github.com/apache/shardingsphere

9. Architecture
M1 S2M2
DBCluster
Sharding-Proxy
Business
Machine A
Sharding-JDBC
Machine B
Business
Sharding-JDBC
Sharding-Scaling
MySQL Client
PostgreSQL Client
S1
Core
Sharding
Transaction
Encryption
Shadow
Governance
OpenTracing
TrafficControl
OnlineChange
Metrics
Sharding-UI

10. Sharding
App2
DB
App1 App3
App2
DB1 DB2 DB3
App1 App3
S
App1
M
App2 App3
write
sync
read
App1
S1
App2 App3
M2
sync
read
S2 S3
M1 M3
write

11. Sharding
App1
M1
App2 App3
S2
M2 M3
S1 S3
Database Middleware
write
sync
read
App2
DB
App1 App3

12. ShardingProxy
Collector
DB2
DB
N
DB1
Storage
Aggregator
Server
…
Collector
DB2
DB
N
DB1
Storage
Aggregator
Server
…
ShardingProxy

13. Use case

14. Requirements
Massive
volume
Storage
Query
efficiency
SQL query
SQL

15. Requirements
More detail: https://github.com/apache/skywalking/issues/4131
Sharding Algorithm
Range (One day one sharding table)
Sharding Key
time_bucket
Sharding Table
segment

16. Table sharding
DB
select count(1) total from segment
where time_bucket >= ? and time_bucket <= ?
order by latency desc
DB0 DB2 DB3 DB9
ShardingProxy
App
10 million
1 million
…
1 million 1 million1 million

17. Data Migration
DB
10 million
DB
0
DB
2
DB
3
DB
9
1 million
…
1 million 1 million1 million
Sharding-Scaling
Data migration
SQL
Log File
Orchestration
Sharding-UI

18. TODO list
Analyze applications
Review query SQL
Discuss sharding strategy
Prepare configuration
Deploy ShardngProxy
Migrate history data
Switch to new DB cluster

19. Comparison
Pros
• Scalability (Part)
• SQL
• Performance (VS ？)
• Reliability
Cons
• TTL
• SQL limits

20. Powered by Apache ShardingSphere
[1] https://shardingsphere.apache.org/community/cn/users/

21. https://github.com/apache/shardingsphere
https://shardingsphere.apache.org/
mailto: dev@shardingsphere.apache.org
Welcome

22. THANK YOU
APACHECON @HOME
Spt, 29th – Oct. 1st 2020
Juan Pan, JD.com,
Apache ShardingSphere PMC