Successfully reported this slideshow.
Your SlideShare is downloading. ×

Another backend storage solution for the APM system

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 22 Ad

More Related Content

Slideshows for you (20)

Similar to Another backend storage solution for the APM system (20)

Advertisement

Recently uploaded (20)

Another backend storage solution for the APM system

  1. 1. Another backend storage solution for the APM system Juan Pan, JD.com, Apache ShardingSphere PMC APACHECON @HOME Spt, 29th – Oct. 1st 2020
  2. 2. As a senior DBA worked at JD.com, the responsibility is to develop the distributed database and middleware, and the automated management platform for database clusters. As a PMC of Apache ShardingSphere, I am willing to contribute to the OS community and explore the area of distributed databases and NewSQL. Juan Pan Senior DBA and PMC of Apache ShardingSphere About me
  3. 3. APM Overview1 2 Backend Storage 目录 CONTENTS 3 Apache ShardingSphere 4 Solution Detail
  4. 4. APM Overview App1 App2 AppN Application hosts … Agent1 Agent2 AgentN Collector DB2 DB N DB1 Storage UI Aggregator Server …
  5. 5. Backend Storage Solutions Elastic Search Cassandra HBase MySQL Memory/H2 Requirements Scalability Simple Query Like SQL Reliability TTL
  6. 6. Requirements Requirements Scalability Simple Query Like SQL Performance Reliability TTL New SQL Scalable SQL Open Source
  7. 7. New SQL A single logical DB New Architecture Transparent Sharding Database as a Service
  8. 8. Apache ShardingSphere Website: https://shardingsphere.apache.org GitHub: https://github.com/apache/shardingsphere
  9. 9. Architecture M1 S2M2 DBCluster Sharding-Proxy Business Machine A Sharding-JDBC Machine B Business Sharding-JDBC Sharding-Scaling MySQL Client PostgreSQL Client S1 Core Sharding Transaction Encryption Shadow Governance OpenTracing TrafficControl OnlineChange Metrics Sharding-UI
  10. 10. Sharding App2 DB App1 App3 App2 DB1 DB2 DB3 App1 App3 S App1 M App2 App3 write sync read App1 S1 App2 App3 M2 sync read S2 S3 M1 M3 write
  11. 11. Sharding App1 M1 App2 App3 S2 M2 M3 S1 S3 Database Middleware write sync read App2 DB App1 App3
  12. 12. ShardingProxy Collector DB2 DB N DB1 Storage Aggregator Server … Collector DB2 DB N DB1 Storage Aggregator Server … ShardingProxy
  13. 13. Use case
  14. 14. Requirements Massive volume Storage Query efficiency SQL query SQL
  15. 15. Requirements More detail: https://github.com/apache/skywalking/issues/4131 Sharding Algorithm Range (One day one sharding table) Sharding Key time_bucket Sharding Table segment
  16. 16. Table sharding DB select count(1) total from segment where time_bucket >= ? and time_bucket <= ? order by latency desc DB0 DB2 DB3 DB9 ShardingProxy App 10 million 1 million … 1 million 1 million1 million
  17. 17. Data Migration DB 10 million DB 0 DB 2 DB 3 DB 9 1 million … 1 million 1 million1 million Sharding-Scaling Data migration SQL Log File Orchestration Sharding-UI
  18. 18. TODO list Analyze applications Review query SQL Discuss sharding strategy Prepare configuration Deploy ShardngProxy Migrate history data Switch to new DB cluster
  19. 19. Comparison Pros • Scalability (Part) • SQL • Performance (VS ?) • Reliability Cons • TTL • SQL limits
  20. 20. Powered by Apache ShardingSphere [1] https://shardingsphere.apache.org/community/cn/users/
  21. 21. https://github.com/apache/shardingsphere https://shardingsphere.apache.org/ mailto: dev@shardingsphere.apache.org Welcome
  22. 22. THANK YOU APACHECON @HOME Spt, 29th – Oct. 1st 2020 Juan Pan, JD.com, Apache ShardingSphere PMC

×