Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Inside CynosDB: MariaDB optimized for the cloud at Tencent

115 views

Published on

Qinglin Zhang, Database Kernel Engineer at Tencent, introduces CynosDB, Tencent's self-developed database for the cloud. CynosDB is based on MariaDB Server, but separates computing and storage. Zhang goes on to provide a detailed explanation of the architecture with a focus on how Tencent implemented the computing and storage layers, and created Tencent’s MariaDB-based “Aurora”.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Inside CynosDB: MariaDB optimized for the cloud at Tencent

  1. 1. Qinglin Zhang TXSQL Team @ Tencent CynosDB @ Tencent Cloud guyue.zql@gmail.com
  2. 2. Preview Tencent Tencent Cloud TXSQL CDB
  3. 3. Agenda • Why CynosDB ? • Architecture • TXStore for CynosDB • TXSQL for CynosDB • Future of CynosDB
  4. 4. Why CynosDB?
  5. 5. User Challenges: ✓ Scalability: fast scale up/down, fast scalable read ✓ Cost: pay on-demand ✓ Replication: binlog, DDL/big transaction ✓ BufferPool: read & write cache, the disadvantage of flushing Traditional MySQL Challenges:
  6. 6. The Architecture of CynosDB
  7. 7. ✓ Separate storage from MySQL ✓ Log is database, write once Architecture slave TXSQL master TXSQL slave TXSQL …CELL CELL CELLCELL TXStore MDS redo
  8. 8. TXStore for CynosDB
  9. 9. TXStore core for CynosDB ✓ Replication protocol ✓ Computational storage ✓ Multi version page
  10. 10. MySQL Master SQL Layer Engine Layer ①send log record LogPool RingQueue MemTable InnoDB Page ③ack ②Replicate log record Primary Storage Node Replica Storage Node Replica Storage Node ④push to memtable ⑤Apply logs to InnoDB Page sync async TXStore core details Disk ②flush to disk
  11. 11. TXStore performance for CynosDB ✓ SPDK(user space api) ✓ RDMA ✓ Reconstruct redo log ✓ … ✓ Jemalloc integrated
  12. 12. TXSQL for CynosDB
  13. 13. The Brief of TXSQL ✓ Tencent MySQL(Compatible with Oracle MySQL5.5/5.6/5.7) ✓ Support Tencent Cloud (TencentDB for CDB) ✓ The Component of TXSQL TXSQL for MySQL TXSQL for NewCDB TXSQL for RocksDB TXSQL CynosDB TXRocks TXTools Backup/Recovery/Checksum
  14. 14. TXSQL core for CynosDB ✓ Crash Recovery ✓ Remote Storage based on TXStore ✓ System tables converted to Innodb ✓ Redo log Replication ✓ Redo for bulk load create index ✓ Parallel redo log copy optimization ✓ MVCC
  15. 15. Parallel redo log copy optimization mtr_commitTrx_1 Trx_2 Trx_m mtr_commit Reduce log_sys->mutex occupancy time to improve perforemance mtr_commit mutex_enter calc lsn mutex_exit copy redo log mutex_enter calc lsn mutex_exit copy redo log mutex_enter calc lsn mutex_exit copy redo log
  16. 16. Parallel redo log replication mtr_commit mtr_commit …… mtr_commit log_sys->write_buf log_write_thread redo_log_receiver_thread redo_log_coor_thread redo_log_worker_thread redo_log_worker_thread redo_log_worker_thread
  17. 17. TXSQL performance for CynosDB ✓ Redo Log optimization for flush ✓ Asyc Thread Pool Commit ✓ Trx sys optimization ✓ … ✓ Jemalloc integrated ✓ Async DDL Replication
  18. 18. Async Thread Pool Commit listener thread (pollfd) ... wroker threadwroker thread high_prio_queue queue handle_event get_event handle_event get_event clientclient …… client thread_group_oversubscribe log_write_up_to log_write_up_to ✓ Sync wait converted to async ✓ Enlarge high-priority-queue ✓ Thread pool & one-thread-per-connection
  19. 19. Async DDL Replication Async DDL Support in CynosDB Frm files stored in Innodb Multi page version from TXStore Synchronized by redo log
  20. 20. master slave read old Table begin end DDL start DDL end read new Table table was locked Asynchronous DDL cost most time of DDL,replica can read cost little time of DDL,master can't read
  21. 21. TXSQL feature for CynosDB ✓ Aggregate pushdown to InnoDB ✓ Enterprise Encryption ✓ INVISIBLE INDEX ✓ Instant DDL Operation ✓ Asynchronous DROP TABLE ✓ …
  22. 22. Instant DDL Operation ✓ Add column at designed order ✓ Change column order ✓ Change default value
  23. 23. Future of CynosDB ✓ CDBTune(Intelligent tuning) ✓ Multi Master ✓ … ✓ PITR

×