SlideShare a Scribd company logo
1 of 24
Download to read offline
Unleashing Data At Scale With TiDB
Sep 09th, 2023
LSPE - 48
Kabilesh PR
Co-Founder, Mydbops
Interested in Open Source technologies
Pingcap Certified TiDB Associate
Tech Speaker/Blogger
AWS certified Database speciality
AWS community builder / SME
Co-Founder, Mydbops
Kabilesh PR
About Me
Mydbops
at a
glance
300+
Clients
3000+
Servers
ISO & PCI DSS
Certified
7 Yrs of
Expertise
AWS Advanced
Consulting Partner
TiDB PingCap
Partner
Mydbops Services
Consulting
Services
Managed
Services
MySQL, MongoDB, PostgreSQL, TiDB & Cassandra
Targeted
Engagement
24 x 7
DBA Team
Agenda
What is TiDB ?
Architecture
Production Readiness
Sharding
What is TiDB ?
An Open Source distributed SQL database by PingCAP
•
Scalable in both ways
•
Supports both Transactional and Analytical (HTAP)
•
Compatible with MySQL protocol
•
Highly available
•
Fault Tolerant
•
MySQL scalability - Bottlenecks
Monolithic - Single node Limitation
Horizontal scaling - Sharding
Architecture
TiDB cluster - SQL layer
Tikv - Storage layer
Placement Driver(PD) - Orchestrator
TiFlash - Analytical columnar storage
Architecture
TiDB - SQL Interface
Stateless
•
Handles Client connections & authentication
•
Parsing & compiling
•
Coverting relational to KV pair (Tikv)
•
Executing SQL
•
Online DDL execution
•
Garbage collections
•
Cache table
•
TiKV - Transactional Storage
Stateful
•
Persistent storage with RocksDB
•
Distributed storage & transaction support
•
Strong Consistency
•
MVCC support
•
Inbuilt HA
•
Co-Processor (PP)
•
TiKV - Regions and Raft
Region - Represents a range of key-value data
•
96 Mb in size
•
A region is replicated to multiple nodes
•
Regions are automatically balanced across TiKV nodes
•
Each region has a leader(1) and followers(2)
•
Leader and Followers form Raft group
•
Leader handles RW ops
•
Followers Participate in Election & Data Reads
•
Services (LB)
Stateful Sets
Operator Maturity
TiKV - Regions and Raft
Placement Driver - PD
Metadata management - Global view
•
Data placement
•
load balancing
•
TransactionID allocation
•
Global Timestamp (TSO)
•
Scheduling by collecting cluster info
•
PD consists of 3 node at-least
•
TiFlash - Analytical Store
Columnar storage
Replicated Async from Tikv
Intelligent engine selection
Optimised for aggregation, join & table scans
MPP (Multiple Parallel Processing)
Detailed Architecture
Sharding
Supports native sharding & distribution
Relational to Key-value transformation
Distributed based on Global key
Global_key = Primary_key + Table_id(generated)
For Non-int primary key row_id is generated
Sharding - Relational to KV
Sharding - Global-key
Sharding - Region split & distribution
Use-Cases
https://www.pingcap.com/customers/
SAAS Application
IOT data , Time series
Ecommerce
Logistics
Gaming
Fintech
Production Readiness
Wire Compatible with MySQL 5.7
UTF8MB4 by default with Binary Collation
Profile your workload with TiDB
Support for CDC with Ti-Binlog
Monitoring integration with Grafana + Prometheus + Alert manager
Full + Incremental , Logical Backup with PITR supported
Limitations
Stored procedures & Functions
Triggers
Events
User-defined functions
Fulltext and Spatial index
Column-level privileges
Reach Us : Info@mydbops.com
Thank You
Image Courtesy : PingCAP

More Related Content

Similar to Data-at-scale-with-TIDB Mydbops Co-Founder Kabilesh PR at LSPE Event

Similar to Data-at-scale-with-TIDB Mydbops Co-Founder Kabilesh PR at LSPE Event (20)

IBM Internet-of-Things architecture and capabilities
IBM Internet-of-Things architecture and capabilitiesIBM Internet-of-Things architecture and capabilities
IBM Internet-of-Things architecture and capabilities
 
NoSQL – Data Center Centric Application Enablement
NoSQL – Data Center Centric Application EnablementNoSQL – Data Center Centric Application Enablement
NoSQL – Data Center Centric Application Enablement
 
Introducing TiDB Operator [Cologne, Germany]
Introducing TiDB Operator [Cologne, Germany]Introducing TiDB Operator [Cologne, Germany]
Introducing TiDB Operator [Cologne, Germany]
 
MySQL NDB Cluster 101
MySQL NDB Cluster 101MySQL NDB Cluster 101
MySQL NDB Cluster 101
 
When Open Source Meets the Enterprise
When Open Source Meets the EnterpriseWhen Open Source Meets the Enterprise
When Open Source Meets the Enterprise
 
MyHeritage backend group - build to scale
MyHeritage backend group - build to scaleMyHeritage backend group - build to scale
MyHeritage backend group - build to scale
 
IBM IoT Architecture and Capabilities at the Edge and Cloud
IBM IoT Architecture and Capabilities at the Edge and Cloud IBM IoT Architecture and Capabilities at the Edge and Cloud
IBM IoT Architecture and Capabilities at the Edge and Cloud
 
MariaDB ColumnStore
MariaDB ColumnStoreMariaDB ColumnStore
MariaDB ColumnStore
 
MYSQL
MYSQLMYSQL
MYSQL
 
Aerospike Hybrid Memory Architecture
Aerospike Hybrid Memory ArchitectureAerospike Hybrid Memory Architecture
Aerospike Hybrid Memory Architecture
 
Introduction to ClustrixDB
Introduction to ClustrixDBIntroduction to ClustrixDB
Introduction to ClustrixDB
 
Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...
Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...
Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...
 
(DAT204) NoSQL? No Worries: Build Scalable Apps on AWS NoSQL Services
(DAT204) NoSQL? No Worries: Build Scalable Apps on AWS NoSQL Services(DAT204) NoSQL? No Worries: Build Scalable Apps on AWS NoSQL Services
(DAT204) NoSQL? No Worries: Build Scalable Apps on AWS NoSQL Services
 
Keynote: Open Source für den geschäftskritischen Einsatz
Keynote: Open Source für den geschäftskritischen EinsatzKeynote: Open Source für den geschäftskritischen Einsatz
Keynote: Open Source für den geschäftskritischen Einsatz
 
Migración desde BBDD propietarias a MariaDB
Migración desde BBDD propietarias a MariaDBMigración desde BBDD propietarias a MariaDB
Migración desde BBDD propietarias a MariaDB
 
Kylin and Druid Presentation
Kylin and Druid PresentationKylin and Druid Presentation
Kylin and Druid Presentation
 
Using Cloud Automation Technologies to Deliver an Enterprise Data Fabric
Using Cloud Automation Technologies to Deliver an Enterprise Data FabricUsing Cloud Automation Technologies to Deliver an Enterprise Data Fabric
Using Cloud Automation Technologies to Deliver an Enterprise Data Fabric
 
TiDB Introduction - Boston MySQL Meetup Group
TiDB Introduction - Boston MySQL Meetup GroupTiDB Introduction - Boston MySQL Meetup Group
TiDB Introduction - Boston MySQL Meetup Group
 
TiDB Introduction
TiDB IntroductionTiDB Introduction
TiDB Introduction
 
TiDB as an HTAP Database
TiDB as an HTAP DatabaseTiDB as an HTAP Database
TiDB as an HTAP Database
 

More from Mydbops

Efficient MySQL Indexing and what's new in MySQL Explain
Efficient MySQL Indexing and what's new in MySQL ExplainEfficient MySQL Indexing and what's new in MySQL Explain
Efficient MySQL Indexing and what's new in MySQL Explain
Mydbops
 

More from Mydbops (20)

Efficient MySQL Indexing and what's new in MySQL Explain
Efficient MySQL Indexing and what's new in MySQL ExplainEfficient MySQL Indexing and what's new in MySQL Explain
Efficient MySQL Indexing and what's new in MySQL Explain
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
PostgreSQL Schema Changes with pg-osc - Mydbops @ PGConf India 2024
PostgreSQL Schema Changes with pg-osc - Mydbops @ PGConf India 2024PostgreSQL Schema Changes with pg-osc - Mydbops @ PGConf India 2024
PostgreSQL Schema Changes with pg-osc - Mydbops @ PGConf India 2024
 
Mastering Aurora PostgreSQL Clusters for Disaster Recovery
Mastering Aurora PostgreSQL Clusters for Disaster RecoveryMastering Aurora PostgreSQL Clusters for Disaster Recovery
Mastering Aurora PostgreSQL Clusters for Disaster Recovery
 
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...
 
AWS RDS in MySQL 2023 Vinoth Kanna @ Mydbops OpenSource Database Meetup 15
AWS RDS in MySQL 2023 Vinoth Kanna @ Mydbops OpenSource Database Meetup 15AWS RDS in MySQL 2023 Vinoth Kanna @ Mydbops OpenSource Database Meetup 15
AWS RDS in MySQL 2023 Vinoth Kanna @ Mydbops OpenSource Database Meetup 15
 
MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...
MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...
MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...
 
Scaling-MongoDB-with-Horizontal-and-Vertical-Sharding Mydbops Opensource Data...
Scaling-MongoDB-with-Horizontal-and-Vertical-Sharding Mydbops Opensource Data...Scaling-MongoDB-with-Horizontal-and-Vertical-Sharding Mydbops Opensource Data...
Scaling-MongoDB-with-Horizontal-and-Vertical-Sharding Mydbops Opensource Data...
 
Mastering MongoDB Atlas: Essentials of Diagnostics and Debugging in the Cloud...
Mastering MongoDB Atlas: Essentials of Diagnostics and Debugging in the Cloud...Mastering MongoDB Atlas: Essentials of Diagnostics and Debugging in the Cloud...
Mastering MongoDB Atlas: Essentials of Diagnostics and Debugging in the Cloud...
 
Data Organisation: Table Partitioning in PostgreSQL
Data Organisation: Table Partitioning in PostgreSQLData Organisation: Table Partitioning in PostgreSQL
Data Organisation: Table Partitioning in PostgreSQL
 
Navigating MongoDB's Queryable Encryption for Ultimate Security - Mydbops
Navigating MongoDB's Queryable Encryption for Ultimate Security - MydbopsNavigating MongoDB's Queryable Encryption for Ultimate Security - Mydbops
Navigating MongoDB's Queryable Encryption for Ultimate Security - Mydbops
 
Mastering Database Migration_ Native replication (8.0) to InnoDB Cluster (8.0...
Mastering Database Migration_ Native replication (8.0) to InnoDB Cluster (8.0...Mastering Database Migration_ Native replication (8.0) to InnoDB Cluster (8.0...
Mastering Database Migration_ Native replication (8.0) to InnoDB Cluster (8.0...
 
Enhancing Security of MySQL Connections using SSL certificates
Enhancing Security of MySQL Connections using SSL certificatesEnhancing Security of MySQL Connections using SSL certificates
Enhancing Security of MySQL Connections using SSL certificates
 
Exploring the Fundamentals of YugabyteDB - Mydbops
Exploring the Fundamentals of YugabyteDB - Mydbops Exploring the Fundamentals of YugabyteDB - Mydbops
Exploring the Fundamentals of YugabyteDB - Mydbops
 
Time series in MongoDB - Mydbops
Time series in MongoDB - Mydbops Time series in MongoDB - Mydbops
Time series in MongoDB - Mydbops
 
Achieving High Availability in PostgreSQL
Achieving High Availability in PostgreSQLAchieving High Availability in PostgreSQL
Achieving High Availability in PostgreSQL
 
Scaling MongoDB with Horizontal and Vertical Sharding
Scaling MongoDB with Horizontal and Vertical Sharding Scaling MongoDB with Horizontal and Vertical Sharding
Scaling MongoDB with Horizontal and Vertical Sharding
 
MySQL Data Encryption at Rest
MySQL Data Encryption at RestMySQL Data Encryption at Rest
MySQL Data Encryption at Rest
 
Top-10-Features-In-MySQL-8.0 - Vinoth Kanna RS - Mydbops Team
Top-10-Features-In-MySQL-8.0 - Vinoth Kanna RS - Mydbops TeamTop-10-Features-In-MySQL-8.0 - Vinoth Kanna RS - Mydbops Team
Top-10-Features-In-MySQL-8.0 - Vinoth Kanna RS - Mydbops Team
 
Achieving compliance With MongoDB Security
Achieving compliance With MongoDB Security Achieving compliance With MongoDB Security
Achieving compliance With MongoDB Security
 

Recently uploaded

Recently uploaded (20)

Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 

Data-at-scale-with-TIDB Mydbops Co-Founder Kabilesh PR at LSPE Event

  • 1. Unleashing Data At Scale With TiDB Sep 09th, 2023 LSPE - 48 Kabilesh PR Co-Founder, Mydbops
  • 2. Interested in Open Source technologies Pingcap Certified TiDB Associate Tech Speaker/Blogger AWS certified Database speciality AWS community builder / SME Co-Founder, Mydbops Kabilesh PR About Me
  • 3. Mydbops at a glance 300+ Clients 3000+ Servers ISO & PCI DSS Certified 7 Yrs of Expertise AWS Advanced Consulting Partner TiDB PingCap Partner
  • 4. Mydbops Services Consulting Services Managed Services MySQL, MongoDB, PostgreSQL, TiDB & Cassandra Targeted Engagement 24 x 7 DBA Team
  • 5. Agenda What is TiDB ? Architecture Production Readiness Sharding
  • 6. What is TiDB ? An Open Source distributed SQL database by PingCAP • Scalable in both ways • Supports both Transactional and Analytical (HTAP) • Compatible with MySQL protocol • Highly available • Fault Tolerant •
  • 7. MySQL scalability - Bottlenecks Monolithic - Single node Limitation Horizontal scaling - Sharding
  • 8. Architecture TiDB cluster - SQL layer Tikv - Storage layer Placement Driver(PD) - Orchestrator TiFlash - Analytical columnar storage
  • 10. TiDB - SQL Interface Stateless • Handles Client connections & authentication • Parsing & compiling • Coverting relational to KV pair (Tikv) • Executing SQL • Online DDL execution • Garbage collections • Cache table •
  • 11. TiKV - Transactional Storage Stateful • Persistent storage with RocksDB • Distributed storage & transaction support • Strong Consistency • MVCC support • Inbuilt HA • Co-Processor (PP) •
  • 12. TiKV - Regions and Raft Region - Represents a range of key-value data • 96 Mb in size • A region is replicated to multiple nodes • Regions are automatically balanced across TiKV nodes • Each region has a leader(1) and followers(2) • Leader and Followers form Raft group • Leader handles RW ops • Followers Participate in Election & Data Reads •
  • 13. Services (LB) Stateful Sets Operator Maturity TiKV - Regions and Raft
  • 14. Placement Driver - PD Metadata management - Global view • Data placement • load balancing • TransactionID allocation • Global Timestamp (TSO) • Scheduling by collecting cluster info • PD consists of 3 node at-least •
  • 15. TiFlash - Analytical Store Columnar storage Replicated Async from Tikv Intelligent engine selection Optimised for aggregation, join & table scans MPP (Multiple Parallel Processing)
  • 17. Sharding Supports native sharding & distribution Relational to Key-value transformation Distributed based on Global key Global_key = Primary_key + Table_id(generated) For Non-int primary key row_id is generated
  • 20. Sharding - Region split & distribution
  • 21. Use-Cases https://www.pingcap.com/customers/ SAAS Application IOT data , Time series Ecommerce Logistics Gaming Fintech
  • 22. Production Readiness Wire Compatible with MySQL 5.7 UTF8MB4 by default with Binary Collation Profile your workload with TiDB Support for CDC with Ti-Binlog Monitoring integration with Grafana + Prometheus + Alert manager Full + Incremental , Logical Backup with PITR supported
  • 23. Limitations Stored procedures & Functions Triggers Events User-defined functions Fulltext and Spatial index Column-level privileges
  • 24. Reach Us : Info@mydbops.com Thank You Image Courtesy : PingCAP