SlideShare a Scribd company logo

How THINQ runs both transactions and analytics at scale

THINQ provides a cloud-based Communications-Platform-as-a-Service (CPaaS) that routes tens of millions of phone calls per day for customers in enterprise and telecommunications industries. In this session Sasha Vaniachine, Senior Database Administrator at THINQ, explains how he combined MariaDB Server and MariaDB ColumnStore to support both high-performance transaction processing and scalable analytics. In addition, he shares some of THINQ's best practices and lessons learned from supporting an ever-increasing database workload that currently exceeds 10,000 transactions per second.

1 of 49
1MySQL Triangle Meetup| 2019-01-24
How runs both transactions and analytics at scale
2MySQL Triangle Meetup| 2019-01-24 2MariaDB OpenWorks Conference | 2019-02-26
Open Source Enthusiast
• I am the most senior DBA at who is fortunate to be among the early
MySQL users
• One of my use cases was listed in An Open Letter to the Community from
MySQL Founders David Axmark & Michael "Monty" Widenius on 2 August
2005
• This year, we are celebrating ten years of MySQL…
• Major free software projects and hugely-popular Web sites such as the Sahana
(disaster recovery system for the tsunami), Ensembl.org and Human Genome Project
(used for cancer research), Wikipedia, Bugzilla, Craigslist, Feedster, Flickr, Freshmeat,
LiveJournal, Neopets, Slashdot, SugarCRM, Technorati, Wordpress, CERNs
ATLAS Experiment -- all taking advantage of MySQL's speed, ease of use, flexibility,
scalability and ecosystem
3MySQL Triangle Meetup| 2019-01-24 3MariaDB OpenWorks Conference | 2019-02-26
Outline
• provides a cloud-based Communications-Platform-as-a-Service
(CPaaS) that routes tens of millions of phone calls per day for
customers in enterprise and telecommunications
• I will present how combined MariaDB ColumnStore/InfiniDB
and MariaDB Galera Cluster to support both high-performance
transaction processing and scalable analytics
• In addition, I will share some of our best practices and lessons learned
from supporting an ever-increasing database workload
• a consequence of continued growth
4MySQL Triangle Meetup| 2019-01-24 4MariaDB OpenWorks Conference | 2019-02-26
What is VoIP and LCR?
• Since 2010 provides Least Cost Routing for VoIP phone calls
• VoIP stands for Voice over Internet Protocol
• An example: Skype
• Uses proprietary protocol to setup calls
• and most phone carriers use a standard Session Initiation Protocol (SIP)
• and many others use an open-source application OpenSIPS to setup calls
• LCR stands for Least Cost Routing
• While setting up VoIP phone call, choose carrier providing least expensive rate
• Providing Least Cost Routing is not trivial – see next slide
5MySQL Triangle Meetup| 2019-01-24 5MariaDB OpenWorks Conference | 2019-02-26
Actively maintained
*stats provided by openhub.net
About CGRateS (2)Least Cost Routing: Solutions by Others
Least Cost Routing: the science of finding the most
cost-effective path to connect your customers’ calls
6MySQL Triangle Meetup| 2019-01-24 6MariaDB OpenWorks Conference | 2019-02-26
Least Cost Routing: Solutions by
• Industry's only toll-
free LCR engine
170+
Countries
60+
Carriers

Recommended

Configuring workload-based storage and topologies
Configuring workload-based storage and topologiesConfiguring workload-based storage and topologies
Configuring workload-based storage and topologiesMariaDB plc
 
How we switched to columnar at SpendHQ
How we switched to columnar at SpendHQHow we switched to columnar at SpendHQ
How we switched to columnar at SpendHQMariaDB plc
 
Migrating from InnoDB and HBase to MyRocks at Facebook
Migrating from InnoDB and HBase to MyRocks at FacebookMigrating from InnoDB and HBase to MyRocks at Facebook
Migrating from InnoDB and HBase to MyRocks at FacebookMariaDB plc
 
How Alibaba Cloud scaled ApsaraDB with MariaDB MaxScale
How Alibaba Cloud scaled ApsaraDB with MariaDB MaxScaleHow Alibaba Cloud scaled ApsaraDB with MariaDB MaxScale
How Alibaba Cloud scaled ApsaraDB with MariaDB MaxScaleMariaDB plc
 
What’s new in Galera 4
What’s new in Galera 4What’s new in Galera 4
What’s new in Galera 4MariaDB plc
 
How MariaDB is approaching DBaaS
How MariaDB is approaching DBaaSHow MariaDB is approaching DBaaS
How MariaDB is approaching DBaaSMariaDB plc
 
Deploying MariaDB databases with containers at Nokia Networks
Deploying MariaDB databases with containers at Nokia NetworksDeploying MariaDB databases with containers at Nokia Networks
Deploying MariaDB databases with containers at Nokia NetworksMariaDB plc
 
Auto Europe's ongoing journey with MariaDB and open source
Auto Europe's ongoing journey with MariaDB and open sourceAuto Europe's ongoing journey with MariaDB and open source
Auto Europe's ongoing journey with MariaDB and open sourceMariaDB plc
 

More Related Content

What's hot

Global Data Replication with Galera for Ansell Guardian®
Global Data Replication with Galera for Ansell Guardian®Global Data Replication with Galera for Ansell Guardian®
Global Data Replication with Galera for Ansell Guardian®MariaDB plc
 
Getting started in the cloud for developers
Getting started in the cloud for developersGetting started in the cloud for developers
Getting started in the cloud for developersMariaDB plc
 
Getting the most out of MariaDB MaxScale
Getting the most out of MariaDB MaxScaleGetting the most out of MariaDB MaxScale
Getting the most out of MariaDB MaxScaleMariaDB plc
 
Inside CynosDB: MariaDB optimized for the cloud at Tencent
Inside CynosDB: MariaDB optimized for the cloud at TencentInside CynosDB: MariaDB optimized for the cloud at Tencent
Inside CynosDB: MariaDB optimized for the cloud at TencentMariaDB plc
 
ClustrixDB: how distributed databases scale out
ClustrixDB: how distributed databases scale outClustrixDB: how distributed databases scale out
ClustrixDB: how distributed databases scale outMariaDB plc
 
Webseminar: MariaDB Enterprise und MariaDB Enterprise Cluster
Webseminar: MariaDB Enterprise und MariaDB Enterprise ClusterWebseminar: MariaDB Enterprise und MariaDB Enterprise Cluster
Webseminar: MariaDB Enterprise und MariaDB Enterprise ClusterMariaDB Corporation
 
M|18 How Facebook Migrated to MyRocks
M|18 How Facebook Migrated to MyRocksM|18 How Facebook Migrated to MyRocks
M|18 How Facebook Migrated to MyRocksMariaDB plc
 
How Pixid dropped Oracle and went hybrid with MariaDB
How Pixid dropped Oracle and went hybrid with MariaDBHow Pixid dropped Oracle and went hybrid with MariaDB
How Pixid dropped Oracle and went hybrid with MariaDBMariaDB plc
 
How QBerg scaled to store data longer, query it faster
How QBerg scaled to store data longer, query it fasterHow QBerg scaled to store data longer, query it faster
How QBerg scaled to store data longer, query it fasterMariaDB plc
 
What to expect from MariaDB Platform X5, part 1
What to expect from MariaDB Platform X5, part 1What to expect from MariaDB Platform X5, part 1
What to expect from MariaDB Platform X5, part 1MariaDB plc
 
M|18 Analyzing Data with the MariaDB AX Platform
M|18 Analyzing Data with the MariaDB AX PlatformM|18 Analyzing Data with the MariaDB AX Platform
M|18 Analyzing Data with the MariaDB AX PlatformMariaDB plc
 
CCV: migrating our payment processing system to MariaDB
CCV: migrating our payment processing system to MariaDBCCV: migrating our payment processing system to MariaDB
CCV: migrating our payment processing system to MariaDBMariaDB plc
 
Choosing the right high availability strategy
Choosing the right high availability strategyChoosing the right high availability strategy
Choosing the right high availability strategyMariaDB plc
 
Introducing the ultimate MariaDB cloud, SkySQL
Introducing the ultimate MariaDB cloud, SkySQLIntroducing the ultimate MariaDB cloud, SkySQL
Introducing the ultimate MariaDB cloud, SkySQLMariaDB plc
 
Writing powerful stored procedures in PL/SQL
Writing powerful stored procedures in PL/SQLWriting powerful stored procedures in PL/SQL
Writing powerful stored procedures in PL/SQLMariaDB plc
 
Faster, better, stronger: The new InnoDB
Faster, better, stronger: The new InnoDBFaster, better, stronger: The new InnoDB
Faster, better, stronger: The new InnoDBMariaDB plc
 
MariaDB und mehr - MariaDB Roadshow Summer 2014 Hamburg Berlin Frankfurt
MariaDB und mehr - MariaDB Roadshow Summer 2014 Hamburg Berlin FrankfurtMariaDB und mehr - MariaDB Roadshow Summer 2014 Hamburg Berlin Frankfurt
MariaDB und mehr - MariaDB Roadshow Summer 2014 Hamburg Berlin FrankfurtMariaDB Corporation
 
How to migrate from Oracle Database with ease
How to migrate from Oracle Database with easeHow to migrate from Oracle Database with ease
How to migrate from Oracle Database with easeMariaDB plc
 
M|18 Creating a Reference Architecture for High Availability at Nokia
M|18 Creating a Reference Architecture for High Availability at NokiaM|18 Creating a Reference Architecture for High Availability at Nokia
M|18 Creating a Reference Architecture for High Availability at NokiaMariaDB plc
 

What's hot (20)

Global Data Replication with Galera for Ansell Guardian®
Global Data Replication with Galera for Ansell Guardian®Global Data Replication with Galera for Ansell Guardian®
Global Data Replication with Galera for Ansell Guardian®
 
Getting started in the cloud for developers
Getting started in the cloud for developersGetting started in the cloud for developers
Getting started in the cloud for developers
 
Getting the most out of MariaDB MaxScale
Getting the most out of MariaDB MaxScaleGetting the most out of MariaDB MaxScale
Getting the most out of MariaDB MaxScale
 
Inside CynosDB: MariaDB optimized for the cloud at Tencent
Inside CynosDB: MariaDB optimized for the cloud at TencentInside CynosDB: MariaDB optimized for the cloud at Tencent
Inside CynosDB: MariaDB optimized for the cloud at Tencent
 
ClustrixDB: how distributed databases scale out
ClustrixDB: how distributed databases scale outClustrixDB: how distributed databases scale out
ClustrixDB: how distributed databases scale out
 
Webseminar: MariaDB Enterprise und MariaDB Enterprise Cluster
Webseminar: MariaDB Enterprise und MariaDB Enterprise ClusterWebseminar: MariaDB Enterprise und MariaDB Enterprise Cluster
Webseminar: MariaDB Enterprise und MariaDB Enterprise Cluster
 
M|18 How Facebook Migrated to MyRocks
M|18 How Facebook Migrated to MyRocksM|18 How Facebook Migrated to MyRocks
M|18 How Facebook Migrated to MyRocks
 
How Pixid dropped Oracle and went hybrid with MariaDB
How Pixid dropped Oracle and went hybrid with MariaDBHow Pixid dropped Oracle and went hybrid with MariaDB
How Pixid dropped Oracle and went hybrid with MariaDB
 
How QBerg scaled to store data longer, query it faster
How QBerg scaled to store data longer, query it fasterHow QBerg scaled to store data longer, query it faster
How QBerg scaled to store data longer, query it faster
 
What to expect from MariaDB Platform X5, part 1
What to expect from MariaDB Platform X5, part 1What to expect from MariaDB Platform X5, part 1
What to expect from MariaDB Platform X5, part 1
 
M|18 Analyzing Data with the MariaDB AX Platform
M|18 Analyzing Data with the MariaDB AX PlatformM|18 Analyzing Data with the MariaDB AX Platform
M|18 Analyzing Data with the MariaDB AX Platform
 
CCV: migrating our payment processing system to MariaDB
CCV: migrating our payment processing system to MariaDBCCV: migrating our payment processing system to MariaDB
CCV: migrating our payment processing system to MariaDB
 
Choosing the right high availability strategy
Choosing the right high availability strategyChoosing the right high availability strategy
Choosing the right high availability strategy
 
MaxScale - The Pluggable Router
MaxScale - The Pluggable RouterMaxScale - The Pluggable Router
MaxScale - The Pluggable Router
 
Introducing the ultimate MariaDB cloud, SkySQL
Introducing the ultimate MariaDB cloud, SkySQLIntroducing the ultimate MariaDB cloud, SkySQL
Introducing the ultimate MariaDB cloud, SkySQL
 
Writing powerful stored procedures in PL/SQL
Writing powerful stored procedures in PL/SQLWriting powerful stored procedures in PL/SQL
Writing powerful stored procedures in PL/SQL
 
Faster, better, stronger: The new InnoDB
Faster, better, stronger: The new InnoDBFaster, better, stronger: The new InnoDB
Faster, better, stronger: The new InnoDB
 
MariaDB und mehr - MariaDB Roadshow Summer 2014 Hamburg Berlin Frankfurt
MariaDB und mehr - MariaDB Roadshow Summer 2014 Hamburg Berlin FrankfurtMariaDB und mehr - MariaDB Roadshow Summer 2014 Hamburg Berlin Frankfurt
MariaDB und mehr - MariaDB Roadshow Summer 2014 Hamburg Berlin Frankfurt
 
How to migrate from Oracle Database with ease
How to migrate from Oracle Database with easeHow to migrate from Oracle Database with ease
How to migrate from Oracle Database with ease
 
M|18 Creating a Reference Architecture for High Availability at Nokia
M|18 Creating a Reference Architecture for High Availability at NokiaM|18 Creating a Reference Architecture for High Availability at Nokia
M|18 Creating a Reference Architecture for High Availability at Nokia
 

Similar to How THINQ runs both transactions and analytics at scale

Is "Free" Good Enough for Your MySQL Environment?
Is "Free" Good Enough for Your MySQL Environment?Is "Free" Good Enough for Your MySQL Environment?
Is "Free" Good Enough for Your MySQL Environment?Datavail
 
Database Virtualization: The Next Wave of Big Data
Database Virtualization: The Next Wave of Big DataDatabase Virtualization: The Next Wave of Big Data
Database Virtualization: The Next Wave of Big Dataexponential-inc
 
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part20812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2Raul Chong
 
Openbar 12 - Leuven - From reactive programming to reactive architecture
Openbar 12 - Leuven - From reactive programming to reactive architectureOpenbar 12 - Leuven - From reactive programming to reactive architecture
Openbar 12 - Leuven - From reactive programming to reactive architectureOpenbar
 
Microservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SREMicroservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SREAraf Karsh Hamid
 
DEVNET-1166 Open SDN Controller APIs
DEVNET-1166	Open SDN Controller APIsDEVNET-1166	Open SDN Controller APIs
DEVNET-1166 Open SDN Controller APIsCisco DevNet
 
Introduction of MariaDB 2017 09
Introduction of MariaDB 2017 09Introduction of MariaDB 2017 09
Introduction of MariaDB 2017 09GOTO Satoru
 
Webinar Slides: MySQL Data Protection: Medical SaaS Manages Sensitive HIPAA C...
Webinar Slides: MySQL Data Protection: Medical SaaS Manages Sensitive HIPAA C...Webinar Slides: MySQL Data Protection: Medical SaaS Manages Sensitive HIPAA C...
Webinar Slides: MySQL Data Protection: Medical SaaS Manages Sensitive HIPAA C...Continuent
 
2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverviewDimas Prasetyo
 
Maplelabs scalable-field-device-cloud-native
Maplelabs scalable-field-device-cloud-nativeMaplelabs scalable-field-device-cloud-native
Maplelabs scalable-field-device-cloud-nativeGaneshkumar Sundararajan
 
Cloud Computing
Cloud ComputingCloud Computing
Cloud ComputingUOS
 
Marketing Automation at Scale: How Marketo Solved Key Data Management Challen...
Marketing Automation at Scale: How Marketo Solved Key Data Management Challen...Marketing Automation at Scale: How Marketo Solved Key Data Management Challen...
Marketing Automation at Scale: How Marketo Solved Key Data Management Challen...Continuent
 
Benchmark Showdown: Which Relational Database is the Fastest on AWS?
Benchmark Showdown: Which Relational Database is the Fastest on AWS?Benchmark Showdown: Which Relational Database is the Fastest on AWS?
Benchmark Showdown: Which Relational Database is the Fastest on AWS?Clustrix
 
Key Database Criteria for Cloud Applications
Key Database Criteria for Cloud ApplicationsKey Database Criteria for Cloud Applications
Key Database Criteria for Cloud ApplicationsNuoDB
 
Serverless: Market Overview and Investment Opportunities
Serverless: Market Overview and Investment OpportunitiesServerless: Market Overview and Investment Opportunities
Serverless: Market Overview and Investment OpportunitiesUnderscore VC
 
The Need For Speed - Strategies to Modernize Your Data Center
The Need For Speed - Strategies to Modernize Your Data CenterThe Need For Speed - Strategies to Modernize Your Data Center
The Need For Speed - Strategies to Modernize Your Data CenterEDB
 
The Crown Jewels: Is Enterprise Data Ready for the Cloud?
The Crown Jewels: Is Enterprise Data Ready for the Cloud?The Crown Jewels: Is Enterprise Data Ready for the Cloud?
The Crown Jewels: Is Enterprise Data Ready for the Cloud?Inside Analysis
 
MySQL InnoDB Cluster - Meetup Oracle MySQL / AFUP Paris
MySQL InnoDB Cluster - Meetup Oracle MySQL / AFUP ParisMySQL InnoDB Cluster - Meetup Oracle MySQL / AFUP Paris
MySQL InnoDB Cluster - Meetup Oracle MySQL / AFUP ParisOlivier DASINI
 

Similar to How THINQ runs both transactions and analytics at scale (20)

Is "Free" Good Enough for Your MySQL Environment?
Is "Free" Good Enough for Your MySQL Environment?Is "Free" Good Enough for Your MySQL Environment?
Is "Free" Good Enough for Your MySQL Environment?
 
MySQL overview
MySQL overviewMySQL overview
MySQL overview
 
Database Virtualization: The Next Wave of Big Data
Database Virtualization: The Next Wave of Big DataDatabase Virtualization: The Next Wave of Big Data
Database Virtualization: The Next Wave of Big Data
 
IBM - Introduction to Cloudant
IBM - Introduction to CloudantIBM - Introduction to Cloudant
IBM - Introduction to Cloudant
 
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part20812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2
 
Openbar 12 - Leuven - From reactive programming to reactive architecture
Openbar 12 - Leuven - From reactive programming to reactive architectureOpenbar 12 - Leuven - From reactive programming to reactive architecture
Openbar 12 - Leuven - From reactive programming to reactive architecture
 
Microservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SREMicroservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SRE
 
DEVNET-1166 Open SDN Controller APIs
DEVNET-1166	Open SDN Controller APIsDEVNET-1166	Open SDN Controller APIs
DEVNET-1166 Open SDN Controller APIs
 
Introduction of MariaDB 2017 09
Introduction of MariaDB 2017 09Introduction of MariaDB 2017 09
Introduction of MariaDB 2017 09
 
Webinar Slides: MySQL Data Protection: Medical SaaS Manages Sensitive HIPAA C...
Webinar Slides: MySQL Data Protection: Medical SaaS Manages Sensitive HIPAA C...Webinar Slides: MySQL Data Protection: Medical SaaS Manages Sensitive HIPAA C...
Webinar Slides: MySQL Data Protection: Medical SaaS Manages Sensitive HIPAA C...
 
2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview
 
Maplelabs scalable-field-device-cloud-native
Maplelabs scalable-field-device-cloud-nativeMaplelabs scalable-field-device-cloud-native
Maplelabs scalable-field-device-cloud-native
 
Cloud Computing
Cloud ComputingCloud Computing
Cloud Computing
 
Marketing Automation at Scale: How Marketo Solved Key Data Management Challen...
Marketing Automation at Scale: How Marketo Solved Key Data Management Challen...Marketing Automation at Scale: How Marketo Solved Key Data Management Challen...
Marketing Automation at Scale: How Marketo Solved Key Data Management Challen...
 
Benchmark Showdown: Which Relational Database is the Fastest on AWS?
Benchmark Showdown: Which Relational Database is the Fastest on AWS?Benchmark Showdown: Which Relational Database is the Fastest on AWS?
Benchmark Showdown: Which Relational Database is the Fastest on AWS?
 
Key Database Criteria for Cloud Applications
Key Database Criteria for Cloud ApplicationsKey Database Criteria for Cloud Applications
Key Database Criteria for Cloud Applications
 
Serverless: Market Overview and Investment Opportunities
Serverless: Market Overview and Investment OpportunitiesServerless: Market Overview and Investment Opportunities
Serverless: Market Overview and Investment Opportunities
 
The Need For Speed - Strategies to Modernize Your Data Center
The Need For Speed - Strategies to Modernize Your Data CenterThe Need For Speed - Strategies to Modernize Your Data Center
The Need For Speed - Strategies to Modernize Your Data Center
 
The Crown Jewels: Is Enterprise Data Ready for the Cloud?
The Crown Jewels: Is Enterprise Data Ready for the Cloud?The Crown Jewels: Is Enterprise Data Ready for the Cloud?
The Crown Jewels: Is Enterprise Data Ready for the Cloud?
 
MySQL InnoDB Cluster - Meetup Oracle MySQL / AFUP Paris
MySQL InnoDB Cluster - Meetup Oracle MySQL / AFUP ParisMySQL InnoDB Cluster - Meetup Oracle MySQL / AFUP Paris
MySQL InnoDB Cluster - Meetup Oracle MySQL / AFUP Paris
 

More from MariaDB plc

MariaDB Paris Workshop 2023 - MaxScale 23.02.x
MariaDB Paris Workshop 2023 - MaxScale 23.02.xMariaDB Paris Workshop 2023 - MaxScale 23.02.x
MariaDB Paris Workshop 2023 - MaxScale 23.02.xMariaDB plc
 
MariaDB Paris Workshop 2023 - Newpharma
MariaDB Paris Workshop 2023 - NewpharmaMariaDB Paris Workshop 2023 - Newpharma
MariaDB Paris Workshop 2023 - NewpharmaMariaDB plc
 
MariaDB Paris Workshop 2023 - Cloud
MariaDB Paris Workshop 2023 - CloudMariaDB Paris Workshop 2023 - Cloud
MariaDB Paris Workshop 2023 - CloudMariaDB plc
 
MariaDB Paris Workshop 2023 - MariaDB Enterprise
MariaDB Paris Workshop 2023 - MariaDB EnterpriseMariaDB Paris Workshop 2023 - MariaDB Enterprise
MariaDB Paris Workshop 2023 - MariaDB EnterpriseMariaDB plc
 
MariaDB Paris Workshop 2023 - Performance Optimization
MariaDB Paris Workshop 2023 - Performance OptimizationMariaDB Paris Workshop 2023 - Performance Optimization
MariaDB Paris Workshop 2023 - Performance OptimizationMariaDB plc
 
MariaDB Paris Workshop 2023 - MaxScale
MariaDB Paris Workshop 2023 - MaxScale MariaDB Paris Workshop 2023 - MaxScale
MariaDB Paris Workshop 2023 - MaxScale MariaDB plc
 
MariaDB Paris Workshop 2023 - novadys presentation
MariaDB Paris Workshop 2023 - novadys presentationMariaDB Paris Workshop 2023 - novadys presentation
MariaDB Paris Workshop 2023 - novadys presentationMariaDB plc
 
MariaDB Paris Workshop 2023 - DARVA presentation
MariaDB Paris Workshop 2023 - DARVA presentationMariaDB Paris Workshop 2023 - DARVA presentation
MariaDB Paris Workshop 2023 - DARVA presentationMariaDB plc
 
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server MariaDB plc
 
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-Backup
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-BackupMariaDB SkySQL Autonome Skalierung, Observability, Cloud-Backup
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-BackupMariaDB plc
 
Einführung : MariaDB Tech und Business Update Hamburg 2023
Einführung : MariaDB Tech und Business Update Hamburg 2023Einführung : MariaDB Tech und Business Update Hamburg 2023
Einführung : MariaDB Tech und Business Update Hamburg 2023MariaDB plc
 
Hochverfügbarkeitslösungen mit MariaDB
Hochverfügbarkeitslösungen mit MariaDBHochverfügbarkeitslösungen mit MariaDB
Hochverfügbarkeitslösungen mit MariaDBMariaDB plc
 
Die Neuheiten in MariaDB Enterprise Server
Die Neuheiten in MariaDB Enterprise ServerDie Neuheiten in MariaDB Enterprise Server
Die Neuheiten in MariaDB Enterprise ServerMariaDB plc
 
Introducing workload analysis
Introducing workload analysisIntroducing workload analysis
Introducing workload analysisMariaDB plc
 
Under the hood: SkySQL monitoring
Under the hood: SkySQL monitoringUnder the hood: SkySQL monitoring
Under the hood: SkySQL monitoringMariaDB plc
 
Introducing the R2DBC async Java connector
Introducing the R2DBC async Java connectorIntroducing the R2DBC async Java connector
Introducing the R2DBC async Java connectorMariaDB plc
 
MariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introductionMariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introductionMariaDB plc
 
The architecture of SkySQL
The architecture of SkySQLThe architecture of SkySQL
The architecture of SkySQLMariaDB plc
 
What to expect from MariaDB Platform X5, part 2
What to expect from MariaDB Platform X5, part 2What to expect from MariaDB Platform X5, part 2
What to expect from MariaDB Platform X5, part 2MariaDB plc
 
Beyond the basics: advanced SQL with MariaDB
Beyond the basics: advanced SQL with MariaDBBeyond the basics: advanced SQL with MariaDB
Beyond the basics: advanced SQL with MariaDBMariaDB plc
 

More from MariaDB plc (20)

MariaDB Paris Workshop 2023 - MaxScale 23.02.x
MariaDB Paris Workshop 2023 - MaxScale 23.02.xMariaDB Paris Workshop 2023 - MaxScale 23.02.x
MariaDB Paris Workshop 2023 - MaxScale 23.02.x
 
MariaDB Paris Workshop 2023 - Newpharma
MariaDB Paris Workshop 2023 - NewpharmaMariaDB Paris Workshop 2023 - Newpharma
MariaDB Paris Workshop 2023 - Newpharma
 
MariaDB Paris Workshop 2023 - Cloud
MariaDB Paris Workshop 2023 - CloudMariaDB Paris Workshop 2023 - Cloud
MariaDB Paris Workshop 2023 - Cloud
 
MariaDB Paris Workshop 2023 - MariaDB Enterprise
MariaDB Paris Workshop 2023 - MariaDB EnterpriseMariaDB Paris Workshop 2023 - MariaDB Enterprise
MariaDB Paris Workshop 2023 - MariaDB Enterprise
 
MariaDB Paris Workshop 2023 - Performance Optimization
MariaDB Paris Workshop 2023 - Performance OptimizationMariaDB Paris Workshop 2023 - Performance Optimization
MariaDB Paris Workshop 2023 - Performance Optimization
 
MariaDB Paris Workshop 2023 - MaxScale
MariaDB Paris Workshop 2023 - MaxScale MariaDB Paris Workshop 2023 - MaxScale
MariaDB Paris Workshop 2023 - MaxScale
 
MariaDB Paris Workshop 2023 - novadys presentation
MariaDB Paris Workshop 2023 - novadys presentationMariaDB Paris Workshop 2023 - novadys presentation
MariaDB Paris Workshop 2023 - novadys presentation
 
MariaDB Paris Workshop 2023 - DARVA presentation
MariaDB Paris Workshop 2023 - DARVA presentationMariaDB Paris Workshop 2023 - DARVA presentation
MariaDB Paris Workshop 2023 - DARVA presentation
 
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server
 
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-Backup
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-BackupMariaDB SkySQL Autonome Skalierung, Observability, Cloud-Backup
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-Backup
 
Einführung : MariaDB Tech und Business Update Hamburg 2023
Einführung : MariaDB Tech und Business Update Hamburg 2023Einführung : MariaDB Tech und Business Update Hamburg 2023
Einführung : MariaDB Tech und Business Update Hamburg 2023
 
Hochverfügbarkeitslösungen mit MariaDB
Hochverfügbarkeitslösungen mit MariaDBHochverfügbarkeitslösungen mit MariaDB
Hochverfügbarkeitslösungen mit MariaDB
 
Die Neuheiten in MariaDB Enterprise Server
Die Neuheiten in MariaDB Enterprise ServerDie Neuheiten in MariaDB Enterprise Server
Die Neuheiten in MariaDB Enterprise Server
 
Introducing workload analysis
Introducing workload analysisIntroducing workload analysis
Introducing workload analysis
 
Under the hood: SkySQL monitoring
Under the hood: SkySQL monitoringUnder the hood: SkySQL monitoring
Under the hood: SkySQL monitoring
 
Introducing the R2DBC async Java connector
Introducing the R2DBC async Java connectorIntroducing the R2DBC async Java connector
Introducing the R2DBC async Java connector
 
MariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introductionMariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introduction
 
The architecture of SkySQL
The architecture of SkySQLThe architecture of SkySQL
The architecture of SkySQL
 
What to expect from MariaDB Platform X5, part 2
What to expect from MariaDB Platform X5, part 2What to expect from MariaDB Platform X5, part 2
What to expect from MariaDB Platform X5, part 2
 
Beyond the basics: advanced SQL with MariaDB
Beyond the basics: advanced SQL with MariaDBBeyond the basics: advanced SQL with MariaDB
Beyond the basics: advanced SQL with MariaDB
 

Recently uploaded

Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCIOWomenMagazine
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleAmir Moghimi
 
Managing multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerManaging multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerThierry Gayet
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System AdministratorsGlobus
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxMindInventory
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with GlobusGlobus
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowNaoki (Neo) SATO
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Dmitry Zinoviev
 
Passbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentPassbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentThierry Gayet
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfssuser82c38d
 
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارانتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارsohilww
 
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio, Inc.
 
Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019VICTOR MAESTRE RAMIREZ
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaSGlobus
 
An Introduction to Globus for Researchers
An Introduction to Globus for ResearchersAn Introduction to Globus for Researchers
An Introduction to Globus for ResearchersGlobus
 
From Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesFrom Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesAnisha Agarwal
 
Open Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsOpen Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsSprings
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration TopicsGlobus
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureHironori Washizaki
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flinkconfluent
 

Recently uploaded (20)

Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdf
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scale
 
Managing multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerManaging multicast/igmp stream on Docker
Managing multicast/igmp stream on Docker
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System Administrators
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptx
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with Globus
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flow
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)
 
Passbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentPassbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managment
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdf
 
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارانتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
 
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
 
Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaS
 
An Introduction to Globus for Researchers
An Introduction to Globus for ResearchersAn Introduction to Globus for Researchers
An Introduction to Globus for Researchers
 
From Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesFrom Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT Services
 
Open Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsOpen Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and Cons
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration Topics
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about Architecture
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 

How THINQ runs both transactions and analytics at scale

  • 1. 1MySQL Triangle Meetup| 2019-01-24 How runs both transactions and analytics at scale
  • 2. 2MySQL Triangle Meetup| 2019-01-24 2MariaDB OpenWorks Conference | 2019-02-26 Open Source Enthusiast • I am the most senior DBA at who is fortunate to be among the early MySQL users • One of my use cases was listed in An Open Letter to the Community from MySQL Founders David Axmark & Michael "Monty" Widenius on 2 August 2005 • This year, we are celebrating ten years of MySQL… • Major free software projects and hugely-popular Web sites such as the Sahana (disaster recovery system for the tsunami), Ensembl.org and Human Genome Project (used for cancer research), Wikipedia, Bugzilla, Craigslist, Feedster, Flickr, Freshmeat, LiveJournal, Neopets, Slashdot, SugarCRM, Technorati, Wordpress, CERNs ATLAS Experiment -- all taking advantage of MySQL's speed, ease of use, flexibility, scalability and ecosystem
  • 3. 3MySQL Triangle Meetup| 2019-01-24 3MariaDB OpenWorks Conference | 2019-02-26 Outline • provides a cloud-based Communications-Platform-as-a-Service (CPaaS) that routes tens of millions of phone calls per day for customers in enterprise and telecommunications • I will present how combined MariaDB ColumnStore/InfiniDB and MariaDB Galera Cluster to support both high-performance transaction processing and scalable analytics • In addition, I will share some of our best practices and lessons learned from supporting an ever-increasing database workload • a consequence of continued growth
  • 4. 4MySQL Triangle Meetup| 2019-01-24 4MariaDB OpenWorks Conference | 2019-02-26 What is VoIP and LCR? • Since 2010 provides Least Cost Routing for VoIP phone calls • VoIP stands for Voice over Internet Protocol • An example: Skype • Uses proprietary protocol to setup calls • and most phone carriers use a standard Session Initiation Protocol (SIP) • and many others use an open-source application OpenSIPS to setup calls • LCR stands for Least Cost Routing • While setting up VoIP phone call, choose carrier providing least expensive rate • Providing Least Cost Routing is not trivial – see next slide
  • 5. 5MySQL Triangle Meetup| 2019-01-24 5MariaDB OpenWorks Conference | 2019-02-26 Actively maintained *stats provided by openhub.net About CGRateS (2)Least Cost Routing: Solutions by Others Least Cost Routing: the science of finding the most cost-effective path to connect your customers’ calls
  • 6. 6MySQL Triangle Meetup| 2019-01-24 6MariaDB OpenWorks Conference | 2019-02-26 Least Cost Routing: Solutions by • Industry's only toll- free LCR engine 170+ Countries 60+ Carriers
  • 7. 7MySQL Triangle Meetup| 2019-01-24 7MariaDB OpenWorks Conference | 2019-02-26 data types, retention periods and volumes SIP Messages CDRs Aggregates days weeks or months permanent Terabytes Terabytes Gigabytes
  • 8. 8MySQL Triangle Meetup| 2019-01-24 8MariaDB OpenWorks Conference | 2019-02-26 Setting up Phone Calls • processes dozens of SIP messages (text format, like HTTP/1.1) to setup one call • For debugging (SIP tracing) all SIP messages are captured in the MariaDB Server database • As a consequence of our continued growth, this results in an ever-increasing database workload to capture more than 75,000 SIP messages per second SIP Messages
  • 9. 9MySQL Triangle Meetup| 2019-01-24 Is there a way to scale up SIP messages inserts in RDBMS rather than inserting rows one by one? SIP Messages
  • 10. 10MySQL Triangle Meetup| 2019-01-24 10MariaDB OpenWorks Conference | 2019-02-26 Scaling up Insert Rate: Multi-Row Inserts • Instead of flushing inserted rows one by one into the database, OpenSIPS will now store rows in memory, and only flush them to DB when a certain number of rows have piled up in memory • The flushing of the rows will be done within a single SQL command 1. On a constant rate of 1500 CPS pushed through the proxy, the load on the mysql daemon was : • OpenSIPS 1.6.4 : 71.5 % • OpenSIPS 1.7.0 : 35.1 % -> BOOST = ~50 % lower load on mysql http://www.opensips.org/About/PerformanceTests-InsertBuffering SIP Messages
  • 11. 11MySQL Triangle Meetup| 2019-01-24 The transactions per second rate has been lowered by inserting multiple rows at once Does the data rate (Mbps) remain the same? How to scale up the I/O rate further? SIP Messages
  • 12. 12MySQL Triangle Meetup| 2019-01-24 12MariaDB OpenWorks Conference | 2019-02-26 Scaling up I/O Rate • As a consequence of our growth we now • optimized my.cnf configuration for InnoDB disk I/O • use faster disks • split (“shard”) the SIP message capture across several MariaDB Servers SIP Messages
  • 13. 13MySQL Triangle Meetup| 2019-01-24 13MariaDB OpenWorks Conference | 2019-02-26 Scaling up Insert Rate: Multi-Row Inserts • Instead of flushing inserted rows one by one into the database, OpenSIPS will now store rows in memory, and only flush them to DB when a certain number of rows have piled up in memory • The flushing of the rows will be done within a single SQL command 1. On a constant rate of 1500 CPS pushed through the proxy, the load on the mysql daemon was : • OpenSIPS 1.6.4 : 71.5 % • OpenSIPS 1.7.0 : 35.1 % -> BOOST = ~50 % lower load on mysql 2. In terms of raw CPS, the maximum number of calls per second that the proxy could handle was : • OpenSIPS 1.6.4 : 1700 CPS • OpenSIPS 1.7.0 : 3000 CPS -> BOOST = ~75 % higher CPS http://www.opensips.org/About/PerformanceTests-InsertBuffering SIP Messages
  • 14. 14MySQL Triangle Meetup| 2019-01-24 14MariaDB OpenWorks Conference | 2019-02-26 Decoupling Database and Application • The increase in OpenSIPS throughput from using multi-row inserts indicates a tight coupling between the database and the application • Decoupling the application from the database requires • a non-blocking MariaDB client library • application re-architecture • The latest version of OpenSIPS has been re-architected to decouple application from database • This presentation shares our experience in case of tight coupling between the database and the application SIP Messages CDRs
  • 15. 15MySQL Triangle Meetup| 2019-01-24 15MariaDB OpenWorks Conference | 2019-02-26 High Availability • A critical application tightly coupled to a database requires High Availability of the database server • For High Availability we use MariaDB Galera Cluster Galera Cluster ➢ Synchronous multi-master cluster ➢ no data loss ➢ no slave lag ➢ no slave failover ➢ For MySQL/InnoDB ➢ 3 or more nodes needed for HA ➢ No single point of failure CDRs
  • 16. 16MySQL Triangle Meetup| 2019-01-24 16MariaDB OpenWorks Conference | 2019-02-26 Call Detail Records • After the phone call is complete, telecommunications providers must store a Call Detail Record (CDR) • In our case, upon the phone call completion the OpenSIPS application generates a corresponding record for CDR accounting • A database query for the predefined accounting table: • INSERT INTO acc … CDRs
  • 17. 17MySQL Triangle Meetup| 2019-01-24 17MariaDB OpenWorks Conference | 2019-02-26 CDR Processing • Records from acc table are processed to generate CDRs and aggregate data further • Traditional solution: • Queue records for processing • Works well on a small scale Flavio E. Goncalves: Building Telephony Systems with OpenSIPS 1.6
  • 18. 18MySQL Triangle Meetup| 2019-01-24 Do you expect a problem with this approach when the CDR processing rate increases as a consequence of our growth? CDRs
  • 19. 19MySQL Triangle Meetup| 2019-01-24 19MariaDB OpenWorks Conference | 2019-02-26 Scaling Up RDBMS Queues • https://www.engineyard.com/blog/5-subtle-ways-youre-using-mysql-as-a-queue-and-why-itll-bite-you • https://www.eschrade.com/page/why-mysql-is-not-a-queue
  • 20. 20MySQL Triangle Meetup| 2019-01-24 Is there a better way to process records that scales better than a queue implemented in RDBMS? CDRs
  • 21. 21MySQL Triangle Meetup| 2019-01-24 21MariaDB OpenWorks Conference | 2019-02-26 Triggers! • Scales much better • For processing and aggregation uses a trigger on the acc table • Upon each acc row INSERT, the trigger increments corresponding rows in the billing table using ON DUPLICATE KEY UPDATE for a unique index on columns used for aggregation such as customer_id, carrier_id https://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql
  • 22. 22MySQL Triangle Meetup| 2019-01-24 22MariaDB OpenWorks Conference | 2019-02-26 Next Scalability Limit We use: • OpenSIPS multi-row inserts to scale up inserts from numerous calls • Triggers to scale up CDR processing • MariaDB Galera cluster for High Availability (HA) • Trigger support is straightforward • Not so trivial to support EVENTs • As a consequence of growth, these solutions hit another scalability limit: • We encountered an increasing rate of deadlocks CDRs Aggregates
  • 23. 23MySQL Triangle Meetup| 2019-01-24 23MariaDB OpenWorks Conference | 2019-02-26 Increasing Rate of Deadlocks Like: • Retry logic is of little or no help here as the deadlock may happen again • Consolidating writes to a single Galera Cluster node is of no help either CDRs Aggregates
  • 24. 24MySQL Triangle Meetup| 2019-01-24 24MariaDB OpenWorks Conference | 2019-02-26 Let us review our aggregation approach CREATE TRIGGER aggregate BEFORE INSERT ON acc FOR EACH ROW BEGIN INSERT INTO billing.aggregates (date, customer_id, carrier_id, calls, ...) VALUES (NEW.date, NEW.customer_id, NEW.carrier_id, IF(NEW.sip_code=200,1,0), # count completed calls ...) ON DUPLICATE KEY UPDATE calls = calls + VALUES(calls), ...; END CDRs AggregatesCREATE TABLE `acc` ( `id` INT NOT NULL AUTO_INCREMENT, `date` DATE, `customer_id` INT, `carrier_id` INT, `sip_code` INT, ... PRIMARY KEY (`id`) ); CREATE TABLE `billing`.`aggregates` ( `id` INT NOT NULL AUTO_INCREMENT, `date` DATE, `customer_id` INT, `carrier_id` INT, `calls` INT, PRIMARY KEY (`id`), UNIQUE KEY `uk_customer_carrier` (`date`,`customer_id`,`carrier_id`) );
  • 25. 25MySQL Triangle Meetup| 2019-01-24 Is it clear why these deadlocks started to happen as a consequence of our growth? CDRs Aggregates
  • 26. 26MySQL Triangle Meetup| 2019-01-24 26MariaDB OpenWorks Conference | 2019-02-26 customer-1, carrier-1 UNIQUE KEY INDEX ROWS customer-1, carrier-2 customer-1, carrier-3 (1) TRANSACTION (2) TRANSACTION customer-1, carrier-4 customer-1, carrier-5 Call 1: customer-1, carrier-1 Call 2: customer-1, carrier-3 Call 3: customer-1, carrier-2 Call 4: customer-2, carrier-1 … Call 100: customer-2, carrier-1 Call 1: customer-1, carrier-5 Call 2: customer-1, carrier-3 Call 3: customer-1, carrier-4 Call 4: customer-1, carrier-1 … Call 100: customer-6, carrier-1 What is causing deadlocks? • Concurrent multi-row INSERT queries may cause a deadlock when two transactions touch the same sets of data in a different order • Such as locking index rows in an opposite order CDRs Aggregates
  • 27. 27MySQL Triangle Meetup| 2019-01-24 27MariaDB OpenWorks Conference | 2019-02-26 Why deadlocks presented a problem? • A deadlocked query on Galera Cluster is like a huge transaction • All later INSERT queries are waiting for it to complete • or to be rolled back • Increasing rate of deadlocks creates a backpressure for our OpenSIPS application • Due to the tight coupling between the application and the database server Impact of Huge Transaction 0 500 1000 1500 2000 2500 3000 3500 4000 4500 Huge Transaction Slave Lag Trx in master 24 secs Trx in slave 9 secs CDRs Aggregates
  • 28. 28MySQL Triangle Meetup| 2019-01-24 Is there a way to avoid such deadlocks? CDRs Aggregates
  • 29. 29MySQL Triangle Meetup| 2019-01-24 29MariaDB OpenWorks Conference | 2019-02-26 Locking in the same order • When modifying different sets of rows in the same table, do those operations in a consistent order each time [MySQL Reference Manual] • Application developers can eliminate all risk of enqueue deadlocks by ensuring that transactions requiring multiple resources always lock them in the same order [Steve Adams] CDRs Aggregates
  • 30. 30MySQL Triangle Meetup| 2019-01-24 30MariaDB OpenWorks Conference | 2019-02-26 customer-1, carrier-1 UNIQUE KEY INDEX ROWS customer-1, carrier-2 customer-1, carrier-3 (1) TRANSACTION (2) TRANSACTION customer-1, carrier-4 customer-1, carrier-5 Call 1: customer-1, carrier-1 Call 2: customer-1, carrier-2 Call 3: customer-1, carrier-3 Call 4: customer-2, carrier-1 … Call 100: customer-2, carrier-1 Call 1: customer-1, carrier-3 Call 2: customer-1, carrier-4 Call 3: customer-1, carrier-5 Call 4: customer-1, carrier-7 … Call 100: customer-6, carrier-9 We need ordered sets of rows • Since OpenSIPS application is open source, we can modify the INSERT statement generated by the OpenSIPS application CDRs Aggregates
  • 31. 31MySQL Triangle Meetup| 2019-01-24 We would like to delegate row ordering to MariaDB, for example 1. Insert unordered data into a temporary table tmp 2. Insert into acc select from tmp order by customer_id, carrier_id Is there a better way? CDRs Aggregates
  • 32. 32MySQL Triangle Meetup| 2019-01-24 32MariaDB OpenWorks Conference | 2019-02-26 Avoiding Deadlocks • We order INSERT rows according to the index columns on-the-fly • by modifying the prepared statement generated by the OpenSIPS application for the acc table INSERT query • With UNION ALL, the modified prepared statement creates a set, which is then ordered by the unique index columns • A use case for the MariaDB UNION ALL optimization row OpenSIPS prepared statement modified prepared statement derived table: select without from insert into acc insert into acc ( ( method, method, customer_id, customer_id, carrier_id, carrier_id, callid, callid, sip_code, sip_code, sip_reason, sip_reason, time, time, duration, duration, setuptime, setuptime, created created ) ) values ( 1 ( select ?, ? AS method, ?, ? AS customer_id, ?, ? AS carrier_id, ?, ? AS callid, ?, ? AS sip_code, ?, ? AS sip_reason, ?, ? AS time, ?, ? AS duration, ?, ? AS setuptime, ? ? AS created ) ) , union all 2 (?,?,?,?,?,?,?,?,?,?) (select ?,?,?,?,?,?,?,?,?,?) , union all … … … 100 (?,?,?,?,?,?,?,?,?,?) (select ?,?,?,?,?,?,?,?,?,?) order by customer_id, carrier_id
  • 33. 33MySQL Triangle Meetup| 2019-01-24 Three use cases for ColumnStore/InfiniDB 1. Conventional use case: analytics on the Call Detail Records The CDRs are wide rows with 50-100 columns, while most analytics queries need just few of those columns Works best for approximately incremental data, such as time series CDRs
  • 34. 34MySQL Triangle Meetup| 2019-01-24 34MariaDB OpenWorks Conference | 2019-02-26 2. Data Aggregation with ColumnStore How many calls happen during the last hour? • A considerable problem here is that this number may change, as Call Detail Records for the past hour may arrive with delay • Under those conditions with InfiniDB/ColumnStore we are able to repeat simple SQL queries for data aggregation • These SQL queries work well, since latest data (for the few select columns) fit in memory of ColumnStore/InfiniDB cluster nodes • In a traditional row-based MariaDB Server this simple approach would require prohibitively more memory to fit whole InnoDB rows CDRs Aggregates
  • 35. 35MySQL Triangle Meetup| 2019-01-24 35MariaDB OpenWorks Conference | 2019-02-26 3. Redundancy for Critical Data • As statistical findings tolerate limited data losses, ACID compliance of the MariaDB ColumnStore (the InfiniDB legacy) is often overlooked • In contrast, the third use case requires both products Telecommunications (IP telephony – SaaS) Database (Hybrid) Transactions Analytics Transactional Capture call detail records Charge by call/message Generate bills Analytical Monitor usage Identify peak periods Estimate costs Self-service analytics Slide by Shane K Johnson, MariaDB CDRs
  • 36. 36MySQL Triangle Meetup| 2019-01-24 36MariaDB OpenWorks Conference | 2019-02-26 Double-Entry Accounting • Since middle ages, accounting uses double-entry system, were at least two accounting entries are required to record each financial transaction • In telecommunications these financial transactions are a part of the Call Detail Record • In 1494 Fra Luca Bartolomeo de Pacioli published a book on the double-entry system of accounting
  • 37. 37MySQL Triangle Meetup| 2019-01-24 37MariaDB OpenWorks Conference | 2019-02-26 Double-Entry Accounting for CDRs • Thanks to the ACID compliance of MariaDB ColumnStore/InfiniDB transactions, thinQ was able to implement in practice double-entry accounting for Call Detail Records • The ColumnStore/InfiniDB transactions enable us to verify/audit customers’ billing done through Call Detail Records stored in MariaDB Server CDRs
  • 38. 38MySQL Triangle Meetup| 2019-01-24 38MariaDB OpenWorks Conference | 2019-02-26 Traditional InfiniDB/CS Data Processing Pipeline OLTP Files/XML Log Files Operational Source Data StagingorODSETL High-speedLoadUtility Ad-Hoc Dashboards Reports Notifications Users Staging Area Data Warehouse Data Warehouse and Metadata Management #6 Load New Data with Minimal Impact OLTP Files/XML Log Files Operational Source Data StagingorODSETL High-speedLoadUtility Ad-Hoc Dashboards Reports Notifications Users Staging Area Data Warehouse Data Warehouse and Metadata Management #6 Load New Data with Minimal Impact OLTP Files/XML Log Files Operational Source Data StagingorODSETL High-speedLoadUtility Ad-Hoc Dashboards Reports Notifications Users Staging Area Data Warehouse Data Warehouse and Metadata Management #6 Load New Data with Minimal Impact InfiniDB HA Staging Area for processed data local to InfiniDB (e.g. costly HA storage) HA Staging Area for raw data (e.g. data streams)
  • 39. 39MySQL Triangle Meetup| 2019-01-24 39MariaDB OpenWorks Conference | 2019-02-26 Heterogeneous Redundancy Assures Scaling • Our experience with using both MariaDB Server and MariaDB ColumnStore/InfiniDB proved crucial in solving business challenges • as two distinct data processing pipelines built with two different technologies encountered scalability limits at different loads • Similar to both mechanical and hydraulic braking in a car • Combining mechanical drum brakes with hydraulic brakes to offer backup braking support in case the car’s hydraulic system fails CDRs
  • 40. 40MySQL Triangle Meetup| 2019-01-24 • To assure resilience, infrastructure is geo-redundant • Our call data is collected in geo-distributed data centers, which complicates analytics • Note the extra cost of large data transfers between data centers • According to the InfiniDB Concepts Guide, User and Performance Modules can be separated out in different data centers and geographic locations • We are pleased that MariaDB ColumnStore retained such feature • The beauty of this is that the analytical data aggregation queries are executed locally, with only small aggregated data are transferred between data centers CDRs
  • 41. 41MySQL Triangle Meetup| 2019-01-24 41MariaDB OpenWorks Conference | 2019-02-26 Tips for Geo-Distributed MariaDB ColumnStore • Three combined UM/PM nodes: PM3 in a different geo-location than PM1 & PM2 • Watch for idle TCP/IP connections dropped by data centers firewalls • Implement a keep-alive ping such as periodic execution of a test data aggregation query • Watch for automatic round-robin distribution of queries: • First query execution is fast (0.3s) – logged on PM1 node • Second execution is fast (0.3s) – logged on PM2 node • Third execution is slow (18s) – logged on PM3 node in a data center separated by 25 ms RTT from PM1 & PM2 • David Thompson (MariaDB VP) kindly provided a workaround for round-robin query distribution: • Change the Columnstore.xml ExeMgr IP addresses to 127.0.0.1 on all three nodes • With these, the geo-distributed MariaDB ColumnStore system operates stably
  • 42. 42MySQL Triangle Meetup| 2019-01-24 We look forward to the new remote mcsimport capabilities of the ColumnStore Bulk Write SDK CDRs
  • 43. 43MySQL Triangle Meetup| 2019-01-24 43MariaDB OpenWorks Conference | 2019-02-26 OLTP Files/XML Log Files Operational Source Data StagingorODSETL High-speedLoadUtility Ad-Hoc Dashboards Reports Notifications Users Staging Area Data Warehouse Data Warehouse and Metadata Management 6 Load New Data with Minimal Impact OLTP Files/XML Log Files Operational Source Data StagingorODSETL High-speedLoadUtility Ad-Hoc Dashboards Reports Notifications Users Staging Area Data Warehouse Data Warehouse and Metadata Management #6 Load New Data with Minimal Impact RemoteHABulkDataLoaders MariaDB ColumnStore HA Staging Area for raw data (e.g. data streams) New ColumnStore Data Processing Pipeline • Improvements in data processing pipeline provided by the remote mcsimport
  • 44. 44MySQL Triangle Meetup| 2019-01-24 44MariaDB OpenWorks Conference | 2019-02-26 vs. Traditional InfiniDB/CS Data Processing Pipeline OLTP Files/XML Log Files Operational Source Data StagingorODSETL High-speedLoadUtility Ad-Hoc Dashboards Reports Notifications Users Staging Area Data Warehouse Data Warehouse and Metadata Management #6 Load New Data with Minimal Impact OLTP Files/XML Log Files Operational Source Data StagingorODSETL High-speedLoadUtility Ad-Hoc Dashboards Reports Notifications Users Staging Area Data Warehouse Data Warehouse and Metadata Management #6 Load New Data with Minimal Impact OLTP Files/XML Log Files Operational Source Data StagingorODSETL High-speedLoadUtility Ad-Hoc Dashboards Reports Notifications Users Staging Area Data Warehouse Data Warehouse and Metadata Management #6 Load New Data with Minimal Impact InfiniDB HA Staging Area for processed data local to InfiniDB (e.g. costly HA storage) HA Staging Area for raw data (e.g. data streams)
  • 45. 45MySQL Triangle Meetup| 2019-01-24 45MariaDB OpenWorks Conference | 2019-02-26 Online Schema Change while Streaming • Tight coupling of streaming applications to CS schema complicates schema changes • Redundant applications for data streaming enable schema changes without data loss • Prepare new application (e.g. mcsimport job.xml) for the new schema • e.g. describe new columns as <DefaultColumn colName="col7"/> • Stop the old application (mcsimport cron job) • Provide enough buffer (staging area) for the data • Use ColumnStore function (it takes time) select calonlinealter('alter table foo add column col7 int;'); alter table foo add column col7 int comment 'schema sync only'; • Start the new application (mcsimport cron job)
  • 46. 46MySQL Triangle Meetup| 2019-01-24 46MariaDB OpenWorks Conference | 2019-02-26 High Availability with ColumnStore Bulk Write SDK • By their nature, data streaming applications run continuously • Redundant applications could increase data streaming uptime, since if one application fails, a second application would still be running • How do you implement HA/failover between data streaming applications using bulk write SDK remotely? • MariaDB developers provide functions to view and clear table locks remotely • In case of MariaDB Server, transaction is rolled back upon client failure • Perhaps the MariaDB Platform X3 may implement a similar behavior for ColumnStore
  • 47. 47MySQL Triangle Meetup| 2019-01-24 47MariaDB OpenWorks Conference | 2019-02-26 More Open Source Benefits • Some ColumnStore features are documented as open source code • A failed cpimport may result in locks that can not be cleared with cleartablelock. Andrew Hutchings (MariaDB CS Lead) pointed to one useful option documented in such a way: • If your ColumnStore installation is running fine now these locks can be removed using a hidden cleartablelock option, '-l'. For example: /usr/local/mariadb/columnstore/bin/cleartablelock -l 1 • The downside is if the table really does exist and there was data to rollback, it cannot be rolled back any more. This is why we don't really publish this option. • I used this option for the table with reference data like customers https://groups.google.com/d/msg/mariadb-columnstore/B0fDukIgUzM/FUGBiZR7AgAJ
  • 48. 48MySQL Triangle Meetup| 2019-01-24 48MariaDB OpenWorks Conference | 2019-02-26 MariaDB ColumnStore Summary • Most analytics queries read only few of table columns • Works best for approximately incremental data, such as time series • Repeated aggregation works well since data could fit in memory • Features often overlooked: • ACID transactions enables heterogeneous redundancy for critical data • Geo-distributed cluster works stably • While streaming, you may change the schema without data loss • With caution, you may use hidden cleartablelock option • Latest bulk write SDK enables remote data upload
  • 49. 49MySQL Triangle Meetup| 2019-01-24 49MariaDB OpenWorks Conference | 2019-02-26 MariaDB TX 3.0 MariaDB Server 10.3 MariaDB MaxScale 2.2 InnoDB/MyRocks MariaDB AX 2.0 MariaDB Server 10.2 MariaDB MaxScale 2.2 ColumnStore 1.2 MariaDB Platform X3 MariaDB MaxScale 2.3 MariaDB Server 10.3 InnoDB/MyRocks MariaDB Server 10.3 ColumnStore 1.3 Conclusion • Wise past technology choices (MariaDB/Galera and InifiniDB) provided with a consolidated roadmap for future upgrades