ProxySQL provides native support for high availability solutions like PXC, InnoDB Cluster, and regular MySQL replication. It can monitor the health of nodes and redirect traffic away from unavailable or stale nodes, improving availability. It supports various topologies out of the box through host groups, health checks, and failure detection. ProxySQL helps implement robust HA architectures by integrating these functions and allowing automatic traffic redirection based on node status.
Best practice-high availability-solution-geo-distributed-finalMarco Tusa
Nowadays implementing different grades of business continuity for the data layer storage is a common requirement. When designing architectures that include MySQL as a data layer, we have different options to cover the required target. Nevertheless we still see a lot of confusion when in the need to properly cover concepts such as High Availability and Disaster Recovery. Confusion that often leads to improper architecture design and wrong solution implementation. This presentation aims to remove that confusion and provide clear guidelines when in the need to design a robust, flexible resilient architecture for your data layer.
Comparing high availability solutions with percona xtradb cluster and percona...Marco Tusa
Percona XtraDB Cluster (PXC) is currently the most popular solution for HA in the MySQL ecosystem, and any solutions Galera-based as PXC have been the only viable option when looking for a high grade of HA using synchronous replication.
But Oracle had intensively worked on making Group Replication more solid and easy to use.
It is time to identify if Group Replication and attached solutions, like InnoDB cluster, can compete or even replace solutions based on Galera.
This presentation will focus on comparing the two solutions and how they behave when serving basic HA problems.
Attendees will be able to get a clearer understanding of which solutions will serve them better, and in which cases.
Message Queuing on a Large Scale: IMVUs stateful real-time message queue for ...Jon Watte
These slides are the ones I presented at the 2011 Game Developer's Conference.
Social game and entertainment company IMVU built a real-time lightweight networked messaging back-end suitable for chat and social gaming. Here's how we did it!
HBaseCon 2015: OpenTSDB and AsyncHBase UpdateHBaseCon
OpenTSDB continues to scale along with HBase. A number of updates have been implemented to push writes over 2 million data points a second. Here we will discuss about HBase schema improvements, including salting, random UI assignment, and using append operations instead of puts. You'll also get AsyncHBase development updates about rate limiting, statistics, and security.
Best practice-high availability-solution-geo-distributed-finalMarco Tusa
Nowadays implementing different grades of business continuity for the data layer storage is a common requirement. When designing architectures that include MySQL as a data layer, we have different options to cover the required target. Nevertheless we still see a lot of confusion when in the need to properly cover concepts such as High Availability and Disaster Recovery. Confusion that often leads to improper architecture design and wrong solution implementation. This presentation aims to remove that confusion and provide clear guidelines when in the need to design a robust, flexible resilient architecture for your data layer.
Comparing high availability solutions with percona xtradb cluster and percona...Marco Tusa
Percona XtraDB Cluster (PXC) is currently the most popular solution for HA in the MySQL ecosystem, and any solutions Galera-based as PXC have been the only viable option when looking for a high grade of HA using synchronous replication.
But Oracle had intensively worked on making Group Replication more solid and easy to use.
It is time to identify if Group Replication and attached solutions, like InnoDB cluster, can compete or even replace solutions based on Galera.
This presentation will focus on comparing the two solutions and how they behave when serving basic HA problems.
Attendees will be able to get a clearer understanding of which solutions will serve them better, and in which cases.
Message Queuing on a Large Scale: IMVUs stateful real-time message queue for ...Jon Watte
These slides are the ones I presented at the 2011 Game Developer's Conference.
Social game and entertainment company IMVU built a real-time lightweight networked messaging back-end suitable for chat and social gaming. Here's how we did it!
HBaseCon 2015: OpenTSDB and AsyncHBase UpdateHBaseCon
OpenTSDB continues to scale along with HBase. A number of updates have been implemented to push writes over 2 million data points a second. Here we will discuss about HBase schema improvements, including salting, random UI assignment, and using append operations instead of puts. You'll also get AsyncHBase development updates about rate limiting, statistics, and security.
Large scale near real-time log indexing with Flume and SolrCloudDataWorks Summit
Apache Flume’s extensible architecture allows Cisco to stream system and application logs from worldwide production data centers to a central Hadoop cluster and Solr. This architecture enables a new level of scalable indexing so that a larger volume of logs is searchable within seconds. Using Solr 4.0′s near real time features together with Hadoop, we can execute mission critical tasks much quicker, improving our ability to meet tight SLAs. At the same time, using the same infrastructure, we can perform large-scale historical analysis and pattern extraction to help further improve our services. This talk will explore our infrastructure and decisions we?ve made to meet key requirements, i.e. high indexing load, high availability and disaster recovery. We will further explore other uses of Flume and SolrCloud within Cisco including dynamic event routing, parsing and multi-tenancy.
Apache Cassandra operations have the reputation to be simple on single datacenter deployments and / or low volume clusters but they become way more complex on high latency multi-datacenter clusters with high volume and / or high throughout: basic Apache Cassandra operations such as repairs, compactions or hints delivery can have dramatic consequences even on a healthy high latency multi-datacenter cluster.
In this presentation, Julien will go through Apache Cassandra mutli-datacenter concepts first then show multi-datacenter operations essentials in details: bootstrapping new nodes and / or datacenter, repairs strategy, Java GC tuning, OS tuning, Apache Cassandra configuration and monitoring.
Based on his 3 years experience managing a multi-datacenter cluster against Apache Cassandra 2.0, 2.1, 2.2 and 3.0, Julien will give you tips on how to anticipate and prevent / mitigate issues related to basic Apache Cassandra operations with a multi-datacenter cluster.
Apache Cassandra operations have the reputation to be quite simple against single datacenter clusters and / or low volume clusters but they become way more complex against high latency multi-datacenter clusters: basic operations such as repair, compaction or hints delivery can have dramatic consequences even on a healthy cluster.
In this presentation, Julien will go through Cassandra operations in details: bootstrapping new nodes and / or datacenter, repair strategies, compaction strategies, GC tuning, OS tuning, large batch of data removal and Apache Cassandra upgrade strategy.
Julien will give you tips and techniques on how to anticipate issues inherent to multi-datacenter cluster: how and what to monitor, hardware and network considerations as well as data model and application level bad design / anti-patterns that can affect your multi-datacenter cluster performances.
Cassandra Summit 2014: Performance Tuning Cassandra in AWSDataStax Academy
Presenters: Michael Nelson, Development Manager at FamilySearch
A recent research project at FamilySearch.org pushed Cassandra to very high scale and performance limits in AWS using a real application. Come see how we achieved 250K reads/sec with latencies under 5 milliseconds on a 400-core cluster holding 6 TB of data while maintaining transactional consistency for users. We'll cover tuning of Cassandra's caches, other server-side settings, client driver, AWS cluster placement and instance types, and the tradeoffs between regular & SSD storage.
This talk is one that I gave to the HPTS workshop in Asilomar in 2009. It describes the ideas behind micro-sharding and outlines how Katta can manage micro-shards.
Some builds and spacing are off because this was exported as power point from Keynote.
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseC4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2nZwuQF.
Peter Mattis talks about how Cockroach Labs addressed the complexity of distributed databases with CockroachDB. He gives a tour of CockroachDB’s internals, covering the usage of Raft for consensus, the challenges of data distribution, distributed transactions, distributed SQL execution, and distributed SQL optimizations. Filmed at qconnewyork.com.
Peter Mattis is the co-founder of Cockroach Labs where he works on a bit of everything, from low-level optimization of code to refining the overall design. He has worked on distributed systems, designing and implementing the original Gmail back-end search and storage system at Google and designing and implementing Colossus, the successor to Google's original distributed file system.
Improving HDFS Availability with Hadoop RPC Quality of ServiceMing Ma
Heavy users monopolizing cluster resources is a frequent cause of slowdown for others. With only one namenode and thousands of datanodes, any poorly written application is a potential distributed denial-of-service attack on namenode. In this talk, you will learn how to prevent slowdown from heavy users and poorly-written applications by enabling IPC Quality of Service (QoS), a new feature in Hadoop 2.6+. On Twitter’s and eBay’s production clusters, we’ve seen response times of 500 milliseconds with QoS off drop to 10 milliseconds with QoS on during heavy usage. We’ll cover how IPC QoS works and share our experience on how to tune performance.
Tales From The Front: An Architecture For Multi-Data Center Scalable Applicat...DataStax Academy
- Quick review of Cassandra functionality that applies to this use case
- Common Data Center and application architectures for highly available inventory applications, and why the were designed that way
- Cassandra implementations vis-a-vis infrastructure capabilities
The impedance mismatch: compromises made to fit into IT infrastructures designed and implemented with an old mindset
Apache Flume is a highly scalable, distributed, fault tolerant data collection framework for Apache Hadoop and Apache HBase. Flume is designed to transfer massive volumes of event data in a highly scalable way into HDFS or HBase. Flume is declarative and easy to configure and can easily be deployed to a large number of machines using configuration management systems like Puppet or Cloudera Manager. In this talk, we will cover the basic components of Flume, configuring and deploying flume. We will also briefly talk about the metrics Flume exposes, and the various ways in which these can be collected. Apache
Flume is a Top Level Project (TLP) at the Apache Software Foundation, and has made several releases since entering incubation in June, 2011. Flume graduated to become a TLP in July, 2012. The current release of Flume is Flume 1.3.1.
Presenter: Hari Shreedharan, PMC Member and Committer, Apache Flume, Software Engineer, Cloudera
Apache Cassandra operations have the reputation to be simple on single datacenter deployments and / or low volume clusters but they become way more complex on high latency multi-datacenter clusters with high volume and / or high throughout: basic Apache Cassandra operations such as repairs, compactions or hints delivery can have dramatic consequences even on a healthy high latency multi-datacenter cluster.
In this presentation, Julien will go through Apache Cassandra mutli-datacenter concepts first then show multi-datacenter operations essentials in details: bootstrapping new nodes and / or datacenter, repairs strategy, Java GC tuning, OS tuning, Apache Cassandra configuration and monitoring.
Based on his 3 years experience managing a multi-datacenter cluster against Apache Cassandra 2.0, 2.1, 2.2 and 3.0, Julien will give you tips on how to anticipate and prevent / mitigate issues related to basic Apache Cassandra operations with a multi-datacenter cluster.
About the Speaker
Julien Anguenot VP Software Engineering, iland Internet Solutions, Corp
Julien currently serves as iland's Vice President of Software Engineering. Prior to joining iland, Mr. Anguenot held tech leadership positions at several open source content management vendors and tech startups in Europe and in the U.S. Julien is a long time Open Source software advocate, contributor and speaker: Zope, ZODB, Nuxeo contributor, Zope and OpenStack foundations member, his talks includes Apache Con, Cassandra summit, OpenStack summit, The WWW Conference or still EuroPython.
Migrare la tua applicazione verso il cloud è estremamente semplice, sulla carta. La dura verità è che l'unico modo per sapere con certezza come si comporterà è testare con attenzione. Estrarre un benchmark on premises è già abbastanza difficile, ma il benchmarking nel cloud può diventare davvero complicato a causa delle restrizioni negli ambienti PaaS e per la mancanza di strumenti.
Raggiungimi in questa sessione e scopri come catturare un carico di lavoro da produzione, riprodurlo nel tuo database cloud e confrontare le prestazioni. Ti illustrerò la metodologia e gli strumenti per portare il tuo database nel cloud senza battere ciglio.
By Gianluca Sartori
"Wire Encryption In HDFS: Protect Your Data From Others, Not Yourself"
ApacheCon 2019, Las Vegas.
SPEAKERS: Chen Liang, Konstantin Shvachko. LinkedIn
Wire data encryption is a key component of the Hadoop Distributed File System (HDFS). HDFS can enforce different levels of data protection, allowing users to specify one based on their own needs. However, such enforcement comes in as an all-or-nothing feature. Namely, wire encryption is enforced either for all accesses or none. Since encryption bears a considerable performance cost, the all-or-nothing condition forces users to choose between 'faster but unencrypted' or 'encrypted but slower' for all clients. In our use case at LinkedIn, we would like to selectively expose fast unencrypted access to fully managed internal clients, which can be trusted, while only expose encrypted access to clients outside of the trusted circle with higher security risks. That way we minimize performance overhead for trusted internal clients while still securing data from potential outside threats. We re-evaluate the RPC encryption mechanism in HDFS. Our design extends HDFS NameNode to run on multiple ports. Depending on the configuration, connecting to different NameNode ports would end up with different levels of encryption protection. This protection then gets enforced for both NameNode RPC and the subsequent data transfers to/from DataNode. System administrators then need to set up a simple firewall rule to allow access to the unencrypted port only for internal clients and expose the encrypted port to the outside clients. This approach comes with minimum operational and performance overhead. The feature has been introduced to Apache Hadoop under HDFS-13541.
Cassandra is pretty awesome, sure I am biased, but it rocks. Always on, tuneable consistency and multi-master architecture? Let’s get our web scale on and build a highly available app that never goes down!
Hold on a second. There is one key piece of the puzzle that has a massive impact on your applications availability: the client driver.
In this talk we will go through the how to best configure your clients to make the most of failure handling and tuneable consistency in Cassandra.
The Hows and Whys of a Distributed SQL Database - Strange Loop 2017Alex Robinson
Until recently, developers have had to deal with some serious tradeoffs when picking a database technology. One could pick a SQL database and deal with their eventual scaling problems or pick a NoSQL database and have to work around their lack of transactions, strong consistency, and/or secondary indexes. However, a new class of distributed database engines is emerging that combines the transactional consistency guarantees of traditional relational databases with the horizontal scalability and high availability of popular NoSQL databases.
In this talk, we'll examine the history of databases to see how we got here, covering the motivations for this new class of systems and why developers should care about them. We'll then take a deep dive into the key design choices behind one open source distributed SQL database, CockroachDB, that enable it to offer such properties and compare them to past SQL and NoSQL designs. We will look specifically at how to achieve the easy deployment and management of a scalable, self-healing, strongly-consistent database with techniques such as dynamic sharding and rebalancing, consensus protocols, lock-free transactions, and more.
OpenStack Days East -- MySQL Options in OpenStackMatt Lord
In most production OpenStack installations, you want the backing metadata store to be highly available. For this, the de facto standard has become MySQL+Galera. In order to help you meet this basic use case even better, I will introduce you to the brand new native MySQL HA solution called MySQL Group Replication. This allows you to easily go from a single instance of MySQL to a MySQL service that's natively distributed and highly available, while eliminating the need for any third party library and implementations.
If you have an extremely large OpenStack installation in production, then you are likely to eventually run into write scaling issues and the metadata store itself can become a bottleneck. For this use case, MySQL NDB Cluster can allow you to linearly scale the metadata store as your needs grow. I will introduce you to the core features of MySQL NDB Cluster--which include in-memory OLTP, transparent sharding, and support for active/active multi-datacenter clusters--that will allow you to meet even the most demanding of use cases with ease.
Large scale near real-time log indexing with Flume and SolrCloudDataWorks Summit
Apache Flume’s extensible architecture allows Cisco to stream system and application logs from worldwide production data centers to a central Hadoop cluster and Solr. This architecture enables a new level of scalable indexing so that a larger volume of logs is searchable within seconds. Using Solr 4.0′s near real time features together with Hadoop, we can execute mission critical tasks much quicker, improving our ability to meet tight SLAs. At the same time, using the same infrastructure, we can perform large-scale historical analysis and pattern extraction to help further improve our services. This talk will explore our infrastructure and decisions we?ve made to meet key requirements, i.e. high indexing load, high availability and disaster recovery. We will further explore other uses of Flume and SolrCloud within Cisco including dynamic event routing, parsing and multi-tenancy.
Apache Cassandra operations have the reputation to be simple on single datacenter deployments and / or low volume clusters but they become way more complex on high latency multi-datacenter clusters with high volume and / or high throughout: basic Apache Cassandra operations such as repairs, compactions or hints delivery can have dramatic consequences even on a healthy high latency multi-datacenter cluster.
In this presentation, Julien will go through Apache Cassandra mutli-datacenter concepts first then show multi-datacenter operations essentials in details: bootstrapping new nodes and / or datacenter, repairs strategy, Java GC tuning, OS tuning, Apache Cassandra configuration and monitoring.
Based on his 3 years experience managing a multi-datacenter cluster against Apache Cassandra 2.0, 2.1, 2.2 and 3.0, Julien will give you tips on how to anticipate and prevent / mitigate issues related to basic Apache Cassandra operations with a multi-datacenter cluster.
Apache Cassandra operations have the reputation to be quite simple against single datacenter clusters and / or low volume clusters but they become way more complex against high latency multi-datacenter clusters: basic operations such as repair, compaction or hints delivery can have dramatic consequences even on a healthy cluster.
In this presentation, Julien will go through Cassandra operations in details: bootstrapping new nodes and / or datacenter, repair strategies, compaction strategies, GC tuning, OS tuning, large batch of data removal and Apache Cassandra upgrade strategy.
Julien will give you tips and techniques on how to anticipate issues inherent to multi-datacenter cluster: how and what to monitor, hardware and network considerations as well as data model and application level bad design / anti-patterns that can affect your multi-datacenter cluster performances.
Cassandra Summit 2014: Performance Tuning Cassandra in AWSDataStax Academy
Presenters: Michael Nelson, Development Manager at FamilySearch
A recent research project at FamilySearch.org pushed Cassandra to very high scale and performance limits in AWS using a real application. Come see how we achieved 250K reads/sec with latencies under 5 milliseconds on a 400-core cluster holding 6 TB of data while maintaining transactional consistency for users. We'll cover tuning of Cassandra's caches, other server-side settings, client driver, AWS cluster placement and instance types, and the tradeoffs between regular & SSD storage.
This talk is one that I gave to the HPTS workshop in Asilomar in 2009. It describes the ideas behind micro-sharding and outlines how Katta can manage micro-shards.
Some builds and spacing are off because this was exported as power point from Keynote.
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseC4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2nZwuQF.
Peter Mattis talks about how Cockroach Labs addressed the complexity of distributed databases with CockroachDB. He gives a tour of CockroachDB’s internals, covering the usage of Raft for consensus, the challenges of data distribution, distributed transactions, distributed SQL execution, and distributed SQL optimizations. Filmed at qconnewyork.com.
Peter Mattis is the co-founder of Cockroach Labs where he works on a bit of everything, from low-level optimization of code to refining the overall design. He has worked on distributed systems, designing and implementing the original Gmail back-end search and storage system at Google and designing and implementing Colossus, the successor to Google's original distributed file system.
Improving HDFS Availability with Hadoop RPC Quality of ServiceMing Ma
Heavy users monopolizing cluster resources is a frequent cause of slowdown for others. With only one namenode and thousands of datanodes, any poorly written application is a potential distributed denial-of-service attack on namenode. In this talk, you will learn how to prevent slowdown from heavy users and poorly-written applications by enabling IPC Quality of Service (QoS), a new feature in Hadoop 2.6+. On Twitter’s and eBay’s production clusters, we’ve seen response times of 500 milliseconds with QoS off drop to 10 milliseconds with QoS on during heavy usage. We’ll cover how IPC QoS works and share our experience on how to tune performance.
Tales From The Front: An Architecture For Multi-Data Center Scalable Applicat...DataStax Academy
- Quick review of Cassandra functionality that applies to this use case
- Common Data Center and application architectures for highly available inventory applications, and why the were designed that way
- Cassandra implementations vis-a-vis infrastructure capabilities
The impedance mismatch: compromises made to fit into IT infrastructures designed and implemented with an old mindset
Apache Flume is a highly scalable, distributed, fault tolerant data collection framework for Apache Hadoop and Apache HBase. Flume is designed to transfer massive volumes of event data in a highly scalable way into HDFS or HBase. Flume is declarative and easy to configure and can easily be deployed to a large number of machines using configuration management systems like Puppet or Cloudera Manager. In this talk, we will cover the basic components of Flume, configuring and deploying flume. We will also briefly talk about the metrics Flume exposes, and the various ways in which these can be collected. Apache
Flume is a Top Level Project (TLP) at the Apache Software Foundation, and has made several releases since entering incubation in June, 2011. Flume graduated to become a TLP in July, 2012. The current release of Flume is Flume 1.3.1.
Presenter: Hari Shreedharan, PMC Member and Committer, Apache Flume, Software Engineer, Cloudera
Apache Cassandra operations have the reputation to be simple on single datacenter deployments and / or low volume clusters but they become way more complex on high latency multi-datacenter clusters with high volume and / or high throughout: basic Apache Cassandra operations such as repairs, compactions or hints delivery can have dramatic consequences even on a healthy high latency multi-datacenter cluster.
In this presentation, Julien will go through Apache Cassandra mutli-datacenter concepts first then show multi-datacenter operations essentials in details: bootstrapping new nodes and / or datacenter, repairs strategy, Java GC tuning, OS tuning, Apache Cassandra configuration and monitoring.
Based on his 3 years experience managing a multi-datacenter cluster against Apache Cassandra 2.0, 2.1, 2.2 and 3.0, Julien will give you tips on how to anticipate and prevent / mitigate issues related to basic Apache Cassandra operations with a multi-datacenter cluster.
About the Speaker
Julien Anguenot VP Software Engineering, iland Internet Solutions, Corp
Julien currently serves as iland's Vice President of Software Engineering. Prior to joining iland, Mr. Anguenot held tech leadership positions at several open source content management vendors and tech startups in Europe and in the U.S. Julien is a long time Open Source software advocate, contributor and speaker: Zope, ZODB, Nuxeo contributor, Zope and OpenStack foundations member, his talks includes Apache Con, Cassandra summit, OpenStack summit, The WWW Conference or still EuroPython.
Migrare la tua applicazione verso il cloud è estremamente semplice, sulla carta. La dura verità è che l'unico modo per sapere con certezza come si comporterà è testare con attenzione. Estrarre un benchmark on premises è già abbastanza difficile, ma il benchmarking nel cloud può diventare davvero complicato a causa delle restrizioni negli ambienti PaaS e per la mancanza di strumenti.
Raggiungimi in questa sessione e scopri come catturare un carico di lavoro da produzione, riprodurlo nel tuo database cloud e confrontare le prestazioni. Ti illustrerò la metodologia e gli strumenti per portare il tuo database nel cloud senza battere ciglio.
By Gianluca Sartori
"Wire Encryption In HDFS: Protect Your Data From Others, Not Yourself"
ApacheCon 2019, Las Vegas.
SPEAKERS: Chen Liang, Konstantin Shvachko. LinkedIn
Wire data encryption is a key component of the Hadoop Distributed File System (HDFS). HDFS can enforce different levels of data protection, allowing users to specify one based on their own needs. However, such enforcement comes in as an all-or-nothing feature. Namely, wire encryption is enforced either for all accesses or none. Since encryption bears a considerable performance cost, the all-or-nothing condition forces users to choose between 'faster but unencrypted' or 'encrypted but slower' for all clients. In our use case at LinkedIn, we would like to selectively expose fast unencrypted access to fully managed internal clients, which can be trusted, while only expose encrypted access to clients outside of the trusted circle with higher security risks. That way we minimize performance overhead for trusted internal clients while still securing data from potential outside threats. We re-evaluate the RPC encryption mechanism in HDFS. Our design extends HDFS NameNode to run on multiple ports. Depending on the configuration, connecting to different NameNode ports would end up with different levels of encryption protection. This protection then gets enforced for both NameNode RPC and the subsequent data transfers to/from DataNode. System administrators then need to set up a simple firewall rule to allow access to the unencrypted port only for internal clients and expose the encrypted port to the outside clients. This approach comes with minimum operational and performance overhead. The feature has been introduced to Apache Hadoop under HDFS-13541.
Cassandra is pretty awesome, sure I am biased, but it rocks. Always on, tuneable consistency and multi-master architecture? Let’s get our web scale on and build a highly available app that never goes down!
Hold on a second. There is one key piece of the puzzle that has a massive impact on your applications availability: the client driver.
In this talk we will go through the how to best configure your clients to make the most of failure handling and tuneable consistency in Cassandra.
The Hows and Whys of a Distributed SQL Database - Strange Loop 2017Alex Robinson
Until recently, developers have had to deal with some serious tradeoffs when picking a database technology. One could pick a SQL database and deal with their eventual scaling problems or pick a NoSQL database and have to work around their lack of transactions, strong consistency, and/or secondary indexes. However, a new class of distributed database engines is emerging that combines the transactional consistency guarantees of traditional relational databases with the horizontal scalability and high availability of popular NoSQL databases.
In this talk, we'll examine the history of databases to see how we got here, covering the motivations for this new class of systems and why developers should care about them. We'll then take a deep dive into the key design choices behind one open source distributed SQL database, CockroachDB, that enable it to offer such properties and compare them to past SQL and NoSQL designs. We will look specifically at how to achieve the easy deployment and management of a scalable, self-healing, strongly-consistent database with techniques such as dynamic sharding and rebalancing, consensus protocols, lock-free transactions, and more.
OpenStack Days East -- MySQL Options in OpenStackMatt Lord
In most production OpenStack installations, you want the backing metadata store to be highly available. For this, the de facto standard has become MySQL+Galera. In order to help you meet this basic use case even better, I will introduce you to the brand new native MySQL HA solution called MySQL Group Replication. This allows you to easily go from a single instance of MySQL to a MySQL service that's natively distributed and highly available, while eliminating the need for any third party library and implementations.
If you have an extremely large OpenStack installation in production, then you are likely to eventually run into write scaling issues and the metadata store itself can become a bottleneck. For this use case, MySQL NDB Cluster can allow you to linearly scale the metadata store as your needs grow. I will introduce you to the core features of MySQL NDB Cluster--which include in-memory OLTP, transparent sharding, and support for active/active multi-datacenter clusters--that will allow you to meet even the most demanding of use cases with ease.
Partitioning is the process of splitting your data into multiple Redis instances, so that every instance will only contain a subset of your keys. The first part of this document will introduce you to the concept of partitioning, the second part will show you the alternatives for Redis partitioning.
Navigating Transactions: ACID Complexity in Modern DatabasesShivji Kumar Jha
Transactions are anything but straightforward, with each database vendor offering its unique interpretation of the term. By scrutinising the internal architectures of these databases, engineers can gain valuable insights, enabling them to write more stable applications.This talk explores the intricacies of transactions, focusing on modern databases. Delving into distributed transactions, we discuss network challenges and cloud deployments in the contemporary era. The session provides a concise examination of the internal architectures of cloud-scale, multi-tenant databases such as Spanner, DynamoDB, and Amazon Aurora.
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...Mydbops
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open Source Database Meetup 15
Shivji explores the evolution of transactions, implementation challenges, and insights into distributed database environments. Whether you're a database enthusiast or a tech enthusiast, this presentation offers valuable insights into the world of database management.
Contents:
• Historical perspective of transactions
• Implementing transactions
• Challenges and trade-offs in ACID properties
• Distributed transactions in modern databases like Amazon Aurora, DynamoDB, and Google Spanner
Key Takeaways:
• Understanding the evolution of transactions in databases
• Insights into the challenges of implementing ACID properties
• Exploration of distributed transaction models in leading database systems
The MySQL High Availability Landscape and where Galera Cluster fits inSakari Keskitalo
The MySQL Server High Availability landscape provides with a slew of tools to help you ensure that your databases keep humming. Such tools are Galera Cluster, however, it is worth looking at semi-synchronous replication with failover tools, and also the usual leader-follower asynchronous replication. Today more workloads are moving to the cloud, and what failover options do you get with Amazon RDS for MySQL or even Aurora? What about the newfangled group replication and InnoDB Cluster? And let us not forget that Galera Cluster has spawned branches too!
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #2: Galera ClusterContinuent
Galera Cluster vs. Continuent Tungsten Clusters
Building a Geo-Scale, Multi-Region and Highly Available MySQL Cloud Back-End
This second installment of our High Noon series of on-demand webinars is focused on Galera Cluster (including MariaDB Cluster & Percona XtraDB Cluster). It looks at some of the key characteristics of Galera Cluster and how it fares as a MySQL HA / DR / Geo-Scale solution, especially when compared to Continuent Tungsten Clustering.
Watch this webinar to learn how to do better MySQL HA / DR / Geo-Scale.
AGENDA
- Goals for the High Noon Webinar Series
- High Noon Series: Tungsten Clustering vs Others
- Galera Cluster (aka MariaDB Cluster & Percona XtraDB Cluster)
- Key Characteristics
- Certification-based Replication
- Galera Multi-Site Requirements
- Limitations Using Galera Cluster
- How to do better MySQL HA / DR / Geo-Scale?
- Galera Cluster vs Tungsten Clustering
- About Continuent & Its Solutions
PRESENTER
Matthew Lang - Customer Success Director – Americas, Continuent - has over 25 years of experience in database administration, database programming, and system architecture, including the creation of a database replication product that is still in use today. He has designed highly available, scaleable systems that have allowed startups to quickly become enterprise organizations, utilizing a variety of technologies including open source projects, virtualization and cloud.
HBaseCon 2013: Using Coprocessors to Index Columns in an Elasticsearch Cluster Cloudera, Inc.
This presentation explores the design and challenges HappiestMinds faced while implementing a storage and search infrastructure for a large publisher where books/documents/artifacts related records are stored in Apache HBase. Upon bulk insert of book records into HBase, the Elasticsearch index is built offline using MapReduce code but there are certain use cases where the records need to be re-indexed in Elasticsearch using Region Observer Coprocessors.
SolrCloud in Public Cloud: Scaling Compute Independently from Storage - Ilan ...Lucidworks
Running SolrCloud in Public Cloud is the future. This presentation and the code that will be contributed back to the community will allow such clusters to be highly efficient, scalable and elastic. Attendees will understand the challenges and potential of sharing index data between servers.
Speakers: Ilan Ginzburg & Yonik Seeley, Salesforce
Everything you always wanted to know about Distributed databases, at devoxx l...javier ramirez
Everything you always wanted to know about Distributed databases, at devoxx london, by javier ramirez, teowaki.
Basic concepts of distributed systems, such as consensus, gossip and infection protocols, vector clocks, sharding storage, so you can create highly available distributed systems
Percona xtra db cluster(pxc) non blocking operations, what you need to know t...Marco Tusa
Performing simple DDL operations as ADD/DROP INDEX in a tightly connected cluster as PXC, can become a nightmare. Metalock will prevent Data modifications for long period of time and to bypass this, we need to become creative, like using Rolling schema upgrade or Percona online-schema-change. With NBO, we will be able to avoid such craziness at least for a simple operation like adding an index. In this brief talk I will illustrate what you should do to see the negative effect of NON using NBO, as well what you should do to use it correctly and what to expect out of it.
The constant pressure to move DATA in containers and Kubernetes is creating a lot of confusion and misunderstanding.
This is particularly dangerous when talking about Relational Database Management System.
MySQL, as well as Oracle, Postgres or SQL Server, is a RDBM, as such subject to the erroneous interpretation caused by this new crazy shining things that will solve all. In this short talk we will clarify, that first of all, we are not looking to something new and second why we need to be very careful when talking about using Kubernetes and containers for RDBMS.
Accessing data through hibernate: what DBAs should tell to developers and vic...Marco Tusa
Accessing data through Hibernate, what DBA should tell to developers by Marco Tusa & Francisco Bordenave
This presentation will go through the simple process of accessing data from a Java application. What actually happens when we use a simple direct connection, and what instead happen using an ORM/Persistent layer like hibernate. How this apparently makes programmers life easier and DBAs days more difficult.
In this presentation I am illustrating how and why InnodDB perform Merge and Split pages. I will also show what are the possible things to do to reduce the impact.
Secure our data is a complex topic. We can build a very strong protection around our data, but nothing will prevent the one WHO could potentially access it to compromise the data integrity or to expose it.
This because we either under estimate the control we can or should impose, or because we think to do not have the tools to perform such control.
Nowadays to be able to control and manage what can access our data is a must, while how to do with standard tools it is a nightmare.
The presentation will guide you in a journey, there you will discover how implementing a quite robust protection, more than what you thought was possible.
Even more, it is possible and your performances will even improve. Cool right?
We will discuss:
- Access using not standard port
- Implement selective query access
- Define accessibility by location/ip/id
- Reduce to minimum cost of filtering
- Automate the query discovery
Are we there Yet?? (The long journey of Migrating from close source to opens...Marco Tusa
Migrating from Oracle to MySQL or another Open source RDBMS like Postgres is not as straightforward as many think if not well guided. Check what it means doing with someone that has done it already.
Mysql8 advance tuning with resource groupMarco Tusa
I have a very noisy secondary application written by a very, very bad developer that accesses my servers, mostly with read queries, and occasionally with write updates. Reads and writes are obsessive and create an impact on the MAIN application. My task is to limit the impact of this secondary application without having the main one affected. To do that I will create two resource groups, one for WRITE and another for READ. The first group, Write_app2, will have no cpu affiliation, but will have lowest priority.
Advance Sharding Solution with ProxySQL
ProxySQL is a very powerful platform that allows us to manipulate and manage our connections and queries in a simple but effective way.
Historically MySQL lacks in sharding capability. This significant missing part had often cause developer do implement sharding at application level, or DBA/SA to move on to another solution.
ProxySQL comes with an elegant and simple solution that allow us to implement sharding capability with MySQL without the need to perform significant, or at all, changes in the code.
This brief presentation will illustrate how to successfully configure and use ProxySQL to perform sharding, from very simple approach based on connection user/ip/port, to complicate ones that see the need to read values inside queries.
Geographically dispersed perconaxtra db cluster deploymentMarco Tusa
Geographically Dispersed Percona XtraDB Cluster Deployment
Percona XtraDB Cluster is a very robust, high performing and widly used solution to answer to High Availability needs. But it can be very challinging when we are in the need to deploy the cluster over a geographically disperse area.
This presentation will briefely discuss what is the right approach to sucessfully deploy PXC when in the need to cover multiple geographical sites, close and far.
- What is PXC and what happens in a set of node when commit
- Let us clarify, geo dispersed
- What to keep in mind then
- how to measure it correctly
- Use the right way (sync/async)
- Use help like replication_manager
After some years, MySQL with Galera became the most common solution for synchronous replication. The cloud (and EC2 in particular) was one of the platforms that most successfully employed MySQL/Galera installations.
This year with Aurora, Amazon introduced an alternative solution that use all the flexibility of AWS and simplicity of RDS.
This presentation describes the behavior of both MySQL/Galera and Aurora, showing the details of how the two different solutions behave when dealing with same load. We will highlight the strong point of each, and which represents the best tool - depending on the needs of the situation.
Attendees will be able to make an informed decision on what kind of solutions will be the most efficient, in respect to their actual requirements.
Presentation shows how ProxySQL can improve the HA in solution like MySQL async and sync replication without the need to increase the platform complexity.
Scaling with sync_replication using Galera and EC2Marco Tusa
Challenging architecture design, and proof of concept on a real case of study using Syncrhomous solution.
Customer asks me to investigate and design MySQL architecture to support his application serving shops around the globe.
Scale out and scale in base to sales seasons.
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
As Europe's leading economic powerhouse and the fourth-largest hashtag#economy globally, Germany stands at the forefront of innovation and industrial might. Renowned for its precision engineering and high-tech sectors, Germany's economic structure is heavily supported by a robust service industry, accounting for approximately 68% of its GDP. This economic clout and strategic geopolitical stance position Germany as a focal point in the global cyber threat landscape.
In the face of escalating global tensions, particularly those emanating from geopolitical disputes with nations like hashtag#Russia and hashtag#China, hashtag#Germany has witnessed a significant uptick in targeted cyber operations. Our analysis indicates a marked increase in hashtag#cyberattack sophistication aimed at critical infrastructure and key industrial sectors. These attacks range from ransomware campaigns to hashtag#AdvancedPersistentThreats (hashtag#APTs), threatening national security and business integrity.
🔑 Key findings include:
🔍 Increased frequency and complexity of cyber threats.
🔍 Escalation of state-sponsored and criminally motivated cyber operations.
🔍 Active dark web exchanges of malicious tools and tactics.
Our comprehensive report delves into these challenges, using a blend of open-source and proprietary data collection techniques. By monitoring activity on critical networks and analyzing attack patterns, our team provides a detailed overview of the threats facing German entities.
This report aims to equip stakeholders across public and private sectors with the knowledge to enhance their defensive strategies, reduce exposure to cyber risks, and reinforce Germany's resilience against cyber threats.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...
Robust ha solutions with proxysql
1. 1
Robust HA Solutions - Native Support
for PXC and InnoDB Cluster in
ProxySQL
Marco Tusa
Percona
2. 2
2
• Open source enthusiast
• Principal architect
• Working in DB world over 25 years
• Open source developer and community contributor
About Me
3. 3
What this talk is about
High level description of High Availability
HA solutions
Misses from current vanilla solutions
Where ProxySQL fits
It is NOT about Performance
It is NOT an in depth description of HA or ProxySQL feature
It is NOT an implementation guide
4. 4
• The need and dimension of HA or DR is related to the real
need of your business.
• We are pathologically online/connected, and often we
expect to have over dimensioned HA or DR.
• Business needs often do not require all the components to
be ALWAYS available.
How We Need to think about HA and DR
5. 5
Why We Need HA and DR
Do:
• Business needs
• Technical challenges
• Supportable solutions
• Knowhow
Don’t:
• Choose based on the “shiny object”
• Pick something you know nothing about
• Choose by yourself and push it up or down
• Use shortcuts, to accelerate deploying
time.
The first step to have a robust solution is to design the right solution for
your business.
6. 6
• High Availability is to
cover service
availability in a location
• 10 Gb Ethernet Best case
scenario
400 mt distance Max
HA vs DR
7. 7
• Disaster Recovery is to assure
we can restore service, in a
geographically different location
• Real speed may vary
• Linear distance ~1000Km
HA vs DR
8. 8
HA vs DR
DO NOT MIX
the solutions in your architecture
http://www.tusacentral.com/joomla/index.php/mysql-blogs/204-how-not-to-do-mysql-high-availability-geographic-node-distribution-with-galera-based-replication-misuse
http://www.tusacentral.com/joomla/index.php/mysql-blogs/205-mysql-high-availability-on-premises-a-geographically-distributed-scenario
9. 9
Replicate data is the key - Sync VS Async
1
Data state
3
Different
Data state
10. 10
Data Replication is the Base
Tightly coupled database clusters
• Datacentric approach (single state
of the data, distributed commit)
• Data is consistent in time cross
nodes
• Replication requires high
performant link
• Geographic distribution is
forbidden
• DR is not supported
Loosely coupled database clusters
• Single node approach (local commit)
• Data state differs by node
• Single node state does not affect the
cluster
• Replication link doesn’t need to be
high performance
• Geographic distribution is allow
• DR is supported
11. 11
In MySQL ecosystem we have different solutions
• NDB
• Galera replication
• InnoDB group replication
• Basic Replication
Different solutions
13. 13
Good thing:
• Very High availability (five 9s)
• Write scale (for real not a fake)
• Read scale
• Data is in sync all the time
Bad things:
• Too be very efficient should
stay in memory as much as possible
• Complex to manage
• Require deep understanding also from application point of view
Understand solutions: NDB
14. 14
Replication by internal design
Good thing:
• Highly available
• Read scale
• Data is in almost in sync
all the time
Bad things:
• Doesn’t Scale writes
• More nodes more internal
overhead
• Network is a very impacting factor
• 1 Master only (PLEASE!!!!!)
Understand solutions: Galera
15. 15
Understand solutions: Group replication
Replication by standard MySQL
Good thing:
• Highly available
• Read scale
• Data is in almost in sync
all the time
• More network tolerant
Bad things:
• Doesn’t Scale writes
• 1 Master only (PLEASE!!!!!)
When something ends to be a new things?
16. 16
Understand solutions: Replication
Replication by internal design
Good thing:
• Each node is independent
• Read scale
• Low network impact
Bad things:
• Stale reads
• Low HA
• Consistent only in the master
• Each node has its own data state
17. 17
1. No embedded R/W split
• Writes and reads are bound to 1 node
• Failing read channel will cause total failure
2. No recognition of possible stale reads
• In case of replication delay, application can get wrong data
3. No node failure identification and no single entry point
• Architecture must implement a double mechanism to check nodes
status (topology manager) and entry point like VIP
• Application must use a connector that allow multiple entry point in case
of issue. (like Java connector)
What they have in common?
18. 18
1. Embedded R/W split by group of servers (Host Group)
• Writes and reads are automatically redirect to working node(s)
2. Identify stale node with the binary log reader
• In case of replication delay, delayed node will not receive the read
requests
3. Embedded node failure, automatic redirection
• Proxy provide native status recognition for Galera and Group
replication
• Only when using basic replication, you need a topology manager
• Application connect to ProxySQL no need to add VIP
What ProxySQL can do to be better
19. 19
Native health checks
read_only
wsrep_local_recv_queue
wsrep_desync
wsrep_reject_queries
wsrep_sst_donor_rejects_queries
primary_partition
ProxySQL for Galera
• writer_hostgroup: the hostgroup ID that refers to the WRITER
• backup_writer_hostgroup: the hostgoup ID referring to the
Hostgorup that will contain the candidate servers
• reader_hostgroup: The reader Hostgroup ID, containing the list of
servers that need to be taken in consideration
• offline_hostgroup: The Hostgroup ID that will eventually contain the
writer that will be put OFFLINE
• active: True[1]/False[0] if this configuration needs to be used or not
• max_writers: This will contain the MAX number of writers you want
to have at the same time. In a sane setup this should be always 1,
but if you want to have multiple writers, you can define it up to the
number of nodes.
• writer_is_also_reader: If true [1] the Writer will NOT be removed
from the reader HG
• max_transactions_behind: The number
of wsrep_local_recv_queue after which the node will be set
OFFLINE. This must be carefully set, observing the node behaviour.
• comment: I suggest to put some meaningful notes to identify what is
what.
20. 20
I have these nodes
192.168.1.205 (Node1)
192.168.1.21 (Node2)
192.168.1.231 (node3)
ProxySQL for Galera
Will use these groups:
Writer HG-> 100
Reader HG-> 101
BackupW HG-> 102
offHG HG-> 9101
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.205’,100,3306,1000);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.205',101,3306,1000);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.21',101,3306,1000);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.231',101,3306,1000);
insert into mysql_galera_hostgroups (writer_hostgroup,backup_writer_hostgroup,reader_hostgroup,
offline_hostgroup,active,max_writers,writer_is_also_reader,max_transactions_behind) values
(100,102,101,9101,1,1,1,16);
24. 22
ProxySQL for Group Replication
Native health checks
Read_only
Viable_candidate
Transaction_behind
• writer_hostgroup: the hostgroup ID that refers to the WRITER
• backup_writer_hostgroup: the hostgoup ID referring to the
Hostgorup that will contain the candidate servers
• reader_hostgroup: The reader Hostgroup ID, containing the list of
servers that need to be taken in consideration
• offline_hostgroup: The Hostgroup ID that will eventually contain the
writer that will be put OFFLINE
• active: True[1]/False[0] if this configuration needs to be used or not
• max_writers: This will contain the MAX number of writers you want
to have at the same time. In a sane setup this should be always 1,
but if you want to have multiple writers, you can define it up to the
number of nodes.
• writer_is_also_reader: If true [1] the Writer will NOT be removed
from the reader HG
• max_transactions_behind: determines the maximum number of
transactions behind the writers that ProxySQL should allow before
shunning the node to prevent stale reads (this is determined by
querying the transactions_behind field of
the sys.gr_member_routing_candidate_status table in MySQL).
• comment: I suggest to put some meaningful notes to identify what is
what.
25. 23
I have these nodes
192.168.4.55 (Node1)
192.168.4.56 (Node2)
192.168.4.57 (node3)
ProxySQL for Group Replication
Will use these groups:
Writer HG-> 400
Reader HG-> 401
BackupW HG-> 402
offHG HG-> 9401
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,max_connections,comment) VALUES ('192.168.4.55',400,3306,10000,2000,'GR1');
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,max_connections,comment) VALUES ('192.168.4.55',401,3306,100,2000,'GR1');
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,max_connections,comment) VALUES ('192.168.4.56',401,3306,10000,2000,'GR2');
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,max_connections,comment) VALUES ('192.168.4.57',401,3306,10000,2000,'GR2');
insert into mysql_group_replication_hostgroups (writer_hostgroup,backup_writer_hostgroup,reader_hostgroup,
offline_hostgroup,active,max_writers,writer_is_also_reader,max_transactions_behind)
values (400,402,401,9401,1,1,1,100);
select * from mysql_server_group_replication_log order by 3 desc,1 limit 3 ;
+--------------+------+------------------+-----------------+------------------+-----------+---------------------+-------+
| hostname | port | time_start_us | success_time_us | viable_candidate | read_only | transactions_behind | error |
+--------------+------+------------------+-----------------+------------------+-----------+---------------------+-------+
| 192.168.4.57 | 3306 | 1569593421324355 | 3085 | YES | YES | 0 | NULL |
| 192.168.4.56 | 3306 | 1569593421321825 | 2889 | YES | YES | 0 | NULL |
| 192.168.4.55 | 3306 | 1569593421321435 | 2764 | YES | NO | 0 | NULL |
+--------------+------+------------------+-----------------+------------------+-----------+---------------------+-------+
34. 27
ProxySQL for MySQL Replication
Native health checks
MySQL ping
read_only
Latency
Stale READS with GTID
delete from mysql_servers where hostgroup_id in (500,501);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,max_connections,comment)
VALUES ('10.0.0.50',500,3306,10000,2000,'DC1 writer');
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,max_connections,comment)
VALUES ('10.0.0.50',501,3306,10000,2000,'DC1 Reader');
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,max_connections,comment)
VALUES ('10.0.0.51',501,3306,10000,2000,'DC1 Reader');
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,max_connections,comment)
VALUES ('10.0.0.52',501,3306,10000,2000,'DC1 Reader');
INSERT INTO mysql_replication_hostgroups VALUES (500,501,'read_only', 'Simple replica');
LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
Easy to monitor
select b.weight, c.* from stats_mysql_connection_pool c left JOIN runtime_mysql_servers b ON
c.hostgroup=b.hostgroup_id and c.srv_host=b.hostname and c.srv_port = b.port where hostgroup in (500,501)
order by hostgroup,srv_host desc;
select * from mysql_server_read_only_log order by time_start_us desc limit 3;
+------------+------+------------------+-----------------+-----------+-------+
| hostname | port | time_start_us | success_time_us | read_only | error |
+------------+------+------------------+-----------------+-----------+-------+
| 10.0.0.52 | 3306 | 1569604127710729 | 895 | 1 | NULL |
| 10.0.0.51 | 3306 | 1569604127697866 | 542 | 1 | NULL |
| 10.0.0.50 | 3306 | 1569604127685005 | 795 | 0 | NULL |
+------------+------+------------------+-----------------+-----------+-------+
39. 29
ProxySQL is cool because:
• Allow you to better distribute the load
• Reduce the service downtime shifting to available node(s)
• In case of writer down reads are served
• With Binlog reader allow u to identify node with stale data also in basic
replication
• More stuff that we are not covering here…
• Multiplexing; sharding; masking; firewalling etc..
In short your HA solution will be more resilient and flexible, which
means that your business will be safer.
Conclusions