At Pinterest, the ProxySQL infrastructure fronts numerous heterogeneous databases. Due to the high number of unique configurations and dynamic nature of cloud deployments, it’s a challenge to reliably provision, change, auto-scale, and monitor ProxySQL servers. Applying Infrastructure as Code principles using Terraform, we made it possible to manage such a large fleet of ProxySQL instances confidently. Come learn how we automated provisioning, testing, and monitoring of ProxySQL at scale.
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfJesmar Cannao'
ProxySQL is a MySQL protocol proxy that provides high availability, scalability, and security for MySQL database systems. It allows clients to connect to ProxySQL, which then evaluates requests and performs actions like routing queries to backend databases, caching reads, connection pooling, and load balancing across servers. ProxySQL's main features include query routing, firewalling, real-time statistics, monitoring, and management of large numbers of backend servers. The presentation discusses using ProxySQL's query routing and rewriting capabilities to mask sensitive data when replicating databases for development environments. It also covers using the REST API and Prometheus integration to configure ProxySQL and monitor metrics without direct SQL access.
This presentation describes how to configure and leverage ProxySQL with
AWS Aurora,
Azure Database for MySQL
and CloudSQL for MySQL.
It details the various benefits, configuration, and monitoring.
The presentation covers improvements made to the redo logs in MySQL 8.0 and their impact on the MySQL performance and Operations. This covers the MySQL version still MySQL 8.0.30.
ProxySQL High Avalability and Configuration Management OverviewRené Cannaò
The document provides an overview of high availability and configuration management options for ProxySQL. It discusses deploying ProxySQL locally on application servers, in a dedicated layer, or using both approaches. When deploying in a dedicated layer, options for high availability include keepalived, load balancers, Consul, and Kubernetes. Configuration can be managed through tools like Ansible, Puppet, or by loading SQL files. ProxySQL Cluster enables syncing configuration across nodes.
Percona XtraDB Cluster vs Galera Cluster vs MySQL Group ReplicationKenny Gryp
This document provides an overview of different database replication technologies including Galera Cluster, Percona XtraDB Cluster, and MySQL Group Replication. It discusses similarities between the technologies such as multi-master replication topologies and consistency models. Key differences are also outlined relating to node provisioning, failure handling, and operational limitations of each solution. Known issues uncovered through quality assurance testing are also briefly mentioned.
Docker Swarm allows managing Docker clusters remotely. The key components are swarm managers, swarm nodes, and a scheduler. Swarm managers oversee nodes in the cluster using Docker APIs. The scheduler uses strategies and filters to determine where to place containers on nodes. Discovery services help register and discover nodes in the cluster.
※다운로드하시면 더 선명한 자료를 보실 수 있습니다.
동접 200만 명이 접속할 수백 대의 게임 서버가 최소한의 MySQL 서버만으로 서비스할 수 있는 구조를 설명합니다.
고성능/고효율의 MySQL 스케일링 기법을 공유합니다. 대규모 게임 서비스에서 이미 검증된 것은 안 비밀~
목차
1. 기본적인 아기텍처
2. ProxySQL을 이용한 더 나은 아키텍처
3. 최종 아키텍처
대상
- 대규모 게임 서비스에 MySQL을 사용한 경험에 관심 있는 분
- ProxySQL에 관심이 있는 서버 개발자 혹은 DBA
- 게임 서버 개발 과정에서 DB 쪽을 유연하게 구성하고 싶은 분
■관련 동영상: https://youtu.be/8Eb_n7JA1yA
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfJesmar Cannao'
ProxySQL is a MySQL protocol proxy that provides high availability, scalability, and security for MySQL database systems. It allows clients to connect to ProxySQL, which then evaluates requests and performs actions like routing queries to backend databases, caching reads, connection pooling, and load balancing across servers. ProxySQL's main features include query routing, firewalling, real-time statistics, monitoring, and management of large numbers of backend servers. The presentation discusses using ProxySQL's query routing and rewriting capabilities to mask sensitive data when replicating databases for development environments. It also covers using the REST API and Prometheus integration to configure ProxySQL and monitor metrics without direct SQL access.
This presentation describes how to configure and leverage ProxySQL with
AWS Aurora,
Azure Database for MySQL
and CloudSQL for MySQL.
It details the various benefits, configuration, and monitoring.
The presentation covers improvements made to the redo logs in MySQL 8.0 and their impact on the MySQL performance and Operations. This covers the MySQL version still MySQL 8.0.30.
ProxySQL High Avalability and Configuration Management OverviewRené Cannaò
The document provides an overview of high availability and configuration management options for ProxySQL. It discusses deploying ProxySQL locally on application servers, in a dedicated layer, or using both approaches. When deploying in a dedicated layer, options for high availability include keepalived, load balancers, Consul, and Kubernetes. Configuration can be managed through tools like Ansible, Puppet, or by loading SQL files. ProxySQL Cluster enables syncing configuration across nodes.
Percona XtraDB Cluster vs Galera Cluster vs MySQL Group ReplicationKenny Gryp
This document provides an overview of different database replication technologies including Galera Cluster, Percona XtraDB Cluster, and MySQL Group Replication. It discusses similarities between the technologies such as multi-master replication topologies and consistency models. Key differences are also outlined relating to node provisioning, failure handling, and operational limitations of each solution. Known issues uncovered through quality assurance testing are also briefly mentioned.
Docker Swarm allows managing Docker clusters remotely. The key components are swarm managers, swarm nodes, and a scheduler. Swarm managers oversee nodes in the cluster using Docker APIs. The scheduler uses strategies and filters to determine where to place containers on nodes. Discovery services help register and discover nodes in the cluster.
※다운로드하시면 더 선명한 자료를 보실 수 있습니다.
동접 200만 명이 접속할 수백 대의 게임 서버가 최소한의 MySQL 서버만으로 서비스할 수 있는 구조를 설명합니다.
고성능/고효율의 MySQL 스케일링 기법을 공유합니다. 대규모 게임 서비스에서 이미 검증된 것은 안 비밀~
목차
1. 기본적인 아기텍처
2. ProxySQL을 이용한 더 나은 아키텍처
3. 최종 아키텍처
대상
- 대규모 게임 서비스에 MySQL을 사용한 경험에 관심 있는 분
- ProxySQL에 관심이 있는 서버 개발자 혹은 DBA
- 게임 서버 개발 과정에서 DB 쪽을 유연하게 구성하고 싶은 분
■관련 동영상: https://youtu.be/8Eb_n7JA1yA
Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?Miguel Araújo
MySQL InnoDB ClusterSet brings multi-datacenter capabilities to our solutions and make it very easy to setup a disaster recovery architecture. Think multiple MySQL InnoDB Clusters into one single database architecture, fully managed from MySQL Shell and with full MySQL Router integration to make it easy to access the entire architecture.
This presentation covers:
- The various features of InnoDB Clusterset
- How to setup MySQL InnoDB ClusterSet
- Ways to migrate from an existing MySQL InnoDB Cluster into MySQL InnoDB ClusterSet
- How to deal with various failures
- The various features of router integration which makes connection to the database architecture easy.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
This document provides an overview and agenda for a Docker networking deep dive presentation. The presentation covers key concepts in Docker networking including libnetwork, the Container Networking Model (CNM), multi-host networking capabilities, service discovery, load balancing, and new features in Docker 1.12 like routing mesh and secured control/data planes. The agenda demonstrates Docker networking use cases like default bridge networks, user-defined bridge networks, and overlay networks. It also covers networking drivers, Docker 1.12 swarm mode networking functionality, and how concepts like routing mesh and load balancing work.
ProxySQL Cluster: Challenges and Solutions To Synchronize Configuration Across Multiple Decentralized Cluster Nodes
As a reverse proxy for MySQL databases, ProxySQL is being used in various infrastructure sizes, and it is not surprising to see deployments with thousands of nodes running. Such large deployments introduce some interesting challenges because nodes can be initialized or destroyed at any time.
This session will describe the challenges in configuring such large deployments of nodes, the most common external tools to configure ProxySQL, and then focus on improved ProxySQL native clustering solutions that allow auto-discovery, bootstrap, and a distributed decentralized reconfiguration.
This presentation covers how app deployment model evolved from bare metal servers to Kubernetes World.
In addition to theoretical information, you will find free KATACODA workshops url to perform practices to understand the details of the each topics.
Optimizing MariaDB for maximum performanceMariaDB plc
When it comes to optimizing the performance of a database, DBAs have to look at everything from the OS to the network. In this session, MariaDB Enterprise Architect Manjot Singh shares best practices for getting the most out of MariaDB. He highlights recommended OS settings, important configuration and tuning parameters, options for improving replication and clustering performance and features such as query result caching.
This is the presentation delivered by Karthik.P.R at MySQL User Camp Bangalore on 09th June 2017. ProxySQL is a high performance MySQL Load Balancer Designed to scale database servers.
How Netflix Tunes EC2 Instances for PerformanceBrendan Gregg
CMP325 talk for AWS re:Invent 2017, by Brendan Gregg. "
At Netflix we make the best use of AWS EC2 instance types and features to create a high performance cloud, achieving near bare metal speed for our workloads. This session will summarize the configuration, tuning, and activities for delivering the fastest possible EC2 instances, and will help other EC2 users improve performance, reduce latency outliers, and make better use of EC2 features. We'll show how we choose EC2 instance types, how we choose between EC2 Xen modes: HVM, PV, and PVHVM, and the importance of EC2 features such SR-IOV for bare-metal performance. SR-IOV is used by EC2 enhanced networking, and recently for the new i3 instance type for enhanced disk performance as well. We'll also cover kernel tuning and observability tools, from basic to advanced. Advanced performance analysis includes the use of Java and Node.js flame graphs, and the new EC2 Performance Monitoring Counter (PMC) feature released this year."
All about Zookeeper and ClickHouse Keeper.pdfAltinity Ltd
ClickHouse clusters depend on ZooKeeper to handle replication and distributed DDL commands. In this Altinity webinar, we’ll explain why ZooKeeper is necessary, how it works, and introduce the new built-in replacement named ClickHouse Keeper. You’ll learn practical tips to care for ZooKeeper in sickness and health. You’ll also learn how/when to use ClickHouse Keeper. We will share our recommendations for keeping that happy as well.
This document provides an overview of HashiCorp Vault for securely storing, accessing, and managing secrets. It discusses how Vault can be used to securely store secrets like API keys, passwords, and certificates. The document outlines Vault's architecture, data storage options, authentication methods, policies for access control, and integrating systems using Vault. It also provides an agenda for a demonstration of Spring Cloud Vault integration for retrieving database credentials from Vault and using them to connect to a MySQL database.
Maxscale switchover, failover, and auto rejoinWagner Bianchi
How the MariaDB Maxscale Switchover, Failover, and Rejoin works under the hood by Esa Korhonen and Wagner Bianchi.
You can watch the video of the presentation at
https://www.linkedin.com/feed/update/urn:li:activity:6381185640607809536
A comprehensive walkthrough of how to manage infrastructure-as-code using Terraform. This presentation includes an introduction to Terraform, a discussion of how to manage Terraform state, how to use Terraform modules, an overview of best practices (e.g. isolation, versioning, loops, if-statements), and a list of gotchas to look out for.
For a written and more in-depth version of this presentation, check out the "Comprehensive Guide to Terraform" blog post series: https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca
Advanced Percona XtraDB Cluster in a nutshell... la suiteKenny Gryp
This document provides a hands-on tutorial for advanced Percona XtraDB Cluster users. It discusses setting up a 3 node PXC cluster environment in VirtualBox and bootstrapping the initial cluster. It then covers topics like avoiding state snapshot transfers when restarting MySQL, recovering from clean and unclean shutdowns, and reproducing and diagnosing different types of conflicts through examples.
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIAltinity Ltd
Graham Mainwaring and Robert Hodges summarize management of ClickHouse on Kubernetes using the ClickHouse Kubernetes Operator and introduce a new UI for it. Presented at the 15 Dec '22 SF Bay Area ClickHouse Meetup.
This document discusses using KVM as the hypervisor with CloudStack. It provides background on KVM, describes how to install and configure it with CloudStack, and covers networking, storage, management, and troubleshooting. Specifically, it outlines installing required components, configuring Linux bridges or Open vSwitch for networking, options for primary storage, and tools for managing virtual machines and troubleshooting KVM.
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...Severalnines
This presentation by Krzysztof Książek at Percona Live 2017 in Santa Clara, California gives detailed descriptions and comparisons of the leading open source database load balancing technologies
24시간 365일 서비스를 위한 MySQL DB 이중화.
MySQL 이중화 방안들에 대해 알아보고 운영하면서 겪은 고민들을 이야기해 봅니다.
목차
1. DB 이중화 필요성
2. 이중화 방안
- HW 이중화
- MySQL Replication 이중화
3. 이중화 운영 장애
4. DNS와 VIP
5. MySQL 이중화 솔루션 비교
대상
- MySQL을 서비스하고 있는 인프라 담당자
- MySQL 이중화에 관심 있는 개발자
In this tutorial, we cover the different deployment possibilities of the MySQL architecture depending on the business requirements for the data. We also deploy some architecture and see how to evolve to the next one.
The tutorial covers the new MySQL Solutions like InnoDB ReplicaSet, InnoDB Cluster, and InnoDB ClusterSet.
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-RegionJi-Woong Choi
OpenStack Ceph & Neutron에 대한 설명을 담고 있습니다.
1. OpenStack
2. How to create instance
3. Ceph
- Ceph
- OpenStack with Ceph
4. Neutron
- Neutron
- How neutron works
5. OpenStack HA- controller- l3 agent
6. OpenStack multi-region
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfAltinity Ltd
Join the Altinity experts as we dig into ClickHouse sharding and replication, showing how they enable clusters that deliver fast queries over petabytes of data. We’ll start with basic definitions of each, then move to practical issues. This includes the setup of shards and replicas, defining schema, choosing sharding keys, loading data, and writing distributed queries. We’ll finish up with tips on performance optimization.
#ClickHouse #datasets #ClickHouseTutorial #opensource #ClickHouseCommunity #Altinity
-----------------
Join ClickHouse Meetups: https://www.meetup.com/San-Francisco-...
Check out more ClickHouse resources: https://altinity.com/resources/
Visit the Altinity Documentation site: https://docs.altinity.com/
Contribute to ClickHouse Knowledge Base: https://kb.altinity.com/
Join the ClickHouse Reddit community: https://www.reddit.com/r/Clickhouse/
----------------
Learn more about Altinity!
Site: https://www.altinity.com
LinkedIn: https://www.linkedin.com/company/alti...
Twitter: https://twitter.com/AltinityDB
Easy Cloud Native Transformation using HashiCorp NomadBram Vogelaar
Nomad is an open source tool for dynamic workload scheduling that can be used to easily transform applications to be cloud native. The document discusses how Nomad was used to deploy a personal blog to AWS, including writing a Nomad job file and using features like spreading tasks across hosts and auto-scaling. It also covers integrating Nomad with other HashiCorp tools like Consul for service discovery, Vault for access control, and the Nomad autoscaler for auto-scaling. Examples are provided of monitoring metrics with Prometheus and visualizing with Grafana.
Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?Miguel Araújo
MySQL InnoDB ClusterSet brings multi-datacenter capabilities to our solutions and make it very easy to setup a disaster recovery architecture. Think multiple MySQL InnoDB Clusters into one single database architecture, fully managed from MySQL Shell and with full MySQL Router integration to make it easy to access the entire architecture.
This presentation covers:
- The various features of InnoDB Clusterset
- How to setup MySQL InnoDB ClusterSet
- Ways to migrate from an existing MySQL InnoDB Cluster into MySQL InnoDB ClusterSet
- How to deal with various failures
- The various features of router integration which makes connection to the database architecture easy.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
This document provides an overview and agenda for a Docker networking deep dive presentation. The presentation covers key concepts in Docker networking including libnetwork, the Container Networking Model (CNM), multi-host networking capabilities, service discovery, load balancing, and new features in Docker 1.12 like routing mesh and secured control/data planes. The agenda demonstrates Docker networking use cases like default bridge networks, user-defined bridge networks, and overlay networks. It also covers networking drivers, Docker 1.12 swarm mode networking functionality, and how concepts like routing mesh and load balancing work.
ProxySQL Cluster: Challenges and Solutions To Synchronize Configuration Across Multiple Decentralized Cluster Nodes
As a reverse proxy for MySQL databases, ProxySQL is being used in various infrastructure sizes, and it is not surprising to see deployments with thousands of nodes running. Such large deployments introduce some interesting challenges because nodes can be initialized or destroyed at any time.
This session will describe the challenges in configuring such large deployments of nodes, the most common external tools to configure ProxySQL, and then focus on improved ProxySQL native clustering solutions that allow auto-discovery, bootstrap, and a distributed decentralized reconfiguration.
This presentation covers how app deployment model evolved from bare metal servers to Kubernetes World.
In addition to theoretical information, you will find free KATACODA workshops url to perform practices to understand the details of the each topics.
Optimizing MariaDB for maximum performanceMariaDB plc
When it comes to optimizing the performance of a database, DBAs have to look at everything from the OS to the network. In this session, MariaDB Enterprise Architect Manjot Singh shares best practices for getting the most out of MariaDB. He highlights recommended OS settings, important configuration and tuning parameters, options for improving replication and clustering performance and features such as query result caching.
This is the presentation delivered by Karthik.P.R at MySQL User Camp Bangalore on 09th June 2017. ProxySQL is a high performance MySQL Load Balancer Designed to scale database servers.
How Netflix Tunes EC2 Instances for PerformanceBrendan Gregg
CMP325 talk for AWS re:Invent 2017, by Brendan Gregg. "
At Netflix we make the best use of AWS EC2 instance types and features to create a high performance cloud, achieving near bare metal speed for our workloads. This session will summarize the configuration, tuning, and activities for delivering the fastest possible EC2 instances, and will help other EC2 users improve performance, reduce latency outliers, and make better use of EC2 features. We'll show how we choose EC2 instance types, how we choose between EC2 Xen modes: HVM, PV, and PVHVM, and the importance of EC2 features such SR-IOV for bare-metal performance. SR-IOV is used by EC2 enhanced networking, and recently for the new i3 instance type for enhanced disk performance as well. We'll also cover kernel tuning and observability tools, from basic to advanced. Advanced performance analysis includes the use of Java and Node.js flame graphs, and the new EC2 Performance Monitoring Counter (PMC) feature released this year."
All about Zookeeper and ClickHouse Keeper.pdfAltinity Ltd
ClickHouse clusters depend on ZooKeeper to handle replication and distributed DDL commands. In this Altinity webinar, we’ll explain why ZooKeeper is necessary, how it works, and introduce the new built-in replacement named ClickHouse Keeper. You’ll learn practical tips to care for ZooKeeper in sickness and health. You’ll also learn how/when to use ClickHouse Keeper. We will share our recommendations for keeping that happy as well.
This document provides an overview of HashiCorp Vault for securely storing, accessing, and managing secrets. It discusses how Vault can be used to securely store secrets like API keys, passwords, and certificates. The document outlines Vault's architecture, data storage options, authentication methods, policies for access control, and integrating systems using Vault. It also provides an agenda for a demonstration of Spring Cloud Vault integration for retrieving database credentials from Vault and using them to connect to a MySQL database.
Maxscale switchover, failover, and auto rejoinWagner Bianchi
How the MariaDB Maxscale Switchover, Failover, and Rejoin works under the hood by Esa Korhonen and Wagner Bianchi.
You can watch the video of the presentation at
https://www.linkedin.com/feed/update/urn:li:activity:6381185640607809536
A comprehensive walkthrough of how to manage infrastructure-as-code using Terraform. This presentation includes an introduction to Terraform, a discussion of how to manage Terraform state, how to use Terraform modules, an overview of best practices (e.g. isolation, versioning, loops, if-statements), and a list of gotchas to look out for.
For a written and more in-depth version of this presentation, check out the "Comprehensive Guide to Terraform" blog post series: https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca
Advanced Percona XtraDB Cluster in a nutshell... la suiteKenny Gryp
This document provides a hands-on tutorial for advanced Percona XtraDB Cluster users. It discusses setting up a 3 node PXC cluster environment in VirtualBox and bootstrapping the initial cluster. It then covers topics like avoiding state snapshot transfers when restarting MySQL, recovering from clean and unclean shutdowns, and reproducing and diagnosing different types of conflicts through examples.
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIAltinity Ltd
Graham Mainwaring and Robert Hodges summarize management of ClickHouse on Kubernetes using the ClickHouse Kubernetes Operator and introduce a new UI for it. Presented at the 15 Dec '22 SF Bay Area ClickHouse Meetup.
This document discusses using KVM as the hypervisor with CloudStack. It provides background on KVM, describes how to install and configure it with CloudStack, and covers networking, storage, management, and troubleshooting. Specifically, it outlines installing required components, configuring Linux bridges or Open vSwitch for networking, options for primary storage, and tools for managing virtual machines and troubleshooting KVM.
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...Severalnines
This presentation by Krzysztof Książek at Percona Live 2017 in Santa Clara, California gives detailed descriptions and comparisons of the leading open source database load balancing technologies
24시간 365일 서비스를 위한 MySQL DB 이중화.
MySQL 이중화 방안들에 대해 알아보고 운영하면서 겪은 고민들을 이야기해 봅니다.
목차
1. DB 이중화 필요성
2. 이중화 방안
- HW 이중화
- MySQL Replication 이중화
3. 이중화 운영 장애
4. DNS와 VIP
5. MySQL 이중화 솔루션 비교
대상
- MySQL을 서비스하고 있는 인프라 담당자
- MySQL 이중화에 관심 있는 개발자
In this tutorial, we cover the different deployment possibilities of the MySQL architecture depending on the business requirements for the data. We also deploy some architecture and see how to evolve to the next one.
The tutorial covers the new MySQL Solutions like InnoDB ReplicaSet, InnoDB Cluster, and InnoDB ClusterSet.
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-RegionJi-Woong Choi
OpenStack Ceph & Neutron에 대한 설명을 담고 있습니다.
1. OpenStack
2. How to create instance
3. Ceph
- Ceph
- OpenStack with Ceph
4. Neutron
- Neutron
- How neutron works
5. OpenStack HA- controller- l3 agent
6. OpenStack multi-region
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfAltinity Ltd
Join the Altinity experts as we dig into ClickHouse sharding and replication, showing how they enable clusters that deliver fast queries over petabytes of data. We’ll start with basic definitions of each, then move to practical issues. This includes the setup of shards and replicas, defining schema, choosing sharding keys, loading data, and writing distributed queries. We’ll finish up with tips on performance optimization.
#ClickHouse #datasets #ClickHouseTutorial #opensource #ClickHouseCommunity #Altinity
-----------------
Join ClickHouse Meetups: https://www.meetup.com/San-Francisco-...
Check out more ClickHouse resources: https://altinity.com/resources/
Visit the Altinity Documentation site: https://docs.altinity.com/
Contribute to ClickHouse Knowledge Base: https://kb.altinity.com/
Join the ClickHouse Reddit community: https://www.reddit.com/r/Clickhouse/
----------------
Learn more about Altinity!
Site: https://www.altinity.com
LinkedIn: https://www.linkedin.com/company/alti...
Twitter: https://twitter.com/AltinityDB
Easy Cloud Native Transformation using HashiCorp NomadBram Vogelaar
Nomad is an open source tool for dynamic workload scheduling that can be used to easily transform applications to be cloud native. The document discusses how Nomad was used to deploy a personal blog to AWS, including writing a Nomad job file and using features like spreading tasks across hosts and auto-scaling. It also covers integrating Nomad with other HashiCorp tools like Consul for service discovery, Vault for access control, and the Nomad autoscaler for auto-scaling. Examples are provided of monitoring metrics with Prometheus and visualizing with Grafana.
Container Orchestration from Theory to PracticeDocker, Inc.
Join Laura Frank and Stephen Day as they explain and examine technical concepts behind container orchestration systems, like distributed consensus, object models, and node topology. These concepts build the foundation of every modern orchestration system, and each technical explanation will be illustrated using Docker’s SwarmKit as a real-world example. Gain a deeper understanding of how orchestration systems like SwarmKit work in practice and walk away with more insights into your production applications.
This document discusses optimizing performance for high-load projects. It summarizes the delivery loads and technologies used for several projects including mGage, mobclix and XXXX. It then discusses optimizations made to improve performance, including using Solr for search, Redis for real-time data, Hadoop for reporting, and various Java optimizations in moving to Java 7. Specific optimizations discussed include reducing garbage collection, improving random number generation, and minimizing I/O operations.
Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski buildacloud
The document provides troubleshooting strategies for CloudStack installations, including network issues, security groups, host connectivity, virtual routers, templates, and log analysis. It discusses common problems such as VLAN misconfigurations, security group rules not being applied, hosts showing in the "avoid set", template preparation errors, and exceptions in the logs. It emphasizes analyzing logs at the management server, hypervisor, and job levels to find the root cause of failures.
thredUP team shares key learnings from after-migration processes. We tell you about what technologies and solutions worked best for us and where we spent time troubleshooting and improving. In particular we have focused on development and staging experience, user authentication, cloud-native CI pipelines, applications telemetry and service mesh. We also share our experience with Kubernetes security hardening, autoscaling and tell you about a new service creation within our infrastructure.
This document discusses PostgreSQL and Solaris as a low-cost platform for medium to large scale critical scenarios. It provides an overview of PostgreSQL, highlighting features like MVCC, PITR, and ACID compliance. It describes how Solaris and PostgreSQL integrate well, with benefits like DTrace support, scalability on multicore/multiprocessor systems, and Solaris Cluster support. Examples are given for installing PostgreSQL on Solaris using different methods, configuring zones for isolation, using ZFS for storage, and monitoring performance with DTrace scripts.
Terraform for azure: the good, the bad and the ugly -Giulio Vian
Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. The presenter discusses the good, bad, and ugly aspects of using Terraform with Azure. The good includes its simple configuration language and ability to integrate with Azure and automate deployments. The bad includes limitations in its language and some errors being difficult to debug. The ugly involves challenges around managing state files and keeping infrastructure definitions well organized. Overall, Terraform provides benefits but also requires understanding its quirks and handling state carefully.
This document discusses managing ECS hosts at Comtravo using Terraform, AWS Lambda, and Step Functions. It describes how Comtravo uses Terraform to maintain six environments with over 500 AWS components each. It also discusses challenges around updating EC2 hosts in an ECS cluster. Specifically, it outlines how they use Terraform to emit a CloudWatch event when a new launch configuration is created. This event triggers a Step Function workflow to safely replace old EC2 instances in the ECS cluster with new instances using the updated launch configuration.
Managing Terraform Module Versioning and Dependencies Nebulaworks
For our February session, we are excited to invite Robert Paprocki, Cloud Engineer at Kong present on managing Terraform Module Versioning and Dependencies.
Key Takeaways
*This talk discusses how the team at Kong Cloud manages Terraform deployments through tooling solutions around dependency management, preaching a culture of creating small-re-usable Terraform modules.
*Leveraging community tools for versioning and common variable management.
*Discusses how these tools and practices are extended to logical software deployments, using Terraform to shape and manage Nomad job definitions.
Workshop Consul .- Service Discovery & Failure DetectionVincent Composieux
This workshop uses a Docker Swarm cluster to deploy a Consul agent and uses Registrator to automatically register Docker containers services into Consul and add a health check on it.
6 Months Sailing with Docker in Production Hung Lin
This document summarizes a company's 6-month experience using Docker in production. They aimed to make local development environments match production, enable one-click deployments, achieve resilience and auto-scaling, centralized logging/monitoring, and add new services easily. They dockerized services, ran Docker everywhere, and aimed to use Docker clusters for high availability. Limitations included Docker image pull issues, instability on non-Linux, immature clusters, and tools requiring environment variables. Tradeoffs included using CloudFormation and duplicating images. Future hopes include improved clusters, networking, state management with Flocker, and tools like Sysdig.
Container orchestration from theory to practiceDocker, Inc.
"Join Laura Frank and Stephen Day as they explain and examine technical concepts behind container orchestration systems, like distributed consensus, object models, and node topology. These concepts build the foundation of every modern orchestration system, and each technical explanation will be illustrated using SwarmKit and Kubernetes as a real-world example. Gain a deeper understanding of how orchestration systems work in practice and walk away with more insights into your production applications."
Terraform, is no doubt very flexible and powerful. The question is, how do we write Terraform code and construct our infrastructure in a reproducible fashion that makes sense? How can we keep code DRY, segment state, and reduce the risk of making changes to our service/stack/infrastructure?
HashiCorp’s infrastructure management tool, Terraform, is no doubt very flexible and powerful. The question is, how do we write Terraform code and construct our infrastructure in a reproducible fashion that makes sense? How can we keep code DRY, segment state, and reduce the risk of making changes to our service/stack/infrastructure?
This talk describes a design pattern to help answer the previous questions. The talk is divided into two sections, with the first section describing and defining the design pattern with a Deployment Example. The second part uses a multi-repository GitHub organization to create a Real World Example of the design pattern.
These slides show how to reduce latency on websites and reduce bandwidth for improved user experience.
Covering network, compression, caching, etags, application optimisation, sphinxsearch, memcache, db optimisation
HKG15-204: OpenStack: 3rd party testing and performance benchmarkingLinaro
HKG15-204: OpenStack: 3rd party testing and performance benchmarking
---------------------------------------------------
Speaker: Andrew McDermott, Clark Laughlin
Date: February 10, 2015
---------------------------------------------------
★ Session Summary ★
Status of Tempest 3rd party testing, discussion on scenarii for Rally benchmarking and hypervisor performance.
--------------------------------------------------
★ Resources ★
Pathable: https://hkg15.pathable.com/meetings/250785
Video: https://www.youtube.com/watch?v=-00rTPCYAyg
Etherpad: http://pad.linaro.org/p/hkg15-204
---------------------------------------------------
★ Event Details ★
Linaro Connect Hong Kong 2015 - #HKG15
February 9-13th, 2015
Regal Airport Hotel Hong Kong Airport
---------------------------------------------------
http://www.linaro.org
http://connect.linaro.org
The document discusses Docker networking and Kubernetes networking concepts. It provides an overview of Docker networking and how containers on the same host can communicate. It then summarizes key Kubernetes concepts like pods, replication controllers, services and networking. It demonstrates how to set up a sample application topology in Kubernetes using replication controllers and services. It also discusses exposing services externally and additional resources for learning about Docker and Kubernetes.
Akka is using the Actors together with STM to create a unified runtime and programming model for scaling both UP (multi-core) and OUT (grid/cloud). Akka provides location transparency by abstracting away both these tangents of scalability by turning them into an ops task. This gives the Akka runtime freedom to do adaptive automatic load-balancing, cluster rebalancing, replication & partitioning
Python has always been an exemplar of good coding style for me. It’s impressive how concise unit tests can be, how flexible mocks are. Really, it’s a great pleasure to write Python code. Up until the point when you need to deploy it. From then on you embark on a rollercoaster to go through a variety of tools and approaches. This article is supposed to help you sail over Python dependency hell, stay alive and still enjoy writing in this awesome language.
The document summarizes how data is organized and accessed efficiently in MySQL databases using indexes. It discusses how B+ trees are used to organize data in indexes to allow efficient lookups, ranges scans, and sorting. It also provides examples of how indexes can improve the performance of common query types like point lookups, range selections, aggregations, sorting, and joins by enabling access plans that utilize the indexes.
The document describes a toolkit for recovering data from MySQL databases using InnoDB tables. It provides an overview of the tools in the toolkit, including stream_parser to parse raw data files into InnoDB pages, sys_parser to recover table structures from those pages, and c_parser and recover_dictionary tools to extract and load table records and reconstruct the InnoDB data dictionary. It demonstrates dropping a sample table and then using the toolkit to recover the table structure and records from the raw data files.
The document provides information about InnoDB tables in MySQL, including their structure on disk and in memory. It demonstrates tools for parsing InnoDB tablespaces and recovering the InnoDB data dictionary from tablespace files to retrieve table definitions and indexes.
This document provides an overview of InnoDB file formats and recovery. It discusses the InnoDB tablespace format, including the shared tablespace that contains the data dictionary and rollback segments, as well as tablespaces per table that contain indexes and external pages. It also describes the InnoDB log files, index structures including the primary and secondary indexes, and typical page and record formats. The goal is to provide background needed to understand InnoDB data recovery.
Recovery of lost or corrupted inno db tables(mysql uc 2010)Aleksandr Kuzminsky
The document discusses recovery of lost or corrupted InnoDB tables in MySQL. It provides an overview of how MySQL stores data in InnoDB, describes typical failure scenarios that can cause data loss or corruption, and introduces an InnoDB recovery tool that can be used to parse InnoDB tablespace pages and apply constraints to recover records from corrupted pages.
Understanding User Behavior with Google Analytics.pdfSEO Article Boost
Unlocking the full potential of Google Analytics is crucial for understanding and optimizing your website’s performance. This guide dives deep into the essential aspects of Google Analytics, from analyzing traffic sources to understanding user demographics and tracking user engagement.
Traffic Sources Analysis:
Discover where your website traffic originates. By examining the Acquisition section, you can identify whether visitors come from organic search, paid campaigns, direct visits, social media, or referral links. This knowledge helps in refining marketing strategies and optimizing resource allocation.
User Demographics Insights:
Gain a comprehensive view of your audience by exploring demographic data in the Audience section. Understand age, gender, and interests to tailor your marketing strategies effectively. Leverage this information to create personalized content and improve user engagement and conversion rates.
Tracking User Engagement:
Learn how to measure user interaction with your site through key metrics like bounce rate, average session duration, and pages per session. Enhance user experience by analyzing engagement metrics and implementing strategies to keep visitors engaged.
Conversion Rate Optimization:
Understand the importance of conversion rates and how to track them using Google Analytics. Set up Goals, analyze conversion funnels, segment your audience, and employ A/B testing to optimize your website for higher conversions. Utilize ecommerce tracking and multi-channel funnels for a detailed view of your sales performance and marketing channel contributions.
Custom Reports and Dashboards:
Create custom reports and dashboards to visualize and interpret data relevant to your business goals. Use advanced filters, segments, and visualization options to gain deeper insights. Incorporate custom dimensions and metrics for tailored data analysis. Integrate external data sources to enrich your analytics and make well-informed decisions.
This guide is designed to help you harness the power of Google Analytics for making data-driven decisions that enhance website performance and achieve your digital marketing objectives. Whether you are looking to improve SEO, refine your social media strategy, or boost conversion rates, understanding and utilizing Google Analytics is essential for your success.
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdfFlorence Consulting
Quattordicesimo Meetup di Milano, tenutosi a Milano il 23 Maggio 2024 dalle ore 17:00 alle ore 18:30 in presenza e da remoto.
Abbiamo parlato di come Axpo Italia S.p.A. ha ridotto il technical debt migrando le proprie APIs da Mule 3.9 a Mule 4.4 passando anche da on-premises a CloudHub 1.0.
Ready to Unlock the Power of Blockchain!Toptal Tech
Imagine a world where data flows freely, yet remains secure. A world where trust is built into the fabric of every transaction. This is the promise of blockchain, a revolutionary technology poised to reshape our digital landscape.
Toptal Tech is at the forefront of this innovation, connecting you with the brightest minds in blockchain development. Together, we can unlock the potential of this transformative technology, building a future of transparency, security, and endless possibilities.
Gen Z and the marketplaces - let's translate their needsLaura Szabó
The product workshop focused on exploring the requirements of Generation Z in relation to marketplace dynamics. We delved into their specific needs, examined the specifics in their shopping preferences, and analyzed their preferred methods for accessing information and making purchases within a marketplace. Through the study of real-life cases , we tried to gain valuable insights into enhancing the marketplace experience for Generation Z.
The workshop was held on the DMA Conference in Vienna June 2024.
11. What module creates
resource "aws_route53_record" "elb" {
name = "${var.backend_type}.proxysql"
type = "CNAME"
ttl = local.dns_ttl
zone_id = data.aws_route53_zone.nlb_zone.id
records = [
aws_lb.proxysql.dns_name
]
}
auto_scaling.tf dns.tf
load_balance.tf outputs.tf
12. What module creates
output "nlb_dns_name" {
value = aws_lb.proxysql.dns_name
}
output "nlb_cname" {
value = aws_route53_record.elb.name
}
auto_scaling.tf dns.tf
load_balance.tf outputs.tf
13. Healthchecks
Two decisions to make:
1. Shall I forward traffic to X?
2. Shall I terminate X?
?
Load
Balancer
?
resource "aws_autoscaling_group" "proxysql_zero" {
…
health_check_type = “EC2|ELB”
…
}
14. ELB type Healthchecks
● Traffic port vs health check port
● TCP connection success?
○ Yes
■ Forward traffic
○ No
■ Do not forward traffic
■ Terminate instance
:8080
:3306
15. EC2 type Healthchecks
● Traffic port == health check port
● TCP connection success?
○ Yes
■ Forward traffic
○ No
■ Do not forward traffic
● ProxySQL healthcheck agent
:8080
:3306
try:
self.check_health(cursor)
self.check_at_least_one_live(cursor)
self.check_whg_count(cursor)
self.check_instance_state(local_instance)
self.mark_healthy(local_instance)
except:
self.mark_unhealthy(local_instance)
16. Zero downtime migration
● Needed when change:
○ AMI, subnet, security group, IAM role
○ ami-12345678 -> ami-87654321
● Blue/green deployment
● Must be tested for a proof
Load
Balancer
20. Zero downtime migrations: How to test
Must be tested (tests/zero_downtime/test_zero_downtime.py)
Plan:
● ProxySQL acts as an HTTP server
● Create pool A
● Start a probe in a loop
● Create pool B
● Watch for probe errors
21. Testing a module: terraform-ci
def test_zero_downtime(ec2_client):
# Pool A
with open(osp.join(terraform_dir, "configuration.tfvars"), "w") as fp:
fp.write(configuration_template.format(payload=”old payload”))
with terraform_apply(terraform_dir) as tf_output:
proxysql_nlb = tf_output["proxysql_nlb"]["value"]
# Probe
que = Queue()
child = Process(target=ping, args=(url, que))
# Pool B
with open(osp.join(terraform_dir, "configuration.tfvars"), "w") as fp:
fp.write(configuration_template.format(payload=”new payload”))
with terraform_apply(terraform_dir):
child.terminate()
result = que.get()
assert result["error"] == 0
terraform-ci · PyPI
39. Source of Truth(s)
● SourceOfTruth(ABC):
● SOTLocalProxy(SourceOfTruth):
● SOTRemoteMySQL(SourceOfTruth):
40. ProxySQL Manager
# proxysql-manager
Usage: proxysql-manager [OPTIONS] COMMAND [ARGS]...
ProxySQL manager helps to manage local ProxySQL instance
Options:
--debug Print debug messages
-q, --quiet Print only errors
--logfile TEXT Log messages to this file [default: /var/log/proxysql-
manager.log]
--version Show the version and exit.
--help Show this message and exit.
Commands:
admin ProxySQL admin credentials
deregister-server Deregister MySQL server from a ProxySQL Source of...
deregister-unknown Deregister non-ZK MySQL servers.
expert ProxySQL manager advanced commands.
generate-config Generate ProxySQL config and save it on disk.
is-server-registered The tool will inspect the latest ProxySQL config...
register-server Register MySQL server in a ProxySQL Source of Truth.
show-backends Show all backend types and their NLB DNS name.
show-instance Show ProxySQL instance for given backend_type
show-instances Same as ``proxysql-manager show-instance`` command.
start-agent Update latest configuration to local ProxySQL.
start-healthcheck Health check for upstream NLB.
sync-sot Synchronize ProxySQL config to remote Source of...
41. ProxySQL Manager: Agent
def _watch_table(self, klass, interval):
"""
Start agent daemon.
:param klass: Table class to watch
:param int interval: Time in seconds between SoT
checks.
"""
while True:
with global_lock():
if klass == PSMysqlUsers:
self._sync_mysql_users_with_knox()
else:
self.update_one_table(klass)
sleep(interval)
42. ProxySQL Manager: Agent (cont)
def update_one_table(self, tblclass):
try:
pslist = self._source_of_truth.read(tblclass, self._backend_type)
if pslist.version and pslist.version > self.versions[tblclass]:
# pslist is empty or pslist has newer version
LOG.info(
"It is a new version %d for table %s",
pslist.version,
tblclass.TABLE,
)
self._local_proxysql.write(pslist)
self.versions[tblclass] = pslist.version
self.update_versions_file(tblclass=tblclass, version=pslist.version)
self._handle_fallback(pslist)
43. ProxySQL Manager: Healthcheck
def watcher(self, credentials):
while True:
try:
with connection.cursor() as cursor:
self.check_health(cursor)
self._check_at_least_one_live(cursor)
self._check_whg_count(cursor)
self._check_instance_state(local_instance)
self._mark_healthy(local_instance)
except (
ProxySQLNotHealthy,
MySQLError,
ConnectionRefusedError,
) as err:
self._unhealthy_count += 1
LOG.error(err)
if self._unhealthy_count > self._healthy_threshold:
self._mark_unhealthy(local_instance)
else:
LOG.warning("Health check failed %d times", self._unhealthy_count)
sleep(self._probe_interval)