This document explains how to set up ProxySQL to log queries from users connecting directly to the database servers. It details installing and configuring ProxySQL to log queries to binary files, using a tool to convert the binary logs to text format, and setting up an ELK stack to index the query logs and make them searchable in Kibana. Filebeat is configured to ship the text query logs to Logstash, which parses them and sends the data to Elasticsearch. Kibana provides a web interface for viewing and analyzing the query logs.
24시간 365일 서비스를 위한 MySQL DB 이중화.
MySQL 이중화 방안들에 대해 알아보고 운영하면서 겪은 고민들을 이야기해 봅니다.
목차
1. DB 이중화 필요성
2. 이중화 방안
- HW 이중화
- MySQL Replication 이중화
3. 이중화 운영 장애
4. DNS와 VIP
5. MySQL 이중화 솔루션 비교
대상
- MySQL을 서비스하고 있는 인프라 담당자
- MySQL 이중화에 관심 있는 개발자
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfJesmar Cannao'
ProxySQL is well-affirmed into thousands of production environments for the features we all know: multiplexing, query routing, and rewriting to name a few.
Let's go through those use cases which maybe are the least common: from keeping malicious eyes away from your production data to rebuilding your non-production environment, from stopping having hanging transactions to monitor your instance, from query firewalling to changing ProxySQL configuration without a single line of SQL!
When the sky is the limit, ProxySQL gives you some boost!
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScaleColin Charles
As proxies (and database routers) go, the first one I ever used was the now deprecated MySQL Proxy. Since then, I've managed to use MariaDB MaxScale quite a bit (including its fork AirBnB MaxScale), played around with ProxySQL in recent time, and also started taking a look at MySQL Router. In this quick 20-minute overview, we'll discuss why these three exist, a feature comparison, and reasons when to use the right tool for the job.
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné
Of course there is no such thing as perfect service discovery, and we will see why in the talk. However, the way ProxySQL is deployed in this case minimizes the risk for split-brains, and this is why I qualify it as almost perfect. But let’s step back a little...
MySQL alone is not a high availability solution. To provide resilience to primary failure, other components need to be integrated with MySQL. At MessageBird, these additional components are ProxySQL and Orchestrator. In this talk, we describe how ProxySQL is architectured to provide close to perfect Service Discovery and how this, combined with Orchestrator, allows for automatic failover. The talk presents the details of the integration of MySQL, ProxySQL and Orchestrator in Google Cloud (and it would be easy to re-implement a similar architecture at other cloud vendors or on-premises). We will also cover lessons learned for the 2 years this architecture has been in production. Come to this talk to learn more about MySQL high availability, ProxySQL and Orchestrator.
24시간 365일 서비스를 위한 MySQL DB 이중화.
MySQL 이중화 방안들에 대해 알아보고 운영하면서 겪은 고민들을 이야기해 봅니다.
목차
1. DB 이중화 필요성
2. 이중화 방안
- HW 이중화
- MySQL Replication 이중화
3. 이중화 운영 장애
4. DNS와 VIP
5. MySQL 이중화 솔루션 비교
대상
- MySQL을 서비스하고 있는 인프라 담당자
- MySQL 이중화에 관심 있는 개발자
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfJesmar Cannao'
ProxySQL is well-affirmed into thousands of production environments for the features we all know: multiplexing, query routing, and rewriting to name a few.
Let's go through those use cases which maybe are the least common: from keeping malicious eyes away from your production data to rebuilding your non-production environment, from stopping having hanging transactions to monitor your instance, from query firewalling to changing ProxySQL configuration without a single line of SQL!
When the sky is the limit, ProxySQL gives you some boost!
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScaleColin Charles
As proxies (and database routers) go, the first one I ever used was the now deprecated MySQL Proxy. Since then, I've managed to use MariaDB MaxScale quite a bit (including its fork AirBnB MaxScale), played around with ProxySQL in recent time, and also started taking a look at MySQL Router. In this quick 20-minute overview, we'll discuss why these three exist, a feature comparison, and reasons when to use the right tool for the job.
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné
Of course there is no such thing as perfect service discovery, and we will see why in the talk. However, the way ProxySQL is deployed in this case minimizes the risk for split-brains, and this is why I qualify it as almost perfect. But let’s step back a little...
MySQL alone is not a high availability solution. To provide resilience to primary failure, other components need to be integrated with MySQL. At MessageBird, these additional components are ProxySQL and Orchestrator. In this talk, we describe how ProxySQL is architectured to provide close to perfect Service Discovery and how this, combined with Orchestrator, allows for automatic failover. The talk presents the details of the integration of MySQL, ProxySQL and Orchestrator in Google Cloud (and it would be easy to re-implement a similar architecture at other cloud vendors or on-premises). We will also cover lessons learned for the 2 years this architecture has been in production. Come to this talk to learn more about MySQL high availability, ProxySQL and Orchestrator.
Understanding ProxySQL internals and then interacting with some common features of ProxySQL such as query rewriting, mirroring, failovers, and ProxySQL Cluster
ProxySQL is a popular database proxy for MySQL/MariaDB servers. This focuses on the possible High availability options for ProxySQL and operations of inbuilt clustering feature in ProxySQL. This tech talk was presented at Mydbops Database Meetup on 27-04-2019 by Aakash M, Database Administrator with Mydbops and Vignesh Prabhu, Database Administrator with Mydbops.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
The presentation covers improvements made to the redo logs in MySQL 8.0 and their impact on the MySQL performance and Operations. This covers the MySQL version still MySQL 8.0.30.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
MariaDB 10.5 binary install (바이너리 설치)
- 네오클로바 DB지원사업부
1. About MariaDB
1.1 MariaDB 개요
1.2 MariaDB as a R-DBMS
1.3 Open Source Database System
2. 설치
2.1 설치 기본 정보
2.2 설치 준비
2.3 MariaDB 설치
2.4 MariaDB 시작 / 접속 / 종료
2.5 추가 설정
MySQL InnoDB Cluster HA Overview & DemoKeith Hollman
Take a look at the High Availability option that you can use with your out-of-the-box MySQL: MySQL InnoDB Cluster. With MySQL Server 8.0, MySQL Shell & MySQL Router you can convert from single-primary to multi-primary and back again, in a single command. Want to know how?
Performance Schema is a powerful diagnostic instrument for:
- Query performance
- Complicated locking issues
- Memory leaks
- Resource usage
- Problematic behavior, caused by inappropriate settings
- More
It comes with hundreds of options which allow precisely tuning what to instrument. More than 100 consumers store collected data.
In this tutorial, we will try all the important instruments out. We will provide a test environment and a few typical problems which could be hardly solved without Performance Schema. You will not only learn how to collect and use this information but have experience with it.
Tutorial at Percona Live Austin 2019
Optimizing MariaDB for maximum performanceMariaDB plc
When it comes to optimizing the performance of a database, DBAs have to look at everything from the OS to the network. In this session, MariaDB Enterprise Architect Manjot Singh shares best practices for getting the most out of MariaDB. He highlights recommended OS settings, important configuration and tuning parameters, options for improving replication and clustering performance and features such as query result caching.
At Pinterest, the ProxySQL infrastructure fronts numerous heterogeneous databases. Due to the high number of unique configurations and dynamic nature of cloud deployments, it’s a challenge to reliably provision, change, auto-scale, and monitor ProxySQL servers. Applying Infrastructure as Code principles using Terraform, we made it possible to manage such a large fleet of ProxySQL instances confidently. Come learn how we automated provisioning, testing, and monitoring of ProxySQL at scale.
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfAltinity Ltd
Join the Altinity experts as we dig into ClickHouse sharding and replication, showing how they enable clusters that deliver fast queries over petabytes of data. We’ll start with basic definitions of each, then move to practical issues. This includes the setup of shards and replicas, defining schema, choosing sharding keys, loading data, and writing distributed queries. We’ll finish up with tips on performance optimization.
#ClickHouse #datasets #ClickHouseTutorial #opensource #ClickHouseCommunity #Altinity
-----------------
Join ClickHouse Meetups: https://www.meetup.com/San-Francisco-...
Check out more ClickHouse resources: https://altinity.com/resources/
Visit the Altinity Documentation site: https://docs.altinity.com/
Contribute to ClickHouse Knowledge Base: https://kb.altinity.com/
Join the ClickHouse Reddit community: https://www.reddit.com/r/Clickhouse/
----------------
Learn more about Altinity!
Site: https://www.altinity.com
LinkedIn: https://www.linkedin.com/company/alti...
Twitter: https://twitter.com/AltinityDB
High Availability Content Caching with NGINXNGINX, Inc.
On-Demand Recording:
https://www.nginx.com/resources/webinars/high-availability-content-caching-nginx/
You trust NGINX to be your web server, but did you know it’s also a high-performance content cache? In fact, the world’s most popular CDNs – CloudFlare, MaxCDN, and Level 3 among them – are built on top of the open source NGINX software.
NGINX content caching can drastically improve the performance of your applications. We’ll start with basic configuration, then move on to advanced concepts and best practices for architecting high availability and capacity in your application infrastructure.
Join this webinar to:
* Enable content caching with the key configuration directives
* Use micro caching with NGINX Plus to cache dynamic content while maintaining low CPU utilization
* Partition your cache across multiple servers for high availability and increased capacity
* Log transactions and troubleshoot your NGINX content cache
Understanding ProxySQL internals and then interacting with some common features of ProxySQL such as query rewriting, mirroring, failovers, and ProxySQL Cluster
ProxySQL is a popular database proxy for MySQL/MariaDB servers. This focuses on the possible High availability options for ProxySQL and operations of inbuilt clustering feature in ProxySQL. This tech talk was presented at Mydbops Database Meetup on 27-04-2019 by Aakash M, Database Administrator with Mydbops and Vignesh Prabhu, Database Administrator with Mydbops.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
The presentation covers improvements made to the redo logs in MySQL 8.0 and their impact on the MySQL performance and Operations. This covers the MySQL version still MySQL 8.0.30.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
MariaDB 10.5 binary install (바이너리 설치)
- 네오클로바 DB지원사업부
1. About MariaDB
1.1 MariaDB 개요
1.2 MariaDB as a R-DBMS
1.3 Open Source Database System
2. 설치
2.1 설치 기본 정보
2.2 설치 준비
2.3 MariaDB 설치
2.4 MariaDB 시작 / 접속 / 종료
2.5 추가 설정
MySQL InnoDB Cluster HA Overview & DemoKeith Hollman
Take a look at the High Availability option that you can use with your out-of-the-box MySQL: MySQL InnoDB Cluster. With MySQL Server 8.0, MySQL Shell & MySQL Router you can convert from single-primary to multi-primary and back again, in a single command. Want to know how?
Performance Schema is a powerful diagnostic instrument for:
- Query performance
- Complicated locking issues
- Memory leaks
- Resource usage
- Problematic behavior, caused by inappropriate settings
- More
It comes with hundreds of options which allow precisely tuning what to instrument. More than 100 consumers store collected data.
In this tutorial, we will try all the important instruments out. We will provide a test environment and a few typical problems which could be hardly solved without Performance Schema. You will not only learn how to collect and use this information but have experience with it.
Tutorial at Percona Live Austin 2019
Optimizing MariaDB for maximum performanceMariaDB plc
When it comes to optimizing the performance of a database, DBAs have to look at everything from the OS to the network. In this session, MariaDB Enterprise Architect Manjot Singh shares best practices for getting the most out of MariaDB. He highlights recommended OS settings, important configuration and tuning parameters, options for improving replication and clustering performance and features such as query result caching.
At Pinterest, the ProxySQL infrastructure fronts numerous heterogeneous databases. Due to the high number of unique configurations and dynamic nature of cloud deployments, it’s a challenge to reliably provision, change, auto-scale, and monitor ProxySQL servers. Applying Infrastructure as Code principles using Terraform, we made it possible to manage such a large fleet of ProxySQL instances confidently. Come learn how we automated provisioning, testing, and monitoring of ProxySQL at scale.
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfAltinity Ltd
Join the Altinity experts as we dig into ClickHouse sharding and replication, showing how they enable clusters that deliver fast queries over petabytes of data. We’ll start with basic definitions of each, then move to practical issues. This includes the setup of shards and replicas, defining schema, choosing sharding keys, loading data, and writing distributed queries. We’ll finish up with tips on performance optimization.
#ClickHouse #datasets #ClickHouseTutorial #opensource #ClickHouseCommunity #Altinity
-----------------
Join ClickHouse Meetups: https://www.meetup.com/San-Francisco-...
Check out more ClickHouse resources: https://altinity.com/resources/
Visit the Altinity Documentation site: https://docs.altinity.com/
Contribute to ClickHouse Knowledge Base: https://kb.altinity.com/
Join the ClickHouse Reddit community: https://www.reddit.com/r/Clickhouse/
----------------
Learn more about Altinity!
Site: https://www.altinity.com
LinkedIn: https://www.linkedin.com/company/alti...
Twitter: https://twitter.com/AltinityDB
High Availability Content Caching with NGINXNGINX, Inc.
On-Demand Recording:
https://www.nginx.com/resources/webinars/high-availability-content-caching-nginx/
You trust NGINX to be your web server, but did you know it’s also a high-performance content cache? In fact, the world’s most popular CDNs – CloudFlare, MaxCDN, and Level 3 among them – are built on top of the open source NGINX software.
NGINX content caching can drastically improve the performance of your applications. We’ll start with basic configuration, then move on to advanced concepts and best practices for architecting high availability and capacity in your application infrastructure.
Join this webinar to:
* Enable content caching with the key configuration directives
* Use micro caching with NGINX Plus to cache dynamic content while maintaining low CPU utilization
* Partition your cache across multiple servers for high availability and increased capacity
* Log transactions and troubleshoot your NGINX content cache
Automatically scaling your Kubernetes workloads - SVC201-S - Chicago AWS SummitAmazon Web Services
As the need for more computing resources has accelerated, so too have the ways in which computing have evolved. The advent of the cloud has allowed us to easily scale to suit our needs, but if we want to keep pace, we need an even more automated way to scale our infrastructure. In this session, we look at automatic scaling using Kubernetes, including how to set it up and, most important, what you should monitor in order to drive your scaling. This session is brought to you by AWS partner, Datadog.
Autoscaling Your Kubernetes Workloads (Sponsored by Datadog) - AWS Summit SydneyAmazon Web Services
As our need for more computing resources has accelerated, so too have the ways in which computing has evolved. The advent of cloud providers like AWS has allowed us to easily scale to suit our needs. But if we want to keep pace, we need an even more automated way to scale our infrastructure. In this session, we’ll look at autoscaling with Kubernetes, how to set it up, and most importantly, what things to monitor in order to drive your autoscaling.
Automatically scaling your Kubernetes workloads - SVC210-S - Santa Clara AWS ...Amazon Web Services
We begin this session by taking a brief tour through the history of infrastructure and the evolution of our ability to scale. This includes what provisioning and scaling look like when working with physical servers. We then discuss the technologies that made automatic scaling possible. We also provide an overview of the most common scaling that is available today. Finally, we discuss how to monitor the things that matter. Using this framework, we can determine what metrics we should scale on for different types of applications and workloads.
Andrew Betts Web Developer, The Financial Times at Fastly Altitude 2016
Running custom code at the Edge using a standard language is one of the biggest advantages of working with Fastly’s CDN. Andrew gives you a tour of all the problems the Financial Times and Nikkei solve in VCL and how their solutions work.
JavaScript and Friends August 20th, 20201 -- MySQL Shell and JavaScriptDave Stokes
The MySQL Shell has a JavaScript mode where you can use JS libraries to access you data and you can also write (and save) your own custom reports (or programs) for future use.
This is the presentation delivered by Karthik.P.R at MySQL User Camp Bangalore on 09th June 2017. ProxySQL is a high performance MySQL Load Balancer Designed to scale database servers.
Automatically Scaling Your Kubernetes Workloads - SVC209-S - Anaheim AWS SummitAmazon Web Services
As our need for more computing resources has accelerated, so too have the ways in which computing has evolved. The cloud has enabled us to easily scale to suit our needs. To keep pace, we need more automated way to scale our infrastructure. In this session, we discuss automatic scaling with Kubernetes, how to set it up, and—most importantly—what to monitor in order to drive your automatic scaling. This session is brought to you by AWS partner, Datadog.
We are using Elasticsearch to power the search feature of our public frontend, serving 10k queries per hour across 8 markets in SEA.
Here we are sharing our experiences of running Elasticsearch on Kubernetes, presenting our general setup, configuration tweaks and possible pitfalls.
Why Managed Service Providers Should Embrace Container TechnologySagi Brody
This talk will demonstrate the importance and value for Managed Service Providers (MSPs) and cloud providers of building their business models around the management of containers. It will also explore the various container technologies being used today and why one might be utilized over another. The object is not to give a technical discussion on the subject, but rather to cover the benefits of Linux containers and how their use can be incorporated into strategies for future business planning and development.
MySQL has a set of utilities written in Python that can do some amazing things for your MySQL instances from setting up replication with automatic fail over to copying database
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
2. Summary
2
This document explains how to use proxysql to leave query logs that users execute
directly by connecting to db.
This document includes the following things:
- How to install proxysql
- How to set up proxysql for query logging
- How to convert binary format query log to text format in proxysql
- How to install ELK stack and set up
4. Set up ProxySQL 4
# rpm -ivh proxysql-2.0.4-1-centos67.x86_64.rpm
# service proxysql start
# mysql -u admin -padmin -h 127.0.0.1 -P6032 --prompt='Admin> '
Install & Start & Connect proxysql
https://github.com/sysown/proxysql/releases/tag/v2.0.4
5. 5
Set up ProxySQL
-- Adding User
insert into mysql_users(username, password, transaction_persistent)
values('test_user','test1234', 0);
load mysql users to runtime;
save mysql users to disk;
-- Adding Server Information
insert into mysql_servers(hostname,hostgroup_id,port,comment)
values('10.xx.xx.01',0,3306,'customerdb');
insert into mysql_servers(hostname,hostgroup_id,port,comment)
values('10.xx.xx.02',1,3306,'productdb');
insert into mysql_servers(hostname,hostgroup_id,port,comment)
values('10.xx.xx.03',2,3306,'orderdb');
load mysql servers to runtime;
save mysql servers to disk;
Set up configuration
6. 6
Set up ProxySQL
-- Set enable query logging
set mysql-eventslog_filename='queries.log';
set mysql-eventslog_filesize=1048576;
load mysql variables to runtime;
save mysql variables to disk;
insert into mysql_query_rules (rule_id, active, match_digest, log,apply)
values (1,1,'.',1,0);
insert into mysql_query_rules
(rule_id, active, flagIN, match_pattern, re_modifiers, replace_pattern, apply)
values (2,1,0,'n','GLOBAL',' ',0);
load mysql query rules to runtime;
save mysql query rules to disk;
Set up configuration
7. 7
Set up ProxySQL
-- Add query rules for query routing
insert into mysql_query_rules(active,match_pattern, destination_hostgroup,apply)
values(1,'^SELECT .* customerdb',0,1);
insert into mysql_query_rules(active,match_pattern, destination_hostgroup,apply)
values(1,'^SELECT .* productdb',1,1);
insert into mysql_query_rules(active,match_pattern, destination_hostgroup,apply)
values(1,'^SELECT .* orderdb',2,1);
load mysql query rules to runtime;
save mysql query rules to disk;
Set up configuration
8. 8
Set up ProxySQL
-- You can check to connect db
# mysql -utest_user -p'test1234' -h10.xx.xx.001 -P6033 -e"SELECT * from customerdb.member limit 10"
# mysql -utest_user -p'test1234' -h10.xx.xx.001 -P6033 -e"SELECT * from orderdb.order limit 10"
-- You can find a query log file(queries.log.0000000x)
# cd /var/lib/proxysql
# ls -al
total 7876
drwxr-xr-x 4 proxysql proxysql 4096 Jun 20 14:47 .
drwxr-xr-x. 27 root root 4096 Jun 19 18:26 ..
-rw------- 1 proxysql proxysql 2177193 Jun 20 14:47 proxysql.log
-rw-r--r-- 1 proxysql proxysql 6 Jun 20 11:18 proxysql.pid
-rw------- 1 proxysql proxysql 2781184 Jun 20 14:47 proxysql_stats.db
-rw------- 1 proxysql proxysql 89474 Jun 20 14:19 queries.log.00000001
Check query logging
9. 9
Convert query log
ProxySQL leaves a query log in binary format.
We need to convert it to text format.
ProxySQL supports tool that convert binary format file to text
format.(eventslog_reader_sample)
# git clone https://github.com/sysown/proxysql.git
# cd proxysql/tools
# ll
total 128
-rwxr-xr-x 1 root root 2896 Jun 11 10:28 check_variables.pl
-rw-r--r-- 1 root root 4527 Jun 11 10:28 eventslog_reader_sample.cpp
-rw-r--r-- 1 root root 122 Jun 11 10:28 Makefile
-rwxr-xr-x 1 root root 16869 Jun 11 10:28 proxysql_galera_checker.sh
-rwxr-xr-x 1 root root 3574 Jun 11 10:28 proxysql_galera_writer.pl
10. 10
Convert query log
You can compile eventslog_reader_sample.cpp file.
And you can convert binary query log file to text format.
# make
# ll
total 128
-rwxr-xr-x 1 root root 2896 Jun 11 10:28 check_variables.pl
-rwxr-xr-x 1 root root 87168 Jun 11 10:41 eventslog_reader_sample
-rw-r--r-- 1 root root 4527 Jun 11 10:28 eventslog_reader_sample.cpp
-rw-r--r-- 1 root root 122 Jun 11 10:28 Makefile
-rwxr-xr-x 1 root root 16869 Jun 11 10:28 proxysql_galera_checker.sh
-rwxr-xr-x 1 root root 3574 Jun 11 10:28 proxysql_galera_writer.pl
# ./eventslog_reader_sample /var/lib/proxysql/queries.log.00000001
or
# ./eventslog_reader_sample /var/lib/proxysql/queries.log.00000001 > result.txt
11. 11
Convert query log
You can see two types of query logs, binary and text.
# tail -n 10 queries.log.00000001
��:�L14SHOW SESSION VARIABLES LIKE 'lower_case_table_names'w�� test_userinformation_schema10.xxx.xxx.x:1475310.xx.xx.xx:3306�KSw/���RSELECT current_user()x��
test_userinformation_schema10.xxx.xxx.xx:1475310.xx.xx.xx:3306��cw�X��Efw�X��
��l��SET CHARACTER SET utf8g�� test_userinformation_schema10.xxx.xxx.xxx:14753�����������pw�X���pw�X����8���Q0.xxx.x:14753x��
test_userinformation_schema10.xxx.xxx.x:1475310.xx.xx.xx:3306�K~w�X���w�X���$�f��SET SQL_SAFE_UPDATES=1o��
test_userinformation_schema10.xxx.xxx.x:14753����������ۏw�X��ۏw�X��yS��g�H�0.xxx.x:14753ION_ID()���
test_userinformation_schema10.xxx.xxx.x:1475310.xx.xx.xx:3306���w�X��h�w�X��Y�Z�~a��%SHOWorderdb10.xxx.xxx.x:14753����������ӹw�X��ӹw�X��0�o�/0.xxx.x:14753=1
# tail -n 10 queries_result.log.00000001
ProxySQL LOG QUERY: thread_id="30" username="test_user" schemaname="information_schema" client="10.xxx.xxx.x:63285" HID=0 server="10.xx.xx.xxx:3306" starttime="2019-06-20 17:35:20.108808" endtime="2019-06-20
17:35:20.109038" duration=230us digest="0xD8CCF296B01AC933"
select *
from orderdb.order
limit 105
ProxySQL LOG QUERY: thread_id="30" username="test_user" schemaname="information_schema" client="10.xxx.xxx.x:63285" HID=0 server="10.xx.xx.xxx:3306" starttime="2019-06-20 17:35:21.300584" endtime="2019-06-20
17:35:21.300738" duration=154us digest="0xD8CCF296B01AC933"
select *
from customerdb.custom
limit 100
12. 12
Make cron job for converting query log file
Query log file will be created sequentially, we need a script for converting binary query
log file to text format file whenever new query log file is created.
BIN TXTqueries.log.00000001 ./result/queries_result.log.00000001
./eventslog_reader_sample queries.log.00000001 > ./result/queries_result.log.00000001
BIN TXTqueries.log.00000002 ./result/queries_result.log.00000002
BIN TXTqueries.log.00000003 ./result/queries_result.log.00000003
13. 13
How user uses with workbench tool
The user who uses to connect directly db can use like this with workbench.
User has to use table name with database name(schema name) in query.
(ex. orderdb.order)
14. 14
Set up FileBeat
Download
# tar -xzvf filebeat-7.1.1-linux-x86_64.tar.gz -C /opt/
# vi filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/lib/proxysql/result/queries_result.log.*
multiline.pattern: '^ProxySQL'
multiline.negate: true
multiline.match: after
output.logstash:
hosts: ["localhost:5044"]
https://www.elastic.co/kr/downloads/beats/filebeat
15. 15
Set up Logstash
Download
# tar -xzvf logstash-7.1.1.tar.gz -C /opt/
# cd config
# vi logstash.conf
input {
beats {
port => 5044
}
}
---continue behind
https://www.elastic.co/kr/downloads/logstash
17. 17
Set up Logstash
# vi logstash.conf
output {
elasticsearch {
hosts => ["http://[elasticsearch_server_ip]:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
18. 18
Set up Elasticsearch
Download
# tar -xzvf elasticsearch-7.1.1-linux-x86_64.tar.gz -C /opt/
# cd config
# vi elasticsearch.yml
cluster.name: db-admin
node.name: first-node
path.data: /elasticsearch/data
path.logs: /elasticsearch/log
bootstrap.system_call_filter: false
network.host: [elasticsearch_server_ip]
http.port: 9200
cluster.initial_master_nodes: "first-node"
https://www.elastic.co/kr/downloads/elasticsearch
19. 19
Set up Kibana
Download
# tar -xzvf kibana-7.1.1-linux-x86_64.tar.gz -C /opt/
# cd config
# vi kibana.yml
server.host: "[kibana_server_ip]"
elasticsearch.hosts: ["http://[elasticsearch_server_ip]:9200"]
https://www.elastic.co/kr/downloads/kibana