The document discusses various tools that can be used for tracing and profiling MySQL, including Linux tools like strace, gdb, ftrace, bpftrace, perf, and dynamic probes. It focuses on perf as one of the best and easiest tools to use for tracing and profiling MySQL in production on Linux. Examples are provided of using perf to add probes to MySQL dynamically to capture SQL queries.
Instant add column for inno db in mariadb 10.3+ (fosdem 2018, second draft)Valerii Kravchuk
My slides for (canceled due to personal issues) talk for FOSDEM 2018 MySQL Devroom. I planned to discuss the history of ALTER TABLE speedup and optimization in MySQL, explain the implementation of instant ADD COLUMN for InnoDB tables in MariaDB 10.3 and compare performance of recent versions of MariaDB 10.2, 10.3, Percona Server 5.7, MyRocks from MariaDB 10.2 and MySQL 8.0.4 while working on multiple step test case incolving ALTER TABLE ... ADD COLUMN.
A lof of links to related manuals, blog posts and resources are presented.
This session is about using GNU debugger (gdb) as a tool to study MySQL internals (namely, InnoDB locks and metadata locks) and as a last resort in cases when server hangs or has to be restarted for other reason. It never hurts to try a trick or two before giving up and restarting.
Sometimes MySQL DBAs have to work with stalled/hanged/unresponsive MySQL instance, where their usual SQL-based tricks do not work any more. Sometimes they can not even connect to check what's going on inside server.
In other cases they know what to do and everything still works, but they have to implement changes to read-only server variables. Server restart is often not an option in production, as it means some downtime and may cause negative performance impact.
In these cases one could do something given read and write access to server memory/internals. Here comes gdb, that, alone with careful reading of the source code helps to often resolve the problems described above. During this session I'll show what can be done with gdb when server already is in
troubles, and how to use gdb to "see" and understand MySQL internals (like InnoDB locks or metadata locks) better.
MariaDB Server on macOS - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
Current MariaDB Server GA versions are formally not supported (and probably not even regularly built or tested) on macOS 10.x and 11.y. But it's relatively easy to set up the environment and build MariaDB Server from current 10.2 - 10.8 GitHub sources, with few minor issues to resolve in the process, depending on macOS and major server version used.
This talk is a summary of my related experience on 10.13 High Sierra that I had a chance to work on recently, with additional quick review of related fixed and open bugs, as well as some unique features like DTrace support that one may benefit from on macOS. Actually, studying DTrace in context of MariaDB Server troubleshooting and performance tuning was one of the goals why I started to use macOS again.
Flame Graphs for MySQL DBAs - FOSDEM 2022 MySQL DevroomValeriy Kravchuk
Flame graph is way to visualize profiling data that allows the most frequent code paths to be identified quickly and accurately. They can be generated using Brendan Gregg's open source programs on github.com/brendangregg/FlameGraph, which create interactive SVG files to be checked in browser. The source of profiling data does not really matter - it can be perf profiler, bpftrace, Performance Schema, EXPLAIN output or any other source that allows to convert the data into the expected format of comma-separated "path" plus metric per line.
Different types of Flame Graphs (CPU, Off-CPU, Memory, Differential etc) are presented. Various tools and approaches to collect profile information of different aspects of MySQL server internal working are presented Several real-life use cases where Flame Graphs helped to understand and solve the problem are discussed.
Instant add column for inno db in mariadb 10.3+ (fosdem 2018, second draft)Valerii Kravchuk
My slides for (canceled due to personal issues) talk for FOSDEM 2018 MySQL Devroom. I planned to discuss the history of ALTER TABLE speedup and optimization in MySQL, explain the implementation of instant ADD COLUMN for InnoDB tables in MariaDB 10.3 and compare performance of recent versions of MariaDB 10.2, 10.3, Percona Server 5.7, MyRocks from MariaDB 10.2 and MySQL 8.0.4 while working on multiple step test case incolving ALTER TABLE ... ADD COLUMN.
A lof of links to related manuals, blog posts and resources are presented.
This session is about using GNU debugger (gdb) as a tool to study MySQL internals (namely, InnoDB locks and metadata locks) and as a last resort in cases when server hangs or has to be restarted for other reason. It never hurts to try a trick or two before giving up and restarting.
Sometimes MySQL DBAs have to work with stalled/hanged/unresponsive MySQL instance, where their usual SQL-based tricks do not work any more. Sometimes they can not even connect to check what's going on inside server.
In other cases they know what to do and everything still works, but they have to implement changes to read-only server variables. Server restart is often not an option in production, as it means some downtime and may cause negative performance impact.
In these cases one could do something given read and write access to server memory/internals. Here comes gdb, that, alone with careful reading of the source code helps to often resolve the problems described above. During this session I'll show what can be done with gdb when server already is in
troubles, and how to use gdb to "see" and understand MySQL internals (like InnoDB locks or metadata locks) better.
MariaDB Server on macOS - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
Current MariaDB Server GA versions are formally not supported (and probably not even regularly built or tested) on macOS 10.x and 11.y. But it's relatively easy to set up the environment and build MariaDB Server from current 10.2 - 10.8 GitHub sources, with few minor issues to resolve in the process, depending on macOS and major server version used.
This talk is a summary of my related experience on 10.13 High Sierra that I had a chance to work on recently, with additional quick review of related fixed and open bugs, as well as some unique features like DTrace support that one may benefit from on macOS. Actually, studying DTrace in context of MariaDB Server troubleshooting and performance tuning was one of the goals why I started to use macOS again.
Flame Graphs for MySQL DBAs - FOSDEM 2022 MySQL DevroomValeriy Kravchuk
Flame graph is way to visualize profiling data that allows the most frequent code paths to be identified quickly and accurately. They can be generated using Brendan Gregg's open source programs on github.com/brendangregg/FlameGraph, which create interactive SVG files to be checked in browser. The source of profiling data does not really matter - it can be perf profiler, bpftrace, Performance Schema, EXPLAIN output or any other source that allows to convert the data into the expected format of comma-separated "path" plus metric per line.
Different types of Flame Graphs (CPU, Off-CPU, Memory, Differential etc) are presented. Various tools and approaches to collect profile information of different aspects of MySQL server internal working are presented Several real-life use cases where Flame Graphs helped to understand and solve the problem are discussed.
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021Valeriy Kravchuk
Tools and approaches based on /proc sampling (like 0x.tools by Tanel Poder or ad hoc scripts) allow to measure individual thread level activity in MySQL server on Linux, like thread sleep states, currently executing system calls and kernel wait locations. If needed you can drill down into CPU usage of any thread or the system as a whole. Historical data can be captured for post factum analysis, without much impact on the system and no need to install or change anything in its configuration. In this presentation I am going to summarize what's possible with /proc and show useful examples for MySQL DBAs.
More on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
bpftrace is a relatively new open source tracer for modern Linux (kernels 5.x.y) that may help to troubleshoot performance issues in production as well as to get insights on how software really works. I use it for a couple of years and would like to present more details on how to do it efficiently, including but not limited to adding user probes to different lines of the code inside functions, checking values of local variables and using bpftrace as a code coverage tool.
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Valeriy Kravchuk
Linux with kernels 2.6+. provides different ways to add user probes to almost every other line of code dynamically, and collect the resulting trace and profiling data in a safe and efficient way. This session discusses basic use of ftrace, perf, bcc tools and bpftrace utility, highlights typical problems MariaDB DBAs and developers may hit while trying to apply them, as well as solutions to some of them.
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)Valeriy Kravchuk
The recently released MariaDB 10.5 GA includes many new, useful features, but I’d like to concentrate on those helping DBAs and support engineers to find out what’s going on when a problem occurs.
Specifically I present and discuss the Performance Schema updates to match MySQL 5.7 instrumentation, new tables in the INFORMATION_SCHEMA to monitor the internals of a generic thread pool and improvements of ANALYZE for statements.
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)Valeriy Kravchuk
eBPF on Linux 4.9+ is probably the best way to study performance problems. Basic usage of ftrace interface, bcc tools and bpftrace, as well as main bpftrace features and commands are presented. Several typical use cases
(including adding dynamic probes to MariaDB servers, working with stack traces and creating Flame Graphs) are discussed.
MySQL indexing is one of the areas where you can achieve the most performance gains. In this presentation we have a short look how we can improve MySQL performance with indexing...
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL/MariaDB replication speed has improved a lot thanks to parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present the different implementations, with their limitations and the corresponding tuning parameters. I cover what to do to make parallel replication faster and what to avoid for maximizing parallel replication benefits. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that take advantage of parallel replication speed improvements.
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL/MariaDB replication speed has improved a lot, thanks to parallel replication. MySQL and MariaDB Server have different types of parallel replication; in this talk, I present the different implementations which will allow us to understand their limitations and tuning parameters. I am covering how to make parallel replication faster and what to avoid for maximizing its benefits. I also present benchmark results from Booking.com workloads. Finally, I discuss some deployments at Booking.com that take advantage of parallel replication speed improvements.
Riding the Binlog: an in Deep Dissection of the Replication StreamJean-François Gagné
Binary Logs are the cornerstone of MySQL Replication, but is it fully understood ? To start apprehending this, we can think of the binary logs as a transport for a Stream of Transactions. Traveling from master to slave, sometimes via Intermediate Masters, this stream evolves: it can shrink by the application of filters, can grow by the addition of slave-local transactions, and two streams can merge by the usage of multi-source replication. After presenting the binary logs Stream Model, the different MySQL use-cases will be mapped to the model, which can serve as a validation of the model. After this validation, the model will be used to make prediction on new use-cases/features that could emerge in the future.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in details the different implementations, with their limitations and the corresponding tuning parameters. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021Valeriy Kravchuk
Tools and approaches based on /proc sampling (like 0x.tools by Tanel Poder or ad hoc scripts) allow to measure individual thread level activity in MySQL server on Linux, like thread sleep states, currently executing system calls and kernel wait locations. If needed you can drill down into CPU usage of any thread or the system as a whole. Historical data can be captured for post factum analysis, without much impact on the system and no need to install or change anything in its configuration. In this presentation I am going to summarize what's possible with /proc and show useful examples for MySQL DBAs.
More on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
bpftrace is a relatively new open source tracer for modern Linux (kernels 5.x.y) that may help to troubleshoot performance issues in production as well as to get insights on how software really works. I use it for a couple of years and would like to present more details on how to do it efficiently, including but not limited to adding user probes to different lines of the code inside functions, checking values of local variables and using bpftrace as a code coverage tool.
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Valeriy Kravchuk
Linux with kernels 2.6+. provides different ways to add user probes to almost every other line of code dynamically, and collect the resulting trace and profiling data in a safe and efficient way. This session discusses basic use of ftrace, perf, bcc tools and bpftrace utility, highlights typical problems MariaDB DBAs and developers may hit while trying to apply them, as well as solutions to some of them.
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)Valeriy Kravchuk
The recently released MariaDB 10.5 GA includes many new, useful features, but I’d like to concentrate on those helping DBAs and support engineers to find out what’s going on when a problem occurs.
Specifically I present and discuss the Performance Schema updates to match MySQL 5.7 instrumentation, new tables in the INFORMATION_SCHEMA to monitor the internals of a generic thread pool and improvements of ANALYZE for statements.
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)Valeriy Kravchuk
eBPF on Linux 4.9+ is probably the best way to study performance problems. Basic usage of ftrace interface, bcc tools and bpftrace, as well as main bpftrace features and commands are presented. Several typical use cases
(including adding dynamic probes to MariaDB servers, working with stack traces and creating Flame Graphs) are discussed.
MySQL indexing is one of the areas where you can achieve the most performance gains. In this presentation we have a short look how we can improve MySQL performance with indexing...
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL/MariaDB replication speed has improved a lot thanks to parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present the different implementations, with their limitations and the corresponding tuning parameters. I cover what to do to make parallel replication faster and what to avoid for maximizing parallel replication benefits. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that take advantage of parallel replication speed improvements.
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL/MariaDB replication speed has improved a lot, thanks to parallel replication. MySQL and MariaDB Server have different types of parallel replication; in this talk, I present the different implementations which will allow us to understand their limitations and tuning parameters. I am covering how to make parallel replication faster and what to avoid for maximizing its benefits. I also present benchmark results from Booking.com workloads. Finally, I discuss some deployments at Booking.com that take advantage of parallel replication speed improvements.
Riding the Binlog: an in Deep Dissection of the Replication StreamJean-François Gagné
Binary Logs are the cornerstone of MySQL Replication, but is it fully understood ? To start apprehending this, we can think of the binary logs as a transport for a Stream of Transactions. Traveling from master to slave, sometimes via Intermediate Masters, this stream evolves: it can shrink by the application of filters, can grow by the addition of slave-local transactions, and two streams can merge by the usage of multi-source replication. After presenting the binary logs Stream Model, the different MySQL use-cases will be mapped to the model, which can serve as a validation of the model. After this validation, the model will be used to make prediction on new use-cases/features that could emerge in the future.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in details the different implementations, with their limitations and the corresponding tuning parameters. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
The presentation introduces tools and best practices that will help you introduce the SRE principles for PostgreSQL monitoring.
What to expect:
- An introduction to tools and best practices that will help you introduce SRE principles for PostgreSQL monitoring
- We define the four SRE “golden signals,” and explain how they’re used in PostgreSQL
- A relatively-detailed look at our software stack used for monitoring
- A look at high-level differences between traditional monitoring stacks and modern solutions
- An introduction to Prometheus and how it works with PostgreSQL
- Specific examples of Alertmanager rules for alerting
- A comprehensive walkthrough of Grafana dashboards
- Details on our logging pipeline, including our ELK stack and a few other Showmax specialties
Talk for PerconaLive 2016 by Brendan Gregg. Video: https://www.youtube.com/watch?v=CbmEDXq7es0 . "Systems performance provides a different perspective for analysis and tuning, and can help you find performance wins for your databases, applications, and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes six important areas of Linux systems performance in 50 minutes: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events), static tracing (tracepoints), and dynamic tracing (kprobes, uprobes), and much advice about what is and isn't important to learn. This talk is aimed at everyone: DBAs, developers, operations, etc, and in any environment running Linux, bare-metal or the cloud."
MariaDB 10.2 New Features for Developers, Administrators and DevOps. Window Functions, Common Table Expressions, Check Constraints, GeoJSON, GIS, JSON, Oracle compatibility and MariaDB Connectors
PG Day'14 Russia, PostgreSQL System Architecture, Heikki Linnakangaspgdayrussia
Доклад был представлен на официальной российской конференции PG Day'14 Russia, посвященной вопросам разработки и эксплуатации PostgreSQL.
Software architecture means a high-level view of the components of the system and their relationships. Understanding how various components work together is crucial if you want to start hacking on PostgreSQL, but also for understanding performance characteristics and run-time behavior of production systems.
Software architecture is not a single diagram, but consists of multiple views of the same system.
Process view: describing the processes at runtime and how they communicate.
Data flow: how a query passes through various parts of the system, and how the results flow back.
Data structures: how the Heap and Indexes are organized.
Source code: what's in the PostgreSQL source treem, and many more.
In this presentation, I will give an overview of the most important views necessary to understand how PostgreSQL works.
These are the slides I used to present "MySQL Performance Schema" at FOSSASIA, 2015 Singapore. It gives an overview of Performance Schema and also explains how it could be used to diagnose issues using few use cases.
Performance Analysis: new tools and concepts from the cloudBrendan Gregg
Talk delivered at SCaLE10x, Los Angeles 2012.
Cloud Computing introduces new challenges for performance
analysis, for both customers and operators of the cloud. Apart from
monitoring a scaling environment, issues within a system can be
complicated when tenants are competing for the same resources, and are
invisible to each other. Other factors include rapidly changing
production code and wildly unpredictable traffic surges. For
performance analysis in the Joyent public cloud, we use a variety of
tools including Dynamic Tracing, which allows us to create custom
tools and metrics and to explore new concepts. In this presentation
I'll discuss a collection of these tools and the metrics that they
measure. While these are DTrace-based, the focus of the talk is on
which metrics are proving useful for analyzing real cloud issues.
기존에 저희 회사에서 사용하던 모니터링은 Zabbix 였습니다.
컨테이너 모니터링 부분으로 옮겨가면서 변화가 필요하였고, 이에 대해서 프로메테우스를 활용한 모니터링 방법을 자연스럽게 고민하게 되었습니다.
이에 이영주님께서 테크세션을 진행하였고, 이에 발표자료를 올립니다.
5개의 부분으로 구성되어 있으며, 세팅 방법에 대한 내용까지 포함합니다.
01. Prometheus?
02. Usage
03. Alertmanager
04. Cluster
05. Performance
Similar to Tracing and profiling my sql (percona live europe 2019) draft_1 (20)
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Łukasz Chruściel
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeAftab Hussain
Understanding variable roles in code has been found to be helpful by students
in learning programming -- could variable roles help deep neural models in
performing coding tasks? We do an exploratory study.
- These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
E-commerce Application Development Company.pdfHornet Dynamics
Your business can reach new heights with our assistance as we design solutions that are specifically appropriate for your goals and vision. Our eCommerce application solutions can digitally coordinate all retail operations processes to meet the demands of the marketplace while maintaining business continuity.
Tracing and profiling my sql (percona live europe 2019) draft_1
1. Tracing and Profiling MySQL
Mostly with Linux tools: from strace and gdb to ftrace, bpftrace,
perf and dynamic probes
Valerii Kravchuk, Principal Support Engineer, MariaDB
vkravchuk@gmail.com
1
2. www.percona.com
Who am I?
Valerii (aka Valeriy) Kravchuk:
● MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005-2012
● Principal Support Engineer in Percona, 2012-2016
● Principal Support Engineer in MariaDB Corporation since March 2016
● http://mysqlentomologist.blogspot.com - my blog about MySQL (a lot about
MySQL bugs, but some HowTos as well)
● https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about
MySQL (mostly bugs…)
● http://bugs.mysql.com - my personal playground
● @mysqlbugs #bugoftheday
● Community Contributor of the Year 2019
● I like FOSDEM and used to participate at Percona Live conferences...
2
3. www.percona.com
Sources of tracing and profiling info for MySQL
● Trace files from -debug binaries, optimizer trace files
● (Extended) slow query log (thanks Percona!)
● show [global] status;
● show engine innodb statusG
● show engine innodb mutex;
● InnoDB-related tables in the INFORMATION_SCHEMA
● userstat (Percona Server and other builds)
● show profiles;
● PERFORMANCE_SCHEMA
● Profilers (even simple like pt-pmp or real like perf)
● OS-level tracing and profiling tools
● tcpdump analysis
3
4. www.percona.com
What is this session about?
● It’s about tracing and profiling MySQL, and mostly some
tools MySQL DBA can use for tracing and profiling in
production on Linux:
○ perf (I think it’s the best and easiest to use now)
○ few words on PMP (pt-pmp)
○ ftrace
○ eBPF and related tools (like bpftrace)
○ … and maybe more...
● Why not about gprof, Callgrind, Massif, dtrace, SystemTap?
● Why not entirely about Performance Schema?
● Performance impact of tracing and profiling
4
5. www.percona.com
Why not about Performance Schema?
● It may be NOT compiled in (see MySQL from Facebook)
● It may be NOT enabled when server was started (see
MariaDB)
● Specific instruments may not be enabled at startup and
then it’s too late (see Bug #68097)
● Sizing instruments properly (memory used and
performance impact vs details collected) may be
problematic (depends on version also)
● Part of the code or 3rd party plugin may not be
instrumented at all or in enough details (see Bug #83912)
● It does not give you a system-wide profiling, just for
selected parts of MySQL server code
● Other people (including myself) talk and write a lot about it
5
6. www.percona.com
Not Enough Details in Performance Schema
Samples: 52K of event 'cpu-clock', Event
count (approx.): 13037500000
Overhead Command Shared Object Symbol
43.75% mysqld mysqld [.]
Item_func_mul::int_op
16.97% mysqld mysqld [.]
Item_func_benchmark::val_int
14.10% mysqld mysqld [.]
Item_int::val_int
13.50% mysqld mysqld [.]
Item_func_numhybrid::val_int
...
2.58% mysqld mysqld [.]
Item_func_numhybrid::result_type
...
30 SELECT `benchmark` ( ?, ... * ? )
(13055172.39?)
30 stage/sql/init (51.56?)
30 stage/sql/checking permissions
(2.27?)
30 stage/sql/Opening tables (1.00?)
30 stage/sql/After opening tables
(0.62?)
30 stage/sql/init (9.32?)
30 stage/sql/optimizing (7.41?)
30 stage/sql/executing (13055061.32?)
30 stage/sql/end (3.98?)
30 stage/sql/query end (2.34?)
30 stage/sql/closing tables (1.73?)
30 stage/sql/freeing items (4.22?)
30 stage/sql/cleaning up (1.13?)
● Yes, this is for primitive select benchmark(500000000,2*2) from Bug #39630
● Performance Schema query is like 20 lines long to make it readable
6
7. www.percona.com
Typical “profiling” query to Performance Schema
● This is how it may look like:
SELECT thread_id, event_id, nesting_event_id, CONCAT( CASE WHEN event_name
LIKE 'stage%' THEN
CONCAT(' ', event_name) WHEN event_name LIKE 'wait%' AND
nesting_event_id IS NOT NULL THEN CONCAT(' ', event_name) ELSE
IF(digest_text IS NOT NULL, SUBSTR(digest_text, 1, 64), event_name) END,
' (',ROUND(timer_wait/1000000000, 2),'ms) ') event
FROM (
(SELECT thread_id,
event_id, event_name, timer_wait, timer_start, nesting_event_id,
digest_text FROM events_statements_history_long)
UNION
(SELECT
thread_id, event_id, event_name, timer_wait, timer_start,
nesting_event_id, NULL FROM events_stages_history_long)
UNION
(SELECT
thread_id, event_id, event_name, timer_wait, timer_start,
nesting_event_id, NULL FROM events_waits_history_long)
) events
ORDER BY thread_id, event_id;
7
9. www.percona.com
So, what do I suggest?
● Use Linux tracing tools!
● Yes, all that “...strace, and ltrace, kprobes, and tracepoints, and uprobes, and
ftrace, and perf, and eBPF”
● Julia Evans explains and illustrates them all here
● Brendan D. Gregg explains them all with a lot of details and numerous
examples:
9
10. www.percona.com
Few words on strace
● strace may help MySQL DBA to find out:
● what files are accessed by the mysqld process or utilities, and in what
order
● why some MySQL-related command (silently) fails or hangs
● why some commands end up with permission denied or other errors
● what signals MySQL server and tools get
● what system calls could took a lot of time when something works slow
● when files are opened and closed, and how much data are read
● where the error log and other logs are really located (we can look for
system calls related to writing to stderr, for example)
● how MySQL really works with files, ports and sockets
● See my blog post for more details
● Use in production as a last resort (2 interrupts per system call, even not
those we care about, may leave traced process hanged)
● strace surely slows server down
10
11. www.percona.com
Few words on DTrace
● DTrace
● If you use Oracle Linux, go for it! Making it work on Fedora 29 took me too
much time to complete for the talk
11
13. www.percona.com
Few words on pt-pmp (Poor Man’s Profiler)
● https://www.percona.com/doc/percona-toolkit/LATEST/pt-pmp.html
pt-pmp [-i 1] [-s 0] [-b mysqld] [-p pidofmysqld] [-l 0] [-k file] [--version]
● It is based on original idea by Domas, http://poormansprofiler.org/. I use
the awk code from the above to analyse backtraces of all threads.
● When mysqld hangs or is slow, you can get some insight quickly - use
pt-pmp to find out why (or what threads mostly do at the moment). For
example, see Bug #92108 (fixed in 5.7.25+, binlog access vs P_S query),
● Yet another example of how it is used: Bug #78277 - InnoDB deadlock,
thread stuck on kernel calls from transparent page compression, “Open”
● Use in production as a last resort (may hang mysqld, --SIGCONT)
● pt-pmp surely slows server down :) Hints:
○ https://bugs.launchpad.net/percona-toolkit/+bug/1320168 - partial
workaround
○ Use quickstack instead of gdb (check this discussion)
13
14. www.percona.com
pt-pmp Applied to “statistics” Case of Bug #83912
MariaDB [test]> select * from t0 where id = 15;
+----+------+--------------------+
| id | c1 | c2 |
+----+------+--------------------+
| 15 | 290 | 0.7441205286831786 |
+----+------+--------------------+
1 row in set ( 52.27 sec)
1
select(libc.so.6), os_thread_sleep(os0thread.cc:303), srv_conc_enter_innodb_wi
th_atomics(srv0conc.cc:298),srv_conc_enter_innodb(srv0conc.cc:298),innobase
_srv_conc_enter_innodb(ha_innodb.cc:1906), ha_innobase::index_read(ha_innodb.
cc:1906),handler::index_read_idx_map(handler.cc:5441),handler::ha_index_rea
d_idx_map(handler.cc:2646),join_read_(handler.cc:2646),join_read__table(han
dler.cc:2646), make_join_statistics(sql_select.cc:3935),JOIN::optimize_inner(
sql_select.cc:1366), JOIN::optimize(sql_select.cc:1045), mysql_select(sql_sele
ct.cc:3430),handle_select(sql_select.cc:372),execute_sqlcom_select(sql_pars
e.cc:5896),mysql_execute_command(sql_parse.cc:2971),mysql_parse(sql_parse.c
c:7319),dispatch_command(sql_parse.cc:1488),do_command(sql_parse.cc:1109),d
o_handle_one_connection(sql_connect.cc:1349),handle_one_connection(sql_conn
ect.cc:1261),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),cl
one(libc.so.6)
...
14
16. www.percona.com
A lot about ftrace
● ftrace
● The way you fundamentally interact with ftrace is:
○ Write to files in /sys/kernel/debug/tracing/
○ Read output from files in /sys/kernel/debug/tracing/
●
16
17. www.percona.com
A lot about eBPF
● eBPF is a tiny language for a VM that can be executed inside Linux Kernel. eBPF instructions can
be JIT-compiled into a native code. eBPF was originally conceived to power tools like tcpdump and
implement programmable network packed dispatch and tracing. Since Linux 4.1, eBPF programs
can be attached to kprobes and later - uprobes, enabling efficient programmable tracing
● Brendan Gregg explained it here:
17
20. www.percona.com
Adding uprobes to MySQL dynamically with perf
● The idea was to add dynamic probe to capture SQL queries
● This was done on Ubuntu 16.04 with recent Percona Server 5.7
● First I had to find out with gdb where is the query (hint: dispatch_command
has com_data parameter):
(gdb) p com_data->com_query.query
$4 = 0x7fb0dba8d021 "select 2"
● Then it’s just as easy as follows:
openxs@ao756:~$ sudo perf probe -x /usr/sbin/mysqld 'dispatch_command
com_data->com_query.query:string'
openxs@ao756:~$ sudo perf record -e 'probe_mysqld:dispatch_command*' -aR
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.676 MB perf.data (3 samples) ]
openxs@ao756:~$ sudo perf script >/tmp/queries.txt
openxs@ao756:~$ sudo perf probe --del dispatch_command
20
21. www.percona.com
Adding uprobes to MySQL dynamically with perf
● We have queries captured with probe added on previous slide:
openxs@ao756:~$ cat /tmp/queries.txt
mysqld 31340 [001] 3888.849079: probe_mysqld:dispatch_command:
(be9250) query="select 100"
mysqld 31340 [001] 3891.648739: probe_mysqld:dispatch_command:
(be9250) query="select user, host from mysql.user"
mysqld 31340 [001] 3895.890141: probe_mysqld:dispatch_command:
(be9250) query="select 2"
● We can control output format, but basically we see binary, PID, CPU where
uprobe was executed on, timestamp (milliseconds since start of record),
probe and variables with format we specified
●
21
22. www.percona.com
perf - Success Stories
● perf (sometimes called perf_events or perf tools, originally Performance
Counters for Linux, PCL) is a new performance analyzing tool for Linux,
available from kernel version 2.6.31 (supported by RHEL6 since 2010)
● It is easier to use .and more popular recently for MySQL than older profilers
● Here is the list of some MySQL bugs by Mark Callaghan confirmed by perf:
○ Bug #69236 - "performance regressions for single-threaded workloads, part 2" -
MySQL 5.6 is spending a lot more time in rec_get_offsets_func,
trx_undo_report_row_operation, btr_cur_optimistic_insert. Same in 5.7.8, “Verified”
○ Bug #74325 - “updates to indexed column much slower in 5.7.5” - nice perf outputs
there. It’s about innodb_fill_factor=100 (that leaves 1/16 free space since 5.7.8).
○ Bug #74280 - “covering secondary scans worse than PK scan at high concurrency” -
the mutex contention that isn't visible in P_S output because the block rw-lock isn't
instrumented. Verified regression since 5.7.5 vs 5.6.x. See also:
Bug #74283 - “Non-covering secondary index scans degrade more in 5.7 than 5.6”
○ http://smalldatum.blogspot.com/2014/10/details-on-range-scan-performance.html - on
two bugs above, perf allows to see the difference
22
23. www.percona.com
perf - Basic Usage
● Check my post, “perf Basics for MySQL Profiling”, for details and references,
but basic minimal steps are:
○ Make sure perf-related packages are installed (perf with RPMs) for your kernel:
sudo apt-get install linux-tools-generic
○ Make sure debug symbols are installed and software is built with -fno-omit-frame-pointer
○ Start data collection for some time using perf record:
sudo perf record -a [-g] [-F99] [-p `pidof mysqld`] sleep 30
Run your problematic load against MySQL server
○ Samples are collected in `pwd`/perf.data by default
○ Process samples and display the profile using perf report:
sudo perf report [-n] [-g] --stdio
● Alternatively, run in foreground and interrupt any time with Ctrl-C:
[root@centos ~]# perf record -ag
^C
● Or run in background and send -SIGINT when done:
[root@centos ~]# perf record -ag &
[1] 2353
[root@centos ~]# kill -sigint 2353
● Let’s see how it works alive… (demo). We’ll see perf top, perf record -g etc
23
25. www.percona.com
Studying Hanging in “statistics” Case(s)
● See my blog post for details and full outputs:
| |--71.70%-- srv_conc_enter_innodb(trx_t*)
| | ha_innobase::index_read(...)
| | handler::index_read_idx_map(...)
| | handler::ha_index_read_idx_map(...)
| | join_read_const(st_join_table*)
| | join_read_const_table(THD*, ...)
| | make_join_statistics(JOIN*, ...)
| | JOIN::optimize_inner()
| | JOIN::optimize()
| | mysql_select(THD*, ...)
...
● We can see that time to do SELECT is mostly spent waiting to enter InnoDB
queue while reading data via index (dive) to get statistics for the optimizer
● We can see where the time is spent by kernel and other processes (-a)
25
26. www.percona.com
Am I crazy doing these?
● Quite possible, maybe I just have too much free time :)
● Or maybe I do not know how to use Performance Schema properly
● But I am not alone...
● MariaDB developers are interested in adding dynamic probes with perf while
working on RocksDB storage engine performance problems
● People use perf probes for tracing Oracle RDBMS! There is enough
instrumentation there for almost everything, but still...
● Dynamic tracers are proven tools for instrumenting OS calls (probes for
measuring I/O latency at microsecond precision, for example)
● Another topic, more complex but also more exciting, is dynamic tracing of
RDBMS userspace. This topic is of growing interest with modern servers
hosting large amounts of RAM and workloads that are often CPU-bound.
There are no “waits”, but time is still spent somewhere!
● For open source RDBMS like MySQL there is no reason NOT to use
dynamic probes while UDST are not on every other line of the code :)
● eBPF and bpftrace make it even easier and safer to do these in production
26