This document discusses various profiling tools that can be used to analyze MySQL performance, including Oprofile, perf, pt-pmp, and the MySQL Performance Schema. It provides examples of how these tools have been used to identify and resolve specific MySQL performance bugs. While the Performance Schema is useful, it does not always provide sufficient detail and other system-wide profilers like Oprofile and perf are still needed in some cases to pinpoint performance issues.
This session is about using GNU debugger (gdb) as a tool to study MySQL internals (namely, InnoDB locks and metadata locks) and as a last resort in cases when server hangs or has to be restarted for other reason. It never hurts to try a trick or two before giving up and restarting.
Sometimes MySQL DBAs have to work with stalled/hanged/unresponsive MySQL instance, where their usual SQL-based tricks do not work any more. Sometimes they can not even connect to check what's going on inside server.
In other cases they know what to do and everything still works, but they have to implement changes to read-only server variables. Server restart is often not an option in production, as it means some downtime and may cause negative performance impact.
In these cases one could do something given read and write access to server memory/internals. Here comes gdb, that, alone with careful reading of the source code helps to often resolve the problems described above. During this session I'll show what can be done with gdb when server already is in
troubles, and how to use gdb to "see" and understand MySQL internals (like InnoDB locks or metadata locks) better.
Flame Graphs for MySQL DBAs - FOSDEM 2022 MySQL DevroomValeriy Kravchuk
Flame graph is way to visualize profiling data that allows the most frequent code paths to be identified quickly and accurately. They can be generated using Brendan Gregg's open source programs on github.com/brendangregg/FlameGraph, which create interactive SVG files to be checked in browser. The source of profiling data does not really matter - it can be perf profiler, bpftrace, Performance Schema, EXPLAIN output or any other source that allows to convert the data into the expected format of comma-separated "path" plus metric per line.
Different types of Flame Graphs (CPU, Off-CPU, Memory, Differential etc) are presented. Various tools and approaches to collect profile information of different aspects of MySQL server internal working are presented Several real-life use cases where Flame Graphs helped to understand and solve the problem are discussed.
This session is about using GNU debugger (gdb) as a tool to study MySQL internals (namely, InnoDB locks and metadata locks) and as a last resort in cases when server hangs or has to be restarted for other reason. It never hurts to try a trick or two before giving up and restarting.
Sometimes MySQL DBAs have to work with stalled/hanged/unresponsive MySQL instance, where their usual SQL-based tricks do not work any more. Sometimes they can not even connect to check what's going on inside server.
In other cases they know what to do and everything still works, but they have to implement changes to read-only server variables. Server restart is often not an option in production, as it means some downtime and may cause negative performance impact.
In these cases one could do something given read and write access to server memory/internals. Here comes gdb, that, alone with careful reading of the source code helps to often resolve the problems described above. During this session I'll show what can be done with gdb when server already is in
troubles, and how to use gdb to "see" and understand MySQL internals (like InnoDB locks or metadata locks) better.
Flame Graphs for MySQL DBAs - FOSDEM 2022 MySQL DevroomValeriy Kravchuk
Flame graph is way to visualize profiling data that allows the most frequent code paths to be identified quickly and accurately. They can be generated using Brendan Gregg's open source programs on github.com/brendangregg/FlameGraph, which create interactive SVG files to be checked in browser. The source of profiling data does not really matter - it can be perf profiler, bpftrace, Performance Schema, EXPLAIN output or any other source that allows to convert the data into the expected format of comma-separated "path" plus metric per line.
Different types of Flame Graphs (CPU, Off-CPU, Memory, Differential etc) are presented. Various tools and approaches to collect profile information of different aspects of MySQL server internal working are presented Several real-life use cases where Flame Graphs helped to understand and solve the problem are discussed.
Instant add column for inno db in mariadb 10.3+ (fosdem 2018, second draft)Valerii Kravchuk
My slides for (canceled due to personal issues) talk for FOSDEM 2018 MySQL Devroom. I planned to discuss the history of ALTER TABLE speedup and optimization in MySQL, explain the implementation of instant ADD COLUMN for InnoDB tables in MariaDB 10.3 and compare performance of recent versions of MariaDB 10.2, 10.3, Percona Server 5.7, MyRocks from MariaDB 10.2 and MySQL 8.0.4 while working on multiple step test case incolving ALTER TABLE ... ADD COLUMN.
A lof of links to related manuals, blog posts and resources are presented.
MariaDB Server on macOS - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
Current MariaDB Server GA versions are formally not supported (and probably not even regularly built or tested) on macOS 10.x and 11.y. But it's relatively easy to set up the environment and build MariaDB Server from current 10.2 - 10.8 GitHub sources, with few minor issues to resolve in the process, depending on macOS and major server version used.
This talk is a summary of my related experience on 10.13 High Sierra that I had a chance to work on recently, with additional quick review of related fixed and open bugs, as well as some unique features like DTrace support that one may benefit from on macOS. Actually, studying DTrace in context of MariaDB Server troubleshooting and performance tuning was one of the goals why I started to use macOS again.
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021Valeriy Kravchuk
Tools and approaches based on /proc sampling (like 0x.tools by Tanel Poder or ad hoc scripts) allow to measure individual thread level activity in MySQL server on Linux, like thread sleep states, currently executing system calls and kernel wait locations. If needed you can drill down into CPU usage of any thread or the system as a whole. Historical data can be captured for post factum analysis, without much impact on the system and no need to install or change anything in its configuration. In this presentation I am going to summarize what's possible with /proc and show useful examples for MySQL DBAs.
More on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
bpftrace is a relatively new open source tracer for modern Linux (kernels 5.x.y) that may help to troubleshoot performance issues in production as well as to get insights on how software really works. I use it for a couple of years and would like to present more details on how to do it efficiently, including but not limited to adding user probes to different lines of the code inside functions, checking values of local variables and using bpftrace as a code coverage tool.
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Valeriy Kravchuk
Linux with kernels 2.6+. provides different ways to add user probes to almost every other line of code dynamically, and collect the resulting trace and profiling data in a safe and efficient way. This session discusses basic use of ftrace, perf, bcc tools and bpftrace utility, highlights typical problems MariaDB DBAs and developers may hit while trying to apply them, as well as solutions to some of them.
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)Valeriy Kravchuk
The recently released MariaDB 10.5 GA includes many new, useful features, but I’d like to concentrate on those helping DBAs and support engineers to find out what’s going on when a problem occurs.
Specifically I present and discuss the Performance Schema updates to match MySQL 5.7 instrumentation, new tables in the INFORMATION_SCHEMA to monitor the internals of a generic thread pool and improvements of ANALYZE for statements.
MySQL indexing is one of the areas where you can achieve the most performance gains. In this presentation we have a short look how we can improve MySQL performance with indexing...
Nginx and friends - putting a turbo button on your siteWim Godden
Whenever you build a dynamic site with user-specific content, each hit to the site causes a hit to the .Net/PHP/Ruby/Python backend, potentially causing scalability issues. In this talk, we’ll look at a new way of handling user-specific content which takes the load away from the backend. Be prepared to learn about some exciting tools that add a ‘turbo’ button to your infrastructure
Indexing with MySQL. What indexes are, where to use them and what to care about.
Indexierung mit MySQL. Was Indizes sind, wo man sie nutzen sollte und worauf man achten sollte.
Instant add column for inno db in mariadb 10.3+ (fosdem 2018, second draft)Valerii Kravchuk
My slides for (canceled due to personal issues) talk for FOSDEM 2018 MySQL Devroom. I planned to discuss the history of ALTER TABLE speedup and optimization in MySQL, explain the implementation of instant ADD COLUMN for InnoDB tables in MariaDB 10.3 and compare performance of recent versions of MariaDB 10.2, 10.3, Percona Server 5.7, MyRocks from MariaDB 10.2 and MySQL 8.0.4 while working on multiple step test case incolving ALTER TABLE ... ADD COLUMN.
A lof of links to related manuals, blog posts and resources are presented.
MariaDB Server on macOS - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
Current MariaDB Server GA versions are formally not supported (and probably not even regularly built or tested) on macOS 10.x and 11.y. But it's relatively easy to set up the environment and build MariaDB Server from current 10.2 - 10.8 GitHub sources, with few minor issues to resolve in the process, depending on macOS and major server version used.
This talk is a summary of my related experience on 10.13 High Sierra that I had a chance to work on recently, with additional quick review of related fixed and open bugs, as well as some unique features like DTrace support that one may benefit from on macOS. Actually, studying DTrace in context of MariaDB Server troubleshooting and performance tuning was one of the goals why I started to use macOS again.
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021Valeriy Kravchuk
Tools and approaches based on /proc sampling (like 0x.tools by Tanel Poder or ad hoc scripts) allow to measure individual thread level activity in MySQL server on Linux, like thread sleep states, currently executing system calls and kernel wait locations. If needed you can drill down into CPU usage of any thread or the system as a whole. Historical data can be captured for post factum analysis, without much impact on the system and no need to install or change anything in its configuration. In this presentation I am going to summarize what's possible with /proc and show useful examples for MySQL DBAs.
More on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
bpftrace is a relatively new open source tracer for modern Linux (kernels 5.x.y) that may help to troubleshoot performance issues in production as well as to get insights on how software really works. I use it for a couple of years and would like to present more details on how to do it efficiently, including but not limited to adding user probes to different lines of the code inside functions, checking values of local variables and using bpftrace as a code coverage tool.
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Valeriy Kravchuk
Linux with kernels 2.6+. provides different ways to add user probes to almost every other line of code dynamically, and collect the resulting trace and profiling data in a safe and efficient way. This session discusses basic use of ftrace, perf, bcc tools and bpftrace utility, highlights typical problems MariaDB DBAs and developers may hit while trying to apply them, as well as solutions to some of them.
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)Valeriy Kravchuk
The recently released MariaDB 10.5 GA includes many new, useful features, but I’d like to concentrate on those helping DBAs and support engineers to find out what’s going on when a problem occurs.
Specifically I present and discuss the Performance Schema updates to match MySQL 5.7 instrumentation, new tables in the INFORMATION_SCHEMA to monitor the internals of a generic thread pool and improvements of ANALYZE for statements.
MySQL indexing is one of the areas where you can achieve the most performance gains. In this presentation we have a short look how we can improve MySQL performance with indexing...
Nginx and friends - putting a turbo button on your siteWim Godden
Whenever you build a dynamic site with user-specific content, each hit to the site causes a hit to the .Net/PHP/Ruby/Python backend, potentially causing scalability issues. In this talk, we’ll look at a new way of handling user-specific content which takes the load away from the backend. Be prepared to learn about some exciting tools that add a ‘turbo’ button to your infrastructure
Indexing with MySQL. What indexes are, where to use them and what to care about.
Indexierung mit MySQL. Was Indizes sind, wo man sie nutzen sollte und worauf man achten sollte.
Modern Linux Performance Tools for Application TroubleshootingTanel Poder
Modern Linux Performance Tools for Application Troubleshooting.
Mostly demos and focused on application/process troubleshooting, not systemwide summaries.
Video: https://www.youtube.com/watch?v=FJW8nGV4jxY and https://www.youtube.com/watch?v=zrr2nUln9Kk . Tutorial slides for O'Reilly Velocity SC 2015, by Brendan Gregg.
There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes.
This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation.
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)Valeriy Kravchuk
eBPF on Linux 4.9+ is probably the best way to study performance problems. Basic usage of ftrace interface, bcc tools and bpftrace, as well as main bpftrace features and commands are presented. Several typical use cases
(including adding dynamic probes to MariaDB servers, working with stack traces and creating Flame Graphs) are discussed.
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak PROIDEA
Speaker: Andrzej Dyjak
Language: English
In recent years security industry started to grow fond of Apple’s iOS and OS X platforms. This talk will cover one of XNU's flagship debugging utilities: DTrace, a dynamic tracing framework for troubleshooting kernel and application problems on production systems in real time. It will be shown how it can be used in order to ease various tasks within the realm of dynamic binary analysis and beyond.
CONFidence: http://confidence.org.pl/
Common Pitfalls for your Drupal Site, and How to Avoid ThemAcquia
Drupal can be a resource-intensive system. Any moderately complicated site will generate a lot of database queries and use a fair amount of memory to build pages to serve to visitors. With some judicious tuning, however, Drupal can perform really well, and at scale.
In this webinar, Drew Webber, Principal Support Engineer at Acquia, will discuss some common pitfalls encountered by sites that struggle in the face of increased traffic. Attendees will walk away with a deeper understanding of:
-The most common problems encountered when it comes to Drupal site performance
-Ways of identifying performance bottlenecks on your Drupal site
-How to avoid these common pitfalls and remedy these issues (often without writing a single line of code!)
-What not to do when building and running your site
The biggest headine at the 2009 Oracle OpenWorld was when Larry Ellison announced that Oracle was entering the hardware business with a pre-built database machine, engineered by Oracle. Since then businesses around the world have started to use these engineered systems. This beginner/intermediate-level session will take you through my first 100 days of starting to administer an Exadata machine and all the roadblocks and all the success I had along this new path.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
In the ever-evolving landscape of technology, enterprise software development is undergoing a significant transformation. Traditional coding methods are being challenged by innovative no-code solutions, which promise to streamline and democratize the software development process.
This shift is particularly impactful for enterprises, which require robust, scalable, and efficient software to manage their operations. In this article, we will explore the various facets of enterprise software development with no-code solutions, examining their benefits, challenges, and the future potential they hold.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppGoogle
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-fusion-buddy-review
AI Fusion Buddy Review: Key Features
✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini
✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique!
✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs!
✅Fully automated AI articles bulk generation!
✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more.
✅With one keyword or URL, generate complete websites, landing pages, and more…
✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7.
✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches.
✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all!
✅Save over $5000 per year and kick out dependency on third parties completely!
✅Brand New App: Not available anywhere else!
✅ Beginner-friendly!
✅ZERO upfront cost or any extra expenses
✅Risk-Free: 30-Day Money-Back Guarantee!
✅Commercial License included!
See My Other Reviews Article:
(1) AI Genie Review: https://sumonreview.com/ai-genie-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
#AIFusionBuddyReview,
#AIFusionBuddyFeatures,
#AIFusionBuddyPricing,
#AIFusionBuddyProsandCons,
#AIFusionBuddyTutorial,
#AIFusionBuddyUserExperience
#AIFusionBuddyforBeginners,
#AIFusionBuddyBenefits,
#AIFusionBuddyComparison,
#AIFusionBuddyInstallation,
#AIFusionBuddyRefundPolicy,
#AIFusionBuddyDemo,
#AIFusionBuddyMaintenanceFees,
#AIFusionBuddyNewbieFriendly,
#WhatIsAIFusionBuddy?,
#HowDoesAIFusionBuddyWorks
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Graspan: A Big Data System for Big Code AnalysisAftab Hussain
We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs.
We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations.
These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18.
- Accepted in ASPLOS ‘17, Xi’an, China.
- Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17.
- Invited for presentation at SoCal PLS ‘16.
- Invited for poster presentation at PLDI SRC ‘16.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Mobile App Development Company In Noida | Drona InfotechDrona Infotech
Looking for a reliable mobile app development company in Noida? Look no further than Drona Infotech. We specialize in creating customized apps for your business needs.
Visit Us For : https://www.dronainfotech.com/mobile-application-development/
Mobile App Development Company In Noida | Drona Infotech
Applying profilers to my sql (fosdem 2017)
1. Applying profilers to MySQL
From PMP to perf, and why Performance Schema is not a
replacement in all cases
Valerii Kravchuk, Principal Support Engineer
valerii.kravchuk@mariadb.com
1
2. www.percona.com
Who am I?
Valerii (aka Valeriy) Kravchuk:
● MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005 - 2012
● Principal Support Engineer in Percona, 2012 - 2016
● Principal Support Engineer in MariaDB Corporation since March 2016
● http://mysqlentomologist.blogspot.com - my blog about MySQL (a lot about
MySQL bugs, but some HowTos as well)
● https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about
MySQL (mostly bugs…)
● http://bugs.mysql.com - my personal playground. 308 bugs reported in total,
20 since February, 2016
● I like FOSDEM, see slides from my previous talks:
○ http://www.slideshare.net/valeriikravchuk1/fosdem2015-gdb-tips-and-tricks-for-my-sql-db-as
○ http://www.slideshare.net/ValeriyKravchuk/more-on-gdb-for-my-sql-db-as-fosdem-2016
2
3. www.percona.com
What is this session about?
● It’s about profiling MySQL and some tools (“profilers”)
MySQL DBA can use for profiling in production on Linux:
○ Oprofile (including opcontrol, opreport and operf)
○ perf (I think it’s the best and easiest to use now)
○ Few words on PMP (pt-pmp)
● Some real life use case, along the lines of Bug #39630, Bug
#68079 (including studying similar use case with MyRocks),
Bug #83912 and maybe more...
● Why not about gprof, Callgrind, Massif, dtrace, SystemTap?
● Why not about Performance Schema?
● Performance impact of profiling
3
4. www.percona.com
Why not about Performance Schema?
● It may be NOT compiled in (see MySQL from Facebook)
● It may be NOT enabled when server was started (see
MariaDB)
● Specific instruments may not be enabled at startup and
then it’s too late (see Bug #68097)
● Sizing instruments properly (memory used and
performance impact vs details collected) may be
problematic (depends on version also)
● Part of the code or 3rd party plugin may not be
instrumented at all or in enough details (see Bug #83912)
● It does not give you a system-wide profiling, just for
selected parts of MySQL server code
● Other people (including myself) talk and write a lot about it
4
5. www.percona.com
Not Enough Details in Performance Schema
Samples: 52K of event 'cpu-clock', Event
count (approx.): 13037500000
Overhead Command Shared Object Symbol
43.75% mysqld mysqld [.]
Item_func_mul::int_op
16.97% mysqld mysqld [.]
Item_func_benchmark::val_int
14.10% mysqld mysqld [.]
Item_int::val_int
13.50% mysqld mysqld [.]
Item_func_numhybrid::val_int
...
2.58% mysqld mysqld [.]
Item_func_numhybrid::result_type
...
30 SELECT `benchmark` ( ?, ... * ? )
(13055172.39?)
30 stage/sql/init (51.56?)
30 stage/sql/checking permissions
(2.27?)
30 stage/sql/Opening tables (1.00?)
30 stage/sql/After opening tables
(0.62?)
30 stage/sql/init (9.32?)
30 stage/sql/optimizing (7.41?)
30 stage/sql/executing (13055061.32?)
30 stage/sql/end (3.98?)
30 stage/sql/query end (2.34?)
30 stage/sql/closing tables (1.73?)
30 stage/sql/freeing items (4.22?)
30 stage/sql/cleaning up (1.13?)
● Yes, this is for primitive select benchmark(500000000,2*2) from Bug #39630
● Performance Schema query is like 25 lines long to make it readable
5
7. www.percona.com
Oprofile - Success Stories
● Oprofile is historically one of the first statistical profiling tools on Linux
(2001)
● It was widely used while working on MySQL performance problems
● Here is the list of some MySQL bugs found or confirmed using oprofile:
○ Bug #33948 - "performance issue when using sysbench benchmark on a multiple-core
system" - table cache, LOCK_open contention etc, resolved in 5.5+ with metadata locks
introduced. See Bug #58037 also.
○ Bug #49047 - "InnoDB deadlock detection is CPU intensive with many locks on a
single row" - note that innodb_deadlock_detect server variable was added recently (in
MySQL 5.7.15!) to disable deadlock detection entirely.
○ Bug #49169 - "read_view_open_now is inefficient with many concurrent sessions" - “It
was fixed as part of the kernel mutex split in 5.6”, or not?
○ Bug #53825 - "Removing locks from queue is very CPU intensive with many locks". It
is still "Verified" and thus, probably is not fixed.
○ Bug #68818 - "Large Value List in WHERE Clause Spends Too Much Time in 'statistics'
State" - oprofile was used to show where exactly in the code (as opposed to just "statistics"
stage) the time was spent
● More examples and details are presented in my blog post
7
8. www.percona.com
Oprofile - Basic Usage of Versions < 1.0.0
● Check my post, “Oprofile Basics for MySQL Profiling”, for details and
references, but basic minimal steps are:
○ Make sure oprofile package is installed: dpkg -l | grep oprofile
○ Load the OProfile module if required and make the OProfile driver interface available:
sudo opcontrol --init
○ Start data collection (daemon is started if was not running):
sudo opcontrol --start --no-vmlinux
○ Run your problematic load against MySQL server
○ Samples are collected in --session-dir (if it was set), or `pwd`/oprofile_data, or, if that
directory does not exist, the standard session-dir of /var/lib/oprofile is used
○ Stop data collection:
sudo opcontrol --stop
○ Flush collected daemon data:
sudo opcontrol --dump
○ Run opreport to see the data collected:
sudo opreport --demangle=smart --symbols --merge tgid path_to_mysqld
○ Check pt-stalk source code for the way --collect-oprofile option is implemented
8
9. www.percona.com
Oprofile - Basic Usage of Versions >= 1.0.0
● No need for daemon (since 0.9.8), no opcontrol. operf allows to profile a
single process (--pid, may work for normal user) or every currently running
process (--system-wide). operf interfaces with the kernel to collect samples
via the Linux Kernel Performance Events Subsystem (see perf).
● Less steps needed:
○ Start system-wide data collection :
sudo operf -s
○ Run your problematic load against MySQL server
○ Interrupt the operf process (Ctrl-C works if it was foreground):
sudo kill -SIGINT pid_of_operf
○ Run opreport to see the data collected:
sudo opreport --demangle=smart --symbols --merge tgid path_to_mysqld
● My naive experience with --pid `pidof mysqld` option on Ubuntu 14.04 and
Fedora 25 was negative, use system-wide profiling
● Check this pull request for pt-stalk (one day it will work with oprofile 1.0.0+)
9
10. www.percona.com
Oprofile - Results You Get (Primitive Example)
● Profiling select benchmark(500000000, 2*2):
openxs@ao756:~$ sudo operf -s
operf: Press Ctl-c or 'kill -SIGINT 12723' to stop profiling
operf: Profiler started
^C
Profiling done.
openxs@ao756:~$ sudo opreport --demangle=smart --symbols --merge tgid
/home/openxs/dbs/maria10.1/bin/mysqld | head -20
Using /home/openxs/oprofile_data/samples/ for samples directory.
warning: /no-vmlinux could not be found.
CPU: Intel Sandy Bridge microarchitecture, speed 1.5e+06 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00
(No unit mask) count 100000
samples % image name symbol name
96078 32.0059 mysqld Item_func_mul::int_op()
51487 17.1516 mysqld Item_hybrid_func::result_type() const
51407 17.1249 mysqld Item_func_hybrid_field_type::val_int()
47570 15.8467 mysqld Item_func_benchmark::val_int()
31575 10.5184 mysqld Item_int::val_int()
10925 3.6394 mysqld Type_handler_int_result::cmp_type() const
10659 3.5508 mysqld Type_handler_int_result::result_type() const
448 0.1492 no-vmlinux /no-vmlinux
...
10
11. www.percona.com
pt-pmp (Poor Man’s Profiler)
● http://www.percona.com/doc/percona-toolkit/2.2/pt-pmp.html
pt-pmp [-i 1] [-s 0] [-b mysqld] [-p pidofmysqld] [-l 0] [-k file] [--version]
● It is based on original idea by Domas, http://poormansprofiler.org/
● One of the recent examples how it is used: Bug #78277 - InnoDB deadlock,
thread stuck on kernel calls from transparent page compression, “Open”
● When mysqld hangs or is slow, you can get some insight quickly: for
example, Bug #75028 (HandlerSocket “hangs” on shutdown)
● When there are stalls, use pt-pmp to find out why (or what threads mostly
do at the moment): Bug #69810
● Use in production as a last resort (may hang mysqld, --SIGCONT)
● pt-pmp surely slows server down :) Hints:
○ https://bugs.launchpad.net/percona-toolkit/+bug/1320168 - partial
workaround
○ Use quickstack instead of gdb (check this discussion)
11
12. www.percona.com
pt-pmp Applied to “statistics” Case of Bug #83912
MariaDB [test]> select * from t0 where id = 15;
+----+------+--------------------+
| id | c1 | c2 |
+----+------+--------------------+
| 15 | 290 | 0.7441205286831786 |
+----+------+--------------------+
1 row in set ( 52.27 sec)
1
select(libc.so.6), os_thread_sleep(os0thread.cc:303), srv_conc_enter_innodb_wi
th_atomics(srv0conc.cc:298),srv_conc_enter_innodb(srv0conc.cc:298),innobase
_srv_conc_enter_innodb(ha_innodb.cc:1906), ha_innobase::index_read(ha_innodb.
cc:1906),handler::index_read_idx_map(handler.cc:5441),handler::ha_index_rea
d_idx_map(handler.cc:2646),join_read_(handler.cc:2646),join_read__table(han
dler.cc:2646), make_join_statistics(sql_select.cc:3935),JOIN::optimize_inner(
sql_select.cc:1366), JOIN::optimize(sql_select.cc:1045), mysql_select(sql_sele
ct.cc:3430),handle_select(sql_select.cc:372),execute_sqlcom_select(sql_pars
e.cc:5896),mysql_execute_command(sql_parse.cc:2971),mysql_parse(sql_parse.c
c:7319),dispatch_command(sql_parse.cc:1488),do_command(sql_parse.cc:1109),d
o_handle_one_connection(sql_connect.cc:1349),handle_one_connection(sql_conn
ect.cc:1261),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),cl
one(libc.so.6)
...
12
14. www.percona.com
perf - Success Stories
● perf (sometimes called perf_events or perf tools, originally Performance
Counters for Linux, PCL) is a new performance analyzing tool for Linux,
available from kernel version 2.6.31 (supported by RHEL6 since 2010)
● It is easier to use (IMHO) and more popular recently for MySQL
● Here is the list of some MySQL bugs by Mark Callaghan confirmed using
perf:
○ Bug #69236 - "performance regressions for single-threaded workloads, part 2" -
MySQL 5.6 is spending a lot more time in rec_get_offsets_func,
trx_undo_report_row_operation, btr_cur_optimistic_insert. Same in 5.7.8, “Verified”
○ Bug #74325 - “updates to indexed column much slower in 5.7.5” - nice perf outputs
there. It’s about innodb_fill_factor=100 (that leaves 1/16 free space since 5.7.8).
○ Bug #74280 - “covering secondary scans worse than PK scan at high concurrency” -
the mutex contention that isn't visible in P_S output because the block rw-lock isn't
instrumented. Verified regression since 5.7.5 vs 5.6.x. See also:
Bug #74283 - “Non-covering secondary index scans degrade more in 5.7 than 5.6”
○ http://smalldatum.blogspot.com/2014/10/details-on-range-scan-performance.html - on
two bugs above, perf allows to see the difference
14
15. www.percona.com
perf - Basic Usage
● Check my post, “perf Basics for MySQL Profiling”, for details and references,
but basic minimal steps are:
○ Make sure perf-related packages are installed (perf with RPMs) for your kernel:
sudo apt-get install linux-tools-generic
○ Make sure debug symbols are installed and software is built with -fno-omit-frame-pointer
○ Start data collection for some time using perf record:
sudo perf record -a [-g] [-F99] [-p `pidof mysqld`] sleep 30
Run your problematic load against MySQL server
○ Samples are collected in `pwd`/perf.data by default
○ Process samples and display the profile using perf report:
sudo perf report [-n] [-g] --stdio
● Alternatively, run in foreground and interrupt any time with Ctrl-C:
[root@centos ~]# perf record -ag
^C
● Or run in background and send -SIGINT when done:
[root@centos ~]# perf record -ag &
[1] 2353
[root@centos ~]# kill -sigint 2353
● Let’s see how it works alive… (demo). We’ll see perf top, perf record -g etc
15
17. www.percona.com
On perf vs Oprofile
● These days perf is, IMHO, better than oprofile/operf for MySQL profiling:
○ perf has a lot of features and requires less steps for basic usage
○ Call graphs are easier to get and more useful with perf
○ Classical oprofile (with daemon etc) has more severe impact on performance
○ In some environments operf just doesn’t work (while perf does):
[root@centos ~]# operf --version
Your kernel's Performance Events Subsystem does not support your processor
type.
Please use the opcontrol command instead of operf.
● Check the following blog post and discussion:
○ “perf: the good, the bad, the ugly”, by Robert Haas
○ Oprofile vs Perf from oprofile-list@lists.sf.net
● --collect-oprofile in pt-stalk may not work well for Galera (see lp:1152571)
● Time to add --collect-perf to pt-stalk!
● PMP is special (ease of use for specific tasks vs performance impact/hangs)
17
18. www.percona.com
Studying Hanging in “statistics” Case(s)
● See my blog post for details and full outputs:
| |--71.70%-- srv_conc_enter_innodb(trx_t*)
| | ha_innobase::index_read(...)
| | handler::index_read_idx_map(...)
| | handler::ha_index_read_idx_map(...)
| | join_read_const(st_join_table*)
| | join_read_const_table(THD*, ...)
| | make_join_statistics(JOIN*, ...)
| | JOIN::optimize_inner()
| | JOIN::optimize()
| | mysql_select(THD*, ...)
...
● We can see that time to do SELECT is mostly spent waiting to enter InnoDB
queue while reading data via index (dive) to get statistics for the optimizer
● We can also see what UPDATEs are doing at the moment (-g caller ?)
● We can see where the time is spent by kernel and other processes (-a)
18
19. www.percona.com
Studying MyRocks Performance - Test Case
● See my blog post “Profiling MyRocks with perf: Good Old Bug #68079
Use Case” for details
● Essentially, why this query:
mysql> explain select count(*), category from task straight_join incident on
task.sys_id=incident.sys_id group by incident.category;
...
| 1 | SIMPLE | task | index | PRIMARY | PRIMARY | 96
| NULL | 8292 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | incident | eq_ref | PRIMARY,incident_category | PRIMARY | 96
| test.task.sys_id | 1 | NULL |
runs faster (scales better?) than this one with MyRocks (unlike with InnoDB):
mysql> explain select count(*), category from task inner join incident on
task.sys_id=incident.sys_id group by incident.category;
...
| 1 | SIMPLE | incident | index | PRIMARY,incident_category |
incident_category | 123 | NULL | 8192 | Using index |
| 1 | SIMPLE | task | eq_ref | PRIMARY | PRIMARY
| 96 | test.incident.sys_id | 1 | Using index |
19