SlideShare a Scribd company logo

FOSDEM 2015: gdb tips and tricks for MySQL DBAs

How gdb can help you to solve MySQL problems...

1 of 13
Download to read offline
gdb tips and tricks
for MySQL DBAs
or
How gdb can help you to solve MySQL problems
Valerii Kravchuk, Principal Support Engineer
valerii.kravchuk@percona.com
www.percona.com
Who am I?
Valerii (aka Valeriy) Kravchuk:
● MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005 - 2012
○ Bugs Verification Team all this time
○ Support issues related to bugs, crashes, InnoDB, performance...
○ Trainings (mostly informal) for new team members
○ All kinds of decision making committees…
● Principal Support Engineer in Percona since September, 2012
○ Doing more or less the same as before, but better (I hope)...
○ Plus I try to speak and write about MySQL in public now
● http://mysqlentomologist.blogspot.com - my blog about MySQL (mostly bugs)
● https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about
MySQL (mostly bugs…)
● http://bugs.mysql.com - my personal playground. 6 bugs reported in 2015
www.percona.com
What is this session about?
● Some historical remarks and URLs to known use
cases/blog posts about gdb and MySQL troubleshooting
● Multi-threaded executables and gdb (threads, frames)
● Basic gdb commands and “tricks”
● Few words on pt-pmp use
● Important MySQL data structures to explore
(mostly THD *)
● A couple of real life use cases, working with core dump
and alive mysqld
● Discussion
www.percona.com
Domas is famous for these tricks...
● http://dom.as/2009/02/15/poor-mans-contention-profiling/
● http://dom.as/2009/07/30/evil-replication-management/
● http://dom.as/2010/01/02/read-ahead/
● http://dom.as/2009/03/14/stupid-innodb-tricks/ - not literally
needed since 5.1.38 and InnoDB plugin, innodb_spin_wait_delay :)
● http://poormansprofiler.org/
● http://dom.as/2009/12/29/when-bad-things-happen/
www.percona.com
More examples of gdb use for MySQL DBAs
● Remember the names:
Domas Mituzas, Shane Bester, Roel Van De Paar, Mark Callaghan,
Aurimas, Zhai Weixiang, ...
● http://www.percona.com/blog/2012/09/09/obtain-last-executed-statement-
from-optimized-core-dump/
● http://www.percona.com/blog/2013/11/11/how-to-extract-all-running-queries-
including-the-last-executed-statement-from-a-core-file/
● http://mysqlbugs.blogspot.com.au/2012/09/how-to-obtain-all-executing-
queries.html
● http://www.percona.com/blog/2010/03/23/too-many-connections-no-problem/
www.percona.com
What you can do with gdb
● Check stack traces (and variables), per thread:
thread apply all bt [full]
● Print variables, up to complex one:
thread 1
print do_command::thd->query_string.string.str
● Set new values for variables (global and per thread, even those formally
read-only in MySQL while it’s running):
set max_connections=5000
set opt_log_slave_updates=1
● Call functions (that may do complex changes):
call rpl_filter->add_do_db(strdup("hehehe"))
● Set breakpoints and watchpoints
● Work interactively or use gdb as a command line utility (-batch)
● Use macros, Python scripting, more…
● All these may not work, fail, hang, crash, produce obscure errors…
● You have to read and understand the source code
www.percona.com
pt-pmp (Poor Man’s Profiler)
● http://www.percona.com/doc/percona-toolkit/2.2/pt-pmp.html
pt-pmp [-i 1] [-s 0] [-b mysqld] [-p pidofmysqld] [-l 0] [-k file] [--version]
● It is based on original idea by Domas (http://poormansprofiler.org/) with
some more bash/awk on top applied
● One of the recent examples how it is used (semi-sync replication
performance): http://bugs.mysql.com/bug.php?id=75570
● When mysqld hangs or is slow, you can get some insight quickly: http://bugs.
mysql.com/bug.php?id=75028 (HandlerSocket “hangs” on shutdown)
● When there are stalls, use pt-pmp to find out why (or what threads mostly do
at the moment): http://bugs.mysql.com/bug.php?id=69810
● pt-pmp surely slows server down :) Hint (workaround is in the bug):
https://bugs.launchpad.net/percona-toolkit/+bug/1320168
www.percona.com
Multi-threaded mysqld and gdb
● process/thread/frame concepts:
(gdb) thread 2
[Switching to thread 2 (Thread 0x7fe771550700 (LWP 2544))]
#0 0x0000000000605774 in Item_func_numhybrid::val_int (
this=<value optimized out>)
at /home/openxs/bzr2/percona-5.6/sql/item_func.cc:1013
1013 }
(gdb) bt
...
#12 0x00000000006f8a45 in dispatch_command (command=COM_QUERY,
thd=0x7fe760f94000, packet=0x7fe77154fac0 "", packet_length=0)
at /home/openxs/bzr2/percona-5.6/sql/sql_parse.cc:1434
...
(gdb) frame 12
#12 0x00000000006f8a45 in dispatch_command (command=COM_QUERY,
thd=0x7fe760f94000, packet=0x7fe77154fac0 "", packet_length=0)
at /home/openxs/bzr2/percona-5.6/sql/sql_parse.cc:1434
warning: Source file is more recent than executable.
1434 mysql_parse(thd, thd->query(), thd->query_length(), &parser_state);
(gdb) p thd->query_string.string.str
$2 = 0x7fe75301d010 "select benchmark(5", '0' <repeats 13 times>, ", 2*2)"
● https://sourceware.org/gdb/onlinedocs/gdb/Variables.html
www.percona.com
THD structure
grep -rn THD sql/sql_class.h
class THD :public MDL_context_owner,
public Statement,
public Open_tables_state
HASH user_vars; // hash for user variables
struct system_variables variables; // Changeable local variables
struct system_status_var status_var; // Per thread statistic vars
struct system_status_var *initial_status_var; /* used by show status */
Security_context main_security_ctx;
…
CSET_STRING query_string; // inherited from Statement...
www.percona.com
THD structure (continued)
(gdb) p thd->main_security_ctx->user
$7 = 0x7fe753019058 "root"
(gdb) p thd->main_security_ctx->host
$8 = {Ptr = 0xc16759 "localhost", str_length = 9, Alloced_length = 0,
alloced = false, str_charset = 0x1393de0}
www.percona.com
Real life case: checking core dump
gdb -ex 'set pagination 0'
…
-ex 'thread apply all bt full'
/path/to/mysqld /var/tmp/core.<pid> | tee core.<pid>.bt
● Make sure you know how to get core when mysqld
crashes:
http://www.percona.com/blog/2011/08/26/getting-mysql-core-file-on-linux/
● Let’s check one example, we need crashing bug for this:
https://bugs.launchpad.net/percona-server/+bug/1384658
www.percona.com
Real life case: attaching to alive mysqld
This is how it goes:
[root@centos openxs]# mysql -uroot -e "show variables like
'innodb_autoinc_lock_mode'"
+--------------------------+-------+
| Variable_name | Value |
+--------------------------+-------+
| innodb_autoinc_lock_mode | 0 |
+--------------------------+-------+
[root@centos openxs]# mysql -uroot -e "set global
innodb_autoinc_lock_mode=1"
ERROR 1238 (HY000) at line 1: Variable 'innodb_autoinc_lock_mode' is a
read only variable
[root@centos openxs]# gdb -ex "set innobase_autoinc_lock_mode=1" -batch -p
`pidof mysqld`
…
[Thread debugging using libthread_db enabled]
0x00007ff31d6830d3 in poll () from /lib64/libc.so.6
… check the variable value again now
[root@centos openxs]# ps aux | grep mysqld
[root@centos openxs]# kill -SIGCONT `pidof mysqld`
www.percona.com
Thank you!
Questions and Answers
Please, report bugs at:
http://bugs.mysql.com
Use “Affects Me” button!

Recommended

Tracing and profiling my sql (percona live europe 2019) draft_1
Tracing and profiling my sql (percona live europe 2019) draft_1Tracing and profiling my sql (percona live europe 2019) draft_1
Tracing and profiling my sql (percona live europe 2019) draft_1Valerii Kravchuk
 
Instant add column for inno db in mariadb 10.3+ (fosdem 2018, second draft)
Instant add column for inno db in mariadb 10.3+ (fosdem 2018, second draft)Instant add column for inno db in mariadb 10.3+ (fosdem 2018, second draft)
Instant add column for inno db in mariadb 10.3+ (fosdem 2018, second draft)Valerii Kravchuk
 
Gdb basics for my sql db as (percona live europe 2019)
Gdb basics for my sql db as (percona live europe 2019)Gdb basics for my sql db as (percona live europe 2019)
Gdb basics for my sql db as (percona live europe 2019)Valerii Kravchuk
 
More on gdb for my sql db as (fosdem 2016)
More on gdb for my sql db as (fosdem 2016)More on gdb for my sql db as (fosdem 2016)
More on gdb for my sql db as (fosdem 2016)Valeriy Kravchuk
 
Gdb basics for my sql db as (openfest 2017) final
Gdb basics for my sql db as (openfest 2017) finalGdb basics for my sql db as (openfest 2017) final
Gdb basics for my sql db as (openfest 2017) finalValeriy Kravchuk
 
Applying profilers to my sql (fosdem 2017)
Applying profilers to my sql (fosdem 2017)Applying profilers to my sql (fosdem 2017)
Applying profilers to my sql (fosdem 2017)Valeriy Kravchuk
 
MariaDB Server on macOS - FOSDEM 2022 MariaDB Devroom
MariaDB Server on macOS -  FOSDEM 2022 MariaDB DevroomMariaDB Server on macOS -  FOSDEM 2022 MariaDB Devroom
MariaDB Server on macOS - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
 
Flame Graphs for MySQL DBAs - FOSDEM 2022 MySQL Devroom
Flame Graphs for MySQL DBAs - FOSDEM 2022 MySQL DevroomFlame Graphs for MySQL DBAs - FOSDEM 2022 MySQL Devroom
Flame Graphs for MySQL DBAs - FOSDEM 2022 MySQL DevroomValeriy Kravchuk
 

More Related Content

What's hot

More on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB Devroom
More on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB DevroomMore on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB Devroom
More on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
 
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021
Linux  /proc filesystem for MySQL DBAs - FOSDEM 2021Linux  /proc filesystem for MySQL DBAs - FOSDEM 2021
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021Valeriy Kravchuk
 
Performance schema in_my_sql_5.6_pluk2013
Performance schema in_my_sql_5.6_pluk2013Performance schema in_my_sql_5.6_pluk2013
Performance schema in_my_sql_5.6_pluk2013Valeriy Kravchuk
 
Mysql 56-experiences-bugs-solutions-50mins
Mysql 56-experiences-bugs-solutions-50minsMysql 56-experiences-bugs-solutions-50mins
Mysql 56-experiences-bugs-solutions-50minsValeriy Kravchuk
 
The New MariaDB Offering - MariaDB 10, MaxScale and more
The New MariaDB Offering - MariaDB 10, MaxScale and moreThe New MariaDB Offering - MariaDB 10, MaxScale and more
The New MariaDB Offering - MariaDB 10, MaxScale and moreMariaDB Corporation
 
MySQL Performance schema missing_manual_flossuk
MySQL Performance schema missing_manual_flossukMySQL Performance schema missing_manual_flossuk
MySQL Performance schema missing_manual_flossukValeriy Kravchuk
 
MySQL Indexierung CeBIT 2014
MySQL Indexierung CeBIT 2014MySQL Indexierung CeBIT 2014
MySQL Indexierung CeBIT 2014FromDual GmbH
 
WiredTiger In-Memory vs WiredTiger B-Tree
WiredTiger In-Memory vs WiredTiger B-TreeWiredTiger In-Memory vs WiredTiger B-Tree
WiredTiger In-Memory vs WiredTiger B-TreeSveta Smirnova
 
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Valeriy Kravchuk
 
PERFORMANCE_SCHEMA and sys schema
PERFORMANCE_SCHEMA and sys schemaPERFORMANCE_SCHEMA and sys schema
PERFORMANCE_SCHEMA and sys schemaFromDual GmbH
 
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)Valeriy Kravchuk
 
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsMySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsJean-François Gagné
 
MySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsMySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
 
MySQL for Oracle DBAs
MySQL for Oracle DBAsMySQL for Oracle DBAs
MySQL for Oracle DBAsFromDual GmbH
 
Make Your Life Easier With Maatkit
Make Your Life Easier With MaatkitMake Your Life Easier With Maatkit
Make Your Life Easier With MaatkitMySQLConference
 
Perl Stored Procedures for MySQL (2009)
Perl Stored Procedures for MySQL (2009)Perl Stored Procedures for MySQL (2009)
Perl Stored Procedures for MySQL (2009)Antony T Curtis
 
Riding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamRiding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamJean-François Gagné
 
MySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsMySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
 

What's hot (20)

More on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB Devroom
More on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB DevroomMore on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB Devroom
More on bpftrace for MariaDB DBAs and Developers - FOSDEM 2022 MariaDB Devroom
 
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021
Linux  /proc filesystem for MySQL DBAs - FOSDEM 2021Linux  /proc filesystem for MySQL DBAs - FOSDEM 2021
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021
 
Performance schema in_my_sql_5.6_pluk2013
Performance schema in_my_sql_5.6_pluk2013Performance schema in_my_sql_5.6_pluk2013
Performance schema in_my_sql_5.6_pluk2013
 
Mysql 56-experiences-bugs-solutions-50mins
Mysql 56-experiences-bugs-solutions-50minsMysql 56-experiences-bugs-solutions-50mins
Mysql 56-experiences-bugs-solutions-50mins
 
The New MariaDB Offering - MariaDB 10, MaxScale and more
The New MariaDB Offering - MariaDB 10, MaxScale and moreThe New MariaDB Offering - MariaDB 10, MaxScale and more
The New MariaDB Offering - MariaDB 10, MaxScale and more
 
MySQL Performance schema missing_manual_flossuk
MySQL Performance schema missing_manual_flossukMySQL Performance schema missing_manual_flossuk
MySQL Performance schema missing_manual_flossuk
 
MySQL Indexierung CeBIT 2014
MySQL Indexierung CeBIT 2014MySQL Indexierung CeBIT 2014
MySQL Indexierung CeBIT 2014
 
WiredTiger In-Memory vs WiredTiger B-Tree
WiredTiger In-Memory vs WiredTiger B-TreeWiredTiger In-Memory vs WiredTiger B-Tree
WiredTiger In-Memory vs WiredTiger B-Tree
 
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
 
PERFORMANCE_SCHEMA and sys schema
PERFORMANCE_SCHEMA and sys schemaPERFORMANCE_SCHEMA and sys schema
PERFORMANCE_SCHEMA and sys schema
 
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
 
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsMySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
 
MySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsMySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitations
 
NoSQL with MySQL
NoSQL with MySQLNoSQL with MySQL
NoSQL with MySQL
 
MySQL for Oracle DBAs
MySQL for Oracle DBAsMySQL for Oracle DBAs
MySQL for Oracle DBAs
 
Make Your Life Easier With Maatkit
Make Your Life Easier With MaatkitMake Your Life Easier With Maatkit
Make Your Life Easier With Maatkit
 
Perl Stored Procedures for MySQL (2009)
Perl Stored Procedures for MySQL (2009)Perl Stored Procedures for MySQL (2009)
Perl Stored Procedures for MySQL (2009)
 
Riding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamRiding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication Stream
 
HA with Galera
HA with GaleraHA with Galera
HA with Galera
 
MySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsMySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitations
 

Similar to FOSDEM 2015: gdb tips and tricks for MySQL DBAs

Common schema my sql uc 2012
Common schema   my sql uc 2012Common schema   my sql uc 2012
Common schema my sql uc 2012Roland Bouman
 
Common schema my sql uc 2012
Common schema   my sql uc 2012Common schema   my sql uc 2012
Common schema my sql uc 2012Roland Bouman
 
BITS: Introduction to MySQL - Introduction and Installation
BITS: Introduction to MySQL - Introduction and InstallationBITS: Introduction to MySQL - Introduction and Installation
BITS: Introduction to MySQL - Introduction and InstallationBITS
 
How to create a useful MySQL bug report
How to create a useful MySQL bug reportHow to create a useful MySQL bug report
How to create a useful MySQL bug reportValerii Kravchuk
 
High-availability with Galera Cluster for MySQL
High-availability with Galera Cluster for MySQLHigh-availability with Galera Cluster for MySQL
High-availability with Galera Cluster for MySQLFromDual GmbH
 
An evening with Postgresql
An evening with PostgresqlAn evening with Postgresql
An evening with PostgresqlJoshua Drake
 
MySQL always-up with Galera Cluster
MySQL always-up with Galera ClusterMySQL always-up with Galera Cluster
MySQL always-up with Galera ClusterFromDual GmbH
 
Doctrine Project
Doctrine ProjectDoctrine Project
Doctrine ProjectDaniel Lima
 
Patroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easyPatroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easyAlexander Kukushkin
 
一种多屏时代的通用 web 应用架构
一种多屏时代的通用 web 应用架构一种多屏时代的通用 web 应用架构
一种多屏时代的通用 web 应用架构勇浩 赖
 
"Swoole: double troubles in c", Alexandr Vronskiy
"Swoole: double troubles in c", Alexandr Vronskiy"Swoole: double troubles in c", Alexandr Vronskiy
"Swoole: double troubles in c", Alexandr VronskiyFwdays
 
Dragoncraft Architectural Overview
Dragoncraft Architectural OverviewDragoncraft Architectural Overview
Dragoncraft Architectural Overviewjessesanford
 
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)Valeriy Kravchuk
 
Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Michael Renner
 
MySQL Utilities -- PyTexas 2015
MySQL Utilities -- PyTexas 2015MySQL Utilities -- PyTexas 2015
MySQL Utilities -- PyTexas 2015Dave Stokes
 

Similar to FOSDEM 2015: gdb tips and tricks for MySQL DBAs (20)

Common schema my sql uc 2012
Common schema   my sql uc 2012Common schema   my sql uc 2012
Common schema my sql uc 2012
 
Common schema my sql uc 2012
Common schema   my sql uc 2012Common schema   my sql uc 2012
Common schema my sql uc 2012
 
Php
PhpPhp
Php
 
BITS: Introduction to MySQL - Introduction and Installation
BITS: Introduction to MySQL - Introduction and InstallationBITS: Introduction to MySQL - Introduction and Installation
BITS: Introduction to MySQL - Introduction and Installation
 
Php
PhpPhp
Php
 
How to create a useful MySQL bug report
How to create a useful MySQL bug reportHow to create a useful MySQL bug report
How to create a useful MySQL bug report
 
High-availability with Galera Cluster for MySQL
High-availability with Galera Cluster for MySQLHigh-availability with Galera Cluster for MySQL
High-availability with Galera Cluster for MySQL
 
An evening with Postgresql
An evening with PostgresqlAn evening with Postgresql
An evening with Postgresql
 
The Accidental DBA
The Accidental DBAThe Accidental DBA
The Accidental DBA
 
MySQL always-up with Galera Cluster
MySQL always-up with Galera ClusterMySQL always-up with Galera Cluster
MySQL always-up with Galera Cluster
 
Doctrine Project
Doctrine ProjectDoctrine Project
Doctrine Project
 
Patroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easyPatroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easy
 
Tp web
Tp webTp web
Tp web
 
一种多屏时代的通用 web 应用架构
一种多屏时代的通用 web 应用架构一种多屏时代的通用 web 应用架构
一种多屏时代的通用 web 应用架构
 
"Swoole: double troubles in c", Alexandr Vronskiy
"Swoole: double troubles in c", Alexandr Vronskiy"Swoole: double troubles in c", Alexandr Vronskiy
"Swoole: double troubles in c", Alexandr Vronskiy
 
Dragoncraft Architectural Overview
Dragoncraft Architectural OverviewDragoncraft Architectural Overview
Dragoncraft Architectural Overview
 
Autolab Workshop
Autolab WorkshopAutolab Workshop
Autolab Workshop
 
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)
 
Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014
 
MySQL Utilities -- PyTexas 2015
MySQL Utilities -- PyTexas 2015MySQL Utilities -- PyTexas 2015
MySQL Utilities -- PyTexas 2015
 

Recently uploaded

Best Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusBest Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusGlobus
 
Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCIOWomenMagazine
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowNaoki (Neo) SATO
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleAmir Moghimi
 
Instrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowInstrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowGlobus
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureHironori Washizaki
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfssuser82c38d
 
Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Fermin Galan
 
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio, Inc.
 
An Introduction to Globus for Researchers
An Introduction to Globus for ResearchersAn Introduction to Globus for Researchers
An Introduction to Globus for ResearchersGlobus
 
From Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesFrom Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesAnisha Agarwal
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxMindInventory
 
Open Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsOpen Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsSprings
 
Passbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentPassbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentThierry Gayet
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Dmitry Zinoviev
 
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...
CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...syedfaisal759877
 
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارانتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارsohilww
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System AdministratorsGlobus
 
Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019VICTOR MAESTRE RAMIREZ
 

Recently uploaded (20)

Best Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusBest Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using Globus
 
Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdf
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flow
 
2024 Trends Transforming Enterprise Resource Planning
2024 Trends Transforming Enterprise Resource Planning2024 Trends Transforming Enterprise Resource Planning
2024 Trends Transforming Enterprise Resource Planning
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scale
 
Instrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowInstrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a Flow
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about Architecture
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdf
 
Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227
 
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
 
An Introduction to Globus for Researchers
An Introduction to Globus for ResearchersAn Introduction to Globus for Researchers
An Introduction to Globus for Researchers
 
From Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesFrom Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT Services
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptx
 
Open Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsOpen Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and Cons
 
Passbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentPassbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managment
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)
 
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...
CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...
 
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارانتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System Administrators
 
Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019
 

FOSDEM 2015: gdb tips and tricks for MySQL DBAs

  • 1. gdb tips and tricks for MySQL DBAs or How gdb can help you to solve MySQL problems Valerii Kravchuk, Principal Support Engineer valerii.kravchuk@percona.com
  • 2. www.percona.com Who am I? Valerii (aka Valeriy) Kravchuk: ● MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005 - 2012 ○ Bugs Verification Team all this time ○ Support issues related to bugs, crashes, InnoDB, performance... ○ Trainings (mostly informal) for new team members ○ All kinds of decision making committees… ● Principal Support Engineer in Percona since September, 2012 ○ Doing more or less the same as before, but better (I hope)... ○ Plus I try to speak and write about MySQL in public now ● http://mysqlentomologist.blogspot.com - my blog about MySQL (mostly bugs) ● https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about MySQL (mostly bugs…) ● http://bugs.mysql.com - my personal playground. 6 bugs reported in 2015
  • 3. www.percona.com What is this session about? ● Some historical remarks and URLs to known use cases/blog posts about gdb and MySQL troubleshooting ● Multi-threaded executables and gdb (threads, frames) ● Basic gdb commands and “tricks” ● Few words on pt-pmp use ● Important MySQL data structures to explore (mostly THD *) ● A couple of real life use cases, working with core dump and alive mysqld ● Discussion
  • 4. www.percona.com Domas is famous for these tricks... ● http://dom.as/2009/02/15/poor-mans-contention-profiling/ ● http://dom.as/2009/07/30/evil-replication-management/ ● http://dom.as/2010/01/02/read-ahead/ ● http://dom.as/2009/03/14/stupid-innodb-tricks/ - not literally needed since 5.1.38 and InnoDB plugin, innodb_spin_wait_delay :) ● http://poormansprofiler.org/ ● http://dom.as/2009/12/29/when-bad-things-happen/
  • 5. www.percona.com More examples of gdb use for MySQL DBAs ● Remember the names: Domas Mituzas, Shane Bester, Roel Van De Paar, Mark Callaghan, Aurimas, Zhai Weixiang, ... ● http://www.percona.com/blog/2012/09/09/obtain-last-executed-statement- from-optimized-core-dump/ ● http://www.percona.com/blog/2013/11/11/how-to-extract-all-running-queries- including-the-last-executed-statement-from-a-core-file/ ● http://mysqlbugs.blogspot.com.au/2012/09/how-to-obtain-all-executing- queries.html ● http://www.percona.com/blog/2010/03/23/too-many-connections-no-problem/
  • 6. www.percona.com What you can do with gdb ● Check stack traces (and variables), per thread: thread apply all bt [full] ● Print variables, up to complex one: thread 1 print do_command::thd->query_string.string.str ● Set new values for variables (global and per thread, even those formally read-only in MySQL while it’s running): set max_connections=5000 set opt_log_slave_updates=1 ● Call functions (that may do complex changes): call rpl_filter->add_do_db(strdup("hehehe")) ● Set breakpoints and watchpoints ● Work interactively or use gdb as a command line utility (-batch) ● Use macros, Python scripting, more… ● All these may not work, fail, hang, crash, produce obscure errors… ● You have to read and understand the source code
  • 7. www.percona.com pt-pmp (Poor Man’s Profiler) ● http://www.percona.com/doc/percona-toolkit/2.2/pt-pmp.html pt-pmp [-i 1] [-s 0] [-b mysqld] [-p pidofmysqld] [-l 0] [-k file] [--version] ● It is based on original idea by Domas (http://poormansprofiler.org/) with some more bash/awk on top applied ● One of the recent examples how it is used (semi-sync replication performance): http://bugs.mysql.com/bug.php?id=75570 ● When mysqld hangs or is slow, you can get some insight quickly: http://bugs. mysql.com/bug.php?id=75028 (HandlerSocket “hangs” on shutdown) ● When there are stalls, use pt-pmp to find out why (or what threads mostly do at the moment): http://bugs.mysql.com/bug.php?id=69810 ● pt-pmp surely slows server down :) Hint (workaround is in the bug): https://bugs.launchpad.net/percona-toolkit/+bug/1320168
  • 8. www.percona.com Multi-threaded mysqld and gdb ● process/thread/frame concepts: (gdb) thread 2 [Switching to thread 2 (Thread 0x7fe771550700 (LWP 2544))] #0 0x0000000000605774 in Item_func_numhybrid::val_int ( this=<value optimized out>) at /home/openxs/bzr2/percona-5.6/sql/item_func.cc:1013 1013 } (gdb) bt ... #12 0x00000000006f8a45 in dispatch_command (command=COM_QUERY, thd=0x7fe760f94000, packet=0x7fe77154fac0 "", packet_length=0) at /home/openxs/bzr2/percona-5.6/sql/sql_parse.cc:1434 ... (gdb) frame 12 #12 0x00000000006f8a45 in dispatch_command (command=COM_QUERY, thd=0x7fe760f94000, packet=0x7fe77154fac0 "", packet_length=0) at /home/openxs/bzr2/percona-5.6/sql/sql_parse.cc:1434 warning: Source file is more recent than executable. 1434 mysql_parse(thd, thd->query(), thd->query_length(), &parser_state); (gdb) p thd->query_string.string.str $2 = 0x7fe75301d010 "select benchmark(5", '0' <repeats 13 times>, ", 2*2)" ● https://sourceware.org/gdb/onlinedocs/gdb/Variables.html
  • 9. www.percona.com THD structure grep -rn THD sql/sql_class.h class THD :public MDL_context_owner, public Statement, public Open_tables_state HASH user_vars; // hash for user variables struct system_variables variables; // Changeable local variables struct system_status_var status_var; // Per thread statistic vars struct system_status_var *initial_status_var; /* used by show status */ Security_context main_security_ctx; … CSET_STRING query_string; // inherited from Statement...
  • 10. www.percona.com THD structure (continued) (gdb) p thd->main_security_ctx->user $7 = 0x7fe753019058 "root" (gdb) p thd->main_security_ctx->host $8 = {Ptr = 0xc16759 "localhost", str_length = 9, Alloced_length = 0, alloced = false, str_charset = 0x1393de0}
  • 11. www.percona.com Real life case: checking core dump gdb -ex 'set pagination 0' … -ex 'thread apply all bt full' /path/to/mysqld /var/tmp/core.<pid> | tee core.<pid>.bt ● Make sure you know how to get core when mysqld crashes: http://www.percona.com/blog/2011/08/26/getting-mysql-core-file-on-linux/ ● Let’s check one example, we need crashing bug for this: https://bugs.launchpad.net/percona-server/+bug/1384658
  • 12. www.percona.com Real life case: attaching to alive mysqld This is how it goes: [root@centos openxs]# mysql -uroot -e "show variables like 'innodb_autoinc_lock_mode'" +--------------------------+-------+ | Variable_name | Value | +--------------------------+-------+ | innodb_autoinc_lock_mode | 0 | +--------------------------+-------+ [root@centos openxs]# mysql -uroot -e "set global innodb_autoinc_lock_mode=1" ERROR 1238 (HY000) at line 1: Variable 'innodb_autoinc_lock_mode' is a read only variable [root@centos openxs]# gdb -ex "set innobase_autoinc_lock_mode=1" -batch -p `pidof mysqld` … [Thread debugging using libthread_db enabled] 0x00007ff31d6830d3 in poll () from /lib64/libc.so.6 … check the variable value again now [root@centos openxs]# ps aux | grep mysqld [root@centos openxs]# kill -SIGCONT `pidof mysqld`
  • 13. www.percona.com Thank you! Questions and Answers Please, report bugs at: http://bugs.mysql.com Use “Affects Me” button!