The document discusses using the PERFORMANCE_SCHEMA feature in MySQL 5.6 to diagnose and improve the performance of a query that is not scaling well. It provides an example query against large tables that does not scale beyond 10 threads. Various tools for performance analysis are discussed, but the PERFORMANCE_SCHEMA is presented as a potentially better option for getting detailed insight into where time is being spent and how to optimize the server. The talk will cover both the capabilities and limitations of the PERFORMANCE_SCHEMA.
This session is about using GNU debugger (gdb) as a tool to study MySQL internals (namely, InnoDB locks and metadata locks) and as a last resort in cases when server hangs or has to be restarted for other reason. It never hurts to try a trick or two before giving up and restarting.
Sometimes MySQL DBAs have to work with stalled/hanged/unresponsive MySQL instance, where their usual SQL-based tricks do not work any more. Sometimes they can not even connect to check what's going on inside server.
In other cases they know what to do and everything still works, but they have to implement changes to read-only server variables. Server restart is often not an option in production, as it means some downtime and may cause negative performance impact.
In these cases one could do something given read and write access to server memory/internals. Here comes gdb, that, alone with careful reading of the source code helps to often resolve the problems described above. During this session I'll show what can be done with gdb when server already is in
troubles, and how to use gdb to "see" and understand MySQL internals (like InnoDB locks or metadata locks) better.
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)Valeriy Kravchuk
The recently released MariaDB 10.5 GA includes many new, useful features, but I’d like to concentrate on those helping DBAs and support engineers to find out what’s going on when a problem occurs.
Specifically I present and discuss the Performance Schema updates to match MySQL 5.7 instrumentation, new tables in the INFORMATION_SCHEMA to monitor the internals of a generic thread pool and improvements of ANALYZE for statements.
Instant add column for inno db in mariadb 10.3+ (fosdem 2018, second draft)Valerii Kravchuk
My slides for (canceled due to personal issues) talk for FOSDEM 2018 MySQL Devroom. I planned to discuss the history of ALTER TABLE speedup and optimization in MySQL, explain the implementation of instant ADD COLUMN for InnoDB tables in MariaDB 10.3 and compare performance of recent versions of MariaDB 10.2, 10.3, Percona Server 5.7, MyRocks from MariaDB 10.2 and MySQL 8.0.4 while working on multiple step test case incolving ALTER TABLE ... ADD COLUMN.
A lof of links to related manuals, blog posts and resources are presented.
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Valeriy Kravchuk
Linux with kernels 2.6+. provides different ways to add user probes to almost every other line of code dynamically, and collect the resulting trace and profiling data in a safe and efficient way. This session discusses basic use of ftrace, perf, bcc tools and bpftrace utility, highlights typical problems MariaDB DBAs and developers may hit while trying to apply them, as well as solutions to some of them.
This session is about using GNU debugger (gdb) as a tool to study MySQL internals (namely, InnoDB locks and metadata locks) and as a last resort in cases when server hangs or has to be restarted for other reason. It never hurts to try a trick or two before giving up and restarting.
Sometimes MySQL DBAs have to work with stalled/hanged/unresponsive MySQL instance, where their usual SQL-based tricks do not work any more. Sometimes they can not even connect to check what's going on inside server.
In other cases they know what to do and everything still works, but they have to implement changes to read-only server variables. Server restart is often not an option in production, as it means some downtime and may cause negative performance impact.
In these cases one could do something given read and write access to server memory/internals. Here comes gdb, that, alone with careful reading of the source code helps to often resolve the problems described above. During this session I'll show what can be done with gdb when server already is in
troubles, and how to use gdb to "see" and understand MySQL internals (like InnoDB locks or metadata locks) better.
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)Valeriy Kravchuk
The recently released MariaDB 10.5 GA includes many new, useful features, but I’d like to concentrate on those helping DBAs and support engineers to find out what’s going on when a problem occurs.
Specifically I present and discuss the Performance Schema updates to match MySQL 5.7 instrumentation, new tables in the INFORMATION_SCHEMA to monitor the internals of a generic thread pool and improvements of ANALYZE for statements.
Instant add column for inno db in mariadb 10.3+ (fosdem 2018, second draft)Valerii Kravchuk
My slides for (canceled due to personal issues) talk for FOSDEM 2018 MySQL Devroom. I planned to discuss the history of ALTER TABLE speedup and optimization in MySQL, explain the implementation of instant ADD COLUMN for InnoDB tables in MariaDB 10.3 and compare performance of recent versions of MariaDB 10.2, 10.3, Percona Server 5.7, MyRocks from MariaDB 10.2 and MySQL 8.0.4 while working on multiple step test case incolving ALTER TABLE ... ADD COLUMN.
A lof of links to related manuals, blog posts and resources are presented.
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Valeriy Kravchuk
Linux with kernels 2.6+. provides different ways to add user probes to almost every other line of code dynamically, and collect the resulting trace and profiling data in a safe and efficient way. This session discusses basic use of ftrace, perf, bcc tools and bpftrace utility, highlights typical problems MariaDB DBAs and developers may hit while trying to apply them, as well as solutions to some of them.
Performance Schema for MySQL TroubleshootingSveta Smirnova
Percona Live (https://www.percona.com/live/data-performance-conference-2016/sessions/performance-schema-mysql-troubleshooting)
The performance schema in MySQL version 5.6, released in February, 2013, is a very powerful tool that can help DBAs discover why even the trickiest performance issues occur. Version 5.7 introduces even more instruments and tables. And while all these give you great power, you can get stuck choosing which instrument to use.
In this session, I will start with a description of a typical problem, then guide you how to use the performance schema to find out what causes the issue, the reason for unwanted behavior and how the received information can help you solve a particular problem.
Traditionally, performance schema sessions teach what is in contained in tables. I will, in contrast, start from a performance issue, then demonstrate which instruments and tables can help solve it. We will discuss how to setup the performance schema so that it has minimal impact on your server.
Percona Live 2016 (https://www.percona.com/live/data-performance-conference-2016/sessions/why-use-explain-formatjson). Although EXPLAIN FORMAT=JSON was first presented a long time ago, there still aren't many resources that explain how and why to use it. The most advertised feature is visual EXPLAIN in MySQL Workbench, but this format can do more than create nice pictures. It prints additional information that can't be found in good old tabular EXPLAIN, and can help to solve many tricky performance issues. In this session, I will not only describe which additional information we can get with the new syntax, but also provide examples showing how to use it to diagnose production issues.
Flame Graphs for MySQL DBAs - FOSDEM 2022 MySQL DevroomValeriy Kravchuk
Flame graph is way to visualize profiling data that allows the most frequent code paths to be identified quickly and accurately. They can be generated using Brendan Gregg's open source programs on github.com/brendangregg/FlameGraph, which create interactive SVG files to be checked in browser. The source of profiling data does not really matter - it can be perf profiler, bpftrace, Performance Schema, EXPLAIN output or any other source that allows to convert the data into the expected format of comma-separated "path" plus metric per line.
Different types of Flame Graphs (CPU, Off-CPU, Memory, Differential etc) are presented. Various tools and approaches to collect profile information of different aspects of MySQL server internal working are presented Several real-life use cases where Flame Graphs helped to understand and solve the problem are discussed.
This slide deck describes the Flexviews materialized view toolkit for MySQL:
http://flexvie.ws
Learn how to use incrementally refreshable materialized views, and how they can improve your performance.
Performance Schema for MySQL TroubleshootingSveta Smirnova
Percona Live (https://www.percona.com/live/data-performance-conference-2016/sessions/performance-schema-mysql-troubleshooting)
The performance schema in MySQL version 5.6, released in February, 2013, is a very powerful tool that can help DBAs discover why even the trickiest performance issues occur. Version 5.7 introduces even more instruments and tables. And while all these give you great power, you can get stuck choosing which instrument to use.
In this session, I will start with a description of a typical problem, then guide you how to use the performance schema to find out what causes the issue, the reason for unwanted behavior and how the received information can help you solve a particular problem.
Traditionally, performance schema sessions teach what is in contained in tables. I will, in contrast, start from a performance issue, then demonstrate which instruments and tables can help solve it. We will discuss how to setup the performance schema so that it has minimal impact on your server.
Percona Live 2016 (https://www.percona.com/live/data-performance-conference-2016/sessions/why-use-explain-formatjson). Although EXPLAIN FORMAT=JSON was first presented a long time ago, there still aren't many resources that explain how and why to use it. The most advertised feature is visual EXPLAIN in MySQL Workbench, but this format can do more than create nice pictures. It prints additional information that can't be found in good old tabular EXPLAIN, and can help to solve many tricky performance issues. In this session, I will not only describe which additional information we can get with the new syntax, but also provide examples showing how to use it to diagnose production issues.
Flame Graphs for MySQL DBAs - FOSDEM 2022 MySQL DevroomValeriy Kravchuk
Flame graph is way to visualize profiling data that allows the most frequent code paths to be identified quickly and accurately. They can be generated using Brendan Gregg's open source programs on github.com/brendangregg/FlameGraph, which create interactive SVG files to be checked in browser. The source of profiling data does not really matter - it can be perf profiler, bpftrace, Performance Schema, EXPLAIN output or any other source that allows to convert the data into the expected format of comma-separated "path" plus metric per line.
Different types of Flame Graphs (CPU, Off-CPU, Memory, Differential etc) are presented. Various tools and approaches to collect profile information of different aspects of MySQL server internal working are presented Several real-life use cases where Flame Graphs helped to understand and solve the problem are discussed.
This slide deck describes the Flexviews materialized view toolkit for MySQL:
http://flexvie.ws
Learn how to use incrementally refreshable materialized views, and how they can improve your performance.
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021Valeriy Kravchuk
Tools and approaches based on /proc sampling (like 0x.tools by Tanel Poder or ad hoc scripts) allow to measure individual thread level activity in MySQL server on Linux, like thread sleep states, currently executing system calls and kernel wait locations. If needed you can drill down into CPU usage of any thread or the system as a whole. Historical data can be captured for post factum analysis, without much impact on the system and no need to install or change anything in its configuration. In this presentation I am going to summarize what's possible with /proc and show useful examples for MySQL DBAs.
These are the slides I used to present "MySQL Performance Schema" at FOSSASIA, 2015 Singapore. It gives an overview of Performance Schema and also explains how it could be used to diagnose issues using few use cases.
Open Source 101 2022 - MySQL Indexes and HistogramsFrederic Descamps
Nobody complains that the database is too fast. But when things slow down, the complaints come quickly. The two most popular approaches to speeding up queries are indexes and histograms. But there are so many options and types on indexes that it can get confusing. Histograms are fairly new to MySQL but they do not work for all types of data. This talk covers how indexes and histograms work and show you how to test just how effective they are so you can measure the performance of your queries.
This presentation was prepared for a Webcast where John Yerhot, Engine Yard US Support Lead, and Chris Kelly, Technical Evangelist at New Relic discussed how you can scale and improve the performance of your Ruby web apps. They shared detailed guidance on issues like:
Caching strategies
Slow database queries
Background processing
Profiling Ruby applications
Picking the right Ruby web server
Sharding data
Attendees will learn how to:
Gain visibility on site performance
Improve scalability and uptime
Find and fix key bottlenecks
See the on-demand replay:
http://pages.engineyard.com/6TipsforImprovingRubyApplicationPerformance.html
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)Valeriy Kravchuk
eBPF on Linux 4.9+ is probably the best way to study performance problems. Basic usage of ftrace interface, bcc tools and bpftrace, as well as main bpftrace features and commands are presented. Several typical use cases
(including adding dynamic probes to MariaDB servers, working with stack traces and creating Flame Graphs) are discussed.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
2. Who am I?
Valerii (aka Valeriy) Kravchuk (you may call me “Larry”):
●
●
●
●
●
●
MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005 - 2012
○ Bugs Verification Team all this time
○ Support issues related to bugs, InnoDB, performance and more
○ Trainings (mostly informal) for new team members
○ All kinds of decision making committees…
Principal Support Engineer in Percona, 2012 - ...
○ Doing more or less the same as before, but better (I hope)...
○ Plus I speak and write about MySQL in public now!
http://mysqlentomologist.blogspot.com - my blog about MySQL bugs
https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about
MySQL (mostly bugs…)
http://bugs.mysql.com - my personal playground
Yes, I am Orthodox, in many senses...
www.percona.com
3. What is this session about?
“The Good, the Bad and the Ugly (Italian title:
Il buono, il brutto, il cattivo) is a 1966 Italian epic
Spaghetti western film directed by Sergio Leone,
starring Clint Eastwood, Lee Van Cleef, and Eli
Wallach in the title roles respectively.
...
It is the third film in the Dollars Trilogy following
A Fistful of Dollars (1964) and For a Few Dollars
More (1965). The plot revolves around three
gunslingers competing to find a fortune in buried
Confederate gold amid the violent chaos of
gunfights, hangings, American Civil War battles
and prison camps.”
http://en.wikipedia.org/wiki/The_Good,_the_Bad_and_the_Ugly
www.percona.com
4. So, what is this session really about?
●
●
●
●
●
How to find the root causes of MySQL server performance problems?
○ http://method-r.com/faq/35-what-is-method-r
○ Where time is spent?
Proper instrumentation of the code, is it really important?
○ http://tkyte.blogspot.com/2005/06/instrumentation.html
○ http://en.wikipedia.org/wiki/Observer_effect_%28physics%29
OS-level debug, trace and profiling tools, aren’t they allow to pinpoint any
performance problem in MySQL?
Instrumentation in MySQL - PERFORMANCE_SCHEMA
○ History and engineers involved
○ Features in MySQL 5.6
○ How to use it?
○ Any alternatives in MySQL world?
○ Sources of additional information and tools
The Good, The Bad and The Ugly in PERFORMANCE_SCHEMA
www.percona.com
5. Method R - common sense approach
1. Identify the most important task (the bottleneck)
2. Measure its response time (R) in detail
3. Optimize that response time in the most (economically)
efficient way
4. Repeat until your system is (economically) optimal
Percona also tries to apply common sense:
http://www.mysqlperformanceblog.com/2008/11/24/how-percona-does-a-mysql-performance-audit/
http://www.mysqlperformanceblog.com/2011/07/27/whats-required-to-tune-mysql/
http://www.mysqlperformanceblog.com/2011/05/05/the-two-even-more-fundamental-performance-metrics/
www.percona.com
6. Instrumentation
“Code instrumentation, what is it? To me, it is the fine art of
making approximately every other line of your developed
code “debug” of some sort. Trace information. With
timestamps. And meaningful information.” - Tom Kyte
“In science, the term observer effect refers to changes that
the act of observation will make on a phenomenon being
observed. This is often the result of instruments that, by
necessity, alter the state of what they measure in some
manner.” - Wikipedia
www.percona.com
7. Trace files and debug printouts?
●
●
●
●
Readily available in MySQL for years...
○ http://dev.mysql.com/doc/refman/5.6/en/making-trace-files.html
○ http://dev.mysql.com/doc/refman/5.6/en/dbug-package.html
○ … --debug=d:t:i:o,/tmp/mysqld.trace
○ -debug binaries only
Additional tracing features are available in MySQL 5.6 for non-debug
binaries also (for optimizer decision making, for example)
○ http://dev.mysql.com/doc/internals/en/tracing-example.html
Still far behind Oracle’s SQL Trace and trace events in general
○ http://www.juliandyke.com/Diagnostics/Events/EventReference.html
Let’s try to produce trace while test case for Bug #68079 is running:
mysql> set session debug='d:t:i:o,/tmp/mysqld.trace';
Query OK, 0 rows affected (0.00 sec)
www.percona.com
9. Example of the problem to solve: Bug #68079
http://bugs.mysql.com/bug.php?id=68079
CREATE TABLE `incident` (
`sys_id` char(32) NOT NULL DEFAULT '',
`category` varchar(40) DEFAULT NULL,
PRIMARY KEY (`sys_id`),
KEY `incident_category` (`category`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
insert into incident(sys_id) values (rand()*1000000);
insert into incident(sys_id) select rand()*1000000 from incident; -- 13 times
CREATE TABLE `task` (
`sys_id` char(32) NOT NULL DEFAULT '',
`u_root_cause` char(32) DEFAULT NULL,
`u_business_impact_description` mediumtext,
`u_business_impact_category` mediumtext,
PRIMARY KEY (`sys_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
insert
insert
insert
update
into task(sys_id) values(rand()*1000000);
into task(sys_id) select rand()*1000000 from task; -- 13 times
into task(sys_id) select sys_id from incident limit 100;
incident set category=rand()*100000;
www.percona.com
10. Example of the problem to solve: Bug #68079
These queries scale well:
select count(*), category from incident group by category;
select * from task where sys_id=rand()*1000000;
This one does not scale even to 10 threads on 12 cores:
select count(*), category from task inner join incident on task.sys_id=incident.sys_id
group by incident.category;
mysql> explain select count(*), category from task inner join incident on task.
sys_id=incident.sys_id group by incident.category;
| 1 | SIMPLE
|
NULL
| 1 | SIMPLE
|
test.incident.sys_id
incident | index | PRIMARY
| 8210 | Using index |
task
| eq_ref | PRIMARY
|
1 | Using index |
| incident_category | 123
|
| PRIMARY
|
| 96
www.percona.com
11. Example of the problem to solve: Bug #68079
mysqlslap -uroot --iterations=10 --concurrency=N --create-schema=test --no-drop --numberof-queries=1000 --query='select count(*), category from task inner join incident on task.
sys_id=incident.sys_id group by incident.category'
where N in (1, 2, 4, 8, 10, 12, 24 …
2*number_of_cores … 1024 … 8192)
Results:
Benchmark
Average number of
Minimum number of
Maximum number of
Number of clients
Average number of
…
Benchmark
Average number of
Minimum number of
Maximum number of
Number of clients
Average number of
seconds
seconds
seconds
running
queries
to run all queries: 8.512 seconds
to run all queries: 8.248 seconds
to run all queries: 8.844 seconds
queries: 8
per client: 125
seconds
seconds
seconds
running
queries
to run all queries: 9.399 seconds
to run all queries: 9.137 seconds
to run all queries: 9.749 seconds
queries: 10
per client: 100
www.percona.com
12. Example of the problem to solve: Bug #68079
Simple (?) join does NOT scale even to 10 threads...
● Why does this happen?
● Where time is spent?
● Is there any way to tune MySQL server to improve
scalability for this query?
● Maybe we need different hardware?
● Any changes at schema level that may help?
● How to get information to answer the questions above?
● This is a performance tuning task to check tools,
approaches and code improvement ideas!
www.percona.com
13. Sources of information on performance
What sources of information and tools we can try to use?
●
●
●
●
●
●
●
●
●
●
●
●
show engine innodb statusG
show engine innodb mutex;
InnoDB-related tables in the INFORMATION_SCHEMA (?)
show global status like ‘innodb%’ (?)
userstat (Percona Server and others) (?)
show profiles
Poor man’s profiler (pt-pmp)
perf
oprofile
DTrace (?)
Maybe… PERFORMANCE_SCHEMA?
Discussion of the need for PERFORMANCE_SCHEMA and alternative
approaches - hence all the above
www.percona.com
14. SHOW ENGINE INNODB STATUS
●
Do you know how to read that?
○
○
○
●
http://dev.mysql.com/doc/refman/5.6/en/innodb-monitors.html - fine manual
http://www.mysqlperformanceblog.com/2006/07/17/show-innodb-status-walk-through/ - the
best thing ever since Worcestershire sauce
http://www.percona.com/doc/percona-server/5.6/diagnostics/innodb_show_status.html - just
better in Percona Server
Not very useful for Bug #68079 on my QuadCore:
Per second averages calculated from the last 36 seconds
----------------BACKGROUND THREAD
----------------srv_master_thread loops: 121 srv_active, 0 srv_shutdown, 2473 srv_idle
srv_master_thread log flush and writes: 2594
---------SEMAPHORES
---------OS WAIT ARRAY INFO: reservation count 9477
OS WAIT ARRAY INFO: signal count 3703
Mutex spin waits 28946906, rounds 25108917, OS waits 8475
RW-shared spins 209, rounds 5999, OS waits 182
RW-excl spins 0, rounds 9, OS waits 0
Spin rounds per wait: 0.87 mutex, 28.70 RW-shared, 9.00 RW-excl
www.percona.com
15. SHOW ENGINE INNODB STATUS
...
Purge done for trx's n:o < 2384 undo n:o < 0 state: running but idle
History list length 20
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 143558, not started
MySQL thread id 1, OS thread handle 0x7f01cc057700, query id 10678 localhost msandbox init
show engine innodb status
...
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
285 OS file reads, 49 OS file writes, 49 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.06 writes/s, 0.06 fsyncs/s
Hash table size 276671, node heap has 0 buffer(s)
0.00 hash searches/s, 142759.09 non-hash searches/s
...
Log sequence number 4732500
...
Last checkpoint at 4732480
0 pending log writes, 0 pending chkp writes
50 log i/o's done, 0.06 log i/o's/second
...
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
...
Main thread process no. 5202, id 139645286401792, state: sleeping
Number of rows inserted 0, updated 0, deleted 0, read 89167561
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 144695.09 reads/s
www.percona.com
16. SHOW ENGINE INNODB MUTEX
●
●
http://dev.mysql.com/doc/refman/5.6/en/show-engine.html
-debug builds give more information
○ count - how many times the mutex was requested
○ spin_waits - how many times the spinlock had to run
○ spin_rounds - the number of spinlock rounds
○ os_waits - indicates the number of operating system waits. The mutex
was not unlocked during the spinlock and it was necessary to yield to
the operating system and wait. You have only this value in nondebug binaries.
○ os_wait_times - indicates the amount of time (in ms) spent in operating
system waits, if the timed_mutexes system variable is 1 (ON). If
timed_mutexes is 0 (OFF, default), timing is disabled, so os_wait_times
is 0.
| InnoDB | rw_lock_mutexes
| count=0, spin_waits=0, spin_rounds=0,
os_waits=0, os_yields=0, os_wait_times=0 |
www.percona.com
17. SHOW ENGINE INNODB MUTEX
●
Looks like we have something immediately useful here for Bug #68079:
mysql [localhost] {msandbox} (test) > show engine innodb mutex;
+--------+-------------------------+----------------+
| Type
| Name
| Status
|
+--------+-------------------------+----------------+
| InnoDB | log0log.cc:737
| os_waits=2
|
| InnoDB | buf0buf.cc:1242
| os_waits=2
|
| InnoDB | combined buf0buf.cc:992 | os_waits=66475 |
| InnoDB | dict0dict.cc:896
| os_waits=1
|
| InnoDB | log0log.cc:799
| os_waits=16 |
| InnoDB | combined buf0buf.cc:993 | os_waits=1
|
+--------+-------------------------+----------------+
6 rows in set (0.02 sec)
vi +992 /home/openxs/bzr2/mysql-5.6/storage/innobase/buf/buf0buf.cc
mutex_create(PFS_NOT_INSTRUMENTED, &block->mutex, SYNC_BUF_BLOCK);
●
●
Hardly is maintained properly in MySQL 5.6. Check my Bug #70819
Percona Server shows mutex name, not file and line number…
| InnoDB | &buf_pool->LRU_list_mutex
| InnoDB | &fil_system->mutex
| InnoDB | combined &block->mutex
| os_waits=13
| os_waits=5073
| os_waits=162396
www.percona.com
18. INFORMATION_SCHEMA?
InnoDB-related tables in the INFORMATION_SCHEMA:
●
●
●
http://dev.mysql.com/doc/refman/5.6/en/innodb-i_s-tables.html - there are
many...
But as we know from INNODB STATUS it’s not about I/O or locks, it seems
they do not add much useful for the case of Bug #68079…
If only INNODB_METRICS - http://dev.mysql.com/doc/refman/5.6/en/innodbmetrics-table.html:
| NAME
| SUBSYSTEM
| COUNT
| MAX_COUNT
| MIN_COUNT
| AVG_COUNT
...
| STATUS
| TYPE
| COMMENT
|
|
|
|
|
|
varchar(193)
varchar(193)
bigint(21)
bigint(21)
bigint(21)
double
|
|
|
|
|
|
NO
NO
NO
YES
YES
YES
| varchar(193) | NO
| varchar(193) | NO
| varchar(193) | NO
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
0
NULL
NULL
NULL
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
www.percona.com
21. Percona Server and userstat?
http://www.percona.com/doc/percona-server/5.6/diagnostics/user_stats.html:
●
●
●
●
●
●
Introduced in Google patch
Additional tables in INFORMATION_SCHEMA:
○ CLIENT_STATISTICS
○ THREAD_STATISTICS
○ USER_STATISTICS
○ TABLE_STATISTICS
○ INDEX_STATISTICS
Easy to configure:
○ http://dom.as/2013/04/17/on-performance-schemas/
Low performance impact we say:
○ http://www.mysqlperformanceblog.com/2012/06/02/how-expensive-is-user_statistics/
Hardly this can help with mutexes or waits…
But some says it adds overhead:
○
“and then, once userstat=1 the regression is horrible.. over 30%... “ - guess who says that and why?
www.percona.com
22. SHOW PROFILE
●
●
●
http://bugs.mysql.com/bug.php?id=24795 - SHOW PROFILE contribution by
Jeremy Cole (MySQL 5.0)
http://dev.mysql.com/doc/refman/5.6/en/show-profile.html - details…
Let’s try to profile the query while test for Bug #68079 is running:
mysql [localhost] {msandbox} (test) > set profiling=1;
Query OK, 0 rows affected, 1 warning (0.02 sec)
mysql [localhost] {msandbox} (test) > show warningsG
*************************** 1. row ***************************
Level: Warning
Code: 1287
Message: '@@profiling' is deprecated and will be removed in a future release.
1 row in set (0.00 sec)
mysql [localhost] {msandbox} (test) > select count(*), category from task inner join
incident on task.sys_id=incident.sys_id group by incident.category;
...
103 rows in set (0.08 sec)
mysql [localhost] {msandbox} (test) > show profiles;
mysql [localhost] {msandbox} (test) > show profile for query 3;
www.percona.com
23. SHOW PROFILE
●
Profile of the query while test for Bug #68079 is running:
| starting
| 0.000063 |
| checking permissions | 0.000004 |
| checking permissions | 0.000004 |
| Opening tables
| 0.000092 |
| init
| 0.000023 |
| System lock
| 0.000007 |
| optimizing
| 0.000009 |
| statistics
| 0.000037 |
| preparing
| 0.000015 |
| Creating tmp table
| 0.000023 |
| Sorting result
| 0.000003 |
| executing
| 0.000002 |
| Sending data
| 0.081722 |
| Creating sort index | 0.000141 |
| end
| 0.000005 |
| removing tmp table
| 0.000008 |
| end
| 0.000004 |
| query end
| 0.000008 |
| closing tables
| 0.000013 |
| freeing items
| 0.000553 |
| cleaning up
| 0.000017 |
+----------------------+----------+
21 rows in set, 1 warning (0.00 sec)
www.percona.com
24. Poor man’s profiler (gdb tricks)
●
●
●
●
http://poormansprofiler.org/
http://www.percona.com/doc/percona-toolkit/2.1/pt-pmp.html
http://www.mysqlperformanceblog.com/2011/12/02/three-ways-that-thepoor-mans-profiler-can-hurt-mysql/
○ mysqld freezes for the duration of the process
○ They say mysqld can crash (?)
○ They say mysqld can be left in an unstable state (?)
I’ve tried recently on QuadCore with test from bug #68079, not impressed:
10 libaio::??,os_aio_linux_collect,os_aio_linux_handle,fil_aio_wait,io_handler_thread,start_thread,clone
2 __memcpy_ssse3,store_key_field::copy_inner,copy,cp_buffer_from_ref,cmp_buffer_with_ref,join_read_key,
sub_select,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_execute_select,mysql_select,
handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,
do_handle_one_connection,handle_one_connection,
pfs_spawn_thread,start_thread,clone
1 update_ref_and_keys,make_join_statistics,JOIN::optimize,mysql_execute_select,mysql_select,
handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,
do_handle_one_connection,handle_one_connection,
pfs_spawn_thread,start_thread,clone
www.percona.com
25. Poor man’s profiler - Domas rulez!
●
●
●
●
So, I was not impressed (found nothing enlightening), but it depends:
○ “Usually pmp (poor man's profiler, making stacktrace snapshots)
producing more relevant information than performance schema reports.”
Google for something like site:bugs.mysql.com Mark Callaghan pmp:
○ http://bugs.mysql.com/bug.php?id=58037 - fixed in 5.5+ with MDL
○ http://bugs.mysql.com/bug.php?id=57021 - it was used to find a problem
in Facebook’s patch…
○ http://bugs.mysql.com/bug.php?id=61545 - still “Verified”, commit stalls
when mysql_binlog_send does disk read
If you are on Linux and don’t run MySQL with real profiler, try use pt-pmp
when server stalls, hangs or seems overloaded and unusable
Key to success for all kinds of profilers - open source of MySQL server!
www.percona.com
26. Let’s try real profiler - perf
●
●
●
https://perf.wiki.kernel.org/index.php/Main_Page:
○ “Among others, it provides per task, per CPU and per-workload
counters, sampling on top of these and source code event annotation”
http://bugs.mysql.com/bug.php?id=69236 - Single-threaded workloads can
be much slower on 5.6 than on 4.0. perf was used to prove that!
Sample session while test for Bug #68079 was running:
[openxs@chief ~]$ perf record -p 3310 sleep 20
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 2.006 MB perf.data (~87654 samples) ]
[openxs@chief ~]$ perf report
...
12.76% mysqld mysqld
12.32% mysqld mysqld
10.39% mysqld mysqld
[.] btr_cur_search_to_nth_level(d
[.] buf_page_get_gen(unsigned lon
[.] my_strnncollsp_utf8
www.percona.com
27. perf output continued...
●
Sample session while test for Bug #68079 was running:
...
7.28%
6.68%
5.76%
4.56%
4.22%
2.87%
2.79%
2.68%
2.33%
1.97%
1.86%
1.83%
1.81%
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
mysqld
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
[.]
rec_get_offsets_func(unsigned
mtr_memo_slot_release_func(mt
page_cur_search_with_match(bu
cmp_dtuple_rec_with_match_low
row_search_for_mysql(unsigned
ut_delay(unsigned long)
pfs_mutex_enter_func(ib_mutex
innobase_mysql_cmp(int, unsig
cmp_whole_field(unsigned long
row_mysql_store_col_in_innoba
get_internal_charset
mutex_spin_wait(ib_mutex_t*,
pfs_mutex_exit_func(ib_mutex_
...
●
●
This was on QuadCore with 4 concurrent threads, MySQL 5.6.14
Now tell me that PERFORMANCE_SCHEMA impact with default settings is
low on MySQL 5.6...
www.percona.com
28. Let’s try another profiler - oprofile
●
●
http://www.fromdual.ch/mysql-oprofile:
○ Nice summary from Oli on how to use it!
Sample session while test for Bug #68079 was running:
[root@chief openxs]# opreport --demangle=smart --symbols --long-filenames --merge tgid
/home/openxs/5.6.14/bin/mysqld | head -n 20
...
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00
(Unhalted core cycles) count 100000
samples %
image name
symbol name
1208437 12.5180 /home/openxs/5.6.14/bin/mysqld btr_cur_search_to_nth_level
(dict_index_t*, unsigned long, dtuple_t const*, unsigned long, unsigned long, btr_cur_t*,
unsigned long, char const*, unsigned long, mtr_t*)
1151258 11.9257 /home/openxs/5.6.14/bin/mysqld buf_page_get_gen(unsigned long, unsigned
long, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned
long, mtr_t*)
1115969 11.5601 /home/openxs/5.6.14/bin/mysqld my_strnncollsp_utf8
...
242608
2.5131 /home/openxs/5.6.14/bin/mysqld pfs_mutex_enter_func(ib_mutex_t*, char
const*, unsigned long)
...
142685
1.4780 /lib64/libc-2.13.so
__memcpy_ssse3
132334
1.3708 /home/openxs/5.6.14/bin/mysqld pfs_mutex_exit_func(ib_mutex_t*)
...
www.percona.com
29. DTrace? One day, maybe...
●
I wish I remember how to use it and have MacOS X or Solaris, or FreeBSD
9.x (on some good hardware) at hand…
●
but I don’t. So, I had not tried to use DTrace to find out what’s wrong with
that Bug #68079.
●
Some useful references, still, if you ever plan to try:
○
○
http://dom.as/tag/dtrace/ - check what Domas used to do with DTrace
○
http://cdn.oreillystatic.com/en/assets/1/event/21/Deep-inspecting%20MySQL%20with%
20DTrace%20Presentation.pdf - from one of User Conferences...
○
●
http://dev.mysql.com/doc/refman/5.6/en/dba-dtrace-server.html - it’s documented in the
manual, including static probes added
http://dev.mysql.com/tech-resources/articles/getting_started_dtrace_saha.html - “native” Sun
engineers also contributed
I really miss Sun...
www.percona.com
30. Time to try PERFORMANCE_SCHEMA
●
●
●
●
We know functions where most of the time was spent (profiling)
We know that some mutex waits were long enough to force OS wait (switch
to other thread)
We need more details, specifically time spent waiting… So, time to use
PERFORMANCE_SCHEMA it seems
Sample output while test for Bug #68079 was running, naive attempt (the
time is in picoseconds):
mysql> select event_name, count_star, sum_timer_wait from performance_schema.
events_waits_summary_global_by_event_name where event_name like
'wait/synch/mutex/innodb%' and count_star > 0 order by sum_timer_wait desc limit 5;
+--------------------------------------------+------------+----------------+
| event_name
| count_star | sum_timer_wait |
+--------------------------------------------+------------+----------------+
| wait/synch/mutex/innodb/os_mutex
|
71754726 | 11380485535846 |
| wait/synch/mutex/innodb/trx_sys_mutex
|
60057 |
4191727883 |
| wait/synch/mutex/innodb/buf_pool_mutex
|
8655 |
468303388 |
| wait/synch/mutex/innodb/mutex_list_mutex
|
10310 |
427385546 |
| wait/synch/mutex/innodb/rw_lock_list_mutex |
8304 |
316946749 |
+--------------------------------------------+------------+----------------+
5 rows in set (0.00 sec)
www.percona.com
31. Basic ideas and terms
PERFORMANCE_SCHEMA (aka PS, P_S, PFS):
● Inspired by Oracle Wait Interface
● Separate database and storage engine
● Records run time statistics via built-in instrumentation points
● Designed to let us get the answers to the following questions:
○ Where was time spent?
○ Who has used most of resources/time?
○ What was executed to to use these?
○ When it was executed?
● Appeared in MySQL 5.5, improved and ON by default in MySQL 5.6, and is
documented and explained in details already:
○ http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html
● Implemented by Marc Alff (http://marcalff.blogspot.com/)
● Deal with instruments (including wait events), consumers, objects (tables for
now) that are monitored, threads (some of them background) that run and
actors (users) that execute SQL statements (in stages).
www.percona.com
32. So, how to use it?
●
●
The amount of information in the manual is huge, it’s not easy to start using P_S
○ I wish it would be as easy as setting TIMED_STATISTICS to TRUE with ALTER
SYSTEM + few queries to V$SESSTATS, V$SYSSTATS or V$SESSION_WAIT
Marc Alff explains in http://marcalff.blogspot.com/2013/04/on-configuringperformance-schema.html:
○ “Define your goals” - this is clear, we have waits and we need to find out what
waits are most important based on time wasted (or waited)
○ “Define what to instrument” - we need to instrument InnoDB waits mutex waits
and internal lock waits, but how to do this? Details next, but:
■ "wait/synch/mutex/%", for mutexes
■ "wait/synch/rwlock/%", for read write locks
○ “Define how much detail to collect” - well, we need counts and time spent for
all InnoDB-related waits
○ “Provide sizing data” - autosizing based on max_connections?
| performance_schema_max_mutex_instances
| performance_schema_max_rwlock_instances
○
| 15600 |
| 9000 |
“Monitor sizing problems” - memory used vs dropped statistics and impact
www.percona.com
33. How instruments are named?
●
●
For those who want solutions, not complains or comments...
http://dev.mysql.com/doc/refman/5.6/en/performance-schema-instrumentnaming.html:
○ "wait/io/file/%" - file io
○ "wait/io/socket/%" - network io
○ "wait/io/table/%" - table io
○ "wait/lock/table/%" - table locks
○ "wait/synch/cond/%" - conditions
○ "wait/synch/mutex/%" - mutexes
○ "wait/synch/rwlock/%" - read write locks
○ "stage/%" - stages of statement execution (see show profile)
○ "idle" - clients idle time.
○ "statement/com/%" - commands in the client/server protocol
○ "statement/sql/%" - SQL queries (and some more statement/…)
Hint: bold means “new in MySQL 5.6”, blue color means enabled by default
www.percona.com
34. Setup for Bug #68079 (waits/…)
●
●
http://mysqlentomologist.blogspot.com/2013/01/how-to-use-performanceschemato-check.html - my experience back at MySQL 5.6 RC stage...
To set up InnoDB mutex waits monitoring via PERFORMANCE_SCHEMA in
general with MySQL 5.6.x you should do at least the following:
1. Start MySQL server with all mutex related instruments enabled at startup
(performance_schema=ON by default on recent 5.6.x), like this:
mysqld_safe --performance_schema_instrument='wait/synch/mutex/innodb/%=on' &
2.
Connect to server and set up proper consumers explicitly, like this:
UPDATE performance_schema.setup_consumers SET enabled = 'YES'
WHERE name like 'events_waits%';
3.
4.
Run your problematic load.
Check waits using whatever tables you need
www.percona.com
35. Summary of my setup experience...
●
●
●
●
It's great to have Oracle engineers reading your Facebook posts and
replying (especially in a real time). Thank you, Mark Leith!
PERFORMANCE_SCHEMA in MySQL 5.6 can be used to study InnoDB
mutex waits
MySQL 5.6 manual is not clear enough about steps and details on
instruments involved to do the study explained above
http://dev.mysql.com/doc/refman/5.6/en/performance-schema-filtering.html:
“Modifications to any of these tables affect monitoring immediately, with the exception of
setup_actors. Modifications to setup_actors affect only foreground threads created
thereafter.”
●
●
Now compare to reality, complain in public, report a bug:
○ http://bugs.mysql.com/bug.php?id=68097 - “Manual does not explain
that some P_S instruments must be enabled at startup”
Get a reply from Dimitri Kravtchuk:
“All instrumentation is dynamic EXCEPT mutexes which are created mainly on the MySQL server
startup, so to get their instrumentation initialized you have to enable it on startup”
www.percona.com
37. ps_helper To the Rescue!
ps_helper:
●
●
●
●
●
●
●
Developed by Mark Leith, MySQL expert @ Oracle
○ worked in MySQL Support team since 2005
○ senior software development manager @Oracle now (MEM, maybe MEB)
○ good Oracle RDBMS background
Easier to use comparing to “raw” PERFORMANCE_SCHEMA
Ready to use collection of views, stored routines and...
Recently even an easy way to create ps, P_S, PFS or whatever synonym you like!
git clone https://github.com/MarkLeith/dbahelper.git dbahelper - if you prefer...
Installed in a separate schema (ps_helper)
IO/latency/waits/statement digests
Links:
● http://www.markleith.co.uk/ps_helper/
● https://github.com/MarkLeith/dbahelper/archive/master.zip
● https://www.slideshare.net/Leithal/performance-schema-andpshelper
www.percona.com
38. ps_helper: basics
Install:
openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot test < ps_helper_56.sql
Check:
mysql> show tables;
+-------------------------------------------------+
| Tables_in_ps_helper
|
+-------------------------------------------------+
| _digest_95th_percentile_by_avg_us
|
| _digest_avg_latency_by_avg_us
|
| check_lost_instrumentation
|
...
53 rows in set (0,00 sec)
mysql> select routine_name from information_schema.routines where
routine_schema='ps_helper';
+-------------------------------+
| routine_name
|
+-------------------------------+
| analyze_statement_digest
|
...
20 rows in set (0,00 sec)
www.percona.com
39. ps_helper: why is it useful?
Have some real fun:
openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot ps_helper <
procedures/create_synonym_db.sql
Warning: Using a password on the command line interface can be insecure.
mysql> call create_synonym_db('performance_schema', 'ps');
Query OK, 0 rows affected (2,35 sec)
mysql> use ps
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+----------------------------------------------------+
| Tables_in_ps
|
+----------------------------------------------------+
| accounts
|
...
Use ps_helper.format_time(SUM(sum_timer_wait)):
● Check Bug #70252 to find out what may happen if you don’t use it
● Check https://www.slideshare.net/Leithal/performance-schema-andpshelper
www.percona.com
40. Some history behind...
● “The Good, The Bad and The Ugly”:
“The film's title has entered the English language as an idiomatic expression. Typically used when describing
something thoroughly, the respective phrases refer to upsides, downsides and the parts that could, or should have
been done better, but were not.” - Wikipedia
● “For Three Men The Civil War Wasn’t Hell. It Was Practice!”
○
○
Guess who are these men, in case of PERFORMANCE_SCHEMA…
Maybe more than three…
● Read the WorkLogs:
○
○
○
http://dev.mysql.com/worklog/task/?id=2360 - “Why we're doing this”
http://dev.mysql.com/worklog/task/?id=4674 - “Setup for Actors”
http://dev.mysql.com/worklog/task/?id=4513 - “Implement it…”
● Where the name comes from?
“Proposals for the schema name were DYNAMIC (but that is a reserved word in standard SQL),
PERFORMANCE_SCHEMA (Jim Winstead's suggestion), PERFORMANCE (Peter's preference), and DBTrace
because it sounds like DTrace. There was a vote (see dev-private thread "WL#2360 Vote: schema name and
table name"). The winner was PERFORMANCE_SCHEMA.” - Peter Gulutzan
www.percona.com
41. PERFORMANCE_SCHEMA in MySQL 5.6
Useful reading (and names to remember):
●
http://marcalff.blogspot.com/2013/04/on-configuring-performance-schema.html - article from Mark
Alff on how to configure PERFORMANCE_SCHEMA
●
http://dev.mysql.com/doc/refman/5.6/en/performance-schema-configuration.html - RTFM
●
https://www.facebook.com/notes/mysql-at-facebook/my-mysql-is-faster-than-yourmysql/10151250402570933 - influence of P_S in 5.6.10 and... 5.1.63 or 5.1.52+FB patch are still
fastest in many cases
●
http://dimitrik.free.fr/blog/archives/2013/07/mysql-performance-why-performance-schemaoverhead.html - overhead explained and measured by Dimitri Kravtchuk:
●
http://www.percona.com/webinars/using-mysql-56-performance-schema-troubleshoot-typicalworkload-bottlenecks - webinar by Peter Zaitsev (May 15, 2013)
●
http://mysql.wisborg.dk/category/mysql/performance_schema/ - useful posts from Jesper Krogh
on the topic. He had created yet another set of tools to make out life easier, ps_tools: http://mysql.
wisborg.dk/2013/10/10/mysql-connect-2013-ps_tools/
●
http://mysqlblog.fivefarmers.com/tag/performance_schema/ - practical use of
PERFORMANCE_SCHEMA, by Todd Farmer
www.percona.com
42. PERFORMANCE_SCHEMA - The Good
●
●
●
●
●
●
●
●
Monitoring is available on all platforms supported by MySQL
Activating the Performance Schema causes no (few) changes in server behavior
○ No persistent on-disk storage
○ Memory allocation at server startup
○ Processing happens mostly at events retrieval
○ No new keywords or threads added
Overhead is small comparing to some other ways to monitor
Problems/failures in P_S code should not prevent server from working normally
It is easy to add new instrumentation points (they say)
Under active development:
○ Very few bugs - 4 active in MySQL 5.6 (of 47) as of November 1, 2013
○ Bug #70018 (PFS Overhead on frequent connect/disconnect) fixed in 5.6.15!
○ Even more features in MySQL 5.7: check Bug #69527 (MDL locks!)
Used by Oracle internally a lot for real things (MEM, performance improvements)
As it’s based on proper ideas, eventually it’s going to become an ultimate solution
for MySQL performance monitoring (and replace everything else)
www.percona.com
44. PERFORMANCE_SCHEMA - The Bad
● Performance overhead in default configuration is notable in
some cases (10-30%, see bug reports and 3rd party studies)
● Additional memory use is notable (see Bug #68514):
○
“Why is this not a bug? On my x86-64 with 5.6.12 when
performance_schema=1 then the extra RAM is:
max_connections * 196kb” - Mark Callaghan
● Mutex and rw-lock waits instrumentation must be enabled at
server startup
● How complete is the instrumentation?
● There are bugs and missing parts in the manual
● There are bugs in implementation
● Is this kind of instrumentation really good? (SQL, no trace)
● Any critical posts and even some bug reports are sometimes
considered as a kind of a personal insult: "godfather" case...
www.percona.com
45. Bugs in P_S and fine manual
Bug #68413 - Facebook: “performance_schema overhead is at least 10%”
Bug #69782 - Old files not being removed from performance_schema.file_instances
Bug #70628 - Wrong instrumentation interface for mysql_cond_timedwait
Bug #70697 - Probably not a bug, but indication that manual for SHOW ENGINE
PERFORMANCE_SCHEMA STATUS is still not clear
Bug #68574 - “No instrumentation for InnoDB files in P_S on windows” - same with
5.6.14, by the way...
Bug #68785 - new features (connection attributes) work well together (or not)...
Bug #66589 - still not fixed (do not use mysqlhotcopy). One of the last bugs I verified in
Oracle...
Bug #68514 - beware of memory usage impact with P_S
Bug #68097 - still some details to document in the manual clearly
www.percona.com
46. PERFORMANCE_SCHEMA - The Ugly
●
●
●
●
●
●
Names: schema name, long table names… you know whom to blame (and thank
for ps_helper). I wish we have public synonyms in MySQL, so I’d create
V$SYSSTATS (Bug #68855). You can create views, but that’s overhead, no?
Why additional views and procedures (ps_helper) are not there by default?
Some important things and storage engines are not properly instrumented yet,
MDL locks, for example… ups, seems fixed in 5.7.3
Heuristics to control size (read Bug #68514 again)… less than obvious and I’d
prefer more control
Make up your mind on what should be in P_S vs I_S vs SHOW statements. Think
about SHOW PROCESSLIST vs I_S.PROCESSLIST vs P_S.THREADS
Want to know what else users care about? Check feature requests for P_S (11
“Verified” as of November 2, 2013), some of them are limitations “by design”:
○
○
○
Bug #69881 - “Try to compute elapsed time for an event which has not completed” - Todd Farmer
Bug #66515 - “I have a problem with PS instrumentation that can only be enabled at server start.” Mark Callaghan
Bug #70383 - “Implement a SELECT-syntax/statement as a synonym for SHOW ENGINE
performance_schema STATUS” - Peter Laursen
www.percona.com
48. Do you still care about Bug #68079?
“mysql-trunk (a.k.a 5.7.3): Scales up to 32 threads reasonably well, the HW has
48 logical CPUs” - Sunny Bains, October 18, 2013
“There is some confusion here, there are two issues in this bug report. One is the
page mutex and the other is the page re-lock. For this query we acquire it in S
mode all the time. If we remove the page mutex overhead then this S mode
rwlock becomes the next bottleneck. However, this doesn't mean that the original
problem is not fixed, it simply means that the fix requires further work that is more
involved, that's all.“ - Sunny Bains, October 30, 2013
“These same numbers can be achieved by using "PARTITION BY KEY()
PARTITIONS N". By using partitions you can reduce the contention on the block
mutex.” - Sunny Bains, October 30, 2013
●
●
Does Oracle care about MySQL?
Was PERFORMANCE_SCHEMA used to find the bottlenecks?
www.percona.com
49. Thank you!
Questions and Answers
Please, report bugs to:
http://bugs.mysql.com
Use “Affects Me” button!
www.percona.com