The document summarizes the Percona Toolkit, which contains free and open source command-line tools for MySQL based on Percona's experience developing best practices. Some of the most popular tools are pt-summary, pt-mysql-summary, pt-stalk, pt-archiver, and pt-query-digest, which allow users to summarize MySQL servers, analyze queries from logs, and check for issues. The toolkit can be installed via package repositories or by downloading individual tools.
This is the presentation delivered by Karthik.P.R at MySQL User Camp Bangalore on 09th June 2017. ProxySQL is a high performance MySQL Load Balancer Designed to scale database servers.
Using MySQL without Maatkit is like taking a photo without removing the camera's lens cap. Professional MySQL experts use this toolkit to help keep complex MySQL installations running smoothly and efficiently. This session will show you practical ways to use Maatkit every day.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
This is the presentation delivered by Karthik.P.R at MySQL User Camp Bangalore on 09th June 2017. ProxySQL is a high performance MySQL Load Balancer Designed to scale database servers.
Using MySQL without Maatkit is like taking a photo without removing the camera's lens cap. Professional MySQL experts use this toolkit to help keep complex MySQL installations running smoothly and efficiently. This session will show you practical ways to use Maatkit every day.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
PostgreSQL 10 had added built-in logical replication which tackles some of the limitations of physical replication and opens up the possibility of promising new areas of replication. In this webinar, we will introduce the concept of logical replication and demonstrate how one can configure a logical replication in no time.
Highlights include:
- Basic architecture, including the publisher and subscriber model
- Configuration, administration, and monitoring
- Limitations and future plans
The webinar will review a multi-layered framework for PostgreSQL security, with a deeper focus on limiting access to the database and data, as well as securing the data.
Using the popular AAA (Authentication, Authorization, Auditing) framework we will cover:
- Best practices for authentication (trust, certificate, MD5, Scram, etc).
- Advanced approaches, such as password profiles.
- Deep dive of authorization and data access control for roles, database objects (tables, etc), view usage, row-level security, and data redaction.
- Auditing, encryption, and SQL injection attack prevention.
Note: this session is delivered in German
Speaker:
Borys Neselovskyi, Sales Engineer, EDB
- MariaDB 소개
- MariaDB 서버 구성 및 아키텍처 이해
- MariaDB 스토리지 엔진
- MariaDB 데이터베이스 관리
- 트랜잭션 / Locking 의 이해
- MariaDB 보안
- 백업과 복구를 통한 데이터베이스 관리
- MariaDB upgrade
- MariaDB 모니터링
- MySQL 에서 MariaDB 로의 전환
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
To get better replication speed and less lag, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK. But fully benefiting from this feature is not as simple as just enabling it.
In this talk, I explain in detail how this feature works. I also cover how to optimize parallel replication and the improvements made in MySQL 8.0 and back-ported in 5.7 (Write Sets), greatly improving the potential for parallel execution on replicas (but needing RBR).
Come to this talk to get all the details about MySQL 5.7 and 8.0 Parallel Replication.
PostgreSQL is a very popular and feature-rich DBMS. At the same time, PostgreSQL has a set of annoying wicked problems, which haven't been resolved in decades. Miraculously, with just a small patch to PostgreSQL core extending this API, it appears possible to solve wicked PostgreSQL problems in a new engine made within an extension.
*If you see the screen is not good condition, downloading please.*
MariaDB Optimization
- 풀 테이블 스캔
- ORDER BY 처리(Using filesort)
- GROUP BY 처리
- DISTINCT 처리
- 임시 테이블(Using Tempoary)
- 인덱스 컨디션 푸시다운(Index Condition Pushdown, ICP)
- 멀티 레인지 리드(Multi Range Read)
- 인덱스 머지(Index merge)
- 테이블 조인
- 서브 쿼리
This is a introduction to PostgreSQL that provides a brief overview of PostgreSQL's architecture, features and ecosystem. It was delivered at NYLUG on Nov 24, 2014.
http://www.meetup.com/nylug-meetings/events/180533472/
Maxscale switchover, failover, and auto rejoinWagner Bianchi
How the MariaDB Maxscale Switchover, Failover, and Rejoin works under the hood by Esa Korhonen and Wagner Bianchi.
You can watch the video of the presentation at
https://www.linkedin.com/feed/update/urn:li:activity:6381185640607809536
The latest version of my PostgreSQL introduction for IL-TechTalks, a free service to introduce the Israeli hi-tech community to new and interesting technologies. In this talk, I describe the history and licensing of PostgreSQL, its built-in capabilities, and some of the new things that were added in the 9.1 and 9.2 releases which make it an attractive option for many applications.
Let's get into several common types of queries that developers struggle with, showing SQL solutions, and then analyze them for optimal efficiency. I'll cover Exclusion Join, Random Selection, Greatest-Per-Group, Dynamic Pivot, and Relational Division.
When does InnoDB lock a row? Multiple rows? Why would it lock a gap? How do transactions affect these scenarios? Locking is one of the more opaque features of MySQL, but it’s very important for both developers and DBA’s to understand if they want their applications to work with high performance and concurrency. This is a creative presentation to illustrate the scenarios for locking in InnoDB and make these scenarios easier to visualize. I'll cover: key locks, table locks, gap locks, shared locks, exclusive locks, intention locks, insert locks, auto-inc locks, and also conditions for deadlocks.
PostgreSQL 10 had added built-in logical replication which tackles some of the limitations of physical replication and opens up the possibility of promising new areas of replication. In this webinar, we will introduce the concept of logical replication and demonstrate how one can configure a logical replication in no time.
Highlights include:
- Basic architecture, including the publisher and subscriber model
- Configuration, administration, and monitoring
- Limitations and future plans
The webinar will review a multi-layered framework for PostgreSQL security, with a deeper focus on limiting access to the database and data, as well as securing the data.
Using the popular AAA (Authentication, Authorization, Auditing) framework we will cover:
- Best practices for authentication (trust, certificate, MD5, Scram, etc).
- Advanced approaches, such as password profiles.
- Deep dive of authorization and data access control for roles, database objects (tables, etc), view usage, row-level security, and data redaction.
- Auditing, encryption, and SQL injection attack prevention.
Note: this session is delivered in German
Speaker:
Borys Neselovskyi, Sales Engineer, EDB
- MariaDB 소개
- MariaDB 서버 구성 및 아키텍처 이해
- MariaDB 스토리지 엔진
- MariaDB 데이터베이스 관리
- 트랜잭션 / Locking 의 이해
- MariaDB 보안
- 백업과 복구를 통한 데이터베이스 관리
- MariaDB upgrade
- MariaDB 모니터링
- MySQL 에서 MariaDB 로의 전환
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
To get better replication speed and less lag, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK. But fully benefiting from this feature is not as simple as just enabling it.
In this talk, I explain in detail how this feature works. I also cover how to optimize parallel replication and the improvements made in MySQL 8.0 and back-ported in 5.7 (Write Sets), greatly improving the potential for parallel execution on replicas (but needing RBR).
Come to this talk to get all the details about MySQL 5.7 and 8.0 Parallel Replication.
PostgreSQL is a very popular and feature-rich DBMS. At the same time, PostgreSQL has a set of annoying wicked problems, which haven't been resolved in decades. Miraculously, with just a small patch to PostgreSQL core extending this API, it appears possible to solve wicked PostgreSQL problems in a new engine made within an extension.
*If you see the screen is not good condition, downloading please.*
MariaDB Optimization
- 풀 테이블 스캔
- ORDER BY 처리(Using filesort)
- GROUP BY 처리
- DISTINCT 처리
- 임시 테이블(Using Tempoary)
- 인덱스 컨디션 푸시다운(Index Condition Pushdown, ICP)
- 멀티 레인지 리드(Multi Range Read)
- 인덱스 머지(Index merge)
- 테이블 조인
- 서브 쿼리
This is a introduction to PostgreSQL that provides a brief overview of PostgreSQL's architecture, features and ecosystem. It was delivered at NYLUG on Nov 24, 2014.
http://www.meetup.com/nylug-meetings/events/180533472/
Maxscale switchover, failover, and auto rejoinWagner Bianchi
How the MariaDB Maxscale Switchover, Failover, and Rejoin works under the hood by Esa Korhonen and Wagner Bianchi.
You can watch the video of the presentation at
https://www.linkedin.com/feed/update/urn:li:activity:6381185640607809536
The latest version of my PostgreSQL introduction for IL-TechTalks, a free service to introduce the Israeli hi-tech community to new and interesting technologies. In this talk, I describe the history and licensing of PostgreSQL, its built-in capabilities, and some of the new things that were added in the 9.1 and 9.2 releases which make it an attractive option for many applications.
Let's get into several common types of queries that developers struggle with, showing SQL solutions, and then analyze them for optimal efficiency. I'll cover Exclusion Join, Random Selection, Greatest-Per-Group, Dynamic Pivot, and Relational Division.
When does InnoDB lock a row? Multiple rows? Why would it lock a gap? How do transactions affect these scenarios? Locking is one of the more opaque features of MySQL, but it’s very important for both developers and DBA’s to understand if they want their applications to work with high performance and concurrency. This is a creative presentation to illustrate the scenarios for locking in InnoDB and make these scenarios easier to visualize. I'll cover: key locks, table locks, gap locks, shared locks, exclusive locks, intention locks, insert locks, auto-inc locks, and also conditions for deadlocks.
Designing an extensible, flexible schema that supports user customization is a common requirement, but it's easy to paint yourself into a corner.
Examples of extensible database requirements:
- A database that allows users to declare new fields on demand.
- Or an e-commerce catalog with many products, each with distinct attributes.
- Or a content management platform that supports extensions for custom data.
The solutions we use to meet these requirements is overly complex and the performance is terrible. How should we find the right balance between schema and schemaless database design?
I'll briefly cover the disadvantages of Entity-Attribute-Value (EAV), a problematic design that's an example of the antipattern called the Inner-Platform Effect, That is, modeling an attribute-management system on top of the RDBMS architecture, which already provides attributes through columns, data types, and constraints.
Then we'll discuss the pros and cons of alternative data modeling patterns, with respect to developer productivity, data integrity, storage efficiency and query performance, and ease of extensibility.
- Class Table Inheritance
- Serialized BLOB
- Inverted Indexing
Finally we'll show tools like pt-online-schema-change and new features of MySQL 5.6 that take the pain out of schema modifications.
Many questions on database newsgroups and forums can be answered with uses of outer joins. Outer joins are part of the standard SQL language and supported by all RDBMS brands. Many programmers are expected to use SQL in their work, but few know how to use outer joins effectively.
Learn to use this powerful feature of SQL, increase your employability, and amaze your friends!
Karwin will explain outer joins, show examples, and demonstrate a Sudoku puzzle solver implemented in a single SQL query.
MySQL users commonly ask: Here's my table, what indexes do I need? Why aren't my indexes helping me? Don't indexes cause overhead? This talk gives you some practical answers, with a step by step method for finding the queries you need to optimize, and choosing the best indexes for them.
We all know how to define database indexes, but which indexes to define remains a mysterious art for most software developers. This talk will use general principles and specific scenarios to give you practical, step-by-step knowledge to turn a performance bottleneck into an epic win!
You find a column named EntityNum in a table you manage, but what data belongs in this column? Not every detail of usage is clear from just SQL data type and constraints. What is the sensible range of values? Unit of measure? How is the column used by applications? Who in the world knows? We need a way to add comments to the database schema, just as we would write comments in application code to document how programmers should use it. But comments are useful only if they're correct and current, and if they're easy to read and to update. Schemadoc is an experimental tool to help in these goals.
MySQL exposes a collection of tunable parameters and indicators that is frankly intimidating. But a poorly tuned MySQL server is a bottleneck for your PHP application scalability. This session shows how to do InnoDB tuning and read the InnoDB status report in MySQL 5.5.
The most massive crime of identity theft in history was perpetrated in 2007 by exploiting an SQL Injection vulnerability. This issue is one of the most common and most serious threats to web application security. In this presentation, you'll see some common myths busted and you'll get a better understanding of defending against SQL injection.
Tree-like data relationships are common, but working with trees in SQL usually requires awkward recursive queries. This talk describes alternative solutions in SQL, including:
- Adjacency List
- Path Enumeration
- Nested Sets
- Closure Table
Code examples will show using these designs in PHP, and offer guidelines for choosing one design over another.
Software developers love tools for coding, debugging, testing, and configuration management. The more these tools improve the How of coding, the more we see that we're behind the curve on improving the What, Why, and When. If you've been on a project that seemed vague, adrift, and endless, this talk can help. Make your projects run SMART.
Presentation given at OSCON 2009 and PostgreSQL West 09. Describes SQL solutions to a selection of object-oriented problems:
- Extensibility
- Polymorphism
- Hierarchies
- Using ORM in MVC application architecture
These slides are excerpted from another presentation, "SQL Antipatterns Strike Back."
A comparison of different solutions for full-text search in web applications using PostgreSQL and other technology. Presented at the PostgreSQL Conference West, in Seattle, October 2009.
Trees In The Database - Advanced data structuresLorenzo Alberton
Storing tree structures in a bi-dimensional table has always been problematic. The simplest tree models are usually quite inefficient, while more complex ones aren't necessarily better. In this talk I briefly go through the most used models (adjacency list, materialized path, nested sets) and introduce some more advanced ones belonging to the nested intervals family (Farey algorithm, Continued Fractions, and other encodings). I describe the advantages and pitfalls of each model, some proprietary solutions (e.g. Oracle's CONNECT BY) and one of the SQL Standard's upcoming features, Common Table Expressions.
Percona Toolkit for Effective MySQL AdministrationMydbops
Percona Tools are one of most widely tool in MySQL industry. It is used for the effective MySQL administartion and handling complex operational tasks in MySQL.
Webinar: Untethering Compute from StorageAvere Systems
Enterprise storage infrastructures are gradually sprawling across the globe and consumers of data increasingly require access to remote storage resources. Solutions for mitigating the pain associated with this growth are out there, but performance varies. This Webinar will take a look at these challenges, review available solutions, and compare tests of performance.
String Comparison Surprises: Did Postgres lose my data?Jeremy Schneider
Comparisons are fundamental to computing - and comparing strings is not nearly as straightforward as you might think. Come learn about the history, nuance and surprises of “putting words in order” that you never knew existed in computer science, and how that nuance impacts both general programming and SQL programming. Next, walk through a few actual scenarios and demonstrations using PostgreSQL as a user and administrator, which you can re-run yourself later for further study, including one way you could easily corrupt your self-managed PostgreSQL database if you aren't prepared. Finally we’ll dive into an explanation of the surprising behaviors we saw in PostgreSQL, and learn more about user and administrative features PostgreSQL provides related to localized string comparison.
High Availability can be a curiously nebulous term, and most people probably don't care about it until they can't access their online banking service, or their plane crashes.
This presentation examines some of the considerations necessary when building highly available computer systems, then focuses on the HA infrastructure software currently available from the Corosync/OpenAIS, Linux-HA and Pacemaker projects.
Originally presented at Linux Users Victoria in April 2010 (http://luv.asn.au/2010/04/06)
All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...Ernie Souhrada
Have you ever thought that your SSD storage just doesn't seem anywhere near as fast as advertised? What if I told you that a couple of small changes to your Linux servers had the potential to more than double the amount of throughput your MySQL servers could handle while simultaneously reducing query response time by around 50 percent? It's not a dream, Neo.
If you're still in a traditional data center and/or your MySQL footprint lives on spinning rust, then you might want to take the blue pill, attend a different session, and believe whatever you want to believe.
However, if your server footprint runs inside Amazon Web Services or if you use SSDs in any fashion, I invite you to take the red pill and come with me on a journey down the rabbit hole, where we'll discuss such things as the Linux block IO driver, CPU starvation, and interrupt handling, using real-world query performance data from the primary MySQL data stores at Pinterest to illustrate the effect.
DAOS - Scale-Out Software-Defined Storage for HPC/Big Data/AI Convergenceinside-BigData.com
In this deck, Johann Lombardi from Intel presents: DAOS - Scale-Out Software-Defined Storage for HPC/Big Data/AI Convergence.
"Intel has been building an entirely open source software ecosystem for data-centric computing, fully optimized for Intel® architecture and non-volatile memory (NVM) technologies, including Intel Optane DC persistent memory and Intel Optane DC SSDs. Distributed Asynchronous Object Storage (DAOS) is the foundation of the Intel exascale storage stack. DAOS is an open source software-defined scale-out object store that provides high bandwidth, low latency, and high I/O operations per second (IOPS) storage containers to HPC applications. It enables next-generation data-centric workflows that combine simulation, data analytics, and AI."
Unlike traditional storage stacks that were primarily designed for rotating media, DAOS is architected from the ground up to make use of new NVM technologies, and it is extremely lightweight because it operates end-to-end in user space with full operating system bypass. DAOS offers a shift away from an I/O model designed for block-based, high-latency storage to one that inherently supports fine- grained data access and unlocks the performance of next- generation storage technologies.
Watch the video: https://youtu.be/wnGBW31yhLM
Learn more: https://www.intel.com/content/www/us/en/high-performance-computing/daos-high-performance-storage-brief.html
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Slides used at CNCF Paris Meetup 02/15/18.
This covers how we setup Prometheus at Deezer and his architecture. We also give some configuration examples and tweaks.
OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...NETWAYS
I gave a talk titled "Continuous Integration in data centers“ at OSDC in 2013, presenting ways how to realize continuous integration/delivery with Jenkins and related tools.Three years later we gained new tools in our continuous delivery pipeline, including Docker, Gerrit and Goss. Over the years we also had to deal with different problems caused by faster release cycles, a growing team and gaining new projects. We therefore established code review in our pipeline, improved our test infrastructure and invested in our infrastructure automation.In this talk I will discuss the lessons we learned over the last years, demonstrate how a proper continuous delivery pipeline can improve your life and how open source tools like Jenkins, Docker and Gerrit can be leveraged for setting up such an environment.
The 5 Minute DBA-DBA Skills for Non-DBApercona2013
The presentation provides you with the essential tips to ensure that your database runs smoothly if you are pressed for time.
Percona provides an in-depth review of your database and recommends appropriate changes by performing a complete MySQL health check in which we identify inefficiencies, find problems before they occur, and ensure that your MySQL database is in the best condition.
Unmasking Careto through Memory Forensics (video in description)Andrew Case
My presentation from SecTor 2014 on analyzing the sophisticated Careto malware with memory forensics & Volatility
Video here: http://2014.video.sector.ca/video/110388398
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Survey of Percona Toolkit
1. Survey of Percona Toolkit:
Command-line Tools for MySQL
Bill Karwin, Percona Inc.
Tuesday, December 4, 2012
2. Percona Toolkit
• Free, open source tools based on Percona’s
experience developing best practices for
repetitive or complex MySQL operations.
• Verify master and replica data consistency.
• Efficiently archive rows.
• Find duplicate indexes.
• Summarize MySQL servers.
• Analyze queries from logs and tcpdump.
• Collect diagnostic information when problems occur.
www.percona.com
Tuesday, December 4, 2012
3. Installing
www.percona.com
Tuesday, December 4, 2012
4. Installing Percona Toolkit
• Percona package repos for RHEL/CentOS or
Debian/Ubuntu:
$ sudo yum install percona-toolkit
$ sudo dpkg install percona-toolkit
• http://www.percona.com/software/repositories
• Other download options available for RPM, DEB,
tarball, or individual tools.
• http://www.percona.com/doc/percona-toolkit/
installation.html
www.percona.com
Tuesday, December 4, 2012
5. Top Nine Popular Tools
• pt-summary • pt-duplicate-key-
checker
• pt-mysql-summary
• pt-table-checksum
• pt-stalk
• pt-table-sync
• pt-archiver
• pt-online-schema-
• pt-query-digest change
www.percona.com
Tuesday, December 4, 2012
6. pt-summary
http://www.percona.com/doc/percona-toolkit/pt-summary.html
www.percona.com
Tuesday, December 4, 2012
7. pt-summary
• Summarize system information in a nice way.
• Useful to verify operating system configuration,
inspect many system attributes quickly.
www.percona.com
Tuesday, December 4, 2012
8. pt-summary
$ pt-summary
# Percona Toolkit System Summary Report ######################
Date | 2011-09-30 17:06:44 UTC (local TZ: PDT -0700)
Hostname | huey.karwin.percona.com
Uptime | 7:45, 1 user, load average: 0.04, 0.01, 0.00
System | innotek GmbH; VirtualBox; v1.2 ()
Service Tag | 0
Platform | Linux
Release | CentOS release 5.6 (Final)
Kernel | 2.6.18-238.19.1.el5
Architecture | CPU = 64-bit, OS = 64-bit
Threading | NPTL 2.5
Compiler | GNU CC version 4.1.2 20080704 (Red Hat 4.1.2-51).
SELinux | Enforcing
Virtualized | VirtualBox
# Processor ##################################################
Processors | physical = 1, cores = 0, virtual = 1, hyperthreading = no
Speeds | 1x2844.667
Models | 1xIntel(R) Core(TM) i7 CPU M 640 @ 2.80GHz
Caches | 1x6144 KB
www.percona.com
Tuesday, December 4, 2012
11. pt-summary
# Network Config #############################################
Controller | Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
Controller | Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
FIN Timeout | net.ipv4.tcp_fin_timeout = 60
Port Range | net.ipv4.ip_local_port_range = 32768 61000
# Interface Statistics #######################################
interface rx_bytes rx_packets rx_errors tx_bytes tx_packets tx_errors
========= ========= ========== ========== ========== ========== ==========
lo 7000 60 0 7000 60 0
eth0 1250000 17500 0 3000000 15000 0
eth1 9000000 12500 0 600000 8000 0
# Network Connections ########################################
Connections from remote IP addresses
192.168.56.1 1
Connections to local IP addresses
192.168.56.111 1
Connections to top 10 local ports
22 1
States of connections
ESTABLISHED 1
LISTEN 5
www.percona.com
Tuesday, December 4, 2012
12. pt-summary
# Top Processes ##############################################
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 15 0 10372 688 572 S 0.0 0.1 0:00.44 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.20 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.03 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:07.58 events/0
6 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
11 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
15 root 10 -5 0 0 0 S 0.0 0.0 0:00.11 kblockd/0
16 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
# Simplified and fuzzy rounded vmstat (wait please) ##########
procs ---swap-- -----io---- ---system---- --------cpu--------
r b si so bi bo ir cs us sy il wa st
2 0 0 0 10 8 1000 30 0 3 97 0 0
0 0 0 0 0 0 1000 30 0 2 98 0 0
0 0 0 0 0 0 1000 30 0 4 96 0 0
0 0 0 0 0 0 1000 30 0 2 98 0 0
0 0 0 0 0 0 1000 35 0 2 98 0 0
# The End ####################################################
www.percona.com
Tuesday, December 4, 2012
13. pt-mysql-summary
http://www.percona.com/doc/percona-toolkit/pt-mysql-summary.html
www.percona.com
Tuesday, December 4, 2012
14. pt-mysql-summary
• Summarize MySQL information in a nice way.
• See current status at a glance.
• Uses live information from running instance, because
it could be different from /etc/my.cnf.
• Organizes information in a consistent order, so you
know where to find it.
www.percona.com
Tuesday, December 4, 2012
15. pt-mysql-summary
$ pt-mysql-summary
# Percona Toolkit MySQL Summary Report #######################
System time | 2011-09-30 17:57:07 UTC (local TZ: PDT -0700)
# Instances ##################################################
Port Data Directory Socket
===== ========================== ======
/var/lib/mysql
# Report On Port 3306 ########################################
User | root@localhost
Time | 2011-09-30 10:57:07 (PDT)
Hostname | huey.karwin.percona.com
Version | 5.1.58-community-log MySQL
Built On | unknown-linux-gnu x86_64
Started | 2011-09-30 10:25 (up 0+00:31:18)
Databases | 5
Datadir | /var/lib/mysql/
Processes | 1 connected, 1 running
Replication | Is not a slave, has 0 slaves connected
Pidfile | /var/lib/mysql/huey.karwin.percona.com.pid (exists)
www.percona.com
Tuesday, December 4, 2012
18. pt-mysql-summary
# Table cache ################################################
Size | 4
Usage | 100%
# Key Percona Server features ################################
Table & Index Stats | Not Supported
Multiple I/O Threads | Enabled
Corruption Resilient | Not Supported
Durable Replication | Not Supported
Import InnoDB Tables | Not Supported
Fast Server Restarts | Not Supported
Enhanced Logging | Not Supported
Replica Perf Logging | Not Supported
Response Time Hist. | Not Supported
Smooth Flushing | Not Supported
HandlerSocket NoSQL | Not Supported
Fast Maatkit Hashes | Unknown
# Query cache ################################################
query_cache_type | ON
Size | 0.0k
Usage | 0%
HitToInsertRatio | 0%
www.percona.com
Tuesday, December 4, 2012
19. pt-mysql-summary
# Schema #####################################################
Would you like to mysqldump -d the schema and analyze it? y/n Y
There are 5 databases. Would you like to dump all, or just one?
Type the name of the database, or press Enter to dump all of them. sakila
Database Tables Views SPs Trigs Funcs FKs Partn
{chosen} 16 7 3 3 22
Database InnoDB MyISAM
{chosen} 15 8
Database BTREE FULLTEXT
{chosen} 63 1
st vt d t y d e s m c i b
mi ai a e e e n e e h n l
am rn t x a c u t d a t o
le cy e t r i m i r b
ls hi t m u
it an i a m
na rt m l i
tm e n
p t
Database === === === === === === === === === === === === === ===
{chosen} 26 45 15 19 4 4 1 7 3 1 2 1 2 1
www.percona.com
Tuesday, December 4, 2012
20. pt-mysql-summary
# Noteworthy Technologies ####################################
Full Text Indexing | Yes
Geospatial Types | No
Foreign Keys | Yes
Partitioning | No
SSL | No
Explicit LOCK TABLES | No
Delayed Insert | No
XA Transactions | No
NDB Cluster | No
Prepared Statements | No
www.percona.com
Tuesday, December 4, 2012
21. pt-mysql-summary
# InnoDB #####################################################
Version | 1.0.17
Buffer Pool Size | 16.0M
Buffer Pool Fill | 45%
Buffer Pool Dirty | 0%
File Per Table | ON
Page Size | 16k
Log File Size | 2 * 5M = 10.0M
Log Buffer Size | 8M
Flush Method | O_DIRECT
Flush Log At Commit | 2
XA Support | ON
Checksums | ON
Doublewrite | ON
R/W I/O Threads | 4 4
I/O Capacity | 200
Thread Concurrency | 0
Concurrency Tickets | 500
Commit Concurrency | 0
Txn Isolation Level | REPEATABLE-READ
Adaptive Flushing | ON
Adaptive Checkpoint | 0
Checkpoint Age | 0k
www.percona.com
Tuesday, December 4, 2012
22. pt-mysql-summary
InnoDB Queue | 0 queries inside InnoDB, 0 queries in queue
Oldest Transaction | 0 Seconds
History List Len | 6
Read Views | 1
Undo Log Entries | 0 transactions, 0 total undo, 0 max undo
Pending I/O Reads | 0 buf pool reads, 0 normal AIO, 0 ibuf AIO, 0 preads
Pending I/O Writes | 0 buf pool (0 LRU, 0 flush list, 0 page); 0 AIO, 0 sync, 0
log IO (0 log, 0 chkp); 0 pwrites
Pending I/O Flushes | 0 buf pool, 0 log
Transaction States | 1xnot started
www.percona.com
Tuesday, December 4, 2012
26. pt-stalk
http://www.percona.com/doc/percona-toolkit/pt-stalk.html
www.percona.com
Tuesday, December 4, 2012
27. pt-stalk
• Wait for a problem to occur, then gather forensic
data about MySQL and the system.
• Good when you want to diagnose a problem, but
you don’t know when it happens.
www.percona.com
Tuesday, December 4, 2012
28. What Data is Collected?
2012_11_30_17_49_13-df 2012_11_30_17_49_13-sysctl
2012_11_30_17_49_13-disk-space 2012_11_30_17_49_13-top
2012_11_30_17_49_13-diskstats 2012_11_30_17_49_13-trigger
2012_11_30_17_49_13-hostname 2012_11_30_17_49_13-variables
2012_11_30_17_49_13-innodbstatus1 2012_11_30_17_49_13-vmstat
2012_11_30_17_49_13-innodbstatus2 2012_11_30_17_49_13-vmstat-overall
2012_11_30_17_49_13-interrupts
2012_11_30_17_49_13-lsof
2012_11_30_17_49_13-meminfo
2012_11_30_17_49_13-mutex-status1
2012_11_30_17_49_13-mutex-status2
2012_11_30_17_49_13-mysqladmin
2012_11_30_17_49_13-netstat
2012_11_30_17_49_13-netstat_s
2012_11_30_17_49_13-opentables1
2012_11_30_17_49_13-opentables2
2012_11_30_17_49_13-output
2012_11_30_17_49_13-pmap
2012_11_30_17_49_13-processlist
2012_11_30_17_49_13-procstat
2012_11_30_17_49_13-procvmstat
2012_11_30_17_49_13-ps
2012_11_30_17_49_13-slabinfo
www.percona.com
Tuesday, December 4, 2012
29. Ad Hoc Usage
$ pt-stalk --no-stalk
2012_11_30_17_49_13 Starting /usr/bin/pt-stalk --function=status --
variable=Threads_running --threshold=25 --match= --cycles=0 --interval=0
--iterations=1 --run-time=30 --sleep=0 --dest=/var/lib/pt-stalk --prefix=
--notify-by-email= --log=/var/log/pt-stalk.log --pid=/var/run/pt-
stalk.pid --plugin=
2012_11_30_17_49_13 Not stalking; collect triggered immediately
2012_11_30_17_49_13 Collect triggered
2012_11_30_17_49_13 Collector PID 2865
2012_11_30_17_49_13 Waiting up to 90 seconds for collectors to finish...
2012_11_30_17_50_43 Killing collector 2865
2012_11_30_17_50_43 Exiting because no more iterations
2012_11_30_17_50_43 /usr/bin/pt-stalk exit status 0
www.percona.com
Tuesday, December 4, 2012
30. Background Usage
• Run pt-stalk run as a daemon and let it wait.
$ pt-stalk --daemonize ...event options...
$ tail -f /var/log/pt-stalk.log
2012_11_30_20_37_30 Check results: Threads_running=1, matched=no, cycles_true=0
2012_11_30_20_37_31 Check results: Threads_running=1, matched=no, cycles_true=0
2012_11_30_20_37_32 Check results: Threads_running=1, matched=no, cycles_true=0
2012_11_30_20_37_33 Check results: Threads_running=1, matched=no, cycles_true=0
...
www.percona.com
Tuesday, December 4, 2012
31. Function, Variable & Threshold
• Function: the information source that pt-stalk polls
for the trigger event.
• status: SHOW GLOBAL STATUS
• processlist: SHOW PROCESSLIST
• filename: you can write a custom shell script
www.percona.com
Tuesday, December 4, 2012
32. Function, Variable & Threshold
• Variable: what to watch in the information source.
• status: watch the named status variable.
• processlist: watch the named column.
www.percona.com
Tuesday, December 4, 2012
33. Function, Variable & Threshold
• Threshold: if the watched value is greater than this
threshold, a collect is triggered.
• status: compare to the value of the given variable.
• processlist: compare to the count of how many
processes show the “match” value in the watched
column.
www.percona.com
Tuesday, December 4, 2012
34. Example: Watch Status
• Watch status and collect information when
Threads_running is 25 or more.
$ pt-stalk --function status
! --variable Threads_running
! --threshold 25
www.percona.com
Tuesday, December 4, 2012
35. Example: Watch Processlist
• Watch processlist and collect information when
there are 10 or more processes with
State=statistics
$ pt-stalk --function processlist
! --variable State
! --match statistics
! --threshold 10
www.percona.com
Tuesday, December 4, 2012
36. Example: Watch Custom Script
• Watch processlist and collect information when
there are 10 or more processes with
State=statistics
$ cat > purge_not_working.sh
trg_plugin() {
mysql $EXT_ARGV -E -e "SHOW ENGINE INNODB
STATUS" | grep "^History list length" | awk
'{print $4}'
}
$ pt-stalk --function purge_not_working.sh
! --threshold 200
www.percona.com
Tuesday, December 4, 2012
37. pt-archiver
http://www.percona.com/doc/percona-toolkit/pt-archiver.html
www.percona.com
Tuesday, December 4, 2012
38. pt-archiver
• Archive rows from a MySQL table into another
table or a file.
• Works incrementally on chunks of rows.
• Deletes data from source safely.
www.percona.com
Tuesday, December 4, 2012
39. Move Data
• Copy data from one MySQL instance to another,
then delete from the source:
$ mysqldump -h huey -d imdb keyword
| mysql -h dewey test
$ pt-archiver --progress 10000
--source h=huey,D=imdb,t=keyword
--dest h=dewey,D=test
--where "1=1" --limit 1000 --commit-each
TIME ELAPSED COUNT
2012-12-03T01:37:36 0 0
2012-12-03T01:37:43 6 10000
. . .
www.percona.com
Tuesday, December 4, 2012
40. Copy Data
• Copy data from one MySQL instance to another,
but do not delete data from the source:
$ pt-archiver --progress 10000 --no-delete
--source h=localhost,D=imdb,t=keyword
--file '%t.csv'
--where "1=1" --limit 1000 --commit-each
TIME ELAPSED COUNT
2012-12-03T01:28:27 0 0
2012-12-03T01:28:28 0 10000
2012-12-03T01:28:28 0 20000
. . .
www.percona.com
Tuesday, December 4, 2012
41. Purge Data
• Delete orphan rows (slowly):
$ pt-archiver --progress 10000 --purge
--source h=huey,D=imdb,t=person_info
--where 'NOT EXISTS(SELECT * FROM name
WHERE id=person_info.person_id)'
www.percona.com
Tuesday, December 4, 2012
42. Limitations
• Destination table must exist.
• Archiving related data across tables is awkward.
• You can use WHERE with subqueries, but not JOIN.
www.percona.com
Tuesday, December 4, 2012
43. pt-query-digest
http://www.percona.com/doc/percona-toolkit/pt-query-digest.html
www.percona.com
Tuesday, December 4, 2012
44. pt-query-digest
• Analyze query execution logs and generate a
query report, filter, replay, or transform queries.
• If you learn only one tool in Percona Toolkit,
make it this one!
• Capture all traffic in the slow query log...
temporarily.
mysql> SET GLOBAL long_query_time=0;
. . . wait for traffic . . .
mysql> SET GLOBAL long_query_time=10;
www.percona.com
Tuesday, December 4, 2012
45. Report Output (1)
$ pt-query-digest /var/lib/mysql/mysql-slow.log
# 20.3s user time, 160ms system time, 29.92M rss, 2.34G vsz
# Current date: Mon Aug 15 15:49:53 2011
# Hostname: huey.percona.com
# Files: shopsite-slow.log
# Overall: 88.68k total, 229 unique, 26.98 QPS, 245.51x concurrency ______
# Time range: 2011-08-15 16:00:43 to 16:55:30
# Attribute total min max avg 95% stddev median
# ============ ======= ======= ======= ======= ======= ======= =======
# Exec time 806989s 2s 160s 9s 30s 11s 3s
# Lock time 8s 21us 9ms 90us 159us 81us 76us
# Rows sent 2.35M 0 368.61k 27.84 49.17 1.95k 0.99
# Rows examine 2.74G 0 737.23k 32.44k 101.89k 41.45k 11.91k
# Query size 37.37M 42 16.77k 441.84 719.66 221.12 400.73
www.percona.com
Tuesday, December 4, 2012
47. Report Output (3)
# Query 1: 6.77 QPS, 181.54x concurrency, ID 0x2C28E6666E1DB80F at byte 37195306
# This item is included in the report because it matches --limit.
# Scores: Apdex = 0.01 [1.0], V/M = 2.34
# Query_time sparkline: | _^|
# Time range: 2011-08-15 16:00:43 to 16:48:34
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 21 19450
# Exec time 64 521215s 2s 55s 27s 40s 8s 26s
# Lock time 29 2s 31us 9ms 120us 185us 112us 108us
# Rows sent 0 18.99k 1 1 1 1 0 1
# Rows examine 70 1.94G 102.83k 105.55k 104.70k 101.89k 0 101.89k
# Query size 19 7.22M 377 424 389.04 420.77 18.23 381.65
# String:
# Databases shopsite
# Users appuser
# Query_time distribution
# 1us
# 10us
# 100us
# 1ms
# 10ms
# 100ms
# 1s #
# 10s+ ################################################################
www.percona.com
Tuesday, December 4, 2012
48. Report Output (4)
# Tables
# SHOW TABLE STATUS FROM `shopsite` LIKE 'campaign_user'G
# SHOW CREATE TABLE `shopsite`.`campaign_user`G
# EXPLAIN /*!50100 PARTITIONS*/
SELECT COUNT(distinct email) AS count
FROM campaign_user
WHERE campaign_id = '1'
AND misc_2 IS NULLG
www.percona.com
Tuesday, December 4, 2012
49. Query Review
• Save each query type seen to a table.
$ pt-query-digest
--review h=dewey,D=percona,t=query_review
/var/lib/mysql/mysql-slow.log
• You can add notes to each query type.
• When you analyze next week’s log, the report
excludes previously reviewed queries.
www.percona.com
Tuesday, December 4, 2012
50. Other Options
• Filtering queries.
• Grouping queries.
• Reading other sources of queries.
• Including query EXPLAIN reports.
• Recording query history for reviews, trending.
• Community tools for browsing and visualizing
query review & query history data:
• https://github.com/kormoc/Query-Digest-UI
• https://github.com/box/Anemometer
www.percona.com
Tuesday, December 4, 2012
51. Cautions
• Processing large logs can be resource-intensive.
• Copy logs to another server to avoid overloading
your production site.
• Reports contain real queries!
• Can expose sensitive information.
www.percona.com
Tuesday, December 4, 2012
52. pt-duplicate-key-checker
http://www.percona.com/doc/percona-toolkit/pt-duplicate-key-checker.html
www.percona.com
Tuesday, December 4, 2012
53. pt-duplicate-key-checker
• Find duplicate indexes and foreign keys on
MySQL tables.
• MySQL permits you to create redundant keys.
• Nearly every database has some.
• Output is a series of ALTER TABLE statements
ready to drop or reform duplicate indexes.
www.percona.com
Tuesday, December 4, 2012
54. pt-duplicate-key-checker
$ pt-duplicate-key-checker
# tezt.media_pictures
#############################################################
# subject_node_id is a left-prefix of INDEX
# Key definitions:
# KEY `subject_node_id` (`subject_node_id`)
# KEY `INDEX` USING BTREE
(`subject_node_id`,`frame_id`,`file_id`,`source_id`),
# Column types:
#! `subject_node_id` int(11) unsigned default null
#! `frame_id` smallint(6) unsigned not null
#! `file_id` int(11) unsigned not null
#! `source_id` int(11) not null
# To remove this duplicate index, execute:
ALTER TABLE `tezt`.`media_pictures` DROP INDEX
`subject_node_id`;
www.percona.com
Tuesday, December 4, 2012
55. pt-duplicate-key-checker
# Db.system_transaction
#############################################################
# FOREIGN KEY A (`table_id`) REFERENCES `Db`.`table` (`id`)
is a duplicate of FOREIGN KEY B (`table_id`) REFERENCES
`Db`.`table` (`id`)
# Key definitions:
# CONSTRAINT `A` FOREIGN KEY (`table_id`) REFERENCES
`table` (`id`)
# CONSTRAINT `B` FOREIGN KEY (`table_id`) REFERENCES
`table` (`id`)
# Column types:
#! `table_id` bigint(20) default null
# To remove this duplicate foreign key, execute:
ALTER TABLE `Db`.`system_transaction` DROP FOREIGN KEY `A`;
# MySQL uses the A index for this foreign key constraint
www.percona.com
Tuesday, December 4, 2012
56. pt-table-checksum
http://www.percona.com/doc/percona-toolkit/pt-table-checksum.html
www.percona.com
Tuesday, December 4, 2012
57. Data Drift
• MySQL slaves may not be perfect replicas.
• Non-deterministic statements.
• Out-of-band changes directly on the slave.
• Slave may lag and fail to keep up.
• No built-in checking.
• Are you using a slave for backups or reporting?
www.percona.com
Tuesday, December 4, 2012
58. pt-table-checksum
• Perform an online replication consistency check,
or checksum MySQL tables efficiently.
• This is the solution to detect data drift.
• Works by calculating checksums against “chunks”
of rows.
• The calculation propagates to slaves.
www.percona.com
Tuesday, December 4, 2012
59. Visualize This
master
slave
replication
www.percona.com
Tuesday, December 4, 2012
60. Visualize This
master
slave
replication
www.percona.com
Tuesday, December 4, 2012
61. Visualize This
master
slave
replication
www.percona.com
Tuesday, December 4, 2012
62. Visualize This
master
slave
replication
www.percona.com
Tuesday, December 4, 2012
63. Visualize This
master
slave
replication
www.percona.com
Tuesday, December 4, 2012
64. Visualize This
master
slave
replication
www.percona.com
Tuesday, December 4, 2012
65. Visualize This
master
slave
replication
www.percona.com
Tuesday, December 4, 2012
67. Let’s Break It
• Delete 5% of data on the slave:
mysql> DELETE FROM title
WHERE RAND()*100 < 5;
Query OK, 77712 rows affected (2.09 sec)
www.percona.com
Tuesday, December 4, 2012
69. Check the Slave(s)
mysql> SELECT db, tbl, SUM(this_cnt) AS total_rows, COUNT(*) AS chunks FROM
percona.checksums WHERE ( master_cnt <> this_cnt OR master_crc <>
this_crc OR ISNULL(master_crc) <> ISNULL(this_crc)) GROUP BY db, tbl;
+------+-------+------------+--------+
| db | tbl | total_rows | chunks |
+------+-------+------------+--------+
| imdb | title | 1466007 | 14 |
+------+-------+------------+--------+
www.percona.com
Tuesday, December 4, 2012
70. pt-table-sync
http://www.percona.com/doc/percona-toolkit/pt-table-sync.html
www.percona.com
Tuesday, December 4, 2012
71. pt-table-sync
• Synchronize MySQL table data efficiently.
• This is the solution to correct data drift.
www.percona.com
Tuesday, December 4, 2012
74. Method 3: Sync Two Hosts
•pt-table-sync won’t let you clobber a slave by syncing
it to some host other than its master.
$ pt-table-sync --verbose --execute h=huey d=dewey --tables imdb.title
Can't make changes on h=dewey because it's a slave. See the documentation
section 'REPLICATION SAFETY' for solutions to this problem. at /usr/bin/pt-
table-sync line 10642.
www.percona.com
Tuesday, December 4, 2012
75. Method 3: Sync Two Hosts
•Now let’s try again, after a RESET SLAVE.
$ pt-table-sync --verbose --execute h=huey h=dewey --tables imdb.title
# Syncing h=dewey
# DELETE REPLACE INSERT UPDATE ALGORITHM START END EXIT DATABASE.TABLE
# 0 0 30867 0 Chunk 13:33:27 13:35:28 2 imdb.title
www.percona.com
Tuesday, December 4, 2012
76. pt-online-schema-change
http://www.percona.com/doc/percona-toolkit/pt-online-schema-change.html
www.percona.com
Tuesday, December 4, 2012
77. pt-online-schema-change
• Perform online, non-blocking ALTER TABLE.
• Captures concurrent updates to a table while
restructuring.
• Some risks and caveats exist; please read the
manual and test carefully.
www.percona.com
Tuesday, December 4, 2012
78. How MySQL Does ALTER TABLE
• Lock the table.
• Make a new, empty the table like the original.
• Modify the columns of the new empty table.
• Copy all rows of data from original to new table.
• Swap the old and new tables.
• Unlock the tables & drop the original.
www.percona.com
Tuesday, December 4, 2012
79. How pt-osc Does ALTER TABLE
• Lock the table.
• Make a new, empty the table like the original.
• Modify the columns of the new empty table.
• Copy all rows of data from original to new table.
• Iterate over the table in chunks, in primary key order.
• Use triggers to capture ongoing changes in the
original, and apply them to the new table.
• Swap the tables, then drop the original.
• Unlock the tables.
www.percona.com
Tuesday, December 4, 2012
80. Visualize This
cast_info
www.percona.com
Tuesday, December 4, 2012
81. Visualize This
cast_info cast_info
new
www.percona.com
Tuesday, December 4, 2012
82. Visualize This
cast_info cast_info
new
www.percona.com
Tuesday, December 4, 2012
83. Visualize This
cast_info after cast_info
trigger new
www.percona.com
Tuesday, December 4, 2012
84. Visualize This
cast_info after cast_info
trigger new
www.percona.com
Tuesday, December 4, 2012
85. Visualize This
cast_info after cast_info
trigger new
www.percona.com
Tuesday, December 4, 2012
86. Visualize This
cast_info after cast_info
trigger new
www.percona.com
Tuesday, December 4, 2012
87. Visualize This
cast_info after cast_info
trigger new
www.percona.com
Tuesday, December 4, 2012
88. Visualize This
cast_info after cast_info
trigger new
www.percona.com
Tuesday, December 4, 2012
89. Visualize This
cast_info after cast_info
trigger new
www.percona.com
Tuesday, December 4, 2012
90. Visualize This
cast_info cast_info
old
www.percona.com
Tuesday, December 4, 2012
91. Visualize This
cast_info cast_info
×
old
www.percona.com
Tuesday, December 4, 2012
92. Example
$ pt-online-schema-change --alter "ADD COLUMN Dummy INT NOT NULL"
h=localhost,D=imdb,t=cast_info
Exiting without altering `imdb`.`cast_info` because neither --dry-run nor --
execute was specified. Please read the tool's documentation carefully
before using this tool.
www.percona.com
Tuesday, December 4, 2012
93. Example
$ pt-online-schema-change --alter "ADD COLUMN Dummy INT NOT NULL"
h=localhost,D=imdb,t=cast_info --execute
Altering `imdb`.`cast_info`...
Creating new table...
Created new table imdb._cast_info_new OK.
Altering new table...
Altered `imdb`.`_cast_info_new` OK.
Creating triggers...
Created triggers OK.
Copying approximately 22545051 rows...
Copying `imdb`.`cast_info`: 10% 04:05 remain
Copying `imdb`.`cast_info`: 19% 04:07 remain
Copying `imdb`.`cast_info`: 28% 03:44 remain
Copying `imdb`.`cast_info`: 37% 03:16 remain
Copying `imdb`.`cast_info`: 47% 02:47 remain
Copying `imdb`.`cast_info`: 56% 02:18 remain
Copying `imdb`.`cast_info`: 64% 01:53 remain
Copying `imdb`.`cast_info`: 73% 01:28 remain
Copying `imdb`.`cast_info`: 82% 00:55 remain
Copying `imdb`.`cast_info`: 91% 00:26 remain
Copied rows OK.
Swapping tables...
Swapped original and new tables OK.
Dropping old table...
Dropped old table `imdb`.`_cast_info_old` OK.
Dropping triggers...
Dropped triggers OK.
Successfully altered `imdb`.`cast_info`.
www.percona.com
Tuesday, December 4, 2012
94. Self-Adjusting
• Copies rows in “chunks” which are sized
dynamically by default.
• The tool throttles itself back if load increases too
much or if any replication slaves are lagging.
• The tool tries to set its lock timeouts to let
applications be more likely to succeed.
www.percona.com
Tuesday, December 4, 2012
95. Limitations
• You can’t alter a table that already has triggers.
• You can’t add a column that is NOT NULL without
also declaring a DEFAULT value.
• You can’t use replication filters.
• If the table is referenced by any foreign keys, you
must choose a method to resolve them.
www.percona.com
Tuesday, December 4, 2012
96. Why Shouldn’t I Use This?
• Is your table small enough that ALTER is already
quick enough?
• Is your change already very quick, for example
DROP KEY in InnoDB plugin?
• Will pt-online-schema-change cause the change
to take too long or increase the load too much?
• Are you using MySQL 5.6, which supports online
schema changes natively?
www.percona.com
Tuesday, December 4, 2012
97. Cautions
• Execute pt-online-schema-change against a test
instance first.
• Back up your data before you execute a change
like this (and verify the backup is viable).
www.percona.com
Tuesday, December 4, 2012
98. Top Nine Popular Tools
• pt-summary • pt-duplicate-key-
checker
• pt-mysql-summary
• pt-table-checksum
• pt-stalk
• pt-table-sync
• pt-archiver
• pt-online-schema-
• pt-query-digest change
www.percona.com
Tuesday, December 4, 2012
101. There Are Many More Tools
pt-align pt-heartbeat pt-sift
pt-config-diff pt-index-usage pt-slave-delay
pt-deadlock-logger pt-ioprofile pt-slave-find
pt-diskstats pt-kill pt-slave-restart
pt-fifo-split pt-mext pt-table-usage
pt-find pt-pmp pt-upgrade
pt-fingerprint pt-query-advisor pt-variable-advisor
pt-fk-error-logger pt-show-grants pt-visual-explain
www.percona.com
Tuesday, December 4, 2012
102. Percona Training for MySQL
Senior Industry Experts
In-Person and Online Classes
Custom Onsite Training
http://percona.com/training
www.percona.com
Tuesday, December 4, 2012
103. Visit Our Sponsors
diamond
platinum
exhibitors
and the rest!
www.percona.com
Tuesday, December 4, 2012
104. SQL Antipatterns
http://www.pragprog.com/titles/bksqla/
www.percona.com
Tuesday, December 4, 2012
105. License and Copyright
Copyright 2012 Bill Karwin
www.slideshare.net/billkarwin
Released under a Creative Commons 3.0 License:
http://creativecommons.org/licenses/by-nc-nd/3.0/
You are free to share - to copy, distribute and
transmit this work, under the following conditions:
Attribution. Noncommercial. No Derivative Works.
You must attribute this work You may not use this work You may not alter, transform,
to Bill Karwin. for commercial purposes. or build upon this work.
www.percona.com
Tuesday, December 4, 2012