Character encoding configuration in MySQL has always been a bit confusing. With too many options to set, unclear relationships between them, and the default settings that make MySQL incompatible with most languages, it is a headache to many users, many of whom end up with broken data. This lecture will provide an overview of the character set support in MySQL, guidelines on how to use it correctly, and will demonstrate several methods of detecting and repairing mangled data.
Performance Schema for MySQL TroubleshootingSveta Smirnova
Percona Live (https://www.percona.com/live/data-performance-conference-2016/sessions/performance-schema-mysql-troubleshooting)
The performance schema in MySQL version 5.6, released in February, 2013, is a very powerful tool that can help DBAs discover why even the trickiest performance issues occur. Version 5.7 introduces even more instruments and tables. And while all these give you great power, you can get stuck choosing which instrument to use.
In this session, I will start with a description of a typical problem, then guide you how to use the performance schema to find out what causes the issue, the reason for unwanted behavior and how the received information can help you solve a particular problem.
Traditionally, performance schema sessions teach what is in contained in tables. I will, in contrast, start from a performance issue, then demonstrate which instruments and tables can help solve it. We will discuss how to setup the performance schema so that it has minimal impact on your server.
Basic MySQL Troubleshooting for Oracle Database AdministratorsSveta Smirnova
This document provides an overview of basic MySQL troubleshooting techniques for Oracle database administrators. It covers MySQL server architecture including connectors, clients, APIs, storage engines, and plugins. It then discusses basic troubleshooting techniques such as error processing, access privileges, and using system variables, performance schema, and EXPLAIN to analyze query execution plans. The document is intended to help Oracle DBAs understand fundamental aspects of MySQL administration.
Performance Schema for MySQL TroubleshootingSveta Smirnova
The Performance Schema provides detailed information for troubleshooting and optimizing MySQL. It collects instrumentation data on server operations, statements, memory usage, locks and connections. The data can be used to identify slow queries, statements not using indexes, memory consumption trends over time, and more. Configuration and enabling specific instruments allows controlling the level of detail collected.
New features in Performance Schema 5.7 in actionSveta Smirnova
The document discusses new features in Performance Schema 5.7, including improved instrumentation for locks, memory usage, stored routines, prepared statements, and variables. It provides examples of using Performance Schema tables like METADATA_LOCKS, TABLE_HANDLES, and prepared_statements_instances to diagnose issues like locks preventing DDL statements from completing and inconsistently timed stored procedure executions. Practices are suggested to identify memory usage and optimize prepared statement performance.
This document provides an agenda and overview for a MySQL Query Tuning 101 presentation. The summary includes:
1. The agenda covers topics like identifying slow queries, using indexes, the EXPLAIN tool, and other optimization techniques.
2. When queries run slow, the presenter will discuss using indexes to improve performance by allowing MySQL to access data more efficiently.
3. The EXPLAIN tool is covered as a way to estimate query execution and see how MySQL utilizes indexes. Different EXPLAIN output will be demonstrated using examples from an employees database.
Introducing new SQL syntax and improving performance with preparse Query Rewr...Sveta Smirnova
This document discusses a new preparse query rewrite plugin for MySQL that allows adding new SQL syntax like the FILTER clause from SQL:2003. The plugin works by catching the query before parsing and rewriting parts using regular expressions. It also describes extending the plugin to support custom optimizer hints by modifying and restoring thread-specific variable values.
Moving to the NoSQL side: MySQL JSON functionsSveta Smirnova
This document provides an overview of JSON functions that have been added to MySQL to support NoSQL and manipulation of JSON documents. It discusses the history and improvements of the JSON functions, provides examples of how various functions such as json_valid, json_contains_key, json_extract, json_append, json_replace, json_set, json_remove, json_search, json_merge, json_depth, and json_count work, and describes how to install and compile the JSON functions. The JSON functions allow users to validate, search, modify, and work with JSON documents directly in MySQL.
Percona Live 2016 (https://www.percona.com/live/data-performance-conference-2016/sessions/why-use-explain-formatjson). Although EXPLAIN FORMAT=JSON was first presented a long time ago, there still aren't many resources that explain how and why to use it. The most advertised feature is visual EXPLAIN in MySQL Workbench, but this format can do more than create nice pictures. It prints additional information that can't be found in good old tabular EXPLAIN, and can help to solve many tricky performance issues. In this session, I will not only describe which additional information we can get with the new syntax, but also provide examples showing how to use it to diagnose production issues.
Performance Schema for MySQL TroubleshootingSveta Smirnova
Percona Live (https://www.percona.com/live/data-performance-conference-2016/sessions/performance-schema-mysql-troubleshooting)
The performance schema in MySQL version 5.6, released in February, 2013, is a very powerful tool that can help DBAs discover why even the trickiest performance issues occur. Version 5.7 introduces even more instruments and tables. And while all these give you great power, you can get stuck choosing which instrument to use.
In this session, I will start with a description of a typical problem, then guide you how to use the performance schema to find out what causes the issue, the reason for unwanted behavior and how the received information can help you solve a particular problem.
Traditionally, performance schema sessions teach what is in contained in tables. I will, in contrast, start from a performance issue, then demonstrate which instruments and tables can help solve it. We will discuss how to setup the performance schema so that it has minimal impact on your server.
Basic MySQL Troubleshooting for Oracle Database AdministratorsSveta Smirnova
This document provides an overview of basic MySQL troubleshooting techniques for Oracle database administrators. It covers MySQL server architecture including connectors, clients, APIs, storage engines, and plugins. It then discusses basic troubleshooting techniques such as error processing, access privileges, and using system variables, performance schema, and EXPLAIN to analyze query execution plans. The document is intended to help Oracle DBAs understand fundamental aspects of MySQL administration.
Performance Schema for MySQL TroubleshootingSveta Smirnova
The Performance Schema provides detailed information for troubleshooting and optimizing MySQL. It collects instrumentation data on server operations, statements, memory usage, locks and connections. The data can be used to identify slow queries, statements not using indexes, memory consumption trends over time, and more. Configuration and enabling specific instruments allows controlling the level of detail collected.
New features in Performance Schema 5.7 in actionSveta Smirnova
The document discusses new features in Performance Schema 5.7, including improved instrumentation for locks, memory usage, stored routines, prepared statements, and variables. It provides examples of using Performance Schema tables like METADATA_LOCKS, TABLE_HANDLES, and prepared_statements_instances to diagnose issues like locks preventing DDL statements from completing and inconsistently timed stored procedure executions. Practices are suggested to identify memory usage and optimize prepared statement performance.
This document provides an agenda and overview for a MySQL Query Tuning 101 presentation. The summary includes:
1. The agenda covers topics like identifying slow queries, using indexes, the EXPLAIN tool, and other optimization techniques.
2. When queries run slow, the presenter will discuss using indexes to improve performance by allowing MySQL to access data more efficiently.
3. The EXPLAIN tool is covered as a way to estimate query execution and see how MySQL utilizes indexes. Different EXPLAIN output will be demonstrated using examples from an employees database.
Introducing new SQL syntax and improving performance with preparse Query Rewr...Sveta Smirnova
This document discusses a new preparse query rewrite plugin for MySQL that allows adding new SQL syntax like the FILTER clause from SQL:2003. The plugin works by catching the query before parsing and rewriting parts using regular expressions. It also describes extending the plugin to support custom optimizer hints by modifying and restoring thread-specific variable values.
Moving to the NoSQL side: MySQL JSON functionsSveta Smirnova
This document provides an overview of JSON functions that have been added to MySQL to support NoSQL and manipulation of JSON documents. It discusses the history and improvements of the JSON functions, provides examples of how various functions such as json_valid, json_contains_key, json_extract, json_append, json_replace, json_set, json_remove, json_search, json_merge, json_depth, and json_count work, and describes how to install and compile the JSON functions. The JSON functions allow users to validate, search, modify, and work with JSON documents directly in MySQL.
Percona Live 2016 (https://www.percona.com/live/data-performance-conference-2016/sessions/why-use-explain-formatjson). Although EXPLAIN FORMAT=JSON was first presented a long time ago, there still aren't many resources that explain how and why to use it. The most advertised feature is visual EXPLAIN in MySQL Workbench, but this format can do more than create nice pictures. It prints additional information that can't be found in good old tabular EXPLAIN, and can help to solve many tricky performance issues. In this session, I will not only describe which additional information we can get with the new syntax, but also provide examples showing how to use it to diagnose production issues.
Performance Schema for MySQL TroubleshootingSveta Smirnova
The Performance Schema in MySQL provides tables and instruments for troubleshooting issues like locks, I/O bottlenecks, slow queries, memory usage, and replication failures. It contains over 500 instruments in MySQL 5.6 and over 800 in 5.7. The tables provide visibility into the internal workings of MySQL to analyze and optimize performance.
This presentation discusses troubleshooting MySQL performance issues. It begins by explaining how to determine if MySQL is the source of slow performance by measuring query response times. The presentation then covers various aspects to check like queries, server options, hardware resources, and replication setup. Specific diagnostic tools are also introduced like EXPLAIN, slow query log, and various system monitoring commands. Finally, it recommends following a structured process of tuning queries, options, and hardware to isolate and resolve any performance problems.
This document discusses troubleshooting MySQL performance issues. It begins with an overview of MySQL server architecture and important components like the optimizer and storage engines. It then covers various diagnostic instruments for troubleshooting like log files, the Information Schema, and the Performance Schema. Specific issues covered include single statement performance using EXPLAIN, internal concurrency issues detected via locks diagnostics from sources like SHOW PROCESSLIST and the Performance Schema. The document provides examples of using these diagnostic tools to analyze and optimize query performance.
Basic MySQL Troubleshooting for Oracle DBAsSveta Smirnova
The document provides an overview of MySQL architecture, configuration, troubleshooting and optimization. It discusses MySQL installation layout, log files, connectors, storage engines, concurrency issues, and diagnostic tools like SHOW PROCESSLIST and INFORMATION_SCHEMA for troubleshooting. The presentation agenda includes introduction, single-client issues, concurrency issues, high availability solutions and tools.
- The document discusses using various Oracle diagnostic tools like AWR, ASH, and SQL Monitoring for SQL tuning. It focuses on scenarios where multiple data sources are needed to fully understand performance issues.
- Historical ASH and AWR data can provide different perspectives on SQL executions over time that help identify problems like long-running queries or concurrent executions. However, ASH only samples a subset of data so it may miss short queries.
- GV$SQL and AWR reports aggregate performance metrics over all executions of a SQL, so they do not show the full user experience if a query runs intermittently. ASH sampled data can help determine how database time relates to clock time in such cases.
-
The document discusses optimizing MySQL/MariaDB query performance through optimizer tuning. It explains that the query optimizer plays a key role in database performance by determining the most efficient query execution plan. Tuning the optimizer can help address performance issues caused by it generating a suboptimal plan. The document provides tips on identifying optimizer-related problems and troubleshooting queries, from examining slow query logs and status variables to testing different index configurations.
The document discusses improvements to the MariaDB query optimizer. It notes that while MySQL is widely used for web applications and OLTP, it is not well-suited for complex analytics queries on large datasets due to issues with disk access strategies and subquery optimizations. MariaDB 5.3 includes multi-range read and batched key access features that improve disk access, reducing query times by over 10x on benchmark tests. It also includes many additional subquery optimization strategies beyond those in earlier MySQL versions.
This document provides a summary of a presentation on practical MySQL tuning. It discusses measuring critical system resources like CPU, memory, I/O and network usage to identify bottlenecks. It also covers rough tuning of MySQL parameters like the InnoDB buffer pool size, log file size and key buffer size. Further tuning includes application optimizations like query tuning with EXPLAIN, index tuning, and schema design. The presentation also discusses scaling MySQL through approaches like caching, sharding, replication and optimizing architecture and data distribution. Regular performance monitoring is emphasized to simulate increased load and aid capacity planning.
The document summarizes several new features in the MariaDB and MySQL query optimizers, including:
1) Subquery optimizations that improved subquery performance from orders of magnitude slower to on par with other databases. These optimizations are available in MariaDB 5.3/5.5 and MySQL 5.6.
2) Batched Key Access, which speeds up large IO-bound joins by accessing keys in batches rather than randomly, improving performance by orders of magnitude.
3) Index Condition Pushdown, which pushes SELECT conditions into indexes to filter records before reading the table, improving performance for IO-bound queries similar to "Using index".
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013Sergey Petrunya
The document discusses techniques for identifying and addressing problems with a database query optimizer. It describes old and new tools for catching slow queries, such as the slow query log, SHOW PROCESSLIST, and the Performance Schema. It also provides examples of using these tools to analyze query plans, identify inefficient plans, and determine if optimizer settings or query structure need to be modified to address performance issues.
This document summarizes new features in the MariaDB 10.0 query optimizer, including:
1. Engine-independent statistics like histograms that are collected via ANALYZE TABLE instead of random sampling.
2. New subquery optimizations that convert EXISTS subqueries to inner joins and trivially correlated EXISTS to IN.
3. EXPLAIN improvements like SHOW EXPLAIN to see EXPLAIN plans for running queries, and logging EXPLAIN output in the slow query log.
This document discusses new SQL syntax and query rewrite plugins in MySQL. It introduces FILTER clauses and custom optimizer hints and describes how to implement them using query rewrite plugins. Key points covered include the plugin interface, parsing and rewriting queries, managing memory, and customizing variables. The goal is to add new SQL features and control query execution through simple syntax extensions. Examples of implementing FILTER clauses and custom optimizer hints are provided to demonstrate how rewrite plugins work.
Introduction into MySQL Query Tuning for Dev[Op]sSveta Smirnova
Percona Live Online 2021 talk: https://www.percona.com/resources/videos/introduction-mysql-query-tuning-for-devops
In this talk I will show how to get started with MySQL Query Tuning. I will make a short introduction into physical table structure and demonstrate how it may influence query execution time.
Then we will discuss basic query tuning instruments and techniques, mainly EXPLAIN command with its latest variations. You will learn how to understand its output and how to rewrite queries or change table structure to achieve better performance.
MariaDB: Engine Independent Table Statistics, including histogramsSergey Petrunya
EITS (Engine Independent Table Statistics) provides new statistics for MariaDB query optimization, including histograms for column value distributions. These statistics must be manually collected with ANALYZE TABLE and enabled to improve estimates for queries using non-indexed columns and ranges. With EITS, the optimizer can better estimate row counts and filtering for joins and WHERE clauses.
Data encoding techniques for reducing energyb consumption in network on-chipLogicMindtech Nologies
VLSI Projects for M. Tech, VLSI Projects in Vijayanagar, VLSI Projects in Bangalore, M. Tech Projects in Vijayanagar, M. Tech Projects in Bangalore, VLSI IEEE projects in Bangalore, IEEE 2015 VLSI Projects, FPGA and Xilinx Projects, FPGA and Xilinx Projects in Bangalore, FPGA and Xilinx Projects in Vijayangar
Performance Schema for MySQL TroubleshootingSveta Smirnova
The Performance Schema in MySQL provides tables and instruments for troubleshooting issues like locks, I/O bottlenecks, slow queries, memory usage, and replication failures. It contains over 500 instruments in MySQL 5.6 and over 800 in 5.7. The tables provide visibility into the internal workings of MySQL to analyze and optimize performance.
This presentation discusses troubleshooting MySQL performance issues. It begins by explaining how to determine if MySQL is the source of slow performance by measuring query response times. The presentation then covers various aspects to check like queries, server options, hardware resources, and replication setup. Specific diagnostic tools are also introduced like EXPLAIN, slow query log, and various system monitoring commands. Finally, it recommends following a structured process of tuning queries, options, and hardware to isolate and resolve any performance problems.
This document discusses troubleshooting MySQL performance issues. It begins with an overview of MySQL server architecture and important components like the optimizer and storage engines. It then covers various diagnostic instruments for troubleshooting like log files, the Information Schema, and the Performance Schema. Specific issues covered include single statement performance using EXPLAIN, internal concurrency issues detected via locks diagnostics from sources like SHOW PROCESSLIST and the Performance Schema. The document provides examples of using these diagnostic tools to analyze and optimize query performance.
Basic MySQL Troubleshooting for Oracle DBAsSveta Smirnova
The document provides an overview of MySQL architecture, configuration, troubleshooting and optimization. It discusses MySQL installation layout, log files, connectors, storage engines, concurrency issues, and diagnostic tools like SHOW PROCESSLIST and INFORMATION_SCHEMA for troubleshooting. The presentation agenda includes introduction, single-client issues, concurrency issues, high availability solutions and tools.
- The document discusses using various Oracle diagnostic tools like AWR, ASH, and SQL Monitoring for SQL tuning. It focuses on scenarios where multiple data sources are needed to fully understand performance issues.
- Historical ASH and AWR data can provide different perspectives on SQL executions over time that help identify problems like long-running queries or concurrent executions. However, ASH only samples a subset of data so it may miss short queries.
- GV$SQL and AWR reports aggregate performance metrics over all executions of a SQL, so they do not show the full user experience if a query runs intermittently. ASH sampled data can help determine how database time relates to clock time in such cases.
-
The document discusses optimizing MySQL/MariaDB query performance through optimizer tuning. It explains that the query optimizer plays a key role in database performance by determining the most efficient query execution plan. Tuning the optimizer can help address performance issues caused by it generating a suboptimal plan. The document provides tips on identifying optimizer-related problems and troubleshooting queries, from examining slow query logs and status variables to testing different index configurations.
The document discusses improvements to the MariaDB query optimizer. It notes that while MySQL is widely used for web applications and OLTP, it is not well-suited for complex analytics queries on large datasets due to issues with disk access strategies and subquery optimizations. MariaDB 5.3 includes multi-range read and batched key access features that improve disk access, reducing query times by over 10x on benchmark tests. It also includes many additional subquery optimization strategies beyond those in earlier MySQL versions.
This document provides a summary of a presentation on practical MySQL tuning. It discusses measuring critical system resources like CPU, memory, I/O and network usage to identify bottlenecks. It also covers rough tuning of MySQL parameters like the InnoDB buffer pool size, log file size and key buffer size. Further tuning includes application optimizations like query tuning with EXPLAIN, index tuning, and schema design. The presentation also discusses scaling MySQL through approaches like caching, sharding, replication and optimizing architecture and data distribution. Regular performance monitoring is emphasized to simulate increased load and aid capacity planning.
The document summarizes several new features in the MariaDB and MySQL query optimizers, including:
1) Subquery optimizations that improved subquery performance from orders of magnitude slower to on par with other databases. These optimizations are available in MariaDB 5.3/5.5 and MySQL 5.6.
2) Batched Key Access, which speeds up large IO-bound joins by accessing keys in batches rather than randomly, improving performance by orders of magnitude.
3) Index Condition Pushdown, which pushes SELECT conditions into indexes to filter records before reading the table, improving performance for IO-bound queries similar to "Using index".
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013Sergey Petrunya
The document discusses techniques for identifying and addressing problems with a database query optimizer. It describes old and new tools for catching slow queries, such as the slow query log, SHOW PROCESSLIST, and the Performance Schema. It also provides examples of using these tools to analyze query plans, identify inefficient plans, and determine if optimizer settings or query structure need to be modified to address performance issues.
This document summarizes new features in the MariaDB 10.0 query optimizer, including:
1. Engine-independent statistics like histograms that are collected via ANALYZE TABLE instead of random sampling.
2. New subquery optimizations that convert EXISTS subqueries to inner joins and trivially correlated EXISTS to IN.
3. EXPLAIN improvements like SHOW EXPLAIN to see EXPLAIN plans for running queries, and logging EXPLAIN output in the slow query log.
This document discusses new SQL syntax and query rewrite plugins in MySQL. It introduces FILTER clauses and custom optimizer hints and describes how to implement them using query rewrite plugins. Key points covered include the plugin interface, parsing and rewriting queries, managing memory, and customizing variables. The goal is to add new SQL features and control query execution through simple syntax extensions. Examples of implementing FILTER clauses and custom optimizer hints are provided to demonstrate how rewrite plugins work.
Introduction into MySQL Query Tuning for Dev[Op]sSveta Smirnova
Percona Live Online 2021 talk: https://www.percona.com/resources/videos/introduction-mysql-query-tuning-for-devops
In this talk I will show how to get started with MySQL Query Tuning. I will make a short introduction into physical table structure and demonstrate how it may influence query execution time.
Then we will discuss basic query tuning instruments and techniques, mainly EXPLAIN command with its latest variations. You will learn how to understand its output and how to rewrite queries or change table structure to achieve better performance.
MariaDB: Engine Independent Table Statistics, including histogramsSergey Petrunya
EITS (Engine Independent Table Statistics) provides new statistics for MariaDB query optimization, including histograms for column value distributions. These statistics must be manually collected with ANALYZE TABLE and enabled to improve estimates for queries using non-indexed columns and ranges. With EITS, the optimizer can better estimate row counts and filtering for joins and WHERE clauses.
Data encoding techniques for reducing energyb consumption in network on-chipLogicMindtech Nologies
VLSI Projects for M. Tech, VLSI Projects in Vijayanagar, VLSI Projects in Bangalore, M. Tech Projects in Vijayanagar, M. Tech Projects in Bangalore, VLSI IEEE projects in Bangalore, IEEE 2015 VLSI Projects, FPGA and Xilinx Projects, FPGA and Xilinx Projects in Bangalore, FPGA and Xilinx Projects in Vijayangar
Data encoding converts data into a signal form for transmission. There are different encoding methods for digital-to-digital, digital-to-analog, analog-to-analog, and analog-to-digital conversion. Common digital encoding techniques include unipolar, bipolar, and polar encoding, which represent binary data using single or multiple voltage levels. Non-return-to-zero and return-to-zero are examples that represent bits through the signal remaining at a level or returning to zero between bits. Manchester encoding and differential Manchester encoding are biphase techniques that provide clocking through signal transitions.
The document provides information about NIICT, a training institute that offers courses in systems and network administration, animation and multimedia. It details some of NIICT's strengths and courses offered, including certifications from Cisco, Microsoft, Red Hat and Solaris. The document also provides background information on Cisco, an overview of Cisco's products and solutions, and descriptions of Cisco certifications including CCNA, CCNP and CCIE.
This chapter discusses various methods for encoding analog and digital signals for transmission, including different conversion schemes such as analog to digital and digital to digital. It examines encoding techniques like unipolar, polar, NRZ-L, NRZ-I, RZ, Manchester, bipolar AMI, B8ZS, and HDB3 encoding. The chapter also includes examples and solutions related to signal encoding.
This document discusses asynchronous and synchronous communication. Asynchronous communication transmits data intermittently without an external clock, with timing encoded in the symbols. It is used for variable bit rate applications like file transfers and email. Synchronous communication transmits data in a steady stream at a constant rate, requiring synchronized clocks, but allows more data transfer. It is used for real-time applications like telephone calls. Both methods have advantages and disadvantages related to overhead, speed and clock synchronization.
Synchronous data transfer involves sharing a common clock between a CPU and I/O interface so that data transfer is coordinated. Asynchronous transfer has independent clocks, so handshaking methods like strobe control and handshaking are used. Strobe control uses a single strobe pulse to indicate valid data. Handshaking adds a second control signal for acknowledgment between units. This ensures the source knows data was received and the destination knows data is available.
This document discusses networking devices and technologies used to connect local area networks (LANs) and wide area networks (WANs). It describes common physical layer components used in Ethernet LANs such as twisted pair cable, fiber optic cable, and connectors. It also discusses serial connection options and devices used for WAN connections including CSU/DSUs and their roles as data terminal equipment (DTE) and data circuit-terminating equipment (DCE).
The document discusses different methods of encoding and modulating digital and analog signals for transmission. It covers digital-to-digital encoding techniques like unipolar, polar, Manchester and differential Manchester encoding. It also discusses analog-to-digital conversion techniques like PAM and PCM. Finally, it discusses analog-to-analog modulation techniques like AM, FM and PM and how they modulate parameters of a carrier signal to transmit an analog signal.
This document provides an overview of MySQL including its architecture, clients, connection layer, SQL layer, storage engines, and installation methods. Key points include:
- MySQL uses a connection thread pool, query cache, parser, optimizer, and various storage engines like InnoDB and MyISAM.
- Common MySQL clients allow executing queries, administering the server, checking tables, backing up data, and more.
- The connection layer handles authentication and the communication protocol.
- The SQL layer performs parsing, optimization, and determining the optimal execution plan.
- Storage engines like InnoDB and MyISAM support different features and have different performance characteristics for storage, indexing, and more.
The document discusses SQL and how to interact with databases using SQL. It provides examples of using SQL commands like CREATE TABLE, INSERT, SELECT, UPDATE, and DELETE to structure and manipulate a database. It also shows how to connect to a MySQL database from the command line or using PHP scripts to execute SQL queries and retrieve/modify data in the database.
While MySQL is a popular and widely used RDBMS, some default features and settings are very foreign in comparison with other commercial RDBMS products. In this discussion, Ronald Bradford will discuss some of the MySQL defaults including a non-transactional state, silent data truncations, date management, and transaction isolation options. These are all critical for data integrity and consistency. He will cover in-depth topics including SQL_MODE that saves the day. He will also cover character sets and collations and the best practices to ensure your UTF8 is stored and retrieved correctly.
This document discusses some idiosyncrasies in MySQL that can cause issues if not properly understood. It covers three main topics:
1. Data integrity - MySQL does not enforce data integrity by default like other RDBMS. Values outside ranges are inserted with warnings instead of errors. Settings like STRICT_ALL_TABLES can help enforce integrity.
2. Transactions - Non-transactional storage engines like MyISAM do not support transactions fully. Changes may not roll back as expected. InnoDB is recommended.
3. Variable scope - Changing global variables like storage_engine does not affect existing tables. New tables will still use prior default unless DB is restarted. Variable changes only affect new connections.
The document provides 15 pro-tips for MySQL users, beginning with basic tips like using the correct MySQL version and understanding EXPLAIN plans, and progressing to more advanced tips involving backups, queries, indexing, and database design. Key recommendations include knowing important configuration settings, backing up at the table level, avoiding SELECT * queries, using triggers for consistency, and separating volatile and non-volatile data.
OSMC 2008 | Monitoring MySQL by Geert VanderkelenNETWAYS
Monitoring MySQL has a long history within Nagios. Several plugins are available already. In addition to that, there are probably lots of plugins that have been developed by the community. We take a look at some of these and discuss what kind of additional useful information could be pulled out of a MySQL Server for monitoring it even better. A simple example on how to write such plugins will be shown, also using NDB API for monitoring MySQL Cluster. Now that MySQL Enterprise Monitor (MEM) is available, we'll go through the possibilities for combining the two platforms. We will also discuss the NDOUtils for storing configuration and event data using MySQL.
This talk starts with a brief overview of MySQL itself: some history, where it's heading too, and why it is so successful.
1. The document discusses MySQL idiosyncrasies related to data integrity, transactions, and variable scope. It provides examples of unexpected behavior when the default storage engine is MyISAM rather than InnoDB, and the use of SQL modes to enforce data integrity.
2. Transactions examples show data is not rolled back as expected when tables are MyISAM rather than InnoDB.
3. Changing global variables like storage engine does not affect existing or new tables, demonstrating variable scope issues in MySQL.
MySQLinsanity! This document provides an overview of Stanley Huang's MySQL performance tuning experience and expertise. It begins with introductions and background on Stanley Huang. It then discusses the typical phases of MySQL performance tuning projects, including SQL tuning and RDBMS tuning. Specific tips are provided around topics like slow query logging, index usage, partitioning, and server configuration. The document concludes with an invitation for questions.
Just about anyone can write a basic SQL query for a table. Not everyone can write a good query though - that takes practice and knowing how to understand what the optimizer is doing with the query. Learn the basics of query optimization so you keep your application engaging the user rather then showing the progress bar as they wait on the database.
How to Avoid Pitfalls in Schema Upgrade with GaleraSveta Smirnova
This document discusses different methods for performing schema upgrades in a Galera cluster, including Total Order Isolation (TOI), Rolling Schema Upgrade (RSU), and the pt-online-schema-change tool. TOI blocks the entire cluster during DDL but ensures consistency, while RSU allows upgrades without blocking the cluster but requires stopping writes and carries risks of inconsistency. Pt-online-schema-change performs non-blocking upgrades using triggers to copy rows to a new table in chunks.
MySQL Kitchen : spice up your everyday SQL queriesDamien Seguy
This document provides an agenda and summary for a MySQL conference session on clever SQL recipes and techniques for MySQL. The session will cover topics like unexpected sorting results, storing IP addresses efficiently, using auto-increment for multiple columns, generating random values, and transposing row data. The presenter is Damien Séguy, a MySQL expert consultant, who will demonstrate various techniques using a sample PHP statistics database schema. Attendees are encouraged to ask questions throughout the presentation.
Big Data Analytics with MariaDB ColumnStoreMariaDB plc
MariaDB ColumnStore is an open source columnar database storage engine that provides high performance analytics capabilities on large datasets using standard SQL. It uses a distributed architecture that stores data by column rather than by row to enable fast queries by only accessing the relevant columns. It can scale horizontally on commodity servers to support analytics workloads on datasets ranging from millions to trillions of rows.
This document provides an overview of MySQL basics including what MySQL is, installing the MySQL server, using the mysql command line client, SQL basics like concepts and terminology for databases, tables and fields, and performing operations like insert, update, delete and select. It also covers retrieving records with select through ordering, limiting and arithmetic functions. The document concludes by explaining how to read and execute SQL statements from a script file.
This document discusses several MySQL idiosyncrasies related to data integrity, transactions, and variable scope that can cause unexpected behavior for developers. Specifically, it notes that MySQL does not enforce data integrity by default, transactions are not supported with all storage engines, and variable settings apply at the session level rather than globally by default. The document provides examples of each issue and recommends configurations like stricter SQL modes and using InnoDB to avoid potential problems.
This document provides instructions for students to obtain fully solved assignments. It tells students to send their semester and specialization name to the email address "help.mbaassignments@gmail.com" or call the phone number 08263069601 to receive solved assignments. It provides this contact information to help students complete their coursework.
Percona Live 4/15/15: Transparent sharding database virtualization engine (DVE)Tesora
Amrith Kumar of Tesora and Peter Boros of Percona present an in-depth exploration of transparent database scale out use the Tesora DVE framework for MySQL.
Applied Partitioning And Scaling Your Database System PresentationRichard Crowley
This document discusses applying partitioning to MySQL OLTP applications. It begins by reviewing classic uses of partitioning in data warehousing applications. It then explains how partitioning can benefit OLTP workloads by reducing seek and scan set sizes and limiting transaction durations. The document provides an example of applying hash partitioning to tables in a retail store application based on store ID. It demonstrates how to expand and merge partitions as stores are added or closed. Performance tests on the partitioned tables show improvements in query times compared to non-partitioned tables. In summary, the document argues that partitioning provides a way to scale MySQL databases for OLTP workloads while reducing maintenance costs.
MySQL 5.7 innodb_enhance_partii_20160527Saewoong Lee
Release Date : 2016.05.27
Version : MySQL 5.7
Index :
- Part I : InnoDB Performance
- Part I : InnoDB Buffer Pool Flushing
- Part I : InnoDB internal Transaction General
- Part I : InnoDB Improved adaptive flushing
- Part II : InnoDB Online DDL
- Part II : Tablespace management
- Part II : InnoDB Bulk Load for Create Index
- Part II : InnoDB Temporary Tables
- Part II : InnoDB Full-Text CJK Support
- Part II : Support Syslog on Linux / Unix OS
- Part II : Performance_schema
- Part II : Useful tips
MySQL 5.7 proposes several changes to improve performance and consistency including:
1. Making replication durable by default by setting sync_binlog and repository options.
2. Deprecating features like INNODB monitor tables and ALTER IGNORE TABLE in favor of newer standards.
3. Simplifying and restricting SQL modes to encourage stricter querying and remove ambiguous options. Explanations for errors and modes will also be improved.
Similar to Character Encoding - MySQL DevRoom - FOSDEM 2015 (20)
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISTier1 app
Are you ready to unlock the secrets hidden within Java thread dumps? Join us for a hands-on session where we'll delve into effective troubleshooting patterns to swiftly identify the root causes of production problems. Discover the right tools, techniques, and best practices while exploring *real-world case studies of major outages* in Fortune 500 enterprises. Engage in interactive lab exercises where you'll have the opportunity to troubleshoot thread dumps and uncover performance issues firsthand. Join us and become a master of Java thread dump analysis!
Boost Your Savings with These Money Management AppsJhone kinadey
A money management app can transform your financial life by tracking expenses, creating budgets, and setting financial goals. These apps offer features like real-time expense tracking, bill reminders, and personalized insights to help you save and manage money effectively. With a user-friendly interface, they simplify financial planning, making it easier to stay on top of your finances and achieve long-term financial stability.
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdfBaha Majid
IBM watsonx Code Assistant for Z, our latest Generative AI-assisted mainframe application modernization solution. Mainframe (IBM Z) application modernization is a topic that every mainframe client is addressing to various degrees today, driven largely from digital transformation. With generative AI comes the opportunity to reimagine the mainframe application modernization experience. Infusing generative AI will enable speed and trust, help de-risk, and lower total costs associated with heavy-lifting application modernization initiatives. This document provides an overview of the IBM watsonx Code Assistant for Z which uses the power of generative AI to make it easier for developers to selectively modernize COBOL business services while maintaining mainframe qualities of service.
Orca: Nocode Graphical Editor for Container OrchestrationPedro J. Molina
Tool demo on CEDI/SISTEDES/JISBD2024 at A Coruña, Spain. 2024.06.18
"Orca: Nocode Graphical Editor for Container Orchestration"
by Pedro J. Molina PhD. from Metadev
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...kalichargn70th171
In today's fiercely competitive mobile app market, the role of the QA team is pivotal for continuous improvement and sustained success. Effective testing strategies are essential to navigate the challenges confidently and precisely. Ensuring the perfection of mobile apps before they reach end-users requires thoughtful decisions in the testing plan.
Streamlining End-to-End Testing Automation with Azure DevOps Build & Release Pipelines
Automating end-to-end (e2e) test for Android and iOS native apps, and web apps, within Azure build and release pipelines, poses several challenges. This session dives into the key challenges and the repeatable solutions implemented across multiple teams at a leading Indian telecom disruptor, renowned for its affordable 4G/5G services, digital platforms, and broadband connectivity.
Challenge #1. Ensuring Test Environment Consistency: Establishing a standardized test execution environment across hundreds of Azure DevOps agents is crucial for achieving dependable testing results. This uniformity must seamlessly span from Build pipelines to various stages of the Release pipeline.
Challenge #2. Coordinated Test Execution Across Environments: Executing distinct subsets of tests using the same automation framework across diverse environments, such as the build pipeline and specific stages of the Release Pipeline, demands flexible and cohesive approaches.
Challenge #3. Testing on Linux-based Azure DevOps Agents: Conducting tests, particularly for web and native apps, on Azure DevOps Linux agents lacking browser or device connectivity presents specific challenges in attaining thorough testing coverage.
This session delves into how these challenges were addressed through:
1. Automate the setup of essential dependencies to ensure a consistent testing environment.
2. Create standardized templates for executing API tests, API workflow tests, and end-to-end tests in the Build pipeline, streamlining the testing process.
3. Implement task groups in Release pipeline stages to facilitate the execution of tests, ensuring consistency and efficiency across deployment phases.
4. Deploy browsers within Docker containers for web application testing, enhancing portability and scalability of testing environments.
5. Leverage diverse device farms dedicated to Android, iOS, and browser testing to cover a wide range of platforms and devices.
6. Integrate AI technology, such as Applitools Visual AI and Ultrafast Grid, to automate test execution and validation, improving accuracy and efficiency.
7. Utilize AI/ML-powered central test automation reporting server through platforms like reportportal.io, providing consolidated and real-time insights into test performance and issues.
These solutions not only facilitate comprehensive testing across platforms but also promote the principles of shift-left testing, enabling early feedback, implementing quality gates, and ensuring repeatability. By adopting these techniques, teams can effectively automate and execute tests, accelerating software delivery while upholding high-quality standards across Android, iOS, and web applications.
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...kalichargn70th171
Visual testing plays a vital role in ensuring that software products meet the aesthetic requirements specified by clients in functional and non-functional specifications. In today's highly competitive digital landscape, users expect a seamless and visually appealing online experience. Visual testing, also known as automated UI testing or visual regression testing, verifies the accuracy of the visual elements that users interact with.
Penify - Let AI do the Documentation, you write the Code.KrishnaveniMohan1
Penify automates the software documentation process for Git repositories. Every time a code modification is merged into "main", Penify uses a Large Language Model to generate documentation for the updated code. This automation covers multiple documentation layers, including InCode Documentation, API Documentation, Architectural Documentation, and PR documentation, each designed to improve different aspects of the development process. By taking over the entire documentation process, Penify tackles the common problem of documentation becoming outdated as the code evolves.
https://www.penify.dev/
The Comprehensive Guide to Validating Audio-Visual Performances.pdfkalichargn70th171
Ensuring the optimal performance of your audio-visual (AV) equipment is crucial for delivering exceptional experiences. AV performance validation is a critical process that verifies the quality and functionality of your AV setup. Whether you're a content creator, a business conducting webinars, or a homeowner creating a home theater, validating your AV performance is essential.
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Paul Brebner
Closing talk for the Performance Engineering track at Community Over Code EU (Bratislava, Slovakia, June 5 2024) https://eu.communityovercode.org/sessions/2024/why-apache-kafka-clusters-are-like-galaxies-and-other-cosmic-kafka-quandaries-explored/ Instaclustr (now part of NetApp) manages 100s of Apache Kafka clusters of many different sizes, for a variety of use cases and customers. For the last 7 years I’ve been focused outwardly on exploring Kafka application development challenges, but recently I decided to look inward and see what I could discover about the performance, scalability and resource characteristics of the Kafka clusters themselves. Using a suite of Performance Engineering techniques, I will reveal some surprising discoveries about cosmic Kafka mysteries in our data centres, related to: cluster sizes and distribution (using Zipf’s Law), horizontal vs. vertical scalability, and predicting Kafka performance using metrics, modelling and regression techniques. These insights are relevant to Kafka developers and operators.
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...Luigi Fugaro
Vector databases are transforming how we handle data, allowing us to search through text, images, and audio by converting them into vectors. Today, we'll dive into the basics of this exciting technology and discuss its potential to revolutionize our next-generation AI applications. We'll examine typical uses for these databases and the essential tools
developers need. Plus, we'll zoom in on the advanced capabilities of vector search and semantic caching in Java, showcasing these through a live demo with Redis libraries. Get ready to see how these powerful tools can change the game!
1. Character encoding
Breaking and unbreaking your data
Maciej Dobrzanski
maciek@psce.com | @mushupl
Brussels, 1 Feb 2015
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
2. Character Encoding
• Binary representation of glyphs
• Each character can be represented by 1 or more bytes
• Popular schemes
• ASCII
• Unicode
• UTF-8, UTF-16, UTF-32
• Language specific character sets
• US (Latin US)
• Europe (Latin 1, Latin 2)
• Asia (EUC-KR, GB18030)
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
3. Character Encoding
• Character set defines the visual interpretation of binary information
• One glyph can be associated with several numeric codes
• One numeric code may be used to represent several different glyphs
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
4. Please state the nature of the emergency
• Application configuration
• Database configuration
• Table/column definitions
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
5. Problem #1: We are all born Swedish
• MySQL uses latin1 by default
• MySQL 5.7 too
• Is anyone actually aware of that?
• Why Swedish?
• latin1_swedish_ci is the default collation
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
6. Problem #1
• Let’s build an application
mysql> SELECT @@global.character_set_server, @@session.character_set_client;
+-------------------------------+--------------------------------+
| @@global.character_set_server | @@session.character_set_client |
+-------------------------------+--------------------------------+
| latin1 | latin1 |
+-------------------------------+--------------------------------+
1 row in set (0.00 sec)
mysql> CREATE SCHEMA fosdem;
Query OK, 1 row affected (0.00 sec)
mysql> USE fosdem;
mysql> CREATE TABLE locations (city VARCHAR(30) NOT NULL);
Query OK, 0 rows affected (0.15 sec)
mysql> SHOW CREATE TABLE locationsG
*************************** 1. row ***************************
Table: locations
Create Table: CREATE TABLE `locations` (
`city` varchar(30) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
16. Problem #2
• Why is the table character set latin1?
mysql> SELECT @@session.character_set_server, @@session.character_set_client;
+--------------------------------+--------------------------------+
| @@session.character_set_server | @@session.character_set_client |
+--------------------------------+--------------------------------+
| utf8 | utf8 |
+--------------------------------+--------------------------------+
1 row in set (0.00 sec)
mysql> USE fosdem;
mysql> SHOW CREATE TABLE peopleG
*************************** 1. row ***************************
Table: people
Create Table: CREATE TABLE `people` (
`first_name` varchar(30) NOT NULL,
`last_name` varchar(30) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
17. Problem #2
• What’s all this, then?
mysql> SHOW SESSION VARIABLES LIKE 'character_set_%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)
mysql> SHOW CREATE DATABASE fosdemG
*************************** 1. row ***************************
Database: fosdem
Create Database: CREATE DATABASE `fosdem` /*!40100 DEFAULT CHARACTER SET latin1 */
1 row in set (0.00 sec)
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
18. Problem #2
• Can we fix this?
mysql> SET NAMES utf8;
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT last_name, HEX(last_name) FROM people;
+------------+----------------------+
| last_name | HEX(last_name) |
+------------+----------------------+
| Lemon | 4C656D6F6E |
| Müller | 4DFC6C6C6572 |
| Dobrza?ski | 446F62727A613F736B69 |
+------------+----------------------+
3 rows in set (0.00 sec)
mysql> SET NAMES latin2;
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT last_name, HEX(last_name) FROM people;
+------------+----------------------+
| last_name | HEX(last_name) |
+------------+----------------------+
| Lemon | 4C656D6F6E |
| Müller | 4DFC6C6C6572 |
| Dobrza?ski | 446F62727A613F736B69 |
+------------+----------------------+
3 rows in set (0.00 sec)
• We can’t! :-(
• 0x3F is '?', so my 'ń' was lost
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
19. Problem #2: The bad news
• It may not be enough to configure the server correctly
• A mismatch between client and server can permantenly break data
• Implicit conversion inside MySQL server
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
20. Problem #2: Settings, defaults, inheritance
• Where do you set character sets in MySQL?
• Sesssion settings
• character_set_server
• character_set_client
• character_set_connection
• character_set_database
• character_set_result
• Schema level defaults – affect new tables
• Table level defaults – affect new columns
• Column charsets
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
21. 01.02.2015 Follow us on Twitter @dbasquare www.psce.com
Problem #2: Settings, defaults, inheritance
master [localhost] {msandbox} ((none)) > SELECT @@global.character_set_server, @@session.character_set_client;
+-------------------------------+--------------------------------+
| @@global.character_set_server | @@session.character_set_client |
+-------------------------------+--------------------------------+
| latin1 | utf8 |
+-------------------------------+--------------------------------+
1 row in set (0.00 sec)
master [localhost] {msandbox} ((none)) > CREATE SCHEMA fosdemG
Query OK, 1 row affected (0.00 sec)
master [localhost] {msandbox} ((none)) > SHOW CREATE SCHEMA fosdemG
*************************** 1. row ***************************
Database: fosdem
Create Database: CREATE DATABASE `fosdem` /*!40100 DEFAULT CHARACTER SET latin1 */
1 row in set (0.00 sec)
22. 01.02.2015 Follow us on Twitter @dbasquare www.psce.com
Problem #2: Settings, defaults, inheritance
master [localhost] {msandbox} ((none)) > USE fosdem;
Database changed
master [localhost] {msandbox} (fosdem) > CREATE TABLE test (a VARCHAR(300), INDEX (a));
Query OK, 0 rows affected (0.62 sec)
master [localhost] {msandbox} (fosdem) > SHOW CREATE TABLE testG
*************************** 1. row ***************************
Table: test
Create Table: CREATE TABLE `test` (
`a` varchar(300) DEFAULT NULL,
KEY `a` (`a`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
23. 01.02.2015 Follow us on Twitter @dbasquare www.psce.com
Problem #2: Settings, defaults, inheritance
master [localhost] {msandbox} (fosdem) > ALTER TABLE test DEFAULT CHARSET = utf8;
Query OK, 0 rows affected (0.08 sec)
Records: 0 Duplicates: 0 Warnings: 0
master [localhost] {msandbox} (fosdem) > SHOW CREATE TABLE testG
*************************** 1. row ***************************
Table: test
Create Table: CREATE TABLE `test` (
`a` varchar(300) CHARACTER SET latin1 DEFAULT NULL,
KEY `a` (`a`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
24. 01.02.2015 Follow us on Twitter @dbasquare www.psce.com
Problem #2: Settings, defaults, inheritance
master [localhost] {msandbox} (fosdem) > ALTER TABLE test ADD b VARCHAR(10);
Query OK, 0 rows affected (0.74 sec)
Records: 0 Duplicates: 0 Warnings: 0
master [localhost] {msandbox} (fosdem) > SHOW CREATE TABLE testG
*************************** 1. row ***************************
Table: test
Create Table: CREATE TABLE `test` (
`a` varchar(300) CHARACTER SET latin1 DEFAULT NULL,
`b` varchar(10) DEFAULT NULL,
KEY `a` (`a`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
25. I f**ckd up. What do I do?
• Let’s start with what you shouldn’t do
• Keep calm and don’t start by changing something
• Analyze the situation
• Why did the problem occur in the first place?
• Reassess the damage
• Is it consistent?
• Are all rows broken in the same way?
• Are some rows bad, but others are okay?
• Are all bad in several different ways?
• Is it actually repearable?
• No character mapping occurred during writes (e.g. unicode over latin1/latin1)
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
26. I f**ckd up. What else I shouldn’t do, then?
• Do not rush things as you may easily go from bad to worse
• Do not start fixing this on a replication slave
• You can’t fix this by fixing tables one by one on a live database
• Unless you really have everything in one table
• Do not use: ALTER TABLE … DEFAULT CHARSET = …
• It only changes the default character set for new columns
• Do not use: ALTER TABLE … CONVERT TO CHARACTER SET …
• It’s not for fixing broken encoding
• Do not use: ALTER TABLE … MODIFY col_name … CHARACTER SET …
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
27. I f**ckd up. So how do I fix it?
• What needs to be fixed?
• Schema defaut character set
• ALTER SCHEMA fosdem DEFAULT CHARSET = utf8
• Tables with text columns: CHAR, VARCHAR, TEXT, TINYTEXT, LONGTEXT
• What about ENUM?
• Use INFORMATION_SCHEMA to grab a list
• What about other tables?
• They too (eventually), but it’s not critical
SELECT CONCAT(c.table_schema, '.', c.table_name) AS candidate_table
FROM information_schema.columns c
WHERE c.table_schema = 'fosdem'
AND c.column_type REGEXP '^(.*CHAR|.*TEXT|ENUM)((.+))?$'
GROUP BY candidate_table;
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
28. I f**ckd up. So how do I fix it?
• Option 1 – requires downtime
• Dump and restore
• Dump the data preserving the bad configuration and drop the old database
bash# mysqldump -u root -p --skip-set-charset --default-character-set=latin1 fosdem >
fosdem.sql
mysql> DROP SCHEMA fosdem;
• Correct table definitions in the dump file
• Edit DEFAULT CHARSET in all CREATE TABLE statements
• Create the database again and import the data back
mysql> CREATE SCHEMA fosdem DEFAULT CHARSET utf8;
bash# mysql -u root -p --default-character-set=utf8 fosdem < fosdem.sql
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
29. I f**ckd up. So how do I fix it?
• Option 2 – requires downtime
• Perform a two step conversion with ALTER TABLE
• Original encoding -> VARBINARY/BLOB -> Target encoding
• Conversion from/to BINARY/BLOB removes character set context
• How?
• Stop applications
• On each tabe, for each text column perform:
ALTER TABLE tbl MODIFY col_name VARBINARY(255);
ALTER TABLE tbl MODIFY col_name VARCHAR(255) CHARACTER SET utf8;
• You may specify multiple columns per ALTER TABLE
• Fix the problems (application and/or db configs)
• Restart applications
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
30. I f**ckd up. So how do I fix it?
• Option 3 – online character set fix; no downtime*
• Thanks to our plugin for pt-online-schema-change
• and a tiny patch for pt-online-schema-change that goes with the plugin
• How?
• Start pt-online-schema-change on all tables – one by one
• Do not rotate tables (--no-swap-tables) or drop pt-osc triggers
• Wait until all tables have been converted
• Stop applications
• Fix the problems (application and/or db configs)
• Rotate tables – takes just 1 minute
• Restart applications
• Et voilà
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
31. GOTCHAs!
• Data space requrements may change during conversion
• Latin1 uses 1 byte per character, utf8 will need to assume 3 bytes
• VARCHAR/TEXT fit up to 64KB – it won’t fit 65536 multi-byte characters
• Key length limit is 767 bytes
• Data type and/or index length changes may be required
• Test and plan this ahead
• There may be more prolems than you think
• Detect irrecoverible problems with a simple stored procedure
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
CREATE FUNCTION `cnv_test_conversion` (`value_before` LONGTEXT, `value_after` LONGTEXT) RETURNS tinyint(1)
BEGIN
RETURN (IFNULL(CONVERT(CONVERT(`value_before` USING latin1) USING binary), "") =
IFNULL(CONVERT(`value_after` USING binary), ""));
END;;
32. 01.02.2015 Follow us on Twitter @dbasquare Need help? Visit www.psce.com
GOTCHAs!
master [localhost] {msandbox} (fosdem) > ALTER TABLE test MODIFY a VARCHAR(300) CHARACTER SET utf8;
Query OK, 0 rows affected, 1 warning (1.23 sec)
Records: 0 Duplicates: 0 Warnings: 1
master [localhost] {msandbox} (fosdem) > SHOW WARNINGSG
*************************** 1. row ***************************
Level: Warning
Code: 1071
Message: Specified key was too long; max key length is 767 bytes
1 row in set (0.00 sec)
master [localhost] {msandbox} (fosdem) > SHOW CREATE TABLE testG
*************************** 1. row ***************************
Table: test
Create Table: CREATE TABLE `test` (
`a` varchar(300) DEFAULT NULL,
`b` varchar(10) DEFAULT NULL,
KEY `a` (`a`(255))
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
33. How to do it right?
• Set character-set-server during initial configuration
• When creating new schemas, always specify the desired charset
• CREATE SCHEMA fosdem DEFAULT CHARSET = utf8
• ALTER SCHEMA fosdem DEFAULT CHARSET = utf8
• When creating new tables, also explicitly specify the charset
• CREATE TABLE people (…) DEFAULT CHARSET = utf8
• And don’t forget to configure applications too
• You can try to force charset on the clients
• init-connect = "SET NAMES utf8"
• It might also break applications that don’t want to talk to MySQL using utf8
01.02.2015 Follow us on Twitter @dbasquare www.psce.com
34. Oh, and one more thing…
01.02.2015 Follow us on Twitter @dbasquare Need help? Visit www.psce.com
35. • We are sharing WebScaleSQL packages with the MySQL Community!
• Check out http://www.psce.com/blog for details
• Follow @dbasquare to receive updates
01.02.2015 Follow us on Twitter @dbasquare 35
WebScaleSQL
What is WebScaleSQL?
WebScaleSQL is a collaboration among engineers from several companies
such as Facebook, Twitter, Google or Linkedin, that face the same challenges
in deploying MySQL at scale, and seek greater performance from a database
technology tailored for their needs.