This document discusses new SQL syntax and query rewrite plugins in MySQL. It introduces FILTER clauses and custom optimizer hints and describes how to implement them using query rewrite plugins. Key points covered include the plugin interface, parsing and rewriting queries, managing memory, and customizing variables. The goal is to add new SQL features and control query execution through simple syntax extensions. Examples of implementing FILTER clauses and custom optimizer hints are provided to demonstrate how rewrite plugins work.

1. Preparse Query Rewrite Plugins
New SQL syntax for fun & performance
September, 13, 2016
Sveta Smirnova

2. •Introducing new SQL syntax
•Working with results
•Variables
•Summary
Table of Contents
2

3. Introducing new SQL syntax

4. • From mascots and from humans
MySQL often receives blames
4

5. • From mascots and from humans
• It cannot make a toast
MySQL often receives blames
4

6. • From mascots and from humans
• It cannot make a toast
•
It does not support some syntax
MySQL often receives blames
4

7. • FILTER clause in MySQL on my home
machine
Or does it?
5

8. •
FILTER clause in MySQL on my home
machine
Or does it?
5

9. •
FILTER clause in MySQL on my home
machine
•
But not in the user manual
Or does it?
5

10. •
With 181 lines of code
• Including comments!
• And new Query Rewrite Plugin interface
How is it done?
6

11. • First introduced in version 5.7.5
•
Was available at MySQL Labs
• Two types of plugins
• Pre-parse
• Post-parse
A little bit of history
7

12. • Part of Audit plugin interface
• Step in at
• MYSQL AUDIT GENERAL ALL
•
MYSQL AUDIT CONNECTION ALL
• MYSQL AUDIT PARSE ALL
MYSQL AUDIT PARSE PREPARSE
MYSQL AUDIT PARSE POSTPARSE
• MYSQL AUDIT AUTHORIZATION ALL
•
...
Today
8

13. #include <mysql/plugin.h>
#include <mysql/plugin_audit.h> - Audit plugin declaration
...
static MYSQL_PLUGIN plugin_info_ptr; - Pointer to the plugin
...
static int filter_plugin_init(MYSQL_PLUGIN plugin_ref); - Plugin initialization
...
static int filter(MYSQL_THD thd, mysql_event_class_t event_class,
const void *event); - Entry point for MYSQL_AUDIT_PARSE_PREPARSE
...
static st_mysql_audit filter_plugin_descriptor;
...
mysql_declare_plugin(filter_plugin);
Plugin skeleton
9

14. static st_mysql_audit filter_plugin_descriptor= {
MYSQL_AUDIT_INTERFACE_VERSION, /* interface version */
NULL,
filter, /* implements FILTER */
// You can also use MYSQL_AUDIT_PARSE_ALL
{ 0, 0, (unsigned long) MYSQL_AUDIT_PARSE_PREPARSE,}
};
Plugin descriptor
10

15. mysql_declare_plugin(filter_plugin)
{
MYSQL_AUDIT_PLUGIN,
&filter_plugin_descriptor,
"filter_plugin",
"Sveta Smirnova",
"FILTER SQL:2003 support for MySQL",
PLUGIN_LICENSE_GPL,
filter_plugin_init,
NULL, /* filter_plugin_deinit - TODO */
0x0001, /* version 0.0.1 */
NULL, /* status variables */
NULL, /* system variables */
NULL, /* config options */
0, /* flags */
}
mysql_declare_plugin_end;
Plugin declaration
11

16. #include <my_thread.h> // my_thread_handle needed by mysql_memory.h
#include <mysql/psi/mysql_memory.h>
...
static PSI_memory_key key_memory_filter;
static PSI_memory_info all_rewrite_memory[]=
{
{ &key_memory_filter, "filter", 0 }
};
static int filter_plugin_init(MYSQL_PLUGIN plugin_ref)
{
plugin_info_ptr= plugin_ref;
const char* category= "sql";
int count;
count= array_elements(all_rewrite_memory);
mysql_memory_register(category, all_rewrite_memory, count);
return 0; /* success */
}
Memory management for plugins
12

17. <filter clause> ::=
FILTER <left paren> WHERE <search condition> <right paren>
(10.9 <aggregate function>, 5WD-02-Foundation-2003-09.pdf, p.505)
Only for aggregate functions:
<computational operation> ::=
AVG | MAX | MIN | SUM | EVERY | ANY
| SOME | COUNT | STDDEV_POP | STDDEV_SAMP
| VAR_SAMP | VAR_POP | COLLECT | FUSION | INTERSECTION
<set quantifier> ::=
DISTINCT
| ALL
MySQL only supports
COUNT | AVG | SUM | MAX | MIN
| STDDEV_POP | STDDEV_SAMP
| VAR_SAMP | VAR_POP
SQL:2003
13

18. • FILTER is practically
CASE WHEN foo THEN bar ELSE NULL
•
So we only need to catch
FUNCTION(var) FILTER(WHERE foo)
• And replace it with CASE
Implementing FILTER clause
14

19. static int filter(MYSQL_THD thd, // MySQL Thread object
mysql_event_class_t event_class, // Class of the event
const void *event // Event itself
)
{
const struct mysql_event_parse *event_parse=
static_cast<const struct mysql_event_parse *>(event);
if (event_parse->event_subclass != MYSQL_AUDIT_PARSE_PREPARSE)
return 0;
string subject= event_parse->query.str; // Original query
string rewritten_query;
//requires std::regex and GCC 4.9+
regex filter_clause_star("(COUNT)((s**s*))s+"
+ "FILTERs*(s*WHEREs+([^)]+)s*)",
ECMAScript | icase);
rewritten_query= regex_replace(subject, filter_clause_star,
"$1(CASE WHEN $3 THEN 1 ELSE NULL END)");
Catching up the query
15

20. void _rewrite_query(const void *event,
const struct mysql_event_parse *event_parse,
char const* new_query
)
{
char *rewritten_query= static_cast<char *>(my_malloc(
key_memory_filter, strlen(new_query) + 1, MYF(0)));
strncpy(rewritten_query, new_query, strlen(new_query));
rewritten_query[strlen(new_query)]= ’0’;
event_parse->rewritten_query->str= rewritten_query; // Rewritten query
event_parse->rewritten_query->length=strlen(new_query);
// You must set this flag to inform MySQL Server what query was rewritten
*((int *)event_parse->flags)|=
(int)MYSQL_AUDIT_PARSE_REWRITE_PLUGIN_QUERY_REWRITTEN;
}
Rewritten query
16

21. Working with results

22. •
Playing with syntax is fun
• But can we introduce something more
MySQL-ish?
Can we do better?
18

23. • MySQL 5.7 has Optimizer Hints
SELECT /*+ NO_RANGE_OPTIMIZATION(t3 PRIMARY, f2_idx) */ f1
FROM t3 WHERE f1 > 30 AND f1 < 33;
SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...;
SELECT /*+ NO_ICP(t1, t2) */ * FROM t1 INNER JOIN t2 WHERE ...;
SELECT /*+ SEMIJOIN(FIRSTMATCH, LOOSESCAN) */ * FROM t1 ...;
EXPLAIN SELECT /*+ NO_ICP(t1) */ * FROM t1 WHERE ...;
Custom hint plugin
19

24. • MySQL 5.7 has Optimizer Hints
• But sometimes thread-specific buffers affect
query execution
•
Common workaround exists:SET tmp_table_size=1073741824;
SELECT * FROM t1 INNER JOIN t2 WHERE ...;
SET tmp_table_size=DEFAULT;
• Workaround requires processing result set
of each of these statements
Custom hint plugin
19

25. • MySQL 5.7 has Optimizer Hints
• But sometimes thread-specific buffers affect
query execution
•
Workaround requires processing result set
of each of these statements
• Percona Server has SET STATEMENT
mysql> SET STATEMENT max_statement_time=1000 FOR SELECT user FROM user;
+------------------+
| user |
+------------------+
| foo |
| root |
...
Custom hint plugin
19

26. • MySQL 5.7 has Optimizer Hints
• But sometimes thread-specific buffers affect
query execution
•
Workaround requires processing result set
of each of these statements
• SET STATEMENT not supported by forksmysql> SET STATEMENT max_statement_time=1000 FOR SELECT user FROM user;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
that corresponds to your MySQL server version for the right syntax to use near
’max_statement_time=1000 FOR SELECT user FROM user’ at line 1
Custom hint plugin
19

27. • MySQL 5.7 has Optimizer Hints
• But sometimes thread-specific buffers affect
query execution
• Workaround requires processing result set
of each of these statements
•
SET STATEMENT not supported by forks
• This is why I extended optimizer hint syntax
SELECT /*+ join_buffer_size=16384 */ f1 FROM t3 WHERE f1 > 30 AND f1 < 33;
SELECT /*+ tmp_table_size=1073741824 BKA(t1) NO_BKA(t2) */ *
FROM t1 INNER JOIN t2 WHERE ...;
Custom hint plugin
19

28. • For Custom Hints we need to:
•
Store previous values of thread variables we
are going to modify
• Modify variables
• Revert them back before sending result
New features
20

29. // map to store modified variables
static map <my_thread_id, map<supported_hints_t, ulonglong> > modified_variables;
...
/* The job */
static int custom_hint(MYSQL_THD thd, mysql_event_class_t event_class,
const void *event)
{
...
// If we have a match store create map of thread variables
std::map<supported_hints_t, ulonglong> current;
...
After processing variables store them in modified_variables map
modified_variables[thd->thread_id()]= current;
...
Store previous values
21

30. • Since we have access to MYSQL THD this
is easy:
switch(get_hint_switch(ssm[1]))
{
case JOIN_BUFFER_SIZE:
current[JOIN_BUFFER_SIZE]= thd->variables.join_buff_size;
thd->variables.join_buff_size= stoull(ssm[2]);
break;
case TMP_TABLE_SIZE:
current[TMP_TABLE_SIZE]= thd->variables.tmp_table_size;
thd->variables.tmp_table_size= stoull(ssm[2]);
break;
...
Modify variables
22

31. • First we need to specify what we need
MYSQL AUDIT GENERAL RESULTstatic st_mysql_audit custom_hint_plugin_descriptor= {
MYSQL_AUDIT_INTERFACE_VERSION, /* interface version */
NULL,
custom_hint, /* implements custom hints */
{ (unsigned long) MYSQL_AUDIT_GENERAL_RESULT, 0,
(unsigned long) MYSQL_AUDIT_PARSE_PREPARSE,
}
};
Revert variables back
23

32. • First we need to specify what we need
MYSQL AUDIT GENERAL RESULT
•
Then revert variables before sending result
if (event_general->event_subclass == MYSQL_AUDIT_GENERAL_RESULT)
{
map<my_thread_id, map<supported_hints_t, ulonglong> >::iterator
current= modified_variables.find(thd->thread_id());
if (current != modified_variables.end())
{
for (map<supported_hints_t, ulonglong>::iterator it=
current->second.begin(); it!= current->second.end(); ++it)
{
switch(it->first)
{
case JOIN_BUFFER_SIZE:
thd->variables.join_buff_size= it->second;
break;
Revert variables back
23

33. •
First we need to specify what we need
MYSQL AUDIT GENERAL RESULT
• Then revert variables before sending result
•
And, finally, erase stored values for current
thread:
modified_variables.erase(current);
Revert variables back
23

34. mysql> flush status;
Query OK, 0 rows affected (0.00 sec)
mysql> select count(*), sum(c) from
-> (select s, count(s) c from joinit where i < 1000000 group by s) t;
+----------+--------+
| count(*) | sum(c) |
+----------+--------+
| 737882 | 737882 |
+----------+--------+
1 row in set (24.70 sec)
mysql> show status like ’Created_tmp_disk_tables’;
+-------------------------+-------+
| Variable_name | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 2 | -- 2 temporary tables on disk
+-------------------------+-------+
1 row in set (0.00 sec)
Before Custom Hint Plugin
24

35. mysql> flush status;
Query OK, 0 rows affected (0.00 sec)
mysql> select /*+ tmp_table_size=134217728 max_heap_table_size=134217728 */
-> count(*), sum(c) from
-> (select s, count(s) c from joinit where i < 1000000 group by s) t;
+----------+--------+
| count(*) | sum(c) |
+----------+--------+
| 737882 | 737882 |
+----------+--------+
1 row in set, 2 warnings (6.21 sec) -- 4 times speed gain!
mysql> show status like ’Created_tmp_disk_tables’;
+-------------------------+-------+
| Variable_name | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 0 | -- No disk-based temporary table!
+-------------------------+-------+
1 row in set (0.00 sec)
Custom Hint Plugin at work
25

36. Variables

37. •
Very simple syntax
•
mysql> BACKUP SERVER;
+---------------------------------+
| Backup finished with status OK! |
+---------------------------------+
| Backup finished with status OK! |
+---------------------------------+
1 row in set, 1 warning (42.92 sec)
BACKUP DATABASE plugin
27

38. •
Very simple syntax
• Supports many tools
• mysqldump
•
mysqlpump
•
mysqlbackup
• xtrabackup
BACKUP DATABASE plugin
27

39. • Very simple syntax
• Supports many tools
•
Needs to pass options
•
Credentials
• Backup directory
• Custom
BACKUP DATABASE plugin
27

40. •
We have access to
• MYSQL_THR->security_context
thd->security_context()->user().str
Password still has to be in the configuration file, under
[client]
or
[toolname]
section
• System variables
Since we are interested in backing up local server we will use
mysqld_unix_port
Customization: credentials
28

41. •
Global variables - Example only!
static MYSQL_SYSVAR_STR(backup_dir, backup_dir_value, PLUGIN_VAR_MEMALLOC,
"Default directory...", NULL, NULL, NULL);
static MYSQL_SYSVAR_ENUM(backup_tool, backup_tool_name,
PLUGIN_VAR_RQCMDARG, "Backup tool. Possible values:
mysqldump|mysqlbackup", NULL, NULL,
MYSQLDUMP, &supported_tools_typelib);
• Thread variables
• Add to plugin declaration
Customization: variables
29

42. •
Global variables - Example only!
• Thread variables
static MYSQL_THDVAR_STR(backup_dir, PLUGIN_VAR_MEMALLOC,
"Default directory...", NULL, NULL, NULL);
static MYSQL_THDVAR_ENUM(backup_tool, PLUGIN_VAR_RQCMDARG,
"Backup tool. Possible values:
mysqldump|mysqlbackup|mysqlpump", NULL, NULL,
MYSQLDUMP, &supported_tools_typelib);
...
•
Add to plugin declaration
Customization: variables
29

43. • Global variables - Example only!
• Thread variables
•
Add to plugin declaration
static struct st_mysql_sys_var *mysqlbackup_plugin_sys_vars[] = {
MYSQL_SYSVAR(backup_dir),
...
MYSQL_SYSVAR(backup_tool_options),
NULL
};
mysql_declare_plugin(mysqlbackup_plugin) {
MYSQL_AUDIT_PLUGIN,
&mysqlbackup_plugin_descriptor,
"mysqlbackup_plugin",
...
NULL, /* status variables */
mysqlbackup_plugin_sys_vars, /* system variables */
...
Customization: variables
29

44. Summary

45. • Custom locks
•
Access to thread- and server-specific
variables
• Fine control at multiple steps of query
execution
• More
More possibilities
31

46. • https://github.com/svetasmirnova/
• filter plugin
• custom hint plugin
•
mysqlbackup plugin
Code
32

47. • MySQL source dir/plugin
•
rewriter
• rewrite example
• Writing Audit Plugins manual
• MySQL Services for Plugins manual
More information
33

48. When: October 3-5, 2016
Where: Amsterdam, Netherlands
Percona Live Europe Open Source Database Conference is the premier event for the
diverse and active open source community, as well as businesses that develop and use
open source software.
Use promo code ParisMeetup to get 25 euros off. Register now
Sponsorship opportunities available as well here.
Join us at Percona Live Europe
34

49. ???
Place for your questions
35

50. http://www.slideshare.net/SvetaSmirnova
https://twitter.com/svetsmirnova
Thank you!
36