Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Query Rewrite Plugin Interface: Writing Your Own Plugin

1,479 views

Published on

This presentation covers MySQL's query rewrite API, and gives practical hands-on tips on how to write your own query rewrite plugin.

Published in: Software
  • Be the first to comment

The Query Rewrite Plugin Interface: Writing Your Own Plugin

  1. 1. The Query Rewrite Plugin Interface: Writing Your Own Plugin Martin Hansson Optimizer Developer martin.hansson@oracle.com Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
  2. 2. Program Agenda  What query rewrites are  The API's  Declaring plugins  Writing plugins  (Plugin services)  Bonus: Writing a Post-Parse Plugin
  3. 3. ServerClient Rewriter What are Query Rewrites? The Philosophy:
  4. 4. What are Query Rewrites? They come in two flavors • Pre-Parse: string-to-string – Low overhead – No structure • Post-Parse: parse tree – Retains structure – Require re-parse (no destructive editing) – Need to traverse parse tree – Only select statements
  5. 5. Program Agenda  What query rewrites are  The API's  Declaring plugins  Writing plugins  (Plugin services)  Bonus: Writing a Post-Parse Plugin
  6. 6. Query Rewrite Plugin API - Overview Network Parser Optimizer Server text LEX QR Plugin(s) Pre-parse QR API Post-parse QR API
  7. 7. mysql> INSTALL PLUGIN rewrite_example SONAME 'rewrite_example.so'; Query OK, 0 rows affected (0,01 sec) mysql> SELECT 'ABC'; +­­­­­+ | abc | +­­­­­+ | abc | +­­­­­+ 1 row in set, 1 warning (0,00 sec) mysql> SHOW WARNINGSG *************************** 1. row ***************************   Level: Note    Code: 1105 Message: Query 'SELECT 'ABC'' rewritten to 'select 'abc'' by a query rewrite plugin *************************** 2. row ***************************   Level: Note    Code: 1105 Message: Query 'SHOW WARNINGS' rewritten to 'show warnings' by a query rewrite  plugin *************************** 3. row *************************** Query Rewrite in Practice
  8. 8. API's Audit API General Query Rewrite API Server Pre-Parse QR API Post-Parse QR API QR Plugin(s)
  9. 9. “Database auditing involves observing a database so as to be aware of the actions of database users. Database administrators and consultants often set up auditing for security purposes, for example, to ensure that those without the permission to access information do not access it.” Source: http://en.wikipedia.org/wiki/Database_audit Audit API
  10. 10. Audit API provides infrastructure for ● Locking ● Caching ● Gathering ● Event model ● Parameter passing
  11. 11. Program Agenda  What query rewrites are  The API's  Declaring plugins  Writing plugins  (Plugin services)  Bonus: Writing a Post-Parse Plugin
  12. 12. static struct st_mysql_audit my_rewrite_descriptor= {   MYSQL_AUDIT_INTERFACE_VERSION,             /* interface version */   NULL,                                      /* release_thd()     */   do_the_rewrite,                            /* event_notify()    */   /* class masks        */   { 0, // General class     0, // Connection class     (unsigned long) MYSQL_AUDIT_PARSE_ALL, // Parser class   } }; MYSQL_AUDIT_PARSE_ALL == MYSQL_AUDIT_PREPARSE | MYSQL_AUDIT_POSTPARSE https://dev.mysql.com/doc/refman/5.7/en/writing-audit-plugins.html Specific Plugin Descriptor
  13. 13. mysql_declare_plugin(audit_log) {   MYSQL_AUDIT_PLUGIN,         /* type                            */   &my_rewrite_descriptor,     /* descriptor                      */   "my_plugin",                /* name                            */   "Me",                       /* author                          */   "My audit plugin",          /* description                     */   PLUGIN_LICENSE_GPL,   my_init,                    /* init function (when loaded)     */   my_deinit,                  /* deinit function (when unloaded) */   0x0003,                     /* version                         */   my_status_vars,             /* status variables                */   my_system_variables,        /* system variables                */   NULL,                       /* reserved                        */   0,                          /* flags                           */ } mysql_declare_plugin_end; https://dev.mysql.com/doc/refman/5.7/en/writing-audit-plugins.html Declaring an Audit Plugin
  14. 14. Program Agenda  What query rewrites are  The API's  Declaring plugins  Writing plugins  (Plugin services)  Bonus: Writing a Post-Parse Plugin
  15. 15. #include <my_global.h>           // Always include this first #include <mysql/plugin.h>        // General plugin header #include <mysql/plugin_audit.h>  // The Query Rewrite API static int plugin_init(MYSQL_PLUGIN) {…} // return 0 for success static int do_the_rewrite(MYSQL_THD,                           mysql_event_class_t event_class,                           const void *event) { …   // return 0 for success } https://dev.mysql.com/doc/refman/5.7/en/writing-audit-plugins.html Skeleton of a Query Rewrite Plugin
  16. 16. static int do_the_rewrite(MYSQL_THD,                           mysql_event_class_t event_class,                           const void *event) {     const mysql_event_parse *event_parse=       static_cast<const mysql_event_parse *>(event);     if (event_parse­>event_subclass == MYSQL_AUDIT_PARSE_PREPARSE)     {        … do pre­parse rewrite ...     } … or …      if (event_parse­>event_subclass == MYSQL_AUDIT_PARSE_POSTPARSE)     {        … do post­parse rewrite ...     }   return 0; /* success */ } Skeleton of a Query Rewrite Plugin (pt. 2)
  17. 17. Skeleton of a Query Rewrite Plugin (Pre-Parse) Doing the Pre-Parse Rewrite static int do_the_rewrite(MYSQL_THD,                           mysql_event_class_t event_class,                           const void *event) {     const mysql_event_parse *event_parse=       static_cast<const mysql_event_parse *>(event);     if (event_parse­>event_subclass == MYSQL_AUDIT_PARSE_PREPARSE)     {       size_t query_length= event_parse­>query.length;       char *rewritten_query=         static_cast<char *>(my_malloc(key_memory_rewrite_example,                                        query_length + 1, MYF(0)));       for (size_t i= 0; i < query_length + 1; ++i)         rewritten_query[i]= tolower(event_parse­>query.str[i]);       event_parse­>rewritten_query­>str= rewritten_query;       event_parse­>rewritten_query­>length= query_length;       *reinterpret_cast<int *>(event_parse­>flags)|=         MYSQL_AUDIT_PARSE_REWRITE_PLUGIN_QUERY_REWRITTEN;     }
  18. 18. Program Agenda  What query rewrites are  The API's  Declaring plugins  Writing plugins  Plugin services  Bonus: Writing a Post-Parse Plugin
  19. 19. What is A Plugin Service? Server Audit API Services Plugins Calls Calls
  20. 20. The Parser Service This Service Lets a Plugin: • Parse a string, get: ● Normalized query ● Query digest • Traverse a parse tree: ● Find positions of literals • Print literals
  21. 21. The Parser Service In code (include/mysql/service_parser.h): kuk int mysql_parser_parse(MYSQL_THD thd, const MYSQL_LEX_STRING query,                        unsigned char is_prepared,                        sql_condition_handler_function handle_cond,                        void *condition_handler_state) MYSQL_LEX_STRING mysql_parser_get_normalized_query(MYSQL_THD thd) int mysql_parser_get_statement_digest(MYSQL_THD thd, uchar *digest) typedef int (*parse_node_visit_function)(MYSQL_ITEM item, unsigned char* arg); int mysql_parser_visit_tree(MYSQL_THD thd,                             parse_node_visit_function processor,                             unsigned char* arg) MYSQL_LEX_STRING mysql_parser_item_string(MYSQL_ITEM item)
  22. 22. The Alloc Service Lets a Plugin: • Allocate • Deallocate • Instrument Code (include/mysql/service_mysql_alloc.h) • my_malloc • my_realloc • my_claim • my_free • my_memdup • my_strdup • my_strndup
  23. 23. Program Agenda  What query rewrites are  The API's  Declaring plugins  Writing plugins  (Plugin services)  Bonus: Writing a Post-Parse Plugin
  24. 24. Skeleton of a Post-Parse Query Rewrite Plugin Doing the Post-Parse Rewrite ...     if (event_parse­>event_subclass == MYSQL_AUDIT_PARSE_POSTPARSE)     {       MYSQL_LEX_STRING first_literal= {NULL, 0};       mysql_parser_visit_tree(thd, catch_literal,                               (unsigned char*)&first_literal);       if (first_literal.str != NULL)       {         size_t query_length=           first_literal.length + event_parse­>query.length + 25;         first_literal.str[first_literal.length]='0';         char *rewritten_query=           static_cast<char *>(my_malloc(key_memory_post_parse_example,                                         query_length,                                         MYF(0)));         sprintf(rewritten_query, "/* First literal: %s */ %s",                 first_literal.str,                 event_parse­>query.str);         MYSQL_LEX_STRING new_query= {rewritten_query, query_length};         mysql_parser_free_string(first_literal);         mysql_parser_parse(thd, new_query, false, handle_parse_error, NULL);         *((int *)event_parse­>flags)|=           (int)MYSQL_AUDIT_PARSE_REWRITE_PLUGIN_QUERY_REWRITTEN;     }
  25. 25. Skeleton of a Post-Parse Query Rewrite Plugin Catching a Literal int catch_literal(MYSQL_ITEM item, unsigned char* arg) {   MYSQL_LEX_STRING *result_string_ptr= (MYSQL_LEX_STRING*)arg;   if (result_string_ptr­>str == NULL)   {     *result_string_ptr= mysql_parser_item_string(item);     return 0;   }   return 1; }
  26. 26. Skeleton of a Post-Parse Query Rewrite Plugin Result mysql> INSTALL PLUGIN post_parse_example SONAME 'post_parse_example.so'; Query OK, 0 rows affected (0,01 sec) mysql> SELECT 'abc', 'def'; +­­­­­+­­­­­+ | abc | def | +­­­­­+­­­­­+ | abc | def | +­­­­­+­­­­­+ 1 row in set, 1 warning (0,00 sec) mysql> SHOW WARNINGSG *************************** 1. row ***************************   Level: Note    Code: 1105 Message: Query 'SELECT 'abc', 'def'' rewritten to '/* First literal: 'abc' */  SELECT 'abc', 'def'' by a query rewrite plugin 1 row in set (0,00 sec)
  27. 27. Links Blog posts: • mysqlserverteam.com/write-yourself-a-query-rewrite-plugin-part-1/ • mysqlserverteam.com/the-query-rewrite-plugins/ In the Manual: • dev.mysql.com/doc/refman/5.7/en/plugin-api.html • dev.mysql.com/doc/refman/5.7/en/plugin-types.html • dev.mysql.com/doc/refman/5.7/en/plugin-services.html • dev.mysql.com/doc/refman/5.7/en/writing-audit-plugins.html • dev.mysql.com/doc/refman/5.7/en/performance-schema-statement- digests.html In the Code: • include/mysql/service_parser.h • include/mysql/services.h
  28. 28. The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

×