0
Programmatic QueriesThings you can code with SQL          Shlomi Noach      openark.org                         Percona Li...
SQL     SQL, or the Structured Query Language, is often      referred to as a declarative language.         ●     From Wi...
Is SQL really declarative?      Even in standard SQL, there are hints of       algorithmic behavior.         ●      Does ...
Query behavior in MySQL      There are certain aspects to query behavior in       MySQL, that imply programmatic nature....
User Defined Variables      Perhaps the most obvious programmatic       feature in MySQL; stored routines aside.      On...
User Defined Variables+­­­­­­­­­+­­­­­­­­­­­­­­­­+| counter | Name           |+­­­­­­­­­+­­­­­­­­­­­­­­­­+|       1 | Kabu...
User Defined Variables      We rely on a programmatic nature of MySQL:         ●      MySQL evaluates the result set row ...
Derived tables      The above query requires us to remember to       SET the @counter variable to zero each time.        ...
Derived tables    A derived table is a query passed in place of a table.    Derived table is materialized before the out...
Order of evaluation      The following query produces the Fibonacci       sequence.      It happens to work.    SELECT  ...
Order of evaluation+­­­­­­­+­­­­­­­­­­­­+­­­­­­­­­­­­+| value | @c1 := @c2 | @c2 := @c3 |+­­­­­­­+­­­­­­­­­­­­+­­­­­­­­­­­...
Order of evaluation      However, it relies on the first column to be       evaluated first, second column to come next  ...
Order of evaluation:                          CASE … WHEN … ELSE      A CASE … WHEN … ELSE statement:                    ...
Order of evaluation:                          CASE … WHEN … ELSE     CASE evaluation algorithm:        ●      Try (evalua...
Order of evaluation      The following query works correctly, and the       order of evaluation is predicted:    SELECT  ...
UNION ALL      A UNION ALL query concatenates results from       multiple query parts.      The declaration of UNION ALL...
UNION ALL execution order      As expected, when no derived tables are       involved, statements are executed by order o...
UNION ALL:                        Time delayed statements      The following reads com_select, sleeps for 10       second...
UNION ALL:                        Time delayed statements      We may not know in advance the order of rows       in the ...
UNION ALL:                        Time delayed statements      The following computes com_select per       second:    SEL...
Monitoring queries     Last example provides SQL with monitoring      capabilities         ●     A META query over META d...
Monitoring MySQL   SELECT     gs0.VARIABLE_NAME,     (gs1.VARIABLE_VALUE ­ gs0.VARIABLE_VALUE),     (gs1.VARIABLE_VALUE ­ ...
Monitoring MySQL   +­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­+­­­­­­+   | VARIABLE_NAME         | diff | rate |   +­­­­­­­­­­­­­­­­­­...
Pop quiz      What would be the result of the following       query?    SELECT NOW()     UNION ALL    SELECT SLEEP(10)   ...
Time evaluation     NOW() indicates the moment the query started (or      routine start of execution)     SYSDATE() indi...
Time evaluation      Assume the following query:    SELECT * FROM t1 JOIN t2 USING (c) JOIN t3 USING (d)      We can get...
Imagine      Suppose LIMIT could accept a non-constant       value; an expression to be re-evaluated.      Imagine the f...
Homework     (Hey, be thankful theres no surprise exam!)     Assume a very long running SELECT query (e.g. a      report...
Thank you!      I blog at http://openark.org      Find open source projects on       http://code.openark.org/forge/    ...
Upcoming SlideShare
Loading in...5
×

Programmatic queries: things you can code with sql

1,157

Published on

My Percona Live London 2011 session about SQL hacks, allowing one to invoke programmatic code from within a query

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,157
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Programmatic queries: things you can code with sql"

  1. 1. Programmatic QueriesThings you can code with SQL Shlomi Noach openark.org Percona Live, London 2011
  2. 2. SQL SQL, or the Structured Query Language, is often referred to as a declarative language. ● From Wikipedia: declarative programming is a  programming paradigm that expresses  the logic of a computation without  describing its control flow. SQL and the relational model are in part based on relational algebra, which allows for mathematical expressions and conclusions on the model.openark.org Programmatic Queries Things you can code with SQL 2Copyright © Shlomi Noach, 2011. All rights reserved
  3. 3. Is SQL really declarative? Even in standard SQL, there are hints of algorithmic behavior. ● Does ORDER BY imply an algorithm? ● Does MySQLs ORDER BY... LIMIT imply an algorithm? ● How about Window Functions & running totals? SELECT SUM(sale_amount) OVER (ORDER BY sale_date)  FROM salesopenark.org Programmatic Queries Things you can code with SQL 3Copyright © Shlomi Noach, 2011. All rights reserved
  4. 4. Query behavior in MySQL There are certain aspects to query behavior in MySQL, that imply programmatic nature. We discuss a few: ● Row evaluation order ● Control flow evaluation order ● Table and query materialization order ● Time & time suspensionopenark.org Programmatic Queries Things you can code with SQL 4Copyright © Shlomi Noach, 2011. All rights reserved
  5. 5. User Defined Variables Perhaps the most obvious programmatic feature in MySQL; stored routines aside. One is allowed to define, assign, calculate & reassign variable values throughout a querys execution. Variables can be used to generate counters, running totals, ranking, sequences, and more. SET @counter := 0; SELECT (@counter := @counter + 1) AS counter, Name    FROM world.City;openark.org Programmatic Queries Things you can code with SQL 5Copyright © Shlomi Noach, 2011. All rights reserved
  6. 6. User Defined Variables+­­­­­­­­­+­­­­­­­­­­­­­­­­+| counter | Name           |+­­­­­­­­­+­­­­­­­­­­­­­­­­+|       1 | Kabul          ||       2 | Qandahar       ||       3 | Herat          ||       4 | Mazar­e­Sharif ||       5 | Amsterdam      ||       6 | Rotterdam      ||       7 | Haag           ||       8 | Utrecht        ||       9 | Eindhoven      ||      10 | Tilburg        |+­­­­­­­­­+­­­­­­­­­­­­­­­­+openark.org Programmatic Queries Things you can code with SQL 6Copyright © Shlomi Noach, 2011. All rights reserved
  7. 7. User Defined Variables We rely on a programmatic nature of MySQL: ● MySQL evaluates the result set row by row ● The order of rows is sometimes easily predictable; other times unexpected The following query will assign lower counter values to larger cities: SET @counter := 0; SELECT (@counter := @counter + 1) AS counter, Name    FROM world.City ORDER BY Population DESC;openark.org Programmatic Queries Things you can code with SQL 7Copyright © Shlomi Noach, 2011. All rights reserved
  8. 8. Derived tables The above query requires us to remember to SET the @counter variable to zero each time. ● We look for a solution that will do it all in one single query. The common solution is to use derived tables: SELECT (@counter := @counter + 1) AS counter, Name    FROM world.City, (SELECT @counter := 0) s_init;openark.org Programmatic Queries Things you can code with SQL 8Copyright © Shlomi Noach, 2011. All rights reserved
  9. 9. Derived tables A derived table is a query passed in place of a table. Derived table is materialized before the outer query is evaluated. ● See sql_base.cc This makes for a programmatic nature of the query, where we can expect certain parts to execute before others. ● In the above example, we used that knowledge to set a session variable. ● Other use cases include: instantiating a subquery; taking advantage of indexes from other tables, etc.openark.org Programmatic Queries Things you can code with SQL 9Copyright © Shlomi Noach, 2011. All rights reserved
  10. 10. Order of evaluation The following query produces the Fibonacci sequence. It happens to work. SELECT    @c3 := @c1 + @c2 AS value,    @c1 := @c2,    @c2 := @c3 FROM    mysql.help_topic,    (SELECT @c1 := 1, @c2 := 0) s_init LIMIT 10;openark.org Programmatic Queries Things you can code with SQL 10Copyright © Shlomi Noach, 2011. All rights reserved
  11. 11. Order of evaluation+­­­­­­­+­­­­­­­­­­­­+­­­­­­­­­­­­+| value | @c1 := @c2 | @c2 := @c3 |+­­­­­­­+­­­­­­­­­­­­+­­­­­­­­­­­­+|     1 |          0 | 1          ||     1 |          1 | 1          ||     2 |          1 | 2          ||     3 |          2 | 3          ||     5 |          3 | 5          ||     8 |          5 | 8          ||    13 |          8 | 13         ||    21 |         13 | 21         ||    34 |         21 | 34         ||    55 |         34 | 55         |+­­­­­­­+­­­­­­­­­­­­+­­­­­­­­­­­­+openark.org Programmatic Queries Things you can code with SQL 11Copyright © Shlomi Noach, 2011. All rights reserved
  12. 12. Order of evaluation However, it relies on the first column to be evaluated first, second column to come next etc. This is not guaranteed, and fails to work on more complex queries. We look for a construct that can guarantee order of evaluation.openark.org Programmatic Queries Things you can code with SQL 12Copyright © Shlomi Noach, 2011. All rights reserved
  13. 13. Order of evaluation: CASE … WHEN … ELSE A CASE … WHEN … ELSE statement: … returns the result for the first  condition that is true. If there was  no matching … the result after ELSE is  returned … [The MySQL Manual] This is a declarative explanation. However, the easiest way to implement this also happens to be the most efficient way.openark.org Programmatic Queries Things you can code with SQL 13Copyright © Shlomi Noach, 2011. All rights reserved
  14. 14. Order of evaluation: CASE … WHEN … ELSE CASE evaluation algorithm: ● Try (evaluate) the first WHEN statement. ● Is it True? Quit. ● Otherwise try the second WHEN. ● Is it True? Quit. ● Otherwise … ● No luck? Evaluate the ELSE statement.  See item_cmpfunc.cc We can utilize this known programmatic nature of evaluation for writing our code.openark.org Programmatic Queries Things you can code with SQL 14Copyright © Shlomi Noach, 2011. All rights reserved
  15. 15. Order of evaluation The following query works correctly, and the order of evaluation is predicted: SELECT    CASE     WHEN (@c3 := @c1 + @c2) IS NULL THEN NULL     WHEN (@c1 := @c2) IS NULL THEN NULL     WHEN (@c2 := @c3) IS NULL THEN NULL     ELSE @c3   END AS seq FROM    mysql.help_topic,    (SELECT @c1 := 1, @c2 := 0) s_init LIMIT 10;openark.org Programmatic Queries Things you can code with SQL 15Copyright © Shlomi Noach, 2011. All rights reserved
  16. 16. UNION ALL A UNION ALL query concatenates results from multiple query parts. The declaration of UNION ALL says nothing about the order of rows in the result set. The MySQL manual explicitly suggests that order of rows is unknown. However the above says nothing about the time at which the statements are executed.openark.org Programmatic Queries Things you can code with SQL 16Copyright © Shlomi Noach, 2011. All rights reserved
  17. 17. UNION ALL execution order As expected, when no derived tables are involved, statements are executed by order of appearance in the UNION ALL query. ● See sql_union.cc SELECT    This will be executed first AS description UNION ALL   SELECT This will be executed second UNION ALL   SELECT This will be executed last ;openark.org Programmatic Queries Things you can code with SQL 17Copyright © Shlomi Noach, 2011. All rights reserved
  18. 18. UNION ALL: Time delayed statements The following reads com_select, sleeps for 10 seconds, then reads com_select again. SELECT VARIABLE_VALUE AS value   FROM INFORMATION_SCHEMA.GLOBAL_STATUS   WHERE VARIABLE_NAME = com_select UNION ALL   SELECT SLEEP(10) FROM DUAL UNION ALL   SELECT VARIABLE_VALUE   FROM INFORMATION_SCHEMA.GLOBAL_STATUS   WHERE VARIABLE_NAME = com_select ;openark.org Programmatic Queries Things you can code with SQL 18Copyright © Shlomi Noach, 2011. All rights reserved
  19. 19. UNION ALL: Time delayed statements We may not know in advance the order of rows in the result set. We can force the order by providing an extra sorting column; but we dont always need to. +­­­­­­­+ | value | +­­­­­­­+ | 10397 | | 0     | | 10562 | +­­­­­­­+openark.org Programmatic Queries Things you can code with SQL 19Copyright © Shlomi Noach, 2011. All rights reserved
  20. 20. UNION ALL: Time delayed statements The following computes com_select per second: SELECT SUM(value) AS diff, SUM(value)/10 AS rate FROM (   SELECT 0 ­ VARIABLE_VALUE AS value     FROM INFORMATION_SCHEMA.GLOBAL_STATUS     WHERE VARIABLE_NAME = com_select   UNION ALL     SELECT SLEEP(10) FROM DUAL   UNION ALL    SELECT 0 + VARIABLE_VALUE     FROM INFORMATION_SCHEMA.GLOBAL_STATUS     WHERE VARIABLE_NAME = com_select ) s_inner;openark.org Programmatic Queries Things you can code with SQL 20Copyright © Shlomi Noach, 2011. All rights reserved
  21. 21. Monitoring queries Last example provides SQL with monitoring capabilities ● A META query over META data Why stop with a single variable? Using our knowledge of evaluation order, we can combine derived tables, time delay & UNION ALL execution order into a general purpose monitoring query. common_schema provides with a view which queries for any changes for status variables.openark.org Programmatic Queries Things you can code with SQL 21Copyright © Shlomi Noach, 2011. All rights reserved
  22. 22. Monitoring MySQL SELECT   gs0.VARIABLE_NAME,   (gs1.VARIABLE_VALUE ­ gs0.VARIABLE_VALUE),   (gs1.VARIABLE_VALUE ­ gs0.VARIABLE_VALUE) / 10 FROM (   SELECT     VARIABLE_NAME, VARIABLE_VALUE     FROM INFORMATION_SCHEMA.GLOBAL_STATUS   UNION ALL     SELECT , SLEEP(10) FROM DUAL   ) AS gs0   JOIN INFORMATION_SCHEMA.GLOBAL_STATUS gs1      USING (VARIABLE_NAME) ;openark.org Programmatic Queries Things you can code with SQL 22Copyright © Shlomi Noach, 2011. All rights reserved
  23. 23. Monitoring MySQL +­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­+­­­­­­+ | VARIABLE_NAME         | diff | rate | +­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­+­­­­­­+ | BYTES_RECEIVED        |   56 |  5.6 | | COM_CALL_PROCEDURE    |  325 | 32.5 | | COM_SELECT            |  162 | 16.2 | | COM_SET_OPTION        |  648 | 64.8 | | HANDLER_READ_RND_NEXT |  587 | 58.7 | | HANDLER_WRITE         |  878 | 87.8 | | QUESTIONS             |    1 |  0.1 | | SELECT_FULL_JOIN      |    1 |  0.1 | | SELECT_SCAN           |    2 |  0.2 | +­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­+­­­­­­+openark.org Programmatic Queries Things you can code with SQL 23Copyright © Shlomi Noach, 2011. All rights reserved
  24. 24. Pop quiz What would be the result of the following query? SELECT NOW()  UNION ALL SELECT SLEEP(10)  UNION ALL SELECT NOW() ;openark.org Programmatic Queries Things you can code with SQL 24Copyright © Shlomi Noach, 2011. All rights reserved
  25. 25. Time evaluation NOW() indicates the moment the query started (or routine start of execution) SYSDATE() indicates the time the operating system reports, moment of function execution. This makes for a programmatic nature. Given a query, how much time does it take to evaluate? ● On your Python/PHP/Perl/... code its easy to read start/end times, and do the math. ● Is it possible to get the time by a query?openark.org Programmatic Queries Things you can code with SQL 25Copyright © Shlomi Noach, 2011. All rights reserved
  26. 26. Time evaluation Assume the following query: SELECT * FROM t1 JOIN t2 USING (c) JOIN t3 USING (d) We can get just the time it took to execute, or we can pass along the time within the query. For example: SELECT TIMESTAMPDIFF(   MICROSECOND, NOW(), MAX(SYSDATE())) / 1000000.0  FROM (   SELECT * FROM t1 JOIN t2 USING (c) JOIN t3 USING (d) ) s_orig;openark.org Programmatic Queries Things you can code with SQL 26Copyright © Shlomi Noach, 2011. All rights reserved
  27. 27. Imagine Suppose LIMIT could accept a non-constant value; an expression to be re-evaluated. Imagine the following: SELECT lots_of_data FROM   many, many_tables WHERE   some_conditions_are_met LIMIT   IF(     TIMESTAMPDIFF(SECOND, NOW(), SYSDATE()) < 60,     99999999999,     0   );openark.org Programmatic Queries Things you can code with SQL 27Copyright © Shlomi Noach, 2011. All rights reserved
  28. 28. Homework (Hey, be thankful theres no surprise exam!) Assume a very long running SELECT query (e.g. a reporting query). ● For simplicity, assume it scans a single table. The query makes for a significant I/O impact. Task: make the query self throttling! ● Step 1: make the query SLEEP() for 5 seconds for every 1,000 rows scanned ● Step 2: make the query SLEEP() for the duration of last 1,000 rows scanned (effectively doubling the total runtime of the query)openark.org Programmatic Queries Things you can code with SQL 28Copyright © Shlomi Noach, 2011. All rights reserved
  29. 29. Thank you! I blog at http://openark.org Find open source projects on http://code.openark.org/forge/ Contact me at shlomi@[you-know-where].org Questions?openark.org Programmatic Queries Things you can code with SQL 29Copyright © Shlomi Noach, 2011. All rights reserved
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×