MySQL creative programming

2,096 views

Published on

cross language creative programming with MySQL

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,096
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
132
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

MySQL creative programming

  1. 1. Creative programming with MySQL Giuseppe Maxia, MySQL Community Team Lead Database Group, Sun Microsystems Inc giuseppe.maxia@sun.com The Data Charmer
  2. 2. Creative programming? 2008 CommunityOne Conference | developers.sun.com/events/communityone | 2
  3. 3. Creative programming? WTF? 2008 CommunityOne Conference | developers.sun.com/events/communityone | 2
  4. 4. Creative programming? WTF? bright ideas 2008 CommunityOne Conference | developers.sun.com/events/communityone | 2
  5. 5. Creative programming? hacking! WTF? bright ideas 2008 CommunityOne Conference | developers.sun.com/events/communityone | 2
  6. 6. About me 22 years in IT consultant MySQL community QA developer MySQL Community Team Lead 2008 CommunityOne Conference | developers.sun.com/events/communityone | 3
  7. 7. About me database hacker creative programmer 2008 CommunityOne Conference | developers.sun.com/events/communityone | 4
  8. 8. Does it look familiar? 2008 CommunityOne Conference | developers.sun.com/events/communityone | 5
  9. 9. Does it look familiar? SELECT * FROM table_name; 2008 CommunityOne Conference | developers.sun.com/events/communityone | 5
  10. 10. Does it look familiar? SELECT * FROM table_name; UPDATE table_name SET x=1 WHERE y=10; 2008 CommunityOne Conference | developers.sun.com/events/communityone | 5
  11. 11. Does it look familiar? SELECT * FROM table_name; UPDATE table_name SET x=1 WHERE y=10; DELETE FROM tab_name WHERE z=1; 2008 CommunityOne Conference | developers.sun.com/events/communityone | 5
  12. 12. Does it look familiar? SELECT * FROM table_name; UPDATE table_name SET x=1 WHERE y=10; DELETE FROM tab_name WHERE z=1; make coffee; 2008 CommunityOne Conference | developers.sun.com/events/communityone | 5
  13. 13. Does it look familiar? SELECT * FROM table_name; UPDATE table_name SET x=1 WHERE y=10; DELETE FROM tab_name WHERE z=1; make coffee; play movie; 2008 CommunityOne Conference | developers.sun.com/events/communityone | 5
  14. 14. Does it look familiar? SELECT * FROM table_name; UPDATE table_name SET x=1 WHERE y=10; DELETE FROM tab_name WHERE z=1; make coffee; ?? play movie; 2008 CommunityOne Conference | developers.sun.com/events/communityone | 5
  15. 15. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 6
  16. 16. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 6
  17. 17. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 6
  18. 18. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops Revisiting Triggers MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 7
  19. 19. What average programmers dislike regular expressions 2008 CommunityOne Conference | developers.sun.com/events/communityone | 8
  20. 20. What average programmers dislike regular expressions quot;I (love|hate)s*([Ff]w*)s*REGEXPquot; 2008 CommunityOne Conference | developers.sun.com/events/communityone | 8
  21. 21. What average programmers dislike regular expressions database backend 2008 CommunityOne Conference | developers.sun.com/events/communityone | 9
  22. 22. What average programmers dislike regular expressions database backend SELECT t.id FROM (SELECT id2 FROM other WHERE b = 2) AS t WHERE a > (SELECT MAX(x) FROM z); 2008 CommunityOne Conference | developers.sun.com/events/communityone | 9
  23. 23. Why many programmers hate database handling? SQL != Java SQL != Perl SQL != PHP SQL != C++ I want to write in my language of choice 2008 CommunityOne Conference | developers.sun.com/events/communityone | 10
  24. 24. database handling bring the logic at application level 2008 CommunityOne Conference | developers.sun.com/events/communityone | 11
  25. 25. database handling database C/C++ Java Perl PHP .NET 2008 CommunityOne Conference | developers.sun.com/events/communityone | 12
  26. 26. database handling too many languages Java class database C/C++ Java Perl Perl library PHP PHP library .NET 2008 CommunityOne Conference | developers.sun.com/events/communityone | 13
  27. 27. database handling set the logic at server level (stored routines) 2008 CommunityOne Conference | developers.sun.com/events/communityone | 14
  28. 28. database handling database stored routines library C/C++ Java Perl PHP .NET 2008 CommunityOne Conference | developers.sun.com/events/communityone | 15
  29. 29. database handling set the logic at protocol level (proxy) 2008 CommunityOne Conference | developers.sun.com/events/communityone | 16
  30. 30. database handling database C/C++ Proxy Java library Perl PHP .NET 2008 CommunityOne Conference | developers.sun.com/events/communityone | 17
  31. 31. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops Revisiting Triggers MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 18
  32. 32. Stored routines - the boring stu SQL standard heavy syntax tricky performance no debug - limited exception handling procedures • can return multiple record sets • can execute dynamic code functions • must return one value • no dynamic code 2008 CommunityOne Conference | developers.sun.com/events/communityone | 19
  33. 33. Stored routines - the boring stu SQL statements IF THEN ELSE .. END IF WHILE .. DO .. END WHILE LOOP .. END LOOP CURSOR CONDITION HANDLER 2008 CommunityOne Conference | developers.sun.com/events/communityone | 20
  34. 34. Stored routines - some hacks enhance the language • arrays • easy loops • global variables • see MySQL general purpose routine library create routines on-the-fly • undocumented hack! • see http://datacharmer.org 2008 CommunityOne Conference | developers.sun.com/events/communityone | 21
  35. 35. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops Revisiting Triggers MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 22
  36. 36. Triggers - the boring stu BEFORE or AFTER event BEFORE INSERT/UPDATE/DELETE AFTER INSERT/UPDATE/DELETE No dynamic code No operations on the same table No multiple triggers per event 2008 CommunityOne Conference | developers.sun.com/events/communityone | 23
  37. 37. Triggers - what for dedicated logging data aggregation transfer of data to other tables/databases calculated fields 2008 CommunityOne Conference | developers.sun.com/events/communityone | 24
  38. 38. Triggers - HACKS! wait. Strong combination with FEDERATED and BLACKHOLE 2008 CommunityOne Conference | developers.sun.com/events/communityone | 25
  39. 39. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops Revisiting Triggers MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 26
  40. 40. Views no storage look like tables can aggregate data can insert data can save time but can also become horribly ineficient 2008 CommunityOne Conference | developers.sun.com/events/communityone | 27
  41. 41. Views - the cool stu can run functions can create sort of triggers on select 2008 CommunityOne Conference | developers.sun.com/events/communityone | 28
  42. 42. Views with functions CREATE FUNCTION f(V VARCHAR(20)) RETURNS INT BEGIN INSERT INTO log_table VALUES (V, NOW()); RETURN 1; END 2008 CommunityOne Conference | developers.sun.com/events/communityone | 29
  43. 43. Views with functions CREATE VIEW v1 AS SELECT x FROM table_name WHERE f('v1') = 1; 2008 CommunityOne Conference | developers.sun.com/events/communityone | 30
  44. 44. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops Revisiting Triggers MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 31
  45. 45. Events - the not-so-boring facts Temporal triggers Operating System independent executes SQL command or routine • AT timestamp • EVERY X time_interval can execute dynamic SQL 2008 CommunityOne Conference | developers.sun.com/events/communityone | 32
  46. 46. Events CREATE EVENT e1 ON SCHEDULE AT NOW() + INTERVAL 10 minute DO CALL clean_employee_recs(); 2008 CommunityOne Conference | developers.sun.com/events/communityone | 33
  47. 47. Events CREATE EVENT e1 ON SCHEDULE EVERY 5 minute DO CALL import_recs_from_file(); 2008 CommunityOne Conference | developers.sun.com/events/communityone | 34
  48. 48. Agenda Database programming blues Stored routines Triggers Events Federated Blackhole Creative loops Revisiting Triggers MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 35
  49. 49. Federated storage engine no storage connects to a table on a remote server since version 5.1.22 can connect to a table on the same server table t1f table t1 federated MyISAM 2008 CommunityOne Conference | developers.sun.com/events/communityone | 36
  50. 50. Federated storage engine - caveats no drop-in replacement for regular tables ineficient aggregates limited push-down conditions 2008 CommunityOne Conference | developers.sun.com/events/communityone | 37
  51. 51. Federated storage engine limited pushdown conditions federated table base table SELECT ID FROM table_name SELECT ID WHERE x = 10 FROM table_name expected returns ~ 100 records filter 1 million records 2008 CommunityOne Conference | developers.sun.com/events/communityone | 38
  52. 52. Federated storage engine workaround pushdown conditions add fake index on column x federated to Federated table table base table SELECT ID FROM table_name SELECT ID WHERE x = 10 FROM table_name WHERE x = 10 expected returns ~ 100 records 100 records 2008 CommunityOne Conference | developers.sun.com/events/communityone | 39
  53. 53. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops Revisiting Triggers MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 40
  54. 54. Blackhole The NON-STORAGE engine like /dev/null in Unix used for triggers and replication relay record record binary log record record triggers 2008 CommunityOne Conference | developers.sun.com/events/communityone | 41
  55. 55. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops Revisiting Triggers MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 42
  56. 56. More exciting triggers Put them together • triggers • Federated engine • Blackhole engine 2008 CommunityOne Conference | developers.sun.com/events/communityone | 43
  57. 57. cascade triggers BASE TABLE before insert after insert before update after update before delete after delete t1 2008 CommunityOne Conference | developers.sun.com/events/communityone | 44
  58. 58. cascade triggers Federated BASE TABLE TABLE before insert before insert after insert after insert before update before update after update after update before delete before delete after delete after delete t1 t1_f1 2008 CommunityOne Conference | developers.sun.com/events/communityone | 44
  59. 59. triggers on request Federated before insert TABLE either BASE TABLE after update t1_f1 after delete Federated before insert TABLE after insert or before update t1 before delete t1_f2 2008 CommunityOne Conference | developers.sun.com/events/communityone | 45
  60. 60. blackhole triggers BASE TABLE blackhole TABLE before insert t1 BASE TABLE t1_b1 t2 2008 CommunityOne Conference | developers.sun.com/events/communityone | 46
  61. 61. remote execution blackhole Federated before insert table TABLE t1 t1_f1 2008 CommunityOne Conference | developers.sun.com/events/communityone | 47
  62. 62. remote execution VIEW Federated function TABLE v1 t1_f1 2008 CommunityOne Conference | developers.sun.com/events/communityone | 48
  63. 63. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops Revisiting Triggers MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 49
  64. 64. loops or why do we have to endure this? loops with CURSORS • dreadful syntax • INEFFICIENT (temporary table created) loops alternatives ... coming soon 2008 CommunityOne Conference | developers.sun.com/events/communityone | 50
  65. 65. loops with cursors :( data to crc flag process cursor temporary loop table handler crc 2008 CommunityOne Conference | developers.sun.com/events/communityone | 51
  66. 66. loops with cursors (I) :( CREATE PROCEDURE p(j INT) BEGIN DECLARE done BOOLEAN default 'false'; DECLARE crc VARCHAR(42) default ''; DECLARE cid INT; DECLARE x CURSOR FOR SELECT id FROM table_name WHERE b j; -- to be continued 2008 CommunityOne Conference | developers.sun.com/events/communityone | 52
  67. 67. loops with cursors (II) :(( DECLARE CONTINUE HANDLER FOR NOT FOUND set done = true; OPEN CURSOR x; search: LOOP FETCH x into cid; IF done THEN LEAVE search; END IF; -- do something with cid SET crc = MD5(CONCAT(crc,cid)) END LOOP; SELECT crc; 2008 CommunityOne Conference | developers.sun.com/events/communityone | 53
  68. 68. loops with blackhole :) data to crc process blackhole table crc 2008 CommunityOne Conference | developers.sun.com/events/communityone | 54
  69. 69. loops with blackhole (I) :) CREATE TABLE t1 ( id VARCHAR(200) ) ENGINE = BLACKHOLE; SET @crc = ''; SET @j = 10; 2008 CommunityOne Conference | developers.sun.com/events/communityone | 55
  70. 70. loops with blackhole (II) :) INSERT INTO t1 SELECT @crc := MD5(CONCAT(@crc, id)) FROM table_name WHERE b @j; SELECT @crc; IT'S FASTER!!! 2008 CommunityOne Conference | developers.sun.com/events/communityone | 56
  71. 71. more loops alternatives use the General Purpose Routine Library • https://sourceforge.net/projects/mysql-sr-lib/ use events! (ON SCHEDULE EVERY 1 SECOND ENDS NOW() + INTERVAL 10 SECOND) use MySQL Proxy (later) 2008 CommunityOne Conference | developers.sun.com/events/communityone | 57
  72. 72. loops with MySQL stored routines library call for_each_table_value_complete( 'db_name', 'table_name', 'id', null, null, '', 'set @crc=MD5(concat(@crc,$I1))', 'set @crc=quot;quot;', 'select @crc','', 'once'); 2008 CommunityOne Conference | developers.sun.com/events/communityone | 58
  73. 73. Agenda Database programming blues Stored routines Triggers Views Events Federated Blackhole Creative loops Revisiting Triggers MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 59
  74. 74. MySQL Proxy So, what about quot;make coeequot; ? 2008 CommunityOne Conference | developers.sun.com/events/communityone | 60
  75. 75. 2008 CommunityOne Conference | developers.sun.com/events/communityone | 61
  76. 76. 2008 CommunityOne Conference | developers.sun.com/events/communityone | 62
  77. 77. 2008 CommunityOne Conference | developers.sun.com/events/communityone | 63
  78. 78. MySQL Proxy 2008 CommunityOne Conference | developers.sun.com/events/communityone | 64
  79. 79. MySQL Proxy principles - hooks 2008 CommunityOne Conference | developers.sun.com/events/communityone | 65
  80. 80. MySQL Proxy principles - hooks PROXY CORE connection hook read query hook read result hook 2008 CommunityOne Conference | developers.sun.com/events/communityone | 65
  81. 81. MySQL Proxy principles - hooks PROXY CORE Lua script function connection function function hook read query function hook read result function hook 2008 CommunityOne Conference | developers.sun.com/events/communityone | 65
  82. 82. MySQL Proxy principles - hooks PROXY CORE Lua script function connection function function hook read query function hook read result function hook 2008 CommunityOne Conference | developers.sun.com/events/communityone | 65
  83. 83. ?? MySQL Proxy principles - Lua { Perl ? PHP? Why not ... Javascript? [whatever]? 2008 CommunityOne Conference | developers.sun.com/events/communityone | 66
  84. 84. MySQL Proxy principles - Lua 2008 CommunityOne Conference | developers.sun.com/events/communityone | 67
  85. 85. MySQL Proxy principles - Lua • SMALL ( 200 KB) • DESIGNED for EMBEDDED systems • Widely used (lighttpd) 2008 CommunityOne Conference | developers.sun.com/events/communityone | 67
  86. 86. MySQL Proxy principles - Lua • SMALL ( 200 KB) • DESIGNED for EMBEDDED systems • Widely used (lighttpd) lighttpd, like MySQL Proxy, was created by Jan Kneschke 2008 CommunityOne Conference | developers.sun.com/events/communityone | 67
  87. 87. MySQL Proxy principles - Lua Very popular among game writers 2008 CommunityOne Conference | developers.sun.com/events/communityone | 68
  88. 88. MySQL Proxy principles - Lua Very popular among game writers 2008 CommunityOne Conference | developers.sun.com/events/communityone | 68
  89. 89. MySQL Proxy principles - Lua Very popular among game writers 2008 CommunityOne Conference | developers.sun.com/events/communityone | 68
  90. 90. injecting queries 2008 CommunityOne Conference | developers.sun.com/events/communityone | 69
  91. 91. injecting queries 2008 CommunityOne Conference | developers.sun.com/events/communityone | 70
  92. 92. Debugging server proxy client diagnostics text 2008 CommunityOne Conference | developers.sun.com/events/communityone | 71
  93. 93. Debugging scripts server proxy proxy diagnostics text client diagnostics text 2008 CommunityOne Conference | developers.sun.com/events/communityone | 72
  94. 94. Chained proxies - double features server proxy proxy pivot tables client loops 2008 CommunityOne Conference | developers.sun.com/events/communityone | 73
  95. 95. Testing - customized packages server proxy client e.g. fake packets connectors 2008 CommunityOne Conference | developers.sun.com/events/communityone | 74
  96. 96. MySQL Proxy - loops revisited # get the script from MySQL Forge # http://forge.mysql.com/tools/tool.php?id=96 mysql-proxy --proxy-lua-script=loop.lua 2008 CommunityOne Conference | developers.sun.com/events/communityone | 75
  97. 97. MySQL Proxy - loops revisited mysql -h 127.0.0.1 -P 4040 FOR VAR 1 3 CREATE TABLE t_$VAR (id int); # creates t_1, t_2, t_3 2008 CommunityOne Conference | developers.sun.com/events/communityone | 76
  98. 98. MySQL Proxy - loops revisited mysql -h 127.0.0.1 -P 4040 FOR VAR ( aa, bb cc) CREATE TABLE t_$VAR (id int); # creates t_aa, t_bb, t_cc 2008 CommunityOne Conference | developers.sun.com/events/communityone | 77
  99. 99. Summary MySQL is open to development creativity Know your bricks • stored routines, triggers, views, events • dynamic engines (Federated, Blackhole) • MySQL Proxy Put them together : be creative 2008 CommunityOne Conference | developers.sun.com/events/communityone | 78
  100. 100. For More Information MySQL Forge ( http://forge.mysql.com ) my blog ( http://datacharmer.blogspot.com ) MySQL blogs ( http://planetmysql.org ) 2008 CommunityOne Conference | developers.sun.com/events/communityone | 79
  101. 101. QA Let's talk! 80

×