This document is a presentation about creative programming with MySQL. It discusses various MySQL features like stored routines, triggers, views, events, federated and blackhole storage engines, and loops. It presents some hacks and creative uses of these features, such as using triggers with federated and blackhole tables to enable remote execution and cascading triggers. The goal is to show programmers ways to overcome limitations and bring more logic into the database layer rather than just the application layer.

1. Creative programming with MySQL
Giuseppe Maxia,
MySQL Community Team Lead
Database Group,
Sun Microsystems Inc
giuseppe.maxia@sun.com The Data Charmer

2. Creative programming?
2008 CommunityOne Conference | developers.sun.com/events/communityone | 2

3. Creative programming?
WTF?
2008 CommunityOne Conference | developers.sun.com/events/communityone | 2

4. Creative programming?
WTF?
bright ideas
2008 CommunityOne Conference | developers.sun.com/events/communityone | 2

5. Creative programming?
hacking!
WTF?
bright ideas
2008 CommunityOne Conference | developers.sun.com/events/communityone | 2

6. About me
22 years in IT
consultant
MySQL community
QA developer
MySQL Community Team Lead
2008 CommunityOne Conference | developers.sun.com/events/communityone | 3

7. About me
database hacker
creative programmer
2008 CommunityOne Conference | developers.sun.com/events/communityone | 4

8. Does it look familiar?
2008 CommunityOne Conference | developers.sun.com/events/communityone | 5

9. Does it look familiar?
SELECT * FROM table_name;
2008 CommunityOne Conference | developers.sun.com/events/communityone | 5

10. Does it look familiar?
SELECT * FROM table_name;
UPDATE table_name SET x=1 WHERE
y=10;
2008 CommunityOne Conference | developers.sun.com/events/communityone | 5

11. Does it look familiar?
SELECT * FROM table_name;
UPDATE table_name SET x=1 WHERE
y=10;
DELETE FROM tab_name WHERE z=1;
2008 CommunityOne Conference | developers.sun.com/events/communityone | 5

12. Does it look familiar?
SELECT * FROM table_name;
UPDATE table_name SET x=1 WHERE
y=10;
DELETE FROM tab_name WHERE z=1;
make coffee;
2008 CommunityOne Conference | developers.sun.com/events/communityone | 5

13. Does it look familiar?
SELECT * FROM table_name;
UPDATE table_name SET x=1 WHERE
y=10;
DELETE FROM tab_name WHERE z=1;
make coffee;
play movie;
2008 CommunityOne Conference | developers.sun.com/events/communityone | 5

14. Does it look familiar?
SELECT * FROM table_name;
UPDATE table_name SET x=1 WHERE
y=10;
DELETE FROM tab_name WHERE z=1;
make coffee;
??
play movie;
2008 CommunityOne Conference | developers.sun.com/events/communityone | 5

15. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 6

16. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 6

17. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 6

18. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
Revisiting Triggers
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 7

19. What average programmers dislike
regular expressions
2008 CommunityOne Conference | developers.sun.com/events/communityone | 8

20. What average programmers dislike
regular expressions
quot;I (love|hate)s*([Ff]w*)s*REGEXPquot;
2008 CommunityOne Conference | developers.sun.com/events/communityone | 8

21. What average programmers dislike
regular expressions
database backend
2008 CommunityOne Conference | developers.sun.com/events/communityone | 9

22. What average programmers dislike
regular expressions
database backend
SELECT t.id
FROM (SELECT id2 FROM other
WHERE b = 2) AS t
WHERE a > (SELECT MAX(x) FROM z);
2008 CommunityOne Conference | developers.sun.com/events/communityone | 9

23. Why many programmers hate database
handling?
SQL != Java
SQL != Perl
SQL != PHP
SQL != C++
I want to write in my language of choice
2008 CommunityOne Conference | developers.sun.com/events/communityone | 10

24. database handling
bring the
logic at
application
level
2008 CommunityOne Conference | developers.sun.com/events/communityone | 11

25. database handling
database
C/C++
Java
Perl
PHP
.NET
2008 CommunityOne Conference | developers.sun.com/events/communityone | 12

26. database handling
too many languages
Java class
database
C/C++
Java
Perl Perl library
PHP PHP library
.NET
2008 CommunityOne Conference | developers.sun.com/events/communityone | 13

27. database handling
set the logic
at server
level (stored
routines)
2008 CommunityOne Conference | developers.sun.com/events/communityone | 14

28. database handling
database
stored
routines
library
C/C++
Java
Perl
PHP
.NET
2008 CommunityOne Conference | developers.sun.com/events/communityone | 15

29. database handling
set the logic
at protocol
level (proxy)
2008 CommunityOne Conference | developers.sun.com/events/communityone | 16

30. database handling
database
C/C++ Proxy
Java library
Perl
PHP
.NET
2008 CommunityOne Conference | developers.sun.com/events/communityone | 17

31. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
Revisiting Triggers
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 18

32. Stored routines - the boring stu
SQL standard
heavy syntax
tricky performance
no debug - limited exception handling
procedures
• can return multiple record sets
• can execute dynamic code
functions
• must return one value
• no dynamic code
2008 CommunityOne Conference | developers.sun.com/events/communityone | 19

33. Stored routines - the boring stu
SQL statements
IF THEN ELSE .. END IF
WHILE .. DO .. END WHILE
LOOP .. END LOOP
CURSOR
CONDITION
HANDLER
2008 CommunityOne Conference | developers.sun.com/events/communityone | 20

34. Stored routines - some hacks
enhance the language
• arrays
• easy loops
• global variables
• see MySQL general purpose routine library
create routines on-the-fly
• undocumented hack!
• see http://datacharmer.org
2008 CommunityOne Conference | developers.sun.com/events/communityone | 21

35. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
Revisiting Triggers
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 22

36. Triggers - the boring stu
BEFORE or AFTER event
BEFORE INSERT/UPDATE/DELETE
AFTER INSERT/UPDATE/DELETE
No dynamic code
No operations on the same table
No multiple triggers per event
2008 CommunityOne Conference | developers.sun.com/events/communityone | 23

37. Triggers - what for
dedicated logging
data aggregation
transfer of data to other tables/databases
calculated fields
2008 CommunityOne Conference | developers.sun.com/events/communityone | 24

38. Triggers - HACKS!
wait.
Strong combination with FEDERATED and
BLACKHOLE
2008 CommunityOne Conference | developers.sun.com/events/communityone | 25

39. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
Revisiting Triggers
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 26

40. Views
no storage
look like tables
can aggregate data
can insert data
can save time
but can also become horribly ineficient
2008 CommunityOne Conference | developers.sun.com/events/communityone | 27

41. Views - the cool stu
can run functions
can create sort of triggers on select
2008 CommunityOne Conference | developers.sun.com/events/communityone | 28

42. Views with functions
CREATE FUNCTION f(V VARCHAR(20))
RETURNS INT
BEGIN
INSERT INTO log_table
VALUES (V, NOW());
RETURN 1;
END
2008 CommunityOne Conference | developers.sun.com/events/communityone | 29

43. Views with functions
CREATE VIEW v1 AS
SELECT x FROM table_name
WHERE f('v1') = 1;
2008 CommunityOne Conference | developers.sun.com/events/communityone | 30

44. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
Revisiting Triggers
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 31

45. Events - the not-so-boring facts
Temporal triggers
Operating System independent
executes SQL command or routine
• AT timestamp
• EVERY X time_interval
can execute dynamic SQL
2008 CommunityOne Conference | developers.sun.com/events/communityone | 32

46. Events
CREATE EVENT e1
ON SCHEDULE
AT NOW() + INTERVAL 10 minute
DO
CALL clean_employee_recs();
2008 CommunityOne Conference | developers.sun.com/events/communityone | 33

47. Events
CREATE EVENT e1
ON SCHEDULE
EVERY 5 minute
DO
CALL import_recs_from_file();
2008 CommunityOne Conference | developers.sun.com/events/communityone | 34

48. Agenda
Database programming blues
Stored routines
Triggers
Events
Federated
Blackhole
Creative loops
Revisiting Triggers
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 35

49. Federated storage engine
no storage
connects to a table on a remote server
since version 5.1.22 can connect to a table on the
same server
table t1f table t1
federated MyISAM
2008 CommunityOne Conference | developers.sun.com/events/communityone | 36

50. Federated storage engine - caveats
no drop-in replacement for regular tables
ineficient aggregates
limited push-down conditions
2008 CommunityOne Conference | developers.sun.com/events/communityone | 37

51. Federated storage engine
limited pushdown conditions
federated
table base
table
SELECT ID
FROM table_name
SELECT ID
WHERE x = 10
FROM table_name
expected
returns
~ 100 records filter 1 million records
2008 CommunityOne Conference | developers.sun.com/events/communityone | 38

52. Federated storage engine
workaround pushdown conditions
add fake index
on column x
federated
to Federated table
table base
table
SELECT ID
FROM table_name
SELECT ID
WHERE x = 10
FROM table_name
WHERE x = 10
expected
returns
~ 100 records
100 records
2008 CommunityOne Conference | developers.sun.com/events/communityone | 39

53. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
Revisiting Triggers
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 40

54. Blackhole
The NON-STORAGE engine
like /dev/null in Unix
used for triggers and replication relay
record
record binary log
record
record
triggers
2008 CommunityOne Conference | developers.sun.com/events/communityone | 41

55. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
Revisiting Triggers
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 42

56. More exciting triggers
Put them together
• triggers
• Federated engine
• Blackhole engine
2008 CommunityOne Conference | developers.sun.com/events/communityone | 43

57. cascade triggers
BASE TABLE
before insert
after insert
before update
after update
before delete
after delete
t1
2008 CommunityOne Conference | developers.sun.com/events/communityone | 44

58. cascade triggers
Federated
BASE TABLE
TABLE
before insert
before insert
after insert
after insert
before update before update
after update after update
before delete before delete
after delete after delete
t1 t1_f1
2008 CommunityOne Conference | developers.sun.com/events/communityone | 44

59. triggers on request Federated before insert
TABLE
either
BASE TABLE after update
t1_f1
after delete
Federated before insert
TABLE after insert
or before update
t1
before delete
t1_f2
2008 CommunityOne Conference | developers.sun.com/events/communityone | 45

60. blackhole triggers
BASE TABLE
blackhole
TABLE
before insert
t1
BASE TABLE
t1_b1
t2
2008 CommunityOne Conference | developers.sun.com/events/communityone | 46

61. remote execution
blackhole Federated
before insert
table TABLE
t1 t1_f1
2008 CommunityOne Conference | developers.sun.com/events/communityone | 47

62. remote execution
VIEW Federated
function
TABLE
v1 t1_f1
2008 CommunityOne Conference | developers.sun.com/events/communityone | 48

63. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
Revisiting Triggers
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 49

64. loops
or why do we have to endure this?
loops with CURSORS
• dreadful syntax
• INEFFICIENT (temporary table created)
loops alternatives ... coming soon
2008 CommunityOne Conference | developers.sun.com/events/communityone | 50

65. loops with cursors :(
data to crc flag
process cursor
temporary
loop
table
handler
crc
2008 CommunityOne Conference | developers.sun.com/events/communityone | 51

66. loops with cursors (I) :(
CREATE PROCEDURE p(j INT)
BEGIN
DECLARE done BOOLEAN default 'false';
DECLARE crc VARCHAR(42) default '';
DECLARE cid INT;
DECLARE x CURSOR FOR
SELECT id FROM
table_name WHERE
b j;
-- to be continued
2008 CommunityOne Conference | developers.sun.com/events/communityone | 52

67. loops with cursors (II) :((
DECLARE CONTINUE HANDLER
FOR NOT FOUND set done = true;
OPEN CURSOR x;
search: LOOP
FETCH x into cid;
IF done THEN
LEAVE search;
END IF;
-- do something with cid
SET crc = MD5(CONCAT(crc,cid))
END LOOP;
SELECT crc;
2008 CommunityOne Conference | developers.sun.com/events/communityone | 53

68. loops with blackhole :)
data to
crc
process
blackhole
table
crc
2008 CommunityOne Conference | developers.sun.com/events/communityone | 54

69. loops with blackhole (I) :)
CREATE TABLE t1 (
id VARCHAR(200)
) ENGINE = BLACKHOLE;
SET @crc = '';
SET @j = 10;
2008 CommunityOne Conference | developers.sun.com/events/communityone | 55

70. loops with blackhole (II) :)
INSERT INTO t1
SELECT @crc :=
MD5(CONCAT(@crc, id))
FROM table_name
WHERE b @j;
SELECT @crc;
IT'S FASTER!!!
2008 CommunityOne Conference | developers.sun.com/events/communityone | 56

71. more loops alternatives
use the General Purpose Routine Library
• https://sourceforge.net/projects/mysql-sr-lib/
use events!
(ON SCHEDULE EVERY 1 SECOND
ENDS NOW() + INTERVAL 10 SECOND)
use MySQL Proxy (later)
2008 CommunityOne Conference | developers.sun.com/events/communityone | 57

72. loops with
MySQL stored routines library
call
for_each_table_value_complete(
'db_name', 'table_name',
'id', null, null, '',
'set @crc=MD5(concat(@crc,$I1))',
'set @crc=quot;quot;',
'select @crc','',
'once');
2008 CommunityOne Conference | developers.sun.com/events/communityone | 58

73. Agenda
Database programming blues
Stored routines
Triggers
Views
Events
Federated
Blackhole
Creative loops
Revisiting Triggers
MySQL Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 59

74. MySQL Proxy
So, what about quot;make coeequot; ?
2008 CommunityOne Conference | developers.sun.com/events/communityone | 60

75. 2008 CommunityOne Conference | developers.sun.com/events/communityone | 61

76. 2008 CommunityOne Conference | developers.sun.com/events/communityone | 62

77. 2008 CommunityOne Conference | developers.sun.com/events/communityone | 63

78. MySQL
Proxy
2008 CommunityOne Conference | developers.sun.com/events/communityone | 64

79. MySQL Proxy principles - hooks
2008 CommunityOne Conference | developers.sun.com/events/communityone | 65

80. MySQL Proxy principles - hooks
PROXY
CORE
connection
hook
read query
hook
read result
hook
2008 CommunityOne Conference | developers.sun.com/events/communityone | 65

81. MySQL Proxy principles - hooks
PROXY
CORE Lua script
function
connection function
function
hook
read query function
hook
read result function
hook
2008 CommunityOne Conference | developers.sun.com/events/communityone | 65

82. MySQL Proxy principles - hooks
PROXY
CORE Lua script
function
connection function
function
hook
read query function
hook
read result function
hook
2008 CommunityOne Conference | developers.sun.com/events/communityone | 65

83. ??
MySQL Proxy principles - Lua
{
Perl ?
PHP?
Why not ...
Javascript?
[whatever]?
2008 CommunityOne Conference | developers.sun.com/events/communityone | 66

84. MySQL Proxy principles - Lua
2008 CommunityOne Conference | developers.sun.com/events/communityone | 67

85. MySQL Proxy principles - Lua
• SMALL ( 200 KB)
• DESIGNED for
EMBEDDED systems
• Widely used (lighttpd)
2008 CommunityOne Conference | developers.sun.com/events/communityone | 67

86. MySQL Proxy principles - Lua
• SMALL ( 200 KB)
• DESIGNED for
EMBEDDED systems
• Widely used (lighttpd)
lighttpd, like MySQL
Proxy, was created by
Jan Kneschke
2008 CommunityOne Conference | developers.sun.com/events/communityone | 67

87. MySQL Proxy principles - Lua
Very popular among
game writers
2008 CommunityOne Conference | developers.sun.com/events/communityone | 68

88. MySQL Proxy principles - Lua
Very popular among
game writers
2008 CommunityOne Conference | developers.sun.com/events/communityone | 68

89. MySQL Proxy principles - Lua
Very popular among
game writers
2008 CommunityOne Conference | developers.sun.com/events/communityone | 68

90. injecting
queries
2008 CommunityOne Conference | developers.sun.com/events/communityone | 69

91. injecting
queries
2008 CommunityOne Conference | developers.sun.com/events/communityone | 70

92. Debugging
server
proxy
client
diagnostics
text
2008 CommunityOne Conference | developers.sun.com/events/communityone | 71

93. Debugging scripts
server
proxy
proxy
diagnostics
text
client
diagnostics
text
2008 CommunityOne Conference | developers.sun.com/events/communityone | 72

94. Chained proxies - double features
server
proxy
proxy
pivot tables
client
loops
2008 CommunityOne Conference | developers.sun.com/events/communityone | 73

95. Testing - customized packages
server
proxy
client
e.g.
fake packets
connectors
2008 CommunityOne Conference | developers.sun.com/events/communityone | 74

96. MySQL Proxy - loops revisited
# get the script from MySQL Forge
# http://forge.mysql.com/tools/tool.php?id=96
mysql-proxy
--proxy-lua-script=loop.lua
2008 CommunityOne Conference | developers.sun.com/events/communityone | 75

97. MySQL Proxy - loops revisited
mysql -h 127.0.0.1 -P 4040
FOR VAR 1 3
CREATE TABLE t_$VAR (id int);
# creates t_1, t_2, t_3
2008 CommunityOne Conference | developers.sun.com/events/communityone | 76

98. MySQL Proxy - loops revisited
mysql -h 127.0.0.1 -P 4040
FOR VAR ( aa, bb cc)
CREATE TABLE t_$VAR (id int);
# creates t_aa, t_bb, t_cc
2008 CommunityOne Conference | developers.sun.com/events/communityone | 77

99. Summary
MySQL is open to development creativity
Know your bricks
• stored routines, triggers, views, events
• dynamic engines (Federated, Blackhole)
• MySQL Proxy
Put them together : be creative
2008 CommunityOne Conference | developers.sun.com/events/communityone | 78

100. For More Information
MySQL Forge ( http://forge.mysql.com )
my blog ( http://datacharmer.blogspot.com )
MySQL blogs ( http://planetmysql.org )
2008 CommunityOne Conference | developers.sun.com/events/communityone | 79

101. QA
Let's talk!
80