https://github.com/rpbouman/mysqlv8udfs
MySQL
User-Defined Functions
...in JavaScript!
https://github.com/rpbouman/mysqlv8udfs
Welcome!
● @rolandbouman
● roland.bouman@gmail.com
● http://rpbouman.blogspot.com/
● http://www.linkedin.com/in/rpbouman
● http://www.slideshare.net/rpbouman
● Ex-MySQL AB, Ex-Sun Microsystems
● Currently at http://www.pentaho.com/
https://github.com/rpbouman/mysqlv8udfs
MySQL Programmability
● SQL
● Persistent Stored Modules (Stored Routines)
● User-defined functions (UDFs)
https://github.com/rpbouman/mysqlv8udfs
MySQLv8UDFs:
JavaScript Programmability
● https://github.com/rpbouman/mysqlv8udfs
● Based on Google's V8
● More than just executing JavaScript:
– Scriptable front for MySQL's native UDF interface
https://github.com/rpbouman/mysqlv8udfs
MySQL stored routines
● “Standard” SQL/PSM syntax
– Scalar functions
– Procedures
– Triggers
– Events
● Stored in the data dictionary
● Interpreted
https://github.com/rpbouman/mysqlv8udfs
MySQL UDFs
● External binary library (typically C/C++):
– Scalar functions
– Aggregate functions
● Registered in the data dictionary
● Compiled Native code
https://github.com/rpbouman/mysqlv8udfs
JavaScript UDFs. Why?
● Started as a UDF example
● Inspired by drizzle's js() function
● Turned out to have real benefits:
– Convenient manipulation of JSON blobs
– Safer and easier than 'real' C/C++ UDFs
– More expressive than SQL/PSM
– Sometimes much faster than stored routines*
https://github.com/rpbouman/mysqlv8udfs
Intermezzo: Easter day
as stored SQL functionCREATE FUNCTION easter_day(dt DATETIME) RETURNS DATE
DETERMINISTIC NO SQL SQL SECURITY INVOKER
COMMENT 'Returns date of easter day for given year'
BEGIN
DECLARE p_year SMALLINT DEFAULT YEAR(dt);
DECLARE a SMALLINT DEFAULT p_year % 19;
DECLARE b SMALLINT DEFAULT p_year DIV 100;
DECLARE c SMALLINT DEFAULT p_year % 100;
DECLARE e SMALLINT DEFAULT b % 4;
DECLARE h SMALLINT DEFAULT (19*a + b - (b DIV 4) - (
(b - ((b + 8) DIV 25) + 1) DIV 3
) + 15) % 30;
DECLARE L SMALLINT DEFAULT (32 + 2*e + 2*(c DIV 4) - h - (c % 4)) % 7;
DECLARE v100 SMALLINT DEFAULT h + L - 7*((a + 11*h + 22*L) DIV 451) + 114;
RETURN STR_TO_DATE(
CONCAT(
p_year
, '-'
, v100 DIV 31
, '-'
, (v100 % 31) + 1
)
, '%Y-%c-%e'
);
END;
https://github.com/rpbouman/mysqlv8udfs
Intermezzo: Easter day
in JavaScript (js UDF)
mysql> SELECT js('
'> var y = parseInt(arguments[0].substr(0,4), 10),
'> a = y % 19, b = Math.floor(y / 100),
'> c = y % 100, d = Math.floor(b / 4),
'> e = b % 4, f = Math.floor((b + 8) / 25),
'> g = Math.floor((b - f + 1) / 3),
'> h = (19 * a + b - d - g + 15) % 30,
'> i = Math.floor(c / 4), k = c % 4,
'> L = (32 + 2 * e + 2 * i - h - k) % 7,
'> m = Math.floor((a + 11 * h + 22 * L) / 451),
'> n = h + L - 7 * m + 114,
'> M = Math.floor(n/31), D = (n%31)+1;
'> if (M < 10) M = "0" + M;
'> if (D < 10) D = "0" + D;
'>
'> y + "-" + M + "-" + D;
'>
'>', NOW());
https://github.com/rpbouman/mysqlv8udfs
Intermezzo: Easter day
as SQL expressionSTR_TO_DATE(CONCAT(YEAR(now()), '-', (((19*(YEAR(now()) % 19) + (YEAR(now()) DIV
100) - ((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now())
DIV 100) + 8) DIV 25) + 1) DIV 3) + 15) % 30) + ((32 + 2*((YEAR(now()) DIV 100) %
4) + 2*((YEAR(now()) % 100) DIV 4) - ((19*(YEAR(now()) % 19) + (YEAR(now()) DIV
100) - ((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now())
DIV 100) + 8) DIV 25) + 1) DIV 3) + 15) % 30) - ((YEAR(now()) % 100) % 4)) % 7) -
7*(((YEAR(now()) % 19) + 11*((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) -
((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100)
+ 8) DIV 25) + 1) DIV 3) + 15) % 30) + 22*((32 + 2*((YEAR(now()) DIV 100) % 4) +
2*((YEAR(now()) % 100) DIV 4) - ((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) -
((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100)
+ 8) DIV 25) + 1) DIV 3) + 15) % 30) - ((YEAR(now()) % 100) % 4)) % 7)) DIV 451) +
114) DIV 31, '-', ((((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) -
((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100)
+ 8) DIV 25) + 1) DIV 3) + 15) % 30) + ((32 + 2*((YEAR(now()) DIV 100) % 4) +
2*((YEAR(now()) % 100) DIV 4) - ((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) -
((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100)
+ 8) DIV 25) + 1) DIV 3) + 15) % 30) - ((YEAR(now()) % 100) % 4)) % 7) -
7*(((YEAR(now()) % 19) + 11*((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) -
((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100)
+ 8) DIV 25) + 1) DIV 3) + 15) % 30) + 22*((32 + 2*((YEAR(now()) DIV 100) % 4) +
2*((YEAR(now()) % 100) DIV 4) - ((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) -
((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100)
+ 8) DIV 25) + 1) DIV 3) + 15) % 30) - ((YEAR(now()) % 100) % 4)) % 7)) DIV 451) +
114) % 31) + 1), '%Y-%c-%e')
https://github.com/rpbouman/mysqlv8udfs
Intermezzo: Easter day
Performance comparison
SQL Expression SQL Stored Function JavaScript UDF
0
2
4
6
8
10
12
14
4.61
12.53
2.66
Easter Day Performance (1.000.000)
time(seconds)
https://github.com/rpbouman/mysqlv8udfs
The mysqlv8udfs project
● Scalar Functions:
– js()
– jsudf()
– jserr()
● Aggregate Functions:
– jsagg()
● Daemon plugin*:
– JS_DAEMON
https://github.com/rpbouman/mysqlv8udfs
The JS_DAEMON Plugin
mysql> SHOW VARIABLES LIKE 'js%';
+-----------------------+--------------------------------------+
| Variable_name | Value |
+-----------------------+--------------------------------------+
| js_daemon_module_path | /home/rbouman/mysql/mysql/lib/plugin |
+-----------------------+--------------------------------------+
1 row in set (0.03 sec)
mysql> SHOW STATUS LIKE 'js%';
+----------------------------------+-----------+
| Variable_name | Value |
+----------------------------------+-----------+
| js_daemon_version | 0.0.1 |
| js_v8_heap_size_limit | 2048 |
| js_v8_heap_size_total | 942944256 |
| js_v8_heap_size_total_executable | 959591424 |
| js_v8_heap_size_used | 892941672 |
| js_v8_is_dead | false |
| js_v8_is_execution_terminating | false |
| js_v8_is_profiler_paused | true |
| js_v8_version | 3.7.12.22 |
+----------------------------------+-----------+
9 rows in set (0.00 sec)
https://github.com/rpbouman/mysqlv8udfs
The js() UDF
● js(script[, arg1, …, argN])
– Execute script
– Return value (as string) of the last js expression
● Optional arguments arg1 … argN
– Accessible via the built-in arguments array
– arg1 accessible as arguments[0] (and so on)
● Script*
– if constant it is compiled only once
– executed for each row
https://github.com/rpbouman/mysqlv8udfs
The js() UDF: Example
mysql> SELECT js('
'> arguments[0] + arguments[1];
'> ', 1, 2) AS example
-> ;
+---------+
| example |
+---------+
| 3 |
+---------+
1 row in set (0.03 sec)
https://github.com/rpbouman/mysqlv8udfs
The MySQL UDF interface
● Simple functions (scalars) and aggregates
● Native library callbacks (calling sequence)
– Plugin directory
– mysql.func table
● Data structures
– Return value: struct UDF_INIT *initid
– Arguments: struct UDF_ARGS *args
CREATE [AGGREGATE] FUNCTION name
RETURNS (STRING | REAL | INTEGER)
SONAME 'libraryfile'
https://github.com/rpbouman/mysqlv8udfs
UDF JavaScript binding
● Scalars: jsudf(), Aggregates: jsagg()
– Return value is always a MySQL STRING
● Script argument:
– Constant. Compiled and immediately executed (1x)
– JavaScript callbacks defined in script are called during
the native UDF calling sequence
● UDF data structures scriptable at runtime:
– Members of struct UDF_INIT appear as js globals
– struct UDF_ARGS as global arguments object
https://github.com/rpbouman/mysqlv8udfs
The jsudf() UDF
● jsudf(script[, arg1, …, argN])
– Call the init() callback (optional)
– For each row, return the result of the udf() callback
– Call the deinit() callback (optional)
init() udf() deinit()More rows? No
Yes
https://github.com/rpbouman/mysqlv8udfs
jsudf() example:
running total
mysql> SELECT amount, jsudf('
-> var total;
-> function init(){
-> console.info("Init");
-> total = 0;
-> }
-> function udf(num){
-> console.info("processing row");
-> return total += num;
-> }
-> function deinit(){
-> console.info("Deinit");
-> }
-> ', amount) AS running_total
-> FROM sakila.payment ORDER BY payment_date
https://github.com/rpbouman/mysqlv8udfs
jsudf() example:
resultset and error log
+--------+--------------------+
| amount | running_total |
+--------+--------------------+
| 2.99 | 2.99 |
| 2.99 | 5.98 |
. ... . ... .
| 4.99 | 67416.5099999921 |
+--------+--------------------+
16049 rows in set (0.29 sec)
2013-09-16 14:31:44 JS_DAEMON [info]: Init
2013-09-16 14:31:44 JS_DAEMON [info]: processing row
.... .. .. .. .. .. .. ...... ...... ..............
2013-09-16 14:31:44 JS_DAEMON [info]: processing row
2013-09-16 14:31:44 JS_DAEMON [info]: Deinit
https://github.com/rpbouman/mysqlv8udfs
jsudf() Argument
processing
● Arguments beyond the initial script argument:
– Values passed to the udf() callback
– UDF_ARGS scriptable as global arguments array
– WARNING: In javascript functions, the local built-in
arguments object refers to actual arguments
– Local arguments mask the global arguments object.
– Use this.arguments to refer to the global array of
argument objects.
● Use init() to validate or pre-process arguments
https://github.com/rpbouman/mysqlv8udfs
jsudf() global arguments
mysql> SELECT jsudf('
-> function udf(){
-> //this.arguments describes the
-> //arguments passed to jsudf
->
-> return JSON.stringify(
-> this.arguments,
-> null, " "
-> );
-> }
-> ', 'string', PI() AS "real", 1, 2.3)
https://github.com/rpbouman/mysqlv8udfs
jsudf() global arguments
[
{
"name": "'string'",
"type": 0,
"max_length": 6,
"maybe_null": false,
"const_item": true,
"value": "string"
},
{
"name": "real",
"type": 1,
"max_length": 8,
"maybe_null": false,
"const_item": true,
"value": 3.141592653589793
},
{
"name": "1",
"type": 2,
"max_length": 1,
"maybe_null": false,
"const_item": true,
"value": 1
},
{
"name": "2.3",
"type": 4,
"max_length": 3,
"maybe_null": false,
"const_item": true,
"value": 2.3
}
]
https://github.com/rpbouman/mysqlv8udfs
jsudf() local arguments
(actual arguments)
mysql> SELECT jsudf('
-> function udf(){
-> //standard, built-in javascript arguments
-> //contains argument values passed to this function
-> return JSON.stringify(
-> arguments,
-> null, " "
-> );
-> }
-> ', 'string', PI() AS "real", 1, 2.3)
{
"0": "string",
"1": 3.141592653589793,
"2": 1,
"3": 2.3
}
https://github.com/rpbouman/mysqlv8udfs
jsudf() named
arguments
SELECT jsudf('
function udf( arg1, arg2, arg3, arg4 ){
//values of named argument are available
//as local variables
return "arg1: " + arg1 +
"n" + "arg2: " + arg2 +
"n" + "arg3: " + arg3 +
"n" + "arg4: " + arg4;
}
', 'string', PI() AS "real", 1, 2.3)
arg1: string
arg2: 3.141592653589793
arg3: 1
arg4: 2.3
https://github.com/rpbouman/mysqlv8udfs
The Argument object
● name: Expression text. If provided, the alias
● type: code indicating the runtime data type
– 0: STRING_RESULT, 1: REAL_RESULT,
4: DECIMAL_RESULT
● max_length: maximum string length
● maybe_null: true if nullable
● const_item: true if value is constant
● value: argument value
https://github.com/rpbouman/mysqlv8udfs
Argument Processing:
Validating count and types
function init(){
var args = this.arguments,
nargs = args.length
;
//validate the number of arguments:
if (nargs != 1) throw "Expected exactly 1 argument";
//validate argument type:
var arg = args[0];
switch (arg.type) {
case REAL_RESULT:
case INT_RESULT:
case DECIMAL_RESULT:
break;
default:
throw "Argument must be numeric";
}
}
https://github.com/rpbouman/mysqlv8udfs
Data type Mapping+------------------------+------------------------+---------------------+-------------+---------+
| Type family | MySQL column data type | MYSQL UDF data type | v8 type | JS Type |
+------------------------+------------------------+---------------------+-------------+---------+
| Integral numbers | BIGINT | INT_RESULT | v8::Integer | Number |
| | INT | | or | |
| | MEDIUMINT | | v8::Number | |
| | SMALLINT | | | |
| | TINYINT | | | |
+------------------------+------------------------+---------------------+-------------| |
| Floating point numbers | DOUBLE | REAL_RESULT | v8::Number | |
| | FLOAT | | | |
+------------------------+------------------------+---------------------+ | |
| Decimal numbers | DECIMAL | DECIMAL_RESULT | | |
+------------------------+------------------------+---------------------+-------------+---------+
| Binary String | BINARY | STRING_RESULT | v8::String | String |
| | BLOB | | | |
| | LONGBLOB | | | |
| | MEDIUMBLOB | | | |
| | VARBINARY | | | |
| | TINYBLOB | | | |
+------------------------+------------------------+ | | |
| Character String | CHAR | | | |
| | LONGTEXT | | | |
| | MEDIUMTEXT | | | |
| | VARCHAR | | | |
| | TEXT | | | |
| | TINYTEXT | | | |
+------------------------+------------------------+ | | |
| Structured String | ENUM | | | |
| | SET | | | |
+------------------------+------------------------+ | | |
| Temporal | DATE | | | |
| | DATETIME | | | |
| | TIME | | | |
| | TIMESTAMP | | | |
+------------------------+------------------------+---------------------+-------------+---------+
https://github.com/rpbouman/mysqlv8udfs
The jsagg() UDF
● jsagg(script[, arg1, …, argN])
– Call the init() callback (optional)
– Calls clear() before processing a group of rows
– For each row in a group, the udf() callback is called
– After processing a group, the agg() is called to return
the aggregate value
– Call the deinit() callback (optional)
https://github.com/rpbouman/mysqlv8udfs
The jsagg() UDF
init()
deinit()
More rows?
No
Yes
clear()
agg()
udf()
More groups?
Yes
No
https://github.com/rpbouman/mysqlv8udfs
jsagg() example:
JSON export
mysql> SELECT jsagg('
-> var rows, args = arguments, n = args.length;
-> function clear(){
-> console.info("clear");
-> rows = [];
-> }
-> function udf(){
-> console.info("udf");
-> var i, arg, row = {};
-> for (i = 0; i < n; i++){
-> arg = args[i];
-> row[arg.name] = arg.value;
-> }
-> rows.push(row);
-> }
-> function agg(){
-> console.info("agg");
-> return JSON.stringify(rows, null, " ");
-> }
-> ', film_id, title, release_year, description) AS json
-> FROM sakila.film GROUP BY rating;
https://github.com/rpbouman/mysqlv8udfs
jsagg() example:
result
[
{
"film_id": 1,
"title": "ACADEMY DINOSAUR",
"release_year": 2006,
"description": "A Epic Drama of ... in The Canadian Rockies"
},
...,
...,
{
"film_id": 1000,
"title": "ZORRO ARK",
"release_year": 2006,
"description": "A Intrepid Panorama of ... in A Monastery"
}
]
https://github.com/rpbouman/mysqlv8udfs
jsagg() example:
error log
2013-09-16 23:36:45 JS_DAEMON [info]: Clear
2013-09-16 23:36:45 JS_DAEMON [info]: Udf
.... .. .. .. .. .. .. ...... .... ...
2013-09-16 23:36:45 JS_DAEMON [info]: Udf
2013-09-16 23:36:45 JS_DAEMON [info]: Agg
2013-09-16 23:36:45 JS_DAEMON [info]: Clear
2013-09-16 23:36:45 JS_DAEMON [info]: Udf
.... .. .. .. .. .. .. ...... .... ...
2013-09-16 23:36:45 JS_DAEMON [info]: Udf
2013-09-16 23:36:45 JS_DAEMON [info]: Agg
.... .. .. .. .. .. .. ...... .... ...
.... .. .. .. .. .. .. ...... .... ...
2013-09-16 23:36:45 JS_DAEMON [info]: Clear
2013-09-16 23:36:45 JS_DAEMON [info]: Udf
.... .. .. .. .. .. .. ...... .... ...
2013-09-16 23:36:45 JS_DAEMON [info]: Udf
2013-09-16 23:36:45 JS_DAEMON [info]: Agg
https://github.com/rpbouman/mysqlv8udfs
JavaScript Environment
● JavaScript Standard built-ins:
– Constructors (Date, RegExp, String etc.)
– Static objects (JSON, Math)
– Misc. functions (decodeURI, parseInt etc.)
● Globals provided by mysqlv8udfs *
– arguments[] array
– Some UDF interface variables and constants
– require() function
– console object
– mysql object
https://github.com/rpbouman/mysqlv8udfs
The require() function
● Inspired by commonjs Module loading
● Signature: require(filename[, reload])
– Loads script file from the js_daemon_module_path
– Executes the script and returns the result
– Script is compiled and cached for reuse
– Pass true as 2nd argument to force reload from file
● js_daemon_module_path
– Read-only system variable of the JS_DAEMON plugin
– Specified at mysqld command line or option file
– Prevent loading arbitrary script files
https://github.com/rpbouman/mysqlv8udfs
require() example:
mysql> SELECT jsagg('
-> require("json_export.js")
-> ', category_id, name) AS json
-> FROM sakila.category
[
{
"category_id": 1,
"name": "Action"
},
...,
{
"category_id": 16,
"name": "Travel"
}
]
https://github.com/rpbouman/mysqlv8udfs
require() example:
json_export.js script
(function json_export(){
var rows, row, i, arg, args = this.arguments, n = args.length;
this.clear = function(){
rows = [];
}
this.udf = function() {
rows.push(row = {});
for (i = 0; i < n; i++) {
arg = args[i];
row[arg.name] = arg.value;
}
}
this.agg = function(){
return JSON.stringify(rows, null, " ");
}
})();
https://github.com/rpbouman/mysqlv8udfs
The console object
● Inspired by console object in web-browsers
● Methods:
– log([arg1, ..., argN])
– info([arg1, ..., argN])
– error([arg1, ..., argN])
– warn([arg1, ..., argN])
● Write arguments to a line on the standard error
stream
– Typically ends up in the mysql error log
● info(), error(), and warn() include a header:
– 2013­09­17 00:50:22 JS_DAEMON [info]: ...
https://github.com/rpbouman/mysqlv8udfs
The mysql object
● Namespace for interacting with MySQL
– Depends on libmysqlclient
mysql
client
  connect()
  
  version
connection
  close()
  commit()
  rollback()
  query()
  setAutocommit()
  charset
  connected
  hostInfo
  InsertId
  protocolVersion
  serverVersion
  statistics
  warnings
query
execute()
result()
done
sql resultset
buffered
done
fieldCount
type: "resultset"
field()
row()
resultinfo
done: true
rowCount
type: "resultinfo" 
types
    0: "decimal"
    1: "tinyint"
  .... .........
  255: "geometry"
https://github.com/rpbouman/mysqlv8udfs
Mysql client example:
inventory_held_by_customer
CREATE FUNCTION inventory_held_by_customer(p_inventory_id INT)
RETURNS INT
READS SQL DATA
BEGIN
DECLARE v_customer_id INT;
DECLARE EXIT HANDLER FOR NOT FOUND RETURN NULL;
SELECT customer_id INTO v_customer_id
FROM rental
WHERE return_date IS NULL
AND inventory_id = p_inventory_id;
RETURN v_customer_id;
END;
https://github.com/rpbouman/mysqlv8udfs
Mysql client example
(function(){
var conn;
this.init = function(){
var args = this.arguments;
if (args.length !== 1 || args[0].type !== INT_RESULT) {
throw "Single integer argument required";
}
conn = mysql.client.connect({
user: "sakila",
password: "sakila",
schema: "sakila"
});
}
this.udf = function(inventory_id){
var query = conn.query(
"SELECT customer_id FROM rental WHERE return_date IS NULL " +
"AND inventory_id = " + inventory_id
);
query.execute();
var result = query.result();
if (result.done) return null;
return result.row()[0];
}
this.deinit = function(){
conn.close();
}
})();
https://github.com/rpbouman/mysqlv8udfs
Oracle JSON functions
(Labs)
● JSON, not JavaScript
– Stick with this If you only need JSON manipulation
● That said....
– Mysqlv8udfs performance is typically slightly better, in
some cases substantially better
– JSON functions have a few bugs which can be easily
worked around using mysqlv8udfs
– Easy to emulate with mysqlv8udfs, see sample
implementations in the js project dir
https://github.com/rpbouman/mysqlv8udfs
JSON functions vs
mysqlV8UDFs example
mysql> SELECT json_append(
-> @json, 'menu', 'popup', 'menuitem', 3,
-> '{"value": "Help", "onclick": "ShowHelp()"}'
-> );
SET @json := '{
"menu": {
"id": "file", "value": "File", "popup": {
"menuitem": [
{"value": "New", "onclick": "CreateNewDoc()"},
{"value": "Open", "onclick": "OpenDoc()"},
{"value": "Close", "onclick": "CloseDoc()"}
]
}
}
}';
mysql> SELECT jsudf(
-> 'require("json_append.js");',
-> @json, 'menu', 'popup', 'menuitem', 3,
-> '{"value": "Help", "onclick": "ShowHelp()"}'
-> );
https://github.com/rpbouman/mysqlv8udfs
JSON functions vs MySQLv8UDFs
append contains key extract merge remove replace valid
0
2
4
6
8
10
12
14
json
v8
Seconds
https://github.com/rpbouman/mysqlv8udfs
Finally...
● Fork it on github. I appreciate your interest!
– https://github.com/rpbouman/mysqlv8udfs
– https://github.com/rpbouman/mysqlv8udfs/wiki
https://github.com/rpbouman/mysqlv8udfs
Questions?

Writing MySQL User-defined Functions in JavaScript

  • 1.
  • 2.
    https://github.com/rpbouman/mysqlv8udfs Welcome! ● @rolandbouman ● roland.bouman@gmail.com ●http://rpbouman.blogspot.com/ ● http://www.linkedin.com/in/rpbouman ● http://www.slideshare.net/rpbouman ● Ex-MySQL AB, Ex-Sun Microsystems ● Currently at http://www.pentaho.com/
  • 3.
    https://github.com/rpbouman/mysqlv8udfs MySQL Programmability ● SQL ●Persistent Stored Modules (Stored Routines) ● User-defined functions (UDFs)
  • 4.
    https://github.com/rpbouman/mysqlv8udfs MySQLv8UDFs: JavaScript Programmability ● https://github.com/rpbouman/mysqlv8udfs ●Based on Google's V8 ● More than just executing JavaScript: – Scriptable front for MySQL's native UDF interface
  • 5.
    https://github.com/rpbouman/mysqlv8udfs MySQL stored routines ●“Standard” SQL/PSM syntax – Scalar functions – Procedures – Triggers – Events ● Stored in the data dictionary ● Interpreted
  • 6.
    https://github.com/rpbouman/mysqlv8udfs MySQL UDFs ● Externalbinary library (typically C/C++): – Scalar functions – Aggregate functions ● Registered in the data dictionary ● Compiled Native code
  • 7.
    https://github.com/rpbouman/mysqlv8udfs JavaScript UDFs. Why? ●Started as a UDF example ● Inspired by drizzle's js() function ● Turned out to have real benefits: – Convenient manipulation of JSON blobs – Safer and easier than 'real' C/C++ UDFs – More expressive than SQL/PSM – Sometimes much faster than stored routines*
  • 8.
    https://github.com/rpbouman/mysqlv8udfs Intermezzo: Easter day asstored SQL functionCREATE FUNCTION easter_day(dt DATETIME) RETURNS DATE DETERMINISTIC NO SQL SQL SECURITY INVOKER COMMENT 'Returns date of easter day for given year' BEGIN DECLARE p_year SMALLINT DEFAULT YEAR(dt); DECLARE a SMALLINT DEFAULT p_year % 19; DECLARE b SMALLINT DEFAULT p_year DIV 100; DECLARE c SMALLINT DEFAULT p_year % 100; DECLARE e SMALLINT DEFAULT b % 4; DECLARE h SMALLINT DEFAULT (19*a + b - (b DIV 4) - ( (b - ((b + 8) DIV 25) + 1) DIV 3 ) + 15) % 30; DECLARE L SMALLINT DEFAULT (32 + 2*e + 2*(c DIV 4) - h - (c % 4)) % 7; DECLARE v100 SMALLINT DEFAULT h + L - 7*((a + 11*h + 22*L) DIV 451) + 114; RETURN STR_TO_DATE( CONCAT( p_year , '-' , v100 DIV 31 , '-' , (v100 % 31) + 1 ) , '%Y-%c-%e' ); END;
  • 9.
    https://github.com/rpbouman/mysqlv8udfs Intermezzo: Easter day inJavaScript (js UDF) mysql> SELECT js(' '> var y = parseInt(arguments[0].substr(0,4), 10), '> a = y % 19, b = Math.floor(y / 100), '> c = y % 100, d = Math.floor(b / 4), '> e = b % 4, f = Math.floor((b + 8) / 25), '> g = Math.floor((b - f + 1) / 3), '> h = (19 * a + b - d - g + 15) % 30, '> i = Math.floor(c / 4), k = c % 4, '> L = (32 + 2 * e + 2 * i - h - k) % 7, '> m = Math.floor((a + 11 * h + 22 * L) / 451), '> n = h + L - 7 * m + 114, '> M = Math.floor(n/31), D = (n%31)+1; '> if (M < 10) M = "0" + M; '> if (D < 10) D = "0" + D; '> '> y + "-" + M + "-" + D; '> '>', NOW());
  • 10.
    https://github.com/rpbouman/mysqlv8udfs Intermezzo: Easter day asSQL expressionSTR_TO_DATE(CONCAT(YEAR(now()), '-', (((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) - ((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100) + 8) DIV 25) + 1) DIV 3) + 15) % 30) + ((32 + 2*((YEAR(now()) DIV 100) % 4) + 2*((YEAR(now()) % 100) DIV 4) - ((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) - ((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100) + 8) DIV 25) + 1) DIV 3) + 15) % 30) - ((YEAR(now()) % 100) % 4)) % 7) - 7*(((YEAR(now()) % 19) + 11*((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) - ((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100) + 8) DIV 25) + 1) DIV 3) + 15) % 30) + 22*((32 + 2*((YEAR(now()) DIV 100) % 4) + 2*((YEAR(now()) % 100) DIV 4) - ((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) - ((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100) + 8) DIV 25) + 1) DIV 3) + 15) % 30) - ((YEAR(now()) % 100) % 4)) % 7)) DIV 451) + 114) DIV 31, '-', ((((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) - ((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100) + 8) DIV 25) + 1) DIV 3) + 15) % 30) + ((32 + 2*((YEAR(now()) DIV 100) % 4) + 2*((YEAR(now()) % 100) DIV 4) - ((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) - ((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100) + 8) DIV 25) + 1) DIV 3) + 15) % 30) - ((YEAR(now()) % 100) % 4)) % 7) - 7*(((YEAR(now()) % 19) + 11*((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) - ((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100) + 8) DIV 25) + 1) DIV 3) + 15) % 30) + 22*((32 + 2*((YEAR(now()) DIV 100) % 4) + 2*((YEAR(now()) % 100) DIV 4) - ((19*(YEAR(now()) % 19) + (YEAR(now()) DIV 100) - ((YEAR(now()) DIV 100) DIV 4) - (((YEAR(now()) DIV 100) - (((YEAR(now()) DIV 100) + 8) DIV 25) + 1) DIV 3) + 15) % 30) - ((YEAR(now()) % 100) % 4)) % 7)) DIV 451) + 114) % 31) + 1), '%Y-%c-%e')
  • 11.
    https://github.com/rpbouman/mysqlv8udfs Intermezzo: Easter day Performancecomparison SQL Expression SQL Stored Function JavaScript UDF 0 2 4 6 8 10 12 14 4.61 12.53 2.66 Easter Day Performance (1.000.000) time(seconds)
  • 12.
    https://github.com/rpbouman/mysqlv8udfs The mysqlv8udfs project ●Scalar Functions: – js() – jsudf() – jserr() ● Aggregate Functions: – jsagg() ● Daemon plugin*: – JS_DAEMON
  • 13.
    https://github.com/rpbouman/mysqlv8udfs The JS_DAEMON Plugin mysql>SHOW VARIABLES LIKE 'js%'; +-----------------------+--------------------------------------+ | Variable_name | Value | +-----------------------+--------------------------------------+ | js_daemon_module_path | /home/rbouman/mysql/mysql/lib/plugin | +-----------------------+--------------------------------------+ 1 row in set (0.03 sec) mysql> SHOW STATUS LIKE 'js%'; +----------------------------------+-----------+ | Variable_name | Value | +----------------------------------+-----------+ | js_daemon_version | 0.0.1 | | js_v8_heap_size_limit | 2048 | | js_v8_heap_size_total | 942944256 | | js_v8_heap_size_total_executable | 959591424 | | js_v8_heap_size_used | 892941672 | | js_v8_is_dead | false | | js_v8_is_execution_terminating | false | | js_v8_is_profiler_paused | true | | js_v8_version | 3.7.12.22 | +----------------------------------+-----------+ 9 rows in set (0.00 sec)
  • 14.
    https://github.com/rpbouman/mysqlv8udfs The js() UDF ●js(script[, arg1, …, argN]) – Execute script – Return value (as string) of the last js expression ● Optional arguments arg1 … argN – Accessible via the built-in arguments array – arg1 accessible as arguments[0] (and so on) ● Script* – if constant it is compiled only once – executed for each row
  • 15.
    https://github.com/rpbouman/mysqlv8udfs The js() UDF:Example mysql> SELECT js(' '> arguments[0] + arguments[1]; '> ', 1, 2) AS example -> ; +---------+ | example | +---------+ | 3 | +---------+ 1 row in set (0.03 sec)
  • 16.
    https://github.com/rpbouman/mysqlv8udfs The MySQL UDFinterface ● Simple functions (scalars) and aggregates ● Native library callbacks (calling sequence) – Plugin directory – mysql.func table ● Data structures – Return value: struct UDF_INIT *initid – Arguments: struct UDF_ARGS *args CREATE [AGGREGATE] FUNCTION name RETURNS (STRING | REAL | INTEGER) SONAME 'libraryfile'
  • 17.
    https://github.com/rpbouman/mysqlv8udfs UDF JavaScript binding ●Scalars: jsudf(), Aggregates: jsagg() – Return value is always a MySQL STRING ● Script argument: – Constant. Compiled and immediately executed (1x) – JavaScript callbacks defined in script are called during the native UDF calling sequence ● UDF data structures scriptable at runtime: – Members of struct UDF_INIT appear as js globals – struct UDF_ARGS as global arguments object
  • 18.
    https://github.com/rpbouman/mysqlv8udfs The jsudf() UDF ●jsudf(script[, arg1, …, argN]) – Call the init() callback (optional) – For each row, return the result of the udf() callback – Call the deinit() callback (optional) init() udf() deinit()More rows? No Yes
  • 19.
    https://github.com/rpbouman/mysqlv8udfs jsudf() example: running total mysql>SELECT amount, jsudf(' -> var total; -> function init(){ -> console.info("Init"); -> total = 0; -> } -> function udf(num){ -> console.info("processing row"); -> return total += num; -> } -> function deinit(){ -> console.info("Deinit"); -> } -> ', amount) AS running_total -> FROM sakila.payment ORDER BY payment_date
  • 20.
    https://github.com/rpbouman/mysqlv8udfs jsudf() example: resultset anderror log +--------+--------------------+ | amount | running_total | +--------+--------------------+ | 2.99 | 2.99 | | 2.99 | 5.98 | . ... . ... . | 4.99 | 67416.5099999921 | +--------+--------------------+ 16049 rows in set (0.29 sec) 2013-09-16 14:31:44 JS_DAEMON [info]: Init 2013-09-16 14:31:44 JS_DAEMON [info]: processing row .... .. .. .. .. .. .. ...... ...... .............. 2013-09-16 14:31:44 JS_DAEMON [info]: processing row 2013-09-16 14:31:44 JS_DAEMON [info]: Deinit
  • 21.
    https://github.com/rpbouman/mysqlv8udfs jsudf() Argument processing ● Argumentsbeyond the initial script argument: – Values passed to the udf() callback – UDF_ARGS scriptable as global arguments array – WARNING: In javascript functions, the local built-in arguments object refers to actual arguments – Local arguments mask the global arguments object. – Use this.arguments to refer to the global array of argument objects. ● Use init() to validate or pre-process arguments
  • 22.
    https://github.com/rpbouman/mysqlv8udfs jsudf() global arguments mysql>SELECT jsudf(' -> function udf(){ -> //this.arguments describes the -> //arguments passed to jsudf -> -> return JSON.stringify( -> this.arguments, -> null, " " -> ); -> } -> ', 'string', PI() AS "real", 1, 2.3)
  • 23.
    https://github.com/rpbouman/mysqlv8udfs jsudf() global arguments [ { "name":"'string'", "type": 0, "max_length": 6, "maybe_null": false, "const_item": true, "value": "string" }, { "name": "real", "type": 1, "max_length": 8, "maybe_null": false, "const_item": true, "value": 3.141592653589793 }, { "name": "1", "type": 2, "max_length": 1, "maybe_null": false, "const_item": true, "value": 1 }, { "name": "2.3", "type": 4, "max_length": 3, "maybe_null": false, "const_item": true, "value": 2.3 } ]
  • 24.
    https://github.com/rpbouman/mysqlv8udfs jsudf() local arguments (actualarguments) mysql> SELECT jsudf(' -> function udf(){ -> //standard, built-in javascript arguments -> //contains argument values passed to this function -> return JSON.stringify( -> arguments, -> null, " " -> ); -> } -> ', 'string', PI() AS "real", 1, 2.3) { "0": "string", "1": 3.141592653589793, "2": 1, "3": 2.3 }
  • 25.
    https://github.com/rpbouman/mysqlv8udfs jsudf() named arguments SELECT jsudf(' functionudf( arg1, arg2, arg3, arg4 ){ //values of named argument are available //as local variables return "arg1: " + arg1 + "n" + "arg2: " + arg2 + "n" + "arg3: " + arg3 + "n" + "arg4: " + arg4; } ', 'string', PI() AS "real", 1, 2.3) arg1: string arg2: 3.141592653589793 arg3: 1 arg4: 2.3
  • 26.
    https://github.com/rpbouman/mysqlv8udfs The Argument object ●name: Expression text. If provided, the alias ● type: code indicating the runtime data type – 0: STRING_RESULT, 1: REAL_RESULT, 4: DECIMAL_RESULT ● max_length: maximum string length ● maybe_null: true if nullable ● const_item: true if value is constant ● value: argument value
  • 27.
    https://github.com/rpbouman/mysqlv8udfs Argument Processing: Validating countand types function init(){ var args = this.arguments, nargs = args.length ; //validate the number of arguments: if (nargs != 1) throw "Expected exactly 1 argument"; //validate argument type: var arg = args[0]; switch (arg.type) { case REAL_RESULT: case INT_RESULT: case DECIMAL_RESULT: break; default: throw "Argument must be numeric"; } }
  • 28.
    https://github.com/rpbouman/mysqlv8udfs Data type Mapping+------------------------+------------------------+---------------------+-------------+---------+ |Type family | MySQL column data type | MYSQL UDF data type | v8 type | JS Type | +------------------------+------------------------+---------------------+-------------+---------+ | Integral numbers | BIGINT | INT_RESULT | v8::Integer | Number | | | INT | | or | | | | MEDIUMINT | | v8::Number | | | | SMALLINT | | | | | | TINYINT | | | | +------------------------+------------------------+---------------------+-------------| | | Floating point numbers | DOUBLE | REAL_RESULT | v8::Number | | | | FLOAT | | | | +------------------------+------------------------+---------------------+ | | | Decimal numbers | DECIMAL | DECIMAL_RESULT | | | +------------------------+------------------------+---------------------+-------------+---------+ | Binary String | BINARY | STRING_RESULT | v8::String | String | | | BLOB | | | | | | LONGBLOB | | | | | | MEDIUMBLOB | | | | | | VARBINARY | | | | | | TINYBLOB | | | | +------------------------+------------------------+ | | | | Character String | CHAR | | | | | | LONGTEXT | | | | | | MEDIUMTEXT | | | | | | VARCHAR | | | | | | TEXT | | | | | | TINYTEXT | | | | +------------------------+------------------------+ | | | | Structured String | ENUM | | | | | | SET | | | | +------------------------+------------------------+ | | | | Temporal | DATE | | | | | | DATETIME | | | | | | TIME | | | | | | TIMESTAMP | | | | +------------------------+------------------------+---------------------+-------------+---------+
  • 29.
    https://github.com/rpbouman/mysqlv8udfs The jsagg() UDF ●jsagg(script[, arg1, …, argN]) – Call the init() callback (optional) – Calls clear() before processing a group of rows – For each row in a group, the udf() callback is called – After processing a group, the agg() is called to return the aggregate value – Call the deinit() callback (optional)
  • 30.
    https://github.com/rpbouman/mysqlv8udfs The jsagg() UDF init() deinit() Morerows? No Yes clear() agg() udf() More groups? Yes No
  • 31.
    https://github.com/rpbouman/mysqlv8udfs jsagg() example: JSON export mysql>SELECT jsagg(' -> var rows, args = arguments, n = args.length; -> function clear(){ -> console.info("clear"); -> rows = []; -> } -> function udf(){ -> console.info("udf"); -> var i, arg, row = {}; -> for (i = 0; i < n; i++){ -> arg = args[i]; -> row[arg.name] = arg.value; -> } -> rows.push(row); -> } -> function agg(){ -> console.info("agg"); -> return JSON.stringify(rows, null, " "); -> } -> ', film_id, title, release_year, description) AS json -> FROM sakila.film GROUP BY rating;
  • 32.
    https://github.com/rpbouman/mysqlv8udfs jsagg() example: result [ { "film_id": 1, "title":"ACADEMY DINOSAUR", "release_year": 2006, "description": "A Epic Drama of ... in The Canadian Rockies" }, ..., ..., { "film_id": 1000, "title": "ZORRO ARK", "release_year": 2006, "description": "A Intrepid Panorama of ... in A Monastery" } ]
  • 33.
    https://github.com/rpbouman/mysqlv8udfs jsagg() example: error log 2013-09-1623:36:45 JS_DAEMON [info]: Clear 2013-09-16 23:36:45 JS_DAEMON [info]: Udf .... .. .. .. .. .. .. ...... .... ... 2013-09-16 23:36:45 JS_DAEMON [info]: Udf 2013-09-16 23:36:45 JS_DAEMON [info]: Agg 2013-09-16 23:36:45 JS_DAEMON [info]: Clear 2013-09-16 23:36:45 JS_DAEMON [info]: Udf .... .. .. .. .. .. .. ...... .... ... 2013-09-16 23:36:45 JS_DAEMON [info]: Udf 2013-09-16 23:36:45 JS_DAEMON [info]: Agg .... .. .. .. .. .. .. ...... .... ... .... .. .. .. .. .. .. ...... .... ... 2013-09-16 23:36:45 JS_DAEMON [info]: Clear 2013-09-16 23:36:45 JS_DAEMON [info]: Udf .... .. .. .. .. .. .. ...... .... ... 2013-09-16 23:36:45 JS_DAEMON [info]: Udf 2013-09-16 23:36:45 JS_DAEMON [info]: Agg
  • 34.
    https://github.com/rpbouman/mysqlv8udfs JavaScript Environment ● JavaScriptStandard built-ins: – Constructors (Date, RegExp, String etc.) – Static objects (JSON, Math) – Misc. functions (decodeURI, parseInt etc.) ● Globals provided by mysqlv8udfs * – arguments[] array – Some UDF interface variables and constants – require() function – console object – mysql object
  • 35.
    https://github.com/rpbouman/mysqlv8udfs The require() function ●Inspired by commonjs Module loading ● Signature: require(filename[, reload]) – Loads script file from the js_daemon_module_path – Executes the script and returns the result – Script is compiled and cached for reuse – Pass true as 2nd argument to force reload from file ● js_daemon_module_path – Read-only system variable of the JS_DAEMON plugin – Specified at mysqld command line or option file – Prevent loading arbitrary script files
  • 36.
    https://github.com/rpbouman/mysqlv8udfs require() example: mysql> SELECTjsagg(' -> require("json_export.js") -> ', category_id, name) AS json -> FROM sakila.category [ { "category_id": 1, "name": "Action" }, ..., { "category_id": 16, "name": "Travel" } ]
  • 37.
    https://github.com/rpbouman/mysqlv8udfs require() example: json_export.js script (functionjson_export(){ var rows, row, i, arg, args = this.arguments, n = args.length; this.clear = function(){ rows = []; } this.udf = function() { rows.push(row = {}); for (i = 0; i < n; i++) { arg = args[i]; row[arg.name] = arg.value; } } this.agg = function(){ return JSON.stringify(rows, null, " "); } })();
  • 38.
    https://github.com/rpbouman/mysqlv8udfs The console object ●Inspired by console object in web-browsers ● Methods: – log([arg1, ..., argN]) – info([arg1, ..., argN]) – error([arg1, ..., argN]) – warn([arg1, ..., argN]) ● Write arguments to a line on the standard error stream – Typically ends up in the mysql error log ● info(), error(), and warn() include a header: – 2013­09­17 00:50:22 JS_DAEMON [info]: ...
  • 39.
    https://github.com/rpbouman/mysqlv8udfs The mysql object ●Namespace for interacting with MySQL – Depends on libmysqlclient mysql client   connect()      version connection   close()   commit()   rollback()   query()   setAutocommit()   charset   connected   hostInfo   InsertId   protocolVersion   serverVersion   statistics   warnings query execute() result() done sql resultset buffered done fieldCount type: "resultset" field() row() resultinfo done: true rowCount type: "resultinfo"  types     0: "decimal"     1: "tinyint"   .... .........   255: "geometry"
  • 40.
    https://github.com/rpbouman/mysqlv8udfs Mysql client example: inventory_held_by_customer CREATEFUNCTION inventory_held_by_customer(p_inventory_id INT) RETURNS INT READS SQL DATA BEGIN DECLARE v_customer_id INT; DECLARE EXIT HANDLER FOR NOT FOUND RETURN NULL; SELECT customer_id INTO v_customer_id FROM rental WHERE return_date IS NULL AND inventory_id = p_inventory_id; RETURN v_customer_id; END;
  • 41.
    https://github.com/rpbouman/mysqlv8udfs Mysql client example (function(){ varconn; this.init = function(){ var args = this.arguments; if (args.length !== 1 || args[0].type !== INT_RESULT) { throw "Single integer argument required"; } conn = mysql.client.connect({ user: "sakila", password: "sakila", schema: "sakila" }); } this.udf = function(inventory_id){ var query = conn.query( "SELECT customer_id FROM rental WHERE return_date IS NULL " + "AND inventory_id = " + inventory_id ); query.execute(); var result = query.result(); if (result.done) return null; return result.row()[0]; } this.deinit = function(){ conn.close(); } })();
  • 42.
    https://github.com/rpbouman/mysqlv8udfs Oracle JSON functions (Labs) ●JSON, not JavaScript – Stick with this If you only need JSON manipulation ● That said.... – Mysqlv8udfs performance is typically slightly better, in some cases substantially better – JSON functions have a few bugs which can be easily worked around using mysqlv8udfs – Easy to emulate with mysqlv8udfs, see sample implementations in the js project dir
  • 43.
    https://github.com/rpbouman/mysqlv8udfs JSON functions vs mysqlV8UDFsexample mysql> SELECT json_append( -> @json, 'menu', 'popup', 'menuitem', 3, -> '{"value": "Help", "onclick": "ShowHelp()"}' -> ); SET @json := '{ "menu": { "id": "file", "value": "File", "popup": { "menuitem": [ {"value": "New", "onclick": "CreateNewDoc()"}, {"value": "Open", "onclick": "OpenDoc()"}, {"value": "Close", "onclick": "CloseDoc()"} ] } } }'; mysql> SELECT jsudf( -> 'require("json_append.js");', -> @json, 'menu', 'popup', 'menuitem', 3, -> '{"value": "Help", "onclick": "ShowHelp()"}' -> );
  • 44.
    https://github.com/rpbouman/mysqlv8udfs JSON functions vsMySQLv8UDFs append contains key extract merge remove replace valid 0 2 4 6 8 10 12 14 json v8 Seconds
  • 45.
    https://github.com/rpbouman/mysqlv8udfs Finally... ● Fork iton github. I appreciate your interest! – https://github.com/rpbouman/mysqlv8udfs – https://github.com/rpbouman/mysqlv8udfs/wiki
  • 46.