In this session, Diego Dupin teaches tips and tricks for using the new Node.js connector for MariaDB. Recent driver updates include exciting new features such as a promise-based API, pipelining and insert streaming. Targeted at beginner to intermediate Node.js developers, this session includes basics for getting started with Node.js before focusing on best practices and more advanced topics. The session finishes with an overview of integration with well-known Node.js frameworks, including the popular objection/relational mapping (ORM) frameworks.
6. QUICK START - INSTALLATION
through npm.
$ npm install mariadb
To use the Connector, you need to import the package into your application code.
const mariadb = require('mariadb'); //promise implementation
or
const mariadb = require('mariadb/callback'); //callback implementation
10. CONNECTION ESSENTIAL API
● connection.query(sql[, values]) → Promise: Executes a query.
● connection.queryStream(sql[, values]) → Emitter: Executes a query, streaming results.
● connection.batch(sql, values) → Promise: fast batch processing.
● connection.beginTransaction() → Promise: Begins a transaction.
● connection.commit() → Promise: Commits the current transaction, if any.
● connection.rollback() → Promise: Rolls back the current transaction, if any.
● connection.ping() → Promise: Sends a 1 byte packet to the database to validate the connection.
● connection.end() → Promise: Gracefully close the connection.
14. POOLING
Basically a database connection cache implementation
Connections are expensive. On local DB:
● 2.4ms for a basic connection
● 0.05ms for a simple query
Problem : correctness and reliability
15. POOLING
Event implementation
- Pool handle new connection creation one by one
- New connection are added to idle connection queue
On connection failure:
- Pool revalidate all other connections
- Continue creating new connections creation one by one
Connection request are queued separately
Idle
Connections
17. POOLING
- Implementation to handle query pikes
Example with a pool that is configured to have a maximum of 50 connections. actual
connection number is 5.
With a basis of a connection creation taking 2.4ms, and query taking 0.05ms (example
on a local server).
Everything is quiet, and then ... Boom! ... 100 queries on the pool at once, wanting a
connection.
20. POOLING
Connection pools SIZING error
Example 10 000 user simultaneously, 20 000 transaction per second.
What value to connectionLimit (max connection number in pool) ?
100 ? 500 ? 1000 ?
21. POOLING - CONFIGURATION
acquireTimeout
t
acquireTimeout Timeout to get a new connection from pool.
Maximum number of connection in pool.
Delay to avoid connection validation
Disabling connection control
connectionLimit
minDelayValidation
noControlAfterUse
22. POOL API
● pool.getConnection() → Promise : Creates a new connection.
● pool.query(sql[, values]) → Promise: Executes a query.
● pool.batch(sql, values) → Promise: Executes a batch
Stats
● pool.activeConnections() → Number: Gets current active connection number.
● pool.totalConnections() → Number: Gets current total connection number.
● pool.idleConnections() → Number: Gets current idle connection number.
● pool.taskQueueSize() → Number: Gets current stacked request.
23. POOLING - CONFIGURATION
const pool = mariadb.createPool({ user: "root", database: "testn", host: "localhost", port: 3306,
sessionVariables: { wait_timeout: 31536000 },
acquireTimeout: 5000, connectionLimit: 8,
});
pool
.getConnection()
.then(conn => {
console.log("Connected successfully to server");
conn.release();
})
.catch(err => {
console.log("Error during connection: " + err.message);
});
Server expect connection to be used (@wait_timeout)
33. MULTI-HOST - ESSENTIAL API
● poolCluster.add(id, config) : add a pool to cluster.
● poolCluster.remove(pattern) : remove and end pool according to pattern.
● poolCluster.end() → Promise : end cluster.
● poolCluster.getConnection(pattern, selector) → Promise : return a connection from
cluster.
● poolCluster.of(pattern, selector) → FilteredPoolCluster : return a subset of cluster.
37. PIPELINING
Pipelining is a technique in which
multiple requests are sent on a single
TCP connection without waiting for the
corresponding responses.
This saves round trip time.
42. Streaming
Goal : Avoid loading all in memory
● Streaming resultset -> avoiding loading large resultset totally into memory
○ By event
○ With pipe
● Streaming sent -> sending buffer by chunk
43. Streaming resultset
BY EVENT :
Limitations :
● Server net_write_timeout.
○ For a command: SET STATEMENT net_write_timeout=10000 FOR XXX
○ For a connection: sessionVariables: { net_write_timeout: 31536000 }
connection
.queryStream("SELECT * FROM mysql.user")
.on("error", err => {})
.on("fields", meta => {})
.on("data", row => {})
.on("end", () => {});
44. Streaming resultset
USING PIPE : const someWriterStream = fs.createWriteStream("./jsonUsers.txt");
const transformStream = new stream.Transform({
objectMode: true,
transform: function transformer(chunk, encoding, callback) {
callback(null, JSON.stringify(chunk));
}
});
const queryStream = connection.queryStream("SELECT * FROM mysql.user");
stream.pipeline(queryStream, transformStream, someWriterStream);
45. Streaming - sending
const https = require("https");
https.get(
"https://node.green/#ES2018-features-Promise-prototype-finally-basic-support", //3Mb page
readableStream => {
connection
.query("INSERT INTO StreamingContent (b) VALUE (?)", [readableStream])
.then(res => {})
.catch(err => {});
});
Limitations :
● Server net_read_timeout : SET STATEMENT net_read_timeout=10000 FOR XXX
○ For a connection: sessionVariables: { net_read_timeout: 10000 }
● max_allowed_packet
53. DEBUGGING
Connection options ‘trace’ default to false, nice to have in development mode.
{ Error: (conn=149, no: 1146, SQLState: 42S02) Table 'testn.unknownTable' doesn't exist
sql: SELECT * FROM unknownTable - parameters:[]
at Object.module.exports.createError (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/misc/errors.js:55:10)
at Packet.readError (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/io/packet.js:506:19)
at Query.readResponsePacket (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/cmd/resultset.js:47:28)
at PacketInputStream.receivePacket (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/io/packet-input-stream.js:73:9)
at PacketInputStream.onData (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/io/packet-input-stream.js:129:20)
at Socket.emit (events.js:197:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:145:17)
fatal: false,
errno: 1146,
sqlState: '42S02',
code: 'ER_NO_SUCH_TABLE' }
54. DEBUGGING
With trace
{ Error: (conn=150, no: 1146, SQLState: 42S02) Table 'testn.unknownTable' doesn't exist
sql: SELECT * FROM unknownTable - parameters:[]
at Object.module.exports.createError (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/misc/errors.js:55:10)
...
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:145:17)
From event:
at /home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/connection.js:166:29
at new Promise (<anonymous>)
at Connection.query (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/connection.js:164:12)
at mariadb.createConnection.then.conn (/home/diego/IdeaProjects/test_node/lib/promise/trace.js:13:8)
at processTicksAndRejections (internal/process/next_tick.js:81:5)
fatal: false,
errno: 1146,
sqlState: '42S02',
code: 'ER_NO_SUCH_TABLE' }
55. AVOID TCP-IP layer for local connection
Connection options ‘socketPath‘ for local server
- UNIX domain
- Windows named pipe
const mariadb = require('mariadb');
mariadb.createConnection({ socketPath: '/tmp/mysql.sock', user: 'root' })
.then(conn => { ... })
.catch(err => { ... });