Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building better Node.js applications on MariaDB

6,511 views

Published on

In this session, Diego Dupin teaches tips and tricks for using the new Node.js connector for MariaDB. Recent driver updates include exciting new features such as a promise-based API, pipelining and insert streaming. Targeted at beginner to intermediate Node.js developers, this session includes basics for getting started with Node.js before focusing on best practices and more advanced topics. The session finishes with an overview of integration with well-known Node.js frameworks, including the popular objection/relational mapping (ORM) frameworks.

Published in: Software
  • Hello! I do no use writing service very often, only when I really have problems. But this one, I like best of all. The team of writers operates very quickly. It's called ⇒ www.HelpWriting.net ⇐ Hope this helps!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi there! I just wanted to share a list of sites that helped me a lot during my studies: .................................................................................................................................... www.EssayWrite.best - Write an essay .................................................................................................................................... www.LitReview.xyz - Summary of books .................................................................................................................................... www.Coursework.best - Online coursework .................................................................................................................................... www.Dissertations.me - proquest dissertations .................................................................................................................................... www.ReMovie.club - Movies reviews .................................................................................................................................... www.WebSlides.vip - Best powerpoint presentations .................................................................................................................................... www.WritePaper.info - Write a research paper .................................................................................................................................... www.EddyHelp.com - Homework help online .................................................................................................................................... www.MyResumeHelp.net - Professional resume writing service .................................................................................................................................. www.HelpWriting.net - Help with writing any papers ......................................................................................................................................... Save so as not to lose
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • The #1 Woodworking Resource With Over 16,000 Plans, Download 50 FREE Plans... =>> https://t.cn/A62Ygslz
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • $25 per hour jobs on Facebook, now hiring! ★★★ http://t.cn/AieXiXbg
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Building better Node.js applications on MariaDB

  1. 1. Node.js Connector Diego DUPIN MariaDB Connector Engineer MariaDB Corporation
  2. 2. PROGRAM - Getting started - Best practice - Advanced topic - ORM
  3. 3. NODE.JS CONNECTORS Existing community connectors ● mysqljs/mysql (npm mysql) (+ Promise-mysql to have promise) ● sidorares/node-mysql2 (npm mysql2) ● mscdex/node-mariasql (npm mariasql) New connector ● MariaDB/mariadb-connector-nodejs (npm mariadb)
  4. 4. WHY A NEW CONNECTOR ● New functionalities ● Better performance Benchmark : “SELECT * FROM mysql.user LIMIT 1” on local DB
  5. 5. Getting started
  6. 6. QUICK START - INSTALLATION through npm. $ npm install mariadb To use the Connector, you need to import the package into your application code. const mariadb = require('mariadb'); //promise implementation or const mariadb = require('mariadb/callback'); //callback implementation
  7. 7. QUICK START - CONNECTION Promise const mariadb = require("mariadb"); mariadb.createConnection({ user: "root", database: "db", host: "localhost", port: 3306 }) .then(conn => { console.log("Connected successful"); conn.end(); }) .catch(err => { console.log("Error: " + err.message); }); const mariadb = require("mariadb/callback"); const conn = mariadb.createConnection({ user: "root", database: "db", host: "localhost", port: 3306 }); conn.connect(err => { if (err) { console.log("Error: " + err.message); } else { console.log("Connected successful"); conn.end(); } }); Callback
  8. 8. QUICK START - INSERT conn.query( "CREATE TEMPORARY TABLE myTable " + "(id int NOT NULL AUTO_INCREMENT, firstName varchar(256), lastName varchar(256), " + " PRIMARY KEY (id))" ) .then(() => { return conn.query("INSERT INTO myTable(firstName, lastName) VALUES (?, ?)", [ "john", "smith" ]); }) .then(res => { console.log(res); //{ affectedRows: 1, insertId: 1, warningStatus: 0 } conn.end(); }) .catch(err => { });
  9. 9. QUICK START - SELECT conn.query( "SELECT ID,COLLATION_NAME FROM INFORMATION_SCHEMA.COLLATIONS " + "WHERE CHARACTER_SET_NAME = ? LIMIT 2", ["utf8mb4"] ) .then(res => { console.log(res); /* resultset is an array of rows, represented by JSON object [ { ID: 45, COLLATION_NAME: 'utf8mb4_general_ci' }, { ID: 46, COLLATION_NAME: 'utf8mb4_bin' } ] */ conn.end(); }) .catch(err => { … });
  10. 10. CONNECTION ESSENTIAL API ● connection.query(sql[, values]) → Promise: Executes a query. ● connection.queryStream(sql[, values]) → Emitter: Executes a query, streaming results. ● connection.batch(sql, values) → Promise: fast batch processing. ● connection.beginTransaction() → Promise: Begins a transaction. ● connection.commit() → Promise: Commits the current transaction, if any. ● connection.rollback() → Promise: Rolls back the current transaction, if any. ● connection.ping() → Promise: Sends a 1 byte packet to the database to validate the connection. ● connection.end() → Promise: Gracefully close the connection.
  11. 11. SSL
  12. 12. SSL One-Way SSL Authentication or Two-Way SSL Authentication const fs = require("fs"); const mariadb = require('mariadb'); //reading certificates from file const serverCert = [fs.readFileSync("server.pem", "utf8")]; //connecting mariadb .createConnection({ user: "myUser", host: "myHost.com", ssl: { ca: serverCert } }).then(conn => {})
  13. 13. Pooling
  14. 14. POOLING Basically a database connection cache implementation Connections are expensive. On local DB: ● 2.4ms for a basic connection ● 0.05ms for a simple query Problem : correctness and reliability
  15. 15. POOLING Event implementation - Pool handle new connection creation one by one - New connection are added to idle connection queue On connection failure: - Pool revalidate all other connections - Continue creating new connections creation one by one Connection request are queued separately Idle Connections
  16. 16. POOLING Connection request Idle ConnectionsConnection request Connection request Request queue
  17. 17. POOLING - Implementation to handle query pikes Example with a pool that is configured to have a maximum of 50 connections. actual connection number is 5. With a basis of a connection creation taking 2.4ms, and query taking 0.05ms (example on a local server). Everything is quiet, and then ... Boom! ... 100 queries on the pool at once, wanting a connection.
  18. 18. POOLING
  19. 19. POOLING
  20. 20. POOLING Connection pools SIZING error Example 10 000 user simultaneously, 20 000 transaction per second. What value to connectionLimit (max connection number in pool) ? 100 ? 500 ? 1000 ?
  21. 21. POOLING - CONFIGURATION acquireTimeout t acquireTimeout Timeout to get a new connection from pool. Maximum number of connection in pool. Delay to avoid connection validation Disabling connection control connectionLimit minDelayValidation noControlAfterUse
  22. 22. POOL API ● pool.getConnection() → Promise : Creates a new connection. ● pool.query(sql[, values]) → Promise: Executes a query. ● pool.batch(sql, values) → Promise: Executes a batch Stats ● pool.activeConnections() → Number: Gets current active connection number. ● pool.totalConnections() → Number: Gets current total connection number. ● pool.idleConnections() → Number: Gets current idle connection number. ● pool.taskQueueSize() → Number: Gets current stacked request.
  23. 23. POOLING - CONFIGURATION const pool = mariadb.createPool({ user: "root", database: "testn", host: "localhost", port: 3306, sessionVariables: { wait_timeout: 31536000 }, acquireTimeout: 5000, connectionLimit: 8, }); pool .getConnection() .then(conn => { console.log("Connected successfully to server"); conn.release(); }) .catch(err => { console.log("Error during connection: " + err.message); }); Server expect connection to be used (@wait_timeout)
  24. 24. MULTI-SERVER
  25. 25. MULTI-HOST - ESSENTIAL API “Cluster” offer handle multiple pools permitting load balancing and high availability const mariadb = require("mariadb"); const cluster = mariadb.createPoolCluster(); cluster.add("master", { host: "mydb1.com", ... }); cluster.add("slave1", { host: "mydb2.com", ... }); cluster.add("slave2", { host: "mydb3.com", ... }); cluster .getConnection("slave*", "RR") //RR: ROUND-ROBIN .then(conn => { }) .catch(err => { });
  26. 26. MULTI-HOST - ESSENTIAL API client Master Slave B Slave A Cluster.getConnection("slave*", "RR")
  27. 27. MULTI-HOST - ESSENTIAL API client Master Slave B Slave A Cluster.getConnection("slave*", "RR")
  28. 28. MULTI-HOST - ESSENTIAL API client Master Slave B Slave A Cluster.getConnection("slave*", "RR")
  29. 29. MULTI-HOST - ESSENTIAL API client Master Slave B Slave A Cluster.getConnection("slave*", "RR")
  30. 30. MULTI-HOST - ESSENTIAL API client Master Slave B Slave A Cluster.getConnection("slave*", "RR")
  31. 31. MULTI-HOST - ESSENTIAL API client Master Slave B Slave A Cluster.getConnection("slave*", "RR")
  32. 32. MULTI-HOST SELECTORS ● RR (round-robin) ● RANDOM ● ORDER
  33. 33. MULTI-HOST - ESSENTIAL API ● poolCluster.add(id, config) : add a pool to cluster. ● poolCluster.remove(pattern) : remove and end pool according to pattern. ● poolCluster.end() → Promise : end cluster. ● poolCluster.getConnection(pattern, selector) → Promise : return a connection from cluster. ● poolCluster.of(pattern, selector) → FilteredPoolCluster : return a subset of cluster.
  34. 34. MULTI-HOST - ESSENTIAL API poolCluster.of(pattern, selector) → FilteredPoolCluster : return a subset of cluster. const cluster = mariadb.createPoolCluster(); cluster.add("master", { host: "mydb1.com", ... }); cluster.add("slave1", { host: "mydb2.com", ... }); cluster.add("slave2", { host: "mydb3.com", ... }); const slaves = cluster.of(/^slave?/, 'RANDOM'); slaves.getConnection().then( ... ); slaves.query(sql, values).then( ... ); slaves.batch(sql, values).then( ... );
  35. 35. New features
  36. 36. Pipelining
  37. 37. PIPELINING Pipelining is a technique in which multiple requests are sent on a single TCP connection without waiting for the corresponding responses. This saves round trip time.
  38. 38. PIPELINING - without const uuid = uuidv1(); //generate new UUID conn .beginTransaction() .then(() => { return conn.query("INSERT INTO basket(basketId) values (?)", uuid); }) .then(() => { return conn.query( "INSERT INTO basket_item(basketId, itemId) VALUES (?, ?)", [ uuid, 100 ]); }) .then(() => { return conn.query( "INSERT INTO basket_item(basketId, itemId) VALUES (?, ?)", [ uuid, 101 ]); }) .then(() => { conn.commit(); }) .catch(err => { conn.rollback(); });
  39. 39. PIPELINING - with const uuid = uuidv1(); //generate new UUID conn .beginTransaction() .then(() => { return Promise.all([ conn.query("INSERT INTO basket(basketId) values (?)", uuid), conn.query( "INSERT INTO basket_item(basketId, itemId) VALUES (?, ?)", [uuid, 100]), conn.query( "INSERT INTO basket_item(basketId, itemId) VALUES (?, ?)", [uuid, 101]) ]); }) .then(() => { conn.commit(); }) .catch(err => { conn.rollback(); });
  40. 40. PIPELINING Local DB: abuse : socket buffering
  41. 41. Streaming
  42. 42. Streaming Goal : Avoid loading all in memory ● Streaming resultset -> avoiding loading large resultset totally into memory ○ By event ○ With pipe ● Streaming sent -> sending buffer by chunk
  43. 43. Streaming resultset BY EVENT : Limitations : ● Server net_write_timeout. ○ For a command: SET STATEMENT net_write_timeout=10000 FOR XXX ○ For a connection: sessionVariables: { net_write_timeout: 31536000 } connection .queryStream("SELECT * FROM mysql.user") .on("error", err => {}) .on("fields", meta => {}) .on("data", row => {}) .on("end", () => {});
  44. 44. Streaming resultset USING PIPE : const someWriterStream = fs.createWriteStream("./jsonUsers.txt"); const transformStream = new stream.Transform({ objectMode: true, transform: function transformer(chunk, encoding, callback) { callback(null, JSON.stringify(chunk)); } }); const queryStream = connection.queryStream("SELECT * FROM mysql.user"); stream.pipeline(queryStream, transformStream, someWriterStream);
  45. 45. Streaming - sending const https = require("https"); https.get( "https://node.green/#ES2018-features-Promise-prototype-finally-basic-support", //3Mb page readableStream => { connection .query("INSERT INTO StreamingContent (b) VALUE (?)", [readableStream]) .then(res => {}) .catch(err => {}); }); Limitations : ● Server net_read_timeout : SET STATEMENT net_read_timeout=10000 FOR XXX ○ For a connection: sessionVariables: { net_read_timeout: 10000 } ● max_allowed_packet
  46. 46. Batching
  47. 47. BATCH INSERT conn.batch("INSERT INTO myTable(firstName, lastName) VALUES (?, ?)", [ ["john", "smith"], ["jack", "brown"] ]); }) .then(res => { }) .catch(err => { }); Benchmark : 100 * insert 100 characters on local DB
  48. 48. ORM - sequelize
  49. 49. ORM $ npm install --save express body-parser sequelize@5.0.0-beta.16 mariadb const Sequelize = require('sequelize'); const sequelize = new Sequelize('testn', 'root', null, { host: 'localhost', dialect: 'mariadb', pool: { max: 5, min: 5, acquire: 30000, idle: 10000 } });
  50. 50. ORM const User = sequelize.define('myUser', { id: { type: Sequelize.INTEGER, primaryKey: true, autoIncrement: true }, firstName: { type: Sequelize.STRING }, lastName: { type: Sequelize.STRING } }); User.sync({ force: true }) .then(() => { return User.create({ firstName: 'John', lastName: 'Hancock' }); });
  51. 51. ORM const express = require('express'); const bodyParser = require('body-parser'); const app = express(); app.use(bodyParser.json()); const port = 3000; app.listen(port, () => { console.log(`Running on http://localhost:${port}`) }); // get all users app.get('/api/users', (req, res) => { User.findAll().then(users => res.json(users)) });
  52. 52. tricks
  53. 53. DEBUGGING Connection options ‘trace’ default to false, nice to have in development mode. { Error: (conn=149, no: 1146, SQLState: 42S02) Table 'testn.unknownTable' doesn't exist sql: SELECT * FROM unknownTable - parameters:[] at Object.module.exports.createError (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/misc/errors.js:55:10) at Packet.readError (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/io/packet.js:506:19) at Query.readResponsePacket (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/cmd/resultset.js:47:28) at PacketInputStream.receivePacket (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/io/packet-input-stream.js:73:9) at PacketInputStream.onData (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/io/packet-input-stream.js:129:20) at Socket.emit (events.js:197:13) at addChunk (_stream_readable.js:288:12) at readableAddChunk (_stream_readable.js:269:11) at Socket.Readable.push (_stream_readable.js:224:10) at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:145:17) fatal: false, errno: 1146, sqlState: '42S02', code: 'ER_NO_SUCH_TABLE' }
  54. 54. DEBUGGING With trace { Error: (conn=150, no: 1146, SQLState: 42S02) Table 'testn.unknownTable' doesn't exist sql: SELECT * FROM unknownTable - parameters:[] at Object.module.exports.createError (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/misc/errors.js:55:10) ... at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:145:17) From event: at /home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/connection.js:166:29 at new Promise (<anonymous>) at Connection.query (/home/diego/IdeaProjects/test_node/node_modules/mariadb/lib/connection.js:164:12) at mariadb.createConnection.then.conn (/home/diego/IdeaProjects/test_node/lib/promise/trace.js:13:8) at processTicksAndRejections (internal/process/next_tick.js:81:5) fatal: false, errno: 1146, sqlState: '42S02', code: 'ER_NO_SUCH_TABLE' }
  55. 55. AVOID TCP-IP layer for local connection Connection options ‘socketPath‘ for local server - UNIX domain - Windows named pipe const mariadb = require('mariadb'); mariadb.createConnection({ socketPath: '/tmp/mysql.sock', user: 'root' }) .then(conn => { ... }) .catch(err => { ... });
  56. 56. METADATA ● Select query = row datas + metadata Metadata = datatype, format, and lots of additional infos: schema, table name, table alias, column name, column alias, … Not always needed conn.query("select * from mysql.user u LIMIT 1") .then(rows => { console.log(rows); // [ // { Host: 'localhost', User: 'root', … } // ] }); conn .query({ sql: "select * from mysql.user u LIMIT 1", rowsAsArray: true }) .then(rows => { console.log(rows); // [ // [ 'localhost', 'root', …] // ] });
  57. 57. What next ? Authentication plugins (Kerberos, ed25519) Performance improvement Failover enhancement Pools improvement
  58. 58. THANK YOU!

×